One of the things the Product team here at LexBlog spends the most time on is our integrations with third-party services that might be useful to our clients. Our single largest and most important integration is with MailChimp.
Mailchimp powers all of our clients’ blog post email notifications, and we have code that powers all parts of that process—from being able to subscribe to the email lists from various places on the blog, to our authors accessing their subscriber lists and stats about their campaign performance on the back-end of our platform.
So imagine our shock when we woke up one morning to discover that MailChimp had completely cut off one of our three major servers from all access to their API. Our clients had no access to their statistics, new users couldn’t subscribe to email notifications and for our blogs that use ‘instant send,’ a feature that allows you to send the email blasts immediately as opposed to the next day, emails weren’t sending at all.
It was a three-alarm fire for us, but the hardest part was that we didn’t know exactly why we were being blocked. From our end, nothing had changed. We hadn’t launched any new features that used the MailChimp API in a long time, and because only one of our three servers was being blocked, it didn’t seem like it was anything in our code specifically that was causing an issue.
After some back-and-forth with MailChimp’s support team, we were told that they weren’t blocking us directly, but rather their security software Akamai was. Our server was being blocked because, according to Akamai, it was sending “path traversal attack attempts” to their API.
Of course, we weren’t doing that intentionally. It was clear that some malicious actor was using our servers and software to send these requests, and we needed to figure out how.
So we started logging every single API call made to MailChimp from the server in question. And sure enough, we were able to see pretty quickly that we were making calls that looked like this:
https://us6.api.mailchimp.com/3.0/lists//..//..//..//..//..//..//..//..//..//..//..//..//..//..//../etc/passwd\0.html/members/
That’s a path traversal attempt.
Essentially a malicious request aimed at accessing directories of a web server “above” the folders that are marked for public accessibility in order to gain control of the server and use it for bad things. To MailChimp, it looked like we were the ones doing this, but it actually was spammers who were using our form to try to gain access to our own server, as opposed to MailChimp’s.
So how was this possible? Long story short, when you use the form on one of our blogs to subscribe to it, there is a hidden field in the form that tells us the ID of the current site’s email list on our MailChimp account. When you submit your email address, that ID gets used to build the URL that we ultimately call to add you to the list, like /lists/e064daac87/members.
The malicious bots somehow figured this out and were injecting that field with the path traversal attempt and then submitting the form, which then got sent along to MailChimp.
In the end, we are lucky that the result of that form was turned into an API call rather than something that was being processed on our own server, which is what the attackers were hoping. But the moral of this whole story, and the eventual fix that we ended up putting in to prevent this from happening in the future, is to properly sanitize and validate any input that is able to be edited by the end user in the DOM.
It’s important not just to sanitize, but to actually enforce things like the length of the string, and not allow characters that wouldn’t be in the ID you are trying to send, like periods and slashes. While on our end we were doing some basic sanitization, we were using a method that was meant for text inputs, meaning those characters were allowed, which opened us up for this attack.
But because we’re always monitoring these things, and have a team dedicated to identifying and fixing such issues, we were able to get things back on track quickly.
If your crew isn’t well-versed in this type of thing, and throwing plugins onto blogs ad-hoc without keeping an eye on them, it could be trouble.
