I'll do my best to respond to both your questions in a single go:
- we did have a rather advanced Captcha system in place, though we missed it on one of the publicly available routes, which we managed to fix rather easily
- you do have to understand that for a startup, at times there are various constraints - like the lack of time and money to justify an enterprise contract with a product like CF
- I do use CF on personal projects that are mostly static pages, so I don't have a particular issue with it
To put it simply it is our naive implementation of "security through obscurity". Since this DDoS attack, we've had probably around 30-40 more of different duration and size, but nothing of this magnitude - hopefully never again. Fortunately with our approach we've been able to keep all of those at bay with zero downtime caused by them. Most likely that won't always be the case :)
If CF works just fine for you, go for it. The morale of this story is that you should prepare at least some basic skillset/knowledge base on fighting a DDoS as they are more and more common in recent years.
Thank you for the thorough and kind response! I had no idea it was so expensive for companies. I'm glad you solved it, I've never suffered an attack of this magnitude, I hope I never have to go through that :D
What was the hotfix that was applied here? did you mean that both frontend and backend were accepting requests directly without going through the API Gateway?
"That night I did a hotfix and patched both our frontend and the backend. It was not acceptable to have requests passing our API Gateway, even hitting our backend and flooding our DB…"
It was more of a mistake on the engineering side of things, one of the API endpoints with anonymous/unauthenticated access allowed was not properly protected and allowing more traffic than it should to the downstream.
Plot twist - we were the FinOps team. We probably circled back and forth at least 4-5 times about if we should just enable AWS Shield and hope they'll deal with it in a timely manner, but we decided not to. Unfortunately, if you're not using Shield (the "managed" version of WAF), they don't cut costs by default. We did reach out to them and they said they'll look into it and that was about it :)
Excuse my ignorance, but why wouldn't a CF challenge solve this on the first day?
There's no ignorance in asking a question.
I'll do my best to respond to both your questions in a single go:
- we did have a rather advanced Captcha system in place, though we missed it on one of the publicly available routes, which we managed to fix rather easily
- you do have to understand that for a startup, at times there are various constraints - like the lack of time and money to justify an enterprise contract with a product like CF
- I do use CF on personal projects that are mostly static pages, so I don't have a particular issue with it
To put it simply it is our naive implementation of "security through obscurity". Since this DDoS attack, we've had probably around 30-40 more of different duration and size, but nothing of this magnitude - hopefully never again. Fortunately with our approach we've been able to keep all of those at bay with zero downtime caused by them. Most likely that won't always be the case :)
If CF works just fine for you, go for it. The morale of this story is that you should prepare at least some basic skillset/knowledge base on fighting a DDoS as they are more and more common in recent years.
Thank you for the thorough and kind response! I had no idea it was so expensive for companies. I'm glad you solved it, I've never suffered an attack of this magnitude, I hope I never have to go through that :D
Thanks for sharing!
with Captcha, sure.
I can definitly relate to that ...
What was the hotfix that was applied here? did you mean that both frontend and backend were accepting requests directly without going through the API Gateway?
"That night I did a hotfix and patched both our frontend and the backend. It was not acceptable to have requests passing our API Gateway, even hitting our backend and flooding our DB…"
It was more of a mistake on the engineering side of things, one of the API endpoints with anonymous/unauthenticated access allowed was not properly protected and allowing more traffic than it should to the downstream.
What a rollercoaster! Sounds like a lot of work for the FinOps team too. Did AWS help cut the costs?
Plot twist - we were the FinOps team. We probably circled back and forth at least 4-5 times about if we should just enable AWS Shield and hope they'll deal with it in a timely manner, but we decided not to. Unfortunately, if you're not using Shield (the "managed" version of WAF), they don't cut costs by default. We did reach out to them and they said they'll look into it and that was about it :)