In the early morning of Friday, March 24th, our systems began to see large spikes of web traffic from various IP addresses. These traffic spikes were targeting our public website (www.knowledgeowl.com\). This high traffic volume overwhelmed the reverse proxy system that we use to serve the public website, the KnowledgeOwl application, and customer knowledge bases. As an immediate mitigation step, we changed the configuration of our public website to no longer use our primary proxy. This seemed to stabilize the application and customer knowledge bases. The traffic spikes continued until late morning.
Our investigations indicated that these connections were not initiated by customers. The associated web requests did not appear to be legitimate. While we cannot say with certainty that these connections were malicious, we are treating this incident as a distributed denial of service (DDoS) attack.
The risk from high-volume traffic spikes like these is almost impossible to completely remove. However, we are reviewing our systems and processes to better handle these kinds of traffic spikes. We have already identified some concrete next steps to reduce the overall risk:
In the short-term, we are taking three steps:
We are exploring ways to further separate our public website from the KnowledgeOwl application and customer knowledge bases.
Friday's incident provided us with data on how to better identify these types of events. We are building that knowledge into our processes moving forward.
We have already begun modifying our proxy and web application firewall configuration to make our traffic management infrastructure a bit more robust. We'll continue to monitor these changes and iterate on them as needed.
This event has highlighted some potential architectural improvements to our infrastructure, mainly around our traffic-management systems. We'll review these changes' feasibility and effectiveness in the coming months.
Above all, we want to thank you for your patience during this incident. We know that we've had a higher number of downtime incidents in the last two months. We know how integral your knowledge base can be to daily operations. Our team is working hard to learn from this experience and to make KnowledgeOwl stronger in the future.