When a longtime associate reached out for advice regarding their new start-up venture, what I thought was just a phone call turned into an all-hands-on-deck response to a major security breach. In describing the situation, my associate disclosed that they no longer had control of their application or host environment, nor did they have access to their source code or data, thus shutting down service for all users. This was all done by a developer inside their company who demanded a hefty ransom.
They needed help, and fast, so I quickly assembled our DevOps team for an in-depth security and risk assessment. Due to the time-critical nature of the situation and multiple challenging factors, we couldn’t guarantee success. However, our objectives were clear:
- Take back control of the application and hosting environment.
- Identify and cut off the backdoor access routes.
- Restore data and privilege access.
This situation was challenging, but we were now working with a brand-new client and needed prior knowledge of the client’s application and network architecture. Some of the most time-consuming technical challenges we faced included:
- Lack of detailed documentation available to help us understand the application flow and system configuration, e.g., no architectural, deployment, or data flow diagrams; no information on how to start or stop the applications running in the servers; no source code or version control of the software used for this application; no documented location for where the database or application code was deployed, or which servers were responsible for which functions or services.
- Lack of administrative login credentials or keys to the servers and database.
- Limited functionality and security features in the client’s existing cloud instance, including no way to automate monitoring of access, lack of a fixed/static IP, servers hosted in different regions, and multiple security loopholes.
Without having the information we needed and the inability to automate specific tasks, much of this challenging work was tedious and slow, including back-tracing error messages and mapping multiple applications (that included some unauthorized ones) running across various servers and open ports.
However, within a few days of working around the clock, we successfully restored and secured the client’s application and platform, reconfigured the access rules and security parameters, and put them back in control of the servers and database. Not only was the situation addressed without paying the ransom, but the client now has better access and more control over their application and code.
In addition, we have submitted a proposal to harden and secure their current architecture following industry best practices – including implementing the cloud provider’s best security practices and CIS and NIST guidelines – to minimize external and internal ‘bad actors’ threats in the future.
A key factor to our success is our ability to work closely with clients in solving challenging technical issues in diverse applications and environments. Our goal, our hope, is to form long-term relationships with clients in which both parties share a vested interest in each other’s success, anchored in our founding values:
- Providing transparency in our abilities and operations.
- Delivering more value than the cost incurred.
- Building mutual trust.
To learn more about our DevOps and cybersecurity capabilities or talk with us about your challenges and/or needs, please reach out to us.
;