At a Glance
Enterprise Technology / Cloud Infrastructure
Global
Recover a critical P1 production outage caused by a development environment decommission then restructure the client's entire Infrastructure-as-Code practice so a Dev action could never take down Production again.
- Incident resolution cut from 12 hours to under 4 .
- Dev-to-Production risk eliminated through pipeline isolation and access control .
- Fully auditable deployments, governed by Git tags rather than open branch merges.
The Challenge
A routine decommission of a development environment took down live production applications. The cause was structural, not accidental: production networking components had been housed in the Dev branch, and Dev pipelines held unchecked permissions to destroy shared resources. When the environment was torn down, shared route tables and AWS Transit Gateway attachments went with it disrupting multiple live applications simultaneously.
The outage exposed a deeper problem. There was no modularisation in the Terraform codebase, no Git tagging, no access control and no release process separating environments. Nothing stood between a developer's routine action and the production estate. Recovery was urgent, but recovery alone would have left the same landmine in place.
Our Approach
Cozentus responded immediately, working on two fronts at once: restore service, then remove the underlying risk.
Recovery came first. We rebuilt the Transit Gateway attachments, route tables and routing entries using Terraform state files, logs and historical documentation, suspended Dev pipeline access to prevent further impact, and restored and tested VPC connectivity across every affected application bringing resolution time down from twelve hours to under four.
With production stable, we addressed the cause. We refactored the Terraform codebase into modular, environment-specific components, consolidated deployments into a single pipeline serving multiple environments, and introduced Git tag-based release controls with approval workflows. Audit logging and least-privilege access control were enabled throughout, turning an error-prone flat codebase into a governed, risk-resilient framework.
- Rapid restoration: TGW attachments, route tables and routing entries rebuilt from Terraform state files and logs, with VPC connectivity validated across all applications
- Immediate risk containment: Dev pipeline access suspended to stop further production impact
- Structural modernisation: modular, environment-specific Terraform; Git tag-based releases with approval workflows; audit logging and least-privilege access
Business Outcomes
- Resolution time reduced from 12 hours to under 4
- Production routing restored with no data loss
- Dev pipeline can no longer impact Production environments properly isolated
- Releases governed by Git tags and approval workflows, not open branch merges
- Modular, scalable Terraform architecture replacing flat, error-prone code
- Full infrastructure visibility changes tracked, reviewed and auditable
- Stronger cross-team collaboration on shared infrastructure
Client Quote
The Cozentus team didn't just help us bounce back from a crisis they completely transformed how we operate… they helped us redesign our infrastructure to be modular and flexible, which means we can adapt quickly to future needs without starting from scratch each time. Their approach to governance was equally impressive now our deployments are fully auditable and tightly controlled.
Related Case Studies
Ready to Transform Your Supply Chain?
Let's discuss how our AI-powered solutions can help you optimize operations and drive measurable results. Our team is ready to understand your unique challenges.