The day I broke production
Mar 2, 2026·1 min read
Early in my time at DevsNest I shipped a change to the checkout widget that looked fine locally, passed review, and took down order tracking for a few hours across every store running it.
The bug was mine and it was dumb: a config value that existed in my environment and not in production. The lesson wasn't "write better code." The lesson was about what I did next.
Small, safe, visible
I stopped shipping big changes. For the next month everything I merged was the smallest useful increment I could carve out, behind a flag where it made sense, with a note in the channel saying what to watch. Nothing heroic — just boring, legible steps that were easy to reverse.
Trust came back faster than I expected, because the team could see it coming back. Every small green deploy was a data point.
What I kept
- The pre-deploy question: what breaks if this value is missing, wrong, or null in production specifically?
- A rollback I've actually tested, not one I assume works.
- Telling people what I changed before they notice something's off.
Breaking production once is almost a rite of passage. Breaking it and going quiet is the part that actually costs you.