Incidents are a new normal.

Johnny Cash once said, “You build on failure. You use it as a stepping stone. Close the door on the past. You don’t try to forget the mistakes, but you don’t dwell on it.” Probably, he didn’t know that this quote will perfectly describe organizations that work according to the “fall fast, fail often” mantra. Incidents are a new normal. Part of the development process. We are seeing an increase in canary releases, feature flags adoption, monitoring to mitigate incidents. While smaller, these are still cuts. How can we avoid our business getting killed by thousands of mitigated problems? There are a lot of techniques at our disposal, but in this presentation, the focus is on not repeating mistakes. I will describe how to benefit from past incidents and encourage engineers to embrace and participate in failures. Share the best practices for gathering and analyzing software metrics and people-related data. Give tips on how to formulate actionable executable actions that prevent repetitive incidents.