This is a true story about the trials and tribulations of an Operations team during a period of extreme growth. A team sinking in endless requests and expedited issues and what happened as they began to measure their capability while pushing back on the barrage of customer demands. We’ll examine the three metrics that mattered, the failures along the way, and the forum used to influence the desired outcome.
A team of forty Ops Engineers was tasked to build out six new data-centers in six different countries within six months. And roll out a new configuration management tool. And deploy features from development teams. And implement stronger security. And support live issues. And implement a new monitoring system. And support various requests from 41 other teams.
Conflicting priorities and over-commitments, exacerbated by a lack of communication led to serious interrupt-driven context switching and missed commitments. Commitments made on behalf of the team by leadership were often not communicated to the team. It wasn’t clear how decisions were made. A dismal hand dealt to the team for sure. How to insert a voice of reason in this madness?
Quantifiable Ammunition - stay calm and collect data!
Decisions needed to be made. How to build and present a case to influence those decisions? Providing relevant metrics in an objective data-driven fashion brought essential visibility and transparency to the problems. Let's look at the diplomatic approach used to communicate the costs and risks of the decisions made.