A fundamental agile principle is “…the team reflects at regular intervals how to become more effective” The SAFe Inspect and Adapt Problem Solving workshop is a wonderful opportunity for everyone on an Agile Release Train (ART) to reflect on becoming more effective. However, what happens when the ART teams are massively distributed, such that the Sun truly never sets on the ART? How do you provide everyone on the ART an opportunity to reflect and collaborate with others who have similar interests, and not just their local cohorts? How do you enable all to participate in the problem-solving session, to raise and solve problems that are important to them, and not just the problems that are important and visible to “home base” or as we called it, the mother ship? This is the situation we faced at a large multi-national energy company preparing to conduct their first SAFe problem-solving workshop. This is our story for how we executed a problem-solving workshop for an ART on which the Sun never set.
1. INTRODUCTION: “The Team Reflects at Regular Intervals How to Become More Effective” – Agile Principles
Agility is not just about continuously learning and adapting the work product, but also reflecting on and adapting the work process itself. Continuous improvement is fundamental to high performing teams and most agile methodologies have a built-in process review like Scrum’s retrospective. The Scaled Agile Framework (SAFe tm) builds on top of this team level view with a problem-solving workshop that is conducted at the end of a Program Increment (big time box) to understand the opportunities for improvement across all teams on the Agile Release Train (ART).
2. BACKGROUND
Our client is a marquee multi-national energy company with operations around the globe and with an ART spanning the globe. While headquartered in US, teams are located across the US and around the world including London, Buenos Aires, Manila, Perth, and Kazakhstan. Literally, the Sun does no set on the program. Our program was moving applications from the on-premise data center to the cloud. While our program was organized on paper as SAFe Solution Train (a train of trains), it operated very much like an oversized single ART, with over 30 teams and with nearly 400 people involved. Our “train” ran 6 two-week iterations, including a 2-week IP sprint. This was our sixth PI and to date, and while the individual teams conducted team level retrospectives, there had not been an overall review of how the train(s) worked together. As the trains were growing rapidly beyond what heroic ad hoc problem solving could resolve, we decided it was important to start systematically “reflecting at regular intervals how to become more effective” and began planning a SAFe problem solving workshop.
3. NO MOTHERSHIP
The SAFe problem-solving workshop is part of the SAFe Inspect and Adapt event. General guidance for the problem-solving workshop is that it is about a two-hour process, where all members of the ART participate. This creates a fantastic opportunity for people to collaborate with others beyond their immediate team members. There is an implied assumption that everyone is in the same room. This, of course, was totally impossible for us, unless we wanted to fly everyone to corporate head office in the US.
A typical solution to this distribution problem is what we sometimes referred to as the “mothership” approach. We could hold the problem-solving session at the head office – the mothership – and use video collaboration tools like WebEx or GotoMeeting or Zoom to engage everyone else. Unfortunately, this approach was most likely to only give us a North American point of view and not a true global view. We wanted to avoid a North America centric problem-solving session for as one plucky Australian noted, more than 50% of the value of the train came from outside of North America. Experience suggests when there is a face to face mothership style meeting with other members engaging online, the online members are not engaged and are at best lurkers.
Conducting a “mothership” problem solving workshop, could have reinforced the perception that head office was the center of the universe as most of the senior staff such as the RTEs, Program Managers, Architects, were located there. Finally, scheduling a single “mothership” session is not respectful of people because we would be asking a fair portion of the train to participate in the middle of their night. Therefore, we did not want to conduct a “mothership” style of problem-solving workshop. We needed an approach that created the same opportunity for everyone to participate.
4. EVERYONE ONLINE
While co-location and face to face conversations are much touted in the agile community, the reality of large-scale systems development is that many people from around the world collaborate to create those large systems. The Agile Principles were written nearly 20 years ago when collaboration technology was at its infancy. Ideally teams that must work closely together are physically close together, but they still need to interact with their global colleagues. Online collaboration is a fact of life and modern tools offer a fair approximation of a physical face to face meeting. With the decision made to conduct the problem-solving workshop online, the next issue was determining how to run the meeting on a program with a never setting Sun.
5. AN AGENDA FOR A GLOBAL WORKSHOP
SAFe outlines a six-step agenda for the two-hour problem-solving workshop:
- Agree on the problem to solve
- Apply root cause analysis (5 Whys)
- Identify the biggest root cause using Pareto analysis
- Restate the problem for the biggest root-cause
- Brainstorm solutions
- Identify improvement backlog items
It was apparent that we were not going to execute this agenda as a two-hour workshop, at least not if we wanted the entire train to actively participate. Instead, we devised a 1 week rolling agenda:
- Dec 12th by this date the teams are expected to have conducted a “mini retrospective” identifying what each team sees as the program level issues.
- Dec 13th Publish and collate Issues discovered during the mini retrospectives.
- Dec 13th Vote on the published issue list to select the top 5 issues.
- Dec 14th Schedule the problem-solving workshop published and name the facilitators.
- Dec 17th Conduct problem solving sessions
- Dec 19th Present a summary of the workshop
5.1 Step 0: Train the Scrum Masters on the Process
We were relying on the Scrum Masters to “fly solo” and work with their teams to facilitate the event. Thus, we trained our Scrum Masters with the intention behind the SAFe problem-solving workshop, our multi-day rolling agenda, and their role in making it happen. This was a two-hour training session with the agenda dates and activities.
5.2 Step 1: Agreeing on the problem to solve.
Step one in the SAFe problem-solving agenda is to come up with the three to five problems that are of the highest interest to everyone on the train. The intention of this step is to give everyone in the room a voice. In a text book problem-solving session, everyone is in the same room and usually writes issues of concern to them on a sticky note. These are posted on a board and everyone dot votes on the top five or so issues. Groups of people with a common interest can then collaborate. This creates a wonderful opportunity for greater social cohesion because people can collaborate with others who share a common interest rather than just their familiars.
Unfortunately, or fortunately, depending on your point of view, corporate IT is conservative While there are numerous cloud-based shared document tools, access to these tools are blocked through the corporate firewall due to security concerns. While this is often annoying, as one IT manager once remarked “we haven’t been in the news, and we don’t intend to be” Conservatism certainly has its virtues, but we needed the equivalent of an electronic flipchart. Fortunately, the organization used Microsoft OneNote which worked quite admirably for us.
Instead of writing issues on post-it notes and sticking the notes onto a flip chart sheet, the Scrum Master worked with their team to capture in Microsoft OneNote the issues the team believed were impeding progress at the “program level”. In our distributed agenda, we gave the Scrum Master three days to gather candidate issues and get them into a OneNote team page. After the issues were captured by the teams in OneNote, the three authors of this paper consolidated the issues and created a list of 20 program level issues. In retrospect it may have been more appropriate to have the teams themselves perform an affinity mapping exercise to consolidate the team issues. However, in our opinion at this time, this would have been a significant coordination effort with very little gain.
For the dot voting, we used PollEv.com and asked people vote on their top issues over a 2-day period. PollEv.com enables people to respond to online questionnaires using either their mobile device or desktop computer. We ran a quick spike to test PollEv.com to create familiarity with the tool by asking people to vote for their favourite science fiction movie. The poll response was at best disappointing, only 20 people responded to the poll or about 5% of the train. While we were disappointed by the lack of interest, we were also thankful that nearly 400 people were not eagerly waiting to collectively jump into the workshop.
Despite the low polling response, this problem identification step was an important step for us because the problems raised were the problems the teams were experiencing and not necessarily the problems program management at the mothership thought were relevant. Without this step, we would have had a very limited view of the problems the widely-distributed teams were experiencing.
The top 5 problems identified were:
- There is no visibility for which team owns certain features (e.g. monitoring and alerts). This has led to duplication of work.
- Dependencies between teams are not clear during sprints.
- Lack of team objectives and identity make it hard to understand what a team does.
- Compliance activities take a long time.
- How should support be structured for cloud migrations?
The benefit of this step was these issues caught head office – the mothership – a bit by surprise. For example, head office had good visibility into team ownership of features and therefore assumed that of course the teams must also have good visibility. By giving voice to all members of the train, we were able to draw attention to a real problem that was not on the management radar.
5.3 Steps 2 to 5: The Workshops
In the textbook version of the problem-solving workshop, after agreeing on the problem to solve, the group immediately rolls into the root cause analysis. That is the benefit of co-location and face to face communication: rapid decision-making action. Distribution across time zones, unfortunately, extends decision making time because of the coordination delays. It took us 3 days to get set up for the root cause analysis. The first day was spent setting up and verifying access to our pages in OneNote. The second day was spent scheduling the workshops. The third day was used to conduct the training to prepare the participants for the workshop.
Scheduling the workshop was at best a compromise between having the whole team present and respect for people. A consequence of having a program on which the Sun never sets is if we wanted to create the opportunity for everyone to simultaneously participate on the issue of their choice, then someone was losing sleep. This is not showing respect for people. The best compromise we came up with, was to schedule three, two-hour workshops throughout one day: one at noon central time (GMT-6), one at 6 pm central, and the final one at 10 pm central. While we had started with 5 issues, we reduced our list to the top three because we did not have enough facilitators to cover 5 workshops.
The intention behind our scheduling was to have at least one workshop scheduled for a time that someone could attend that would be reasonably convenient for them in their time zone. Of course, the topic for the reasonably convenient workshop may not be of interest to the participant. In addition, for someone who had a keen interest in a specific problem that was scheduled at an inconvenient time may have to choose between sleep and collaborating. Not ideal, but at least that would be their choice.
We continued to use Microsoft OneNote as our collaboration tool. In a OneNote document we created three sections, one for each problem and set up the SAFe fishbone diagram for each. OneNote allows multiple individuals to simultaneously create and edit content on the page; very much an electronic flip chart. The workshops were conducted in WebEx and we had two facilitators per workshop. One facilitator was the “driver” actively engaging and facilitating the session, while the other was the “navigator” keeping an eye on the chat window and engaging with individuals through chat.
Figure 1 SAFe Fishbone Diagram
Participation was voluntary for this first problem-solving session because we only needed to validate whether our agenda and tooling worked. While we were disappointed by the low participation rate of 20-30 per workshop, we were also grateful that we did not have to facilitate an interactive online workshop with 100+ people in it with our initial attempt in combining all the different technologies.
We timeboxed the root cause analysis to 20 minutes. Participants were initially a little hesitant to engage with the fishbone diagram but that is what the facilitators are for: to help participants move out of their comfort zones. Soon, issues began to, almost magically, appear on the shared page. It was fun to watch as participants engaged in the root cause analysis.
After root cause analysis, we moved to the next agenda item – identify the biggest root cause. We identified the biggest root cause by requesting participants “dot vote” on the fishbone diagram and simply place an “X” on what they believed was the biggest root cause. This was in a word, messy. It would certainly have not work well if we had a large group to work with. For future workshops we would have to transcribe the analysis to another OneNote page for the dot voting.
Once we identified the biggest root causes, we moved onto re-writing the problem statement. The SAFe training materials remind people that a problem well-stated is a problem half solved. In one workshop, the original problem “lack of team objectives make it hard to understand what a team does” was re-written as “I don’t know what other teams are doing and therefore I do not know who I depend on and therefore who I need to talk to” As facilitators, we probably overstepped our boundaries: rather than asking “powerful questions” we almost took the wheel ourselves. It is one thing to ask people to post their thoughts on a fishbone diagram. It is quite another to get people to collaboratively write a statement online. Part of our motivation to “grab the wheel” was to get something done within the timebox. This behaviour on our parts is something we will have to be more cautious about in future. We also took note that future participants will be more familiar with the process and will hopefully be less hesitant to participate.
After restating the problem, we moved to the next agenda item and brainstormed solutions. We simply used a blank page in OneNote to let everyone write their solutions, and then we followed up with a dot vote to pick the actions for us to take. These actions were either implemented as new “working agreements” or added to the program backlog:
- Establish a regular meeting between business owners and their POs where the business owners can make their goals clear to the PO
- Highlight the team’s objectives and benefits during PI Planning
- Scrum Masters add their team objectives to their team descriptions in MS Teams
- Build and maintain a SAFe program board
A day after the workshop we consolidated the contributions and outcomes in the problem-solving workshop page in OneNote and broadcast a summary to all members of the train.
6. LESSONS LEARNED
This experience highlighted the importance of the problem-solving workshop and creating an opportunity for all voices to be heard. This was the sixth PI for these trains and yet this was their first problem solving workshop. The workshop revealed problems that the members of the trains were experiencing but were not on the management radar. Even with the best of intentions, on a very large distributed train, it is all too easy to become disconnected from the needs of the far-flung teams. This problem-solving workshop is a massive opportunity to mitigate this “mothership” syndrome. Our experience demonstrates the value of a globally distributed problem-solving workshop that creates equal opportunity for all voices to be heard.
While we were able to validate our global agenda, the next lesson learned is running a highly distributed workshop is a significant logistical undertaking. Potentially two orders of magnitude more planning than a comparable co-located workshop. The logistics for running the workshop had long been an impediment to scheduling the workshop. For a large distributed train, there will be considerable effort required to prepare and coordinate all teams around the globe. SAFe suggests the workshop only requires two hours. It took us over a week to plan and execute the workshop. One person was almost fully dedicated to this effort. The price of a large distributed team is an order of magnitude increase in both coordination effort and coordination delays. The value in learning what is really impeding work can be priceless.
Some other lessons learned:
- Surprise – a large logistically complex workshop will not happen unless leadership drives it.
- People do not mind losing sleep to solve a problem if the problem is of interest to them and it is their choice to participate or not.
- The problem causing the teams the most pain are often not what management thinks are the problems causing the teams the most pain.
- Managing the logistics of a globally distributed workshop are easily an order of magnitude more time consuming and complex than running a local face-to-face workshop.
- Even primitive collaboration tools can help you run a distributed problem-solving workshop(s).
- People require additional training ahead of time to run an effective distributed problem-solving workshop
Was it worth it? Yes, for if the Sun never sets on your program, then you owe it to everyone in the program to discover what their concerns and issues are and not what the mothership thinks they are.
7. ACKNOWLEDGEMENTS
We would to thank Lise Hvatum our shepherd whose guidance and recommendation was greatly appreciated. Also we wish to express our gratitude to Rebecca Wirfs-Brock for her support and help.
