In this conversation, Agile Experience Reports Podcast Host Rebecca Wirfs-Brock talks with Sarah Baca. They dig into the challenges in determining whether scrum masters are doing a good job. What exactly does it mean to do a good job in a highly contextualized role?
In my current role at Express Scripts, I serve as one of the people leaders for our 100 Scrum Masters. Since I began this role, the biggest challenge in my mind was: How do we know our Scrum Masters are doing a good job? And what does it mean to do a good job in our highly contextual roles as Scrum Masters? In this experience report, I will share how we are trying to answer this question, and how it is evolving as we learn more.
In my role as a people leader, I want our Scrum Masters to be as effective as they can be so that our teams can function well and create software that helps our patients. But how do we know if a Scrum Master is doing a good job? What does that mean in our roles when so much depends on the dynamics and personalities of team members, the leaders in other organizations, and the established systems around the teams?
At Express Scripts we have about 100 Scrum Masters and 20 agile coaches. I’m one of the four people leaders who lead the Scrum Masters. When I first came into this role I questioned how I could bring value to my team of about 20 people. A manager’s role is rarely defined in most agile literature, and a Scrum Master manager most definitely isn’t. I’ve always been passionate about motivating and supporting people. So how could I translate that to my new role? That has been an interesting and sometimes painful lesson that I continue to learn.
3. YOUR STORY
When I was a Scrum Master, an important part of my job was understanding where my team was at, the next step to coach towards, and how to keep track that progress. If I understand the team’s pain, it helps me develop empathy so I can work with them as we grow together. In my role as a Scrum Master people leader (SMPL), it’s my job to support and help other Scrum Masters and their teams through that same journey. I also apply the same skills at a higher level, with the leadership of other groups, such as Engineering and Product Management.
I work with three other SMPLs, each of us having 20-30 contractors/employees on our teams. The four of us work together closely, bouncing ideas off each other and challenging each other. Any good ideas in this report came as a result of our collaboration.
When I first started in this role, I thought it was my job to measure if Scrum Masters were competent enough to do their jobs. I still believe it’s my responsibility, if it becomes clear that someone isn’t serving their team effectively, to work with the team to solve that problem. That sometimes includes ending a Scrum Master’s time with us, and the difficult conversations that are part of that. However, I’ve learned now that if I can support, encourage, and focus on creating a system where Scrum Masters can be successful, I byproduct is that I grow to understand their competency level. And by approaching it that way, I can help them grow, whereas if I am solely judging them, I’ve closed off any trust that might cause them to open up to me.
A tangential topic to Scrum Master competency is team maturity. Our company has gone through several cycles of trying to measure team maturity. As I was learning that judging Scrum Master competency wasn’t helpful, I also learned that it’s not helpful to judge team maturity. What is helpful is having a common framework to track how we can best help the teams over time, which also makes Scrum Master competency and team maturity apparent. If I come from a place of assessing and supporting instead of judging, I am much more helpful. This paper goes into more detail on how I grew (and continue to grow) a deeper understanding of what is helpful to focus on, instead of spending my time and energy measuring and judging those around me.
3.2 Team-Based Assessments
We have tried several different ways to measure team maturity and Scrum Master competency over the last few years.
When our transformation first began, a large consultancy was hired to train the teams. They created a survey for the team to take together that they called the Agile Maturity Assessment (AMA). The entire team was expected to take the survey, it took an hour or two to complete, and they were expected to take it every two weeks.
The AMA was in Excel and had a scale for each item.
Table 1. Agile Maturity Assessment Scale
There were both foundational and advanced categories for the questions. The team would go through each and rate themselves for each item and Excel would average the totals for each category. Items included topics like: “at least two sprints worth of work is in the backlog” and “team is not creating separate stories for dev and QA in their backlog.”
Some benefits of this survey were that it made it possible to assess maturity over time. It also made it clear what “good” looked like, so the team knew the desired state, based on their interpretation of the category. The team was able to have conversations while going through the AMA about how to get their score higher and what behaviors might be necessary to get the team to a level five.
There were several drawbacks to the AMA. When team members rated themselves, they might not feel safe to be honest if there were negative consequences for lower numbers or for not moving to higher numbers quickly. The team members also might not know what good really looked like and might interpret the sub-category differently than an experienced coach would. It also took several hours of time from the entire team, making it a very expensive assessment.
Back when I was a Scrum Master, I would sit with the members of my two teams and take this assessment. Often we would cheat and I would do it with the development leads representing the team because the team felt like doing the assessment was a waste of time. Of course, the survey couldn’t cover everything, so there would be areas where I knew the team needed to grow, even if we looked like we were high-performing according to the scale. The team would grumble and complain about all the time they lost when they did take this assessment, and I got the impression that they felt like the agile transformation office didn’t care about them—only what scores they got. Our leadership did care, but all they knew about our leadership was that they wanted the teams to take this survey.
Our next iteration of the assessment was called the Agile Health Assessment, and was based on Modern Agile (Joshua Kerievsky). This assessment was taken individually and there were only fifteen questions, so it took much less time to complete. Also, team members completed this assessment individually and when it was convenient for them, versus a long meeting.
This is the survey as our team members saw it:
To continue in our agile transformation, we need your thoughts and feedback. The Agility Health Assessment (AHA) does not rate your skill level or team competency. Instead, it serves as a forum to express how you feel about our progress in our agile journey. The data from the AHA is aggregated to show how the team members feel about team agility.
We know your time is valuable, so the survey is only 15 questions long and it is supposed to be taken individually. We take personal privacy seriously and have done our best to aggregate the data. Thank you for taking the time to respond openly and honestly.
Let’s get started…
Please answer these questions from your experience and perspective in the Business Portfolio and Technical Domain. You may be asked to complete this survey more than once, to keep these combinations separated.
Options are: Always; Most of the time; Sometimes; Rarely; Never
- We understand clearly what we will work on next and why:
- We recognize good work and celebrate success:
- We actively experiment with our product, process and craftmanship to improve our solutions and our team:
- All team members express their opinions/concerns:
- We measure the impact of WIP limits on our cycle time and adjust accordingly:
- We adjust and abandon opportunities and solutions upon feedback:
- We trust each other:
- We make everyone around us awesome:
- We frequently deliver value to our end users:
- We are motivated by our work and have the opportunity to improve our craftsmanship:
- We reflect on how to become more effective then tune and adjust our behavior accordingly:
- We are not afraid to fail and we learn from our failures:
- We can change our code easily and safely when a new change is requested:
- We can describe how our work improves the lives of ESI patients and clients:
- We make informed decisions upon listening to different opinions than our own:
We Value your feedback! Please share your comments below…
The results were exported to a spider graph for easy viewing.
Figure 1. Agile Health Assessment Spidergraph
The challenge with this survey was that the team was still assessing themselves, and many of the team members didn’t know what “good” looked like. We saw teams rating themselves high on categories like collaboration, when Scrum Masters and coaches were seeing siloed behavior.
The teams took this survey after I had moved to a leadership role, so I didn’t take it with any teams. But I did talk to my Scrum Masters about their teams taking it, and even though it was quick, they had a really hard time getting their teams to take it. The best solution they found was getting teams to take it during retrospectives, which of course meant they were losing time they would have spent reflecting on needed team improvements. Because it didn’t give an accurate representation of how the teams were doing, the Scrum Masters were frustrated. They still had no data to use for knowing how best to help the teams.
Because the team members didn’t know what “good” looked like the survey showed teams were very mature when we knew they weren’t. We needed a better solution.
3.3 Assessment Based on the Twelve Agile Principles
During this time, some team members suggested a report card, so teams could be compared and measured. Our leadership was able to help those team members understand that a report card and comparisons aren’t helpful, but did ask us to create a way to assess the teams so we could show growth over time.
One of my peers, Wendy Bennett, had already been using an assessment with her team using red, yellow, and green and the twelve agile principles. We used this as the foundation to create the next iteration of an assessment. We called it the Twelve Agile Principles (TAP) Assessment. We created guidelines and asked Scrum Masters to measure their teams in green, yellow, red, with an arrow trending up (getting better), sideways (staying the same), or down (getting worse). By having the Scrum Masters fill out the TAP Assessment, we could determine the understanding level of the Scrum Masters—such as what kind of evidence they used to rate a category green versus yellow. We also could discuss with the Scrum Masters how they planned to teach and coach their team to get to green. This exposed information about their skillset while also giving us a way to assess the teams.
During our review of the TAP assessments with the Scrum Masters, we were pleased to discover that the conversations were even richer than we had thought they would be. Where before conversations with our Scrum Masters might turn into a status report or a gripe session, now the conversations were focused on the behaviors of the team and how the Scrum Master was serving the team to help them be more effective. Some Scrum Masters might say that their team was green, but when we discussed the evidence it was clear that the team wasn’t mature enough to be green in that category. This became a teaching moment with those Scrum Masters, and gave us an opportunity to encourage Scrum Master to continue coaching their teams to become even better.
One thing that surprised us was that some of our Scrum Masters did not initially feel safe to share when their team was struggling. They would fill out the TAP assessment showing the team was all green, even for teams that we knew never had face-to-face conversations or never spoke to their customers. This gave us a great opportunity to ensure our Scrum Masters knew that we valued honesty over perfection, and that our role as leaders is to support them and their teams, not to judge or punish them.
Those conversations started to improve the system around the Scrum Masters. We began to realize that we were more effective when we work to “improve the system, not the rules, nor the people. When you set the right constraints, rules and people will take care of themselves. .” (Appelo, p. 246-7).
Jurgen Appelo suggests the following guidelines for setting those constraints (p. 246-247):
- “Allow constraints for competence to emerge through positive feedback loops.” The TAP assessments have multiple positive feedback loops. We review them quarterly with the Scrum Masters, the Scrum Masters can review them with the teams, and we can review them with our leadership. During these reviews we are soliciting feedback from multiple stakeholders, potentially improving with each loop. We also meet to review them at the sub-domain level, to find patterns and give an opportunity for Scrum Masters to collaborate when they are experiencing similar challenges.
- “Introduce memeplexes instead of individual ideas to accelerate the adoption of good practices.” Our TAP assessment created a standard way to make sure we were all using good practices for our conversations, rather than talking about random topics.
- “Allow people to have ‘barely enough’ competence levels in some areas so that they can focus on things they are good at.” By going through the TAP assessments, we were able to determine where the Scrum Masters were not as strong, and connect them to others who were strong in those areas. This spread knowledge, created opportunities for collaboration, and allowed people to do what they were best at.
- “Big problems start as small problems. Minor carelessness ultimately leads to total quality disasters. Don’t spend your time only on big problems because you are allowing the small ones to become big as well.” The TAP assessments helped us discover challenges before they grew larger. They also helped us see systemic challenges more clearly. For example, if every Scrum Master in a sub-domain was trying to work with a challenging engineering manager, we could coach them all in how best to give feedback and help them escalate to leadership if that was necessary.
We haven’t found the one true way to assess our Scrum Masters and their teams. But I do think that using these elements to create a framework for a conversation has been highly valuable. The colors and arrows are helpful for tracking progress over time, and for sharing information at a macro level to our leadership. But the true value are the conversations we have with our Scrum Masters, and the focus of all of us to support the teams as they own their growth.
3.4 Assessment Results
Our Scrum Masters are now using the TAP assessment to have conversations with us and their teams. By combining the TAP assessment with other engineering data, our Scrum Masters have several powerful tools to reference in conversations with their teams. It’s important that the teams grow to understand and own where they are with the TAP assessment. Their Scrum Master can then use the patterns in that data over time to show trends and support them as they grow.
The difficult thing is no one metric that can tell us if we’re “mature” or not. It’s too contextual and too complex to measure that way. But by using the TAP assessment over time and combining that with engineering data, such as cycle time, we can get an idea of how the team is growing, how the Scrum Masters are helping, and what direction our organizational growth is going.
The red, green, and yellow arrows help us follow that growth over time, but they are not the highest value—the highest value comes from the conversations and discoveries that emerge.
In addition to the TAP assessment, we have a few other ways that we are gathering other qualitative and quantitative data to track and encourage growth.
3.5 Coaching Plans
As we review the TAP assessment with the Scrum Masters, each Scrum Master creates or updates a coaching plan. They maintain their plan in between TAP assessment reviews with their people leader, and use it to track how they are helping teams over time.
The coaching plan has a column for:
- What behavior the SM is seeing (evidence)
- What beliefs the team has that is causing this behavior (inference)
- What behavior the SM thinks should be happening (to check knowledge)
- What the SM will be doing/coaching toward to get closer to what they think should be happening (impact)
The most important elements of these coaching plans are evidence, inference, and impact. This follows the clean feedback model, created by Caitlin Walker.
3.6 Servant Leadership Survey
We combine this with another tool that we started using, a survey that goes out to the team members. We call this survey the Servant Leadership Survey and send it out quarterly.
These are the questions on the Servant Leadership Survey. All response options are: Strongly agree, Agree, Neutral, Disagree, and Strongly Disagree.
- My Scrum Master promotes a culture where my voice is heard and it’s okay to disagree.
- Team members feel safe to speak up during regular team check-ins (stand up, etc.) without prompting.
- The entire team has the opportunity to become familiar with the work before they start working on it.
- I would recommend this Scrum Master to another team (we promise we won’t move them!).
- My Scrum Master encourages me and my team to grow and continually improve.
- My Scrum Master raises awareness and facilitates resolution of impediments, encouraging the team to take ownership as appropriate.
- My Scrum Master encourages collaboration across roles and across different teams.
- My team has a regularly-updated working agreement that is owned by the team.
- My Scrum Master protects the team while inviting them to advocate for themselves.
- My Scrum Master makes suggestions rather than telling the team what to do.
We know that this survey has similar problems to the other surveys—different understanding of what each phrase means, a possible fear of recourse if they score negatively, etc. We decided that risk was worth it, as this is a simplified anonymous 360 for our Scrum Masters, and we want that feedback even if it’s not perfect.
Understanding the results of the servant leadership survey isn’t a simple process for the Scrum Master people leaders. Sometimes a team will love their Scrum Master because the Scrum Master doesn’t challenge them. Sometimes a team will dislike their Scrum Master because the Scrum Master has just started challenging them, and they’re don’t like it. We use this as another feedback loop during our one-on-one conversations with our Scrum Masters, so we can help them grow and learn. We also use it as a feedback loop from the team to us and the Scrum Masters, so we can catch challenges before they get bigger.
3.7 Communities of Practice
We also started several Scrum-Master-led communities of practice. These are small groups (called guilds) that the Scrum Masters use to learn from each other. Currently we have a coaching guild; a “stump the coach” guild, where the Scrum Masters interview coaches or another Scrum Master; a Kanban guild; a technology guild; a clean language and systemic modeling guild; and more that pop up when people become interested in a topic, and dissolve when interest in that topic wanes.
All of these methods give us more information so we can support our Scrum Masters and teams. This creates a better organizational culture and it also helps our teams so we can help our patients.
4. WHAT I LEARNED
While I was working on understanding how to assess the teams’ maturity and scrum master competency, some of our coaches started teaching a leadership course. Part of this course included a coaching session with a co-active coach, Jennifer Davis. She asked me to coach me in a short coaching demo at the front of the room and I agreed. During this session, she kept digging into why I felt that I needed to measure my team. Why did I feel like it was so important to assess them? What was I hoping to gain by measuring where we were in our journey? What exactly was I trying to accomplish? It was then that I realized that the value of those assessments went deeper for me. What I was trying to prove wasn’t really about my team or our agile journey. What I was trying to prove was that I had value as a leader and that I knew what I was doing. By trying to judge others, I was asserting my superiority and establishing my worth. The entire course opened my eyes so that when it was time for that coaching session, I was able to see what had been in front of me all along.
If I could release that need to prove that I was a valuable person and that I was somehow better than everyone else, I could spend that time and energy on supporting a system that serves our Scrum Masters. If I could focus on the conversations from the TAP assessment instead of seeing it as a way to judge team maturity, I could spend my time helping Scrum Masters decide on next steps to help the team. When I support instead of judge, I’m able to be more effective because my relationships with those around me are stronger. I’m not focused on proving how good I am—I’m focused on helping those around me to become better.
I hadn’t done that yet because letting down the protection of judging others left me vulnerable for others to see that I might not know what I was doing, and I might not actually bring value. That was scary to me, so I had worked to design elaborate ways of judging others to protect myself. Those protections had left me much less effective and had actually hurt the people I was trying to help. By having the courage to release the need to judge others, I could become the leader I had wanted to pretend I was.
If I had not taken that leadership class, I would have been able to step back during this journey and recognize how I was using this need to judge as a way to prop up my self esteem. Because of that personal realization, I saw my role in a whole new light.
Now instead of focusing on assessing and measuring Scrum Masters, I work with them to use the TAP assessment as a guide to determine where to focus first. We work together with leaders in other groups to try experiments that might help our teams. We ask the leaders in those groups “What challenges are you experiencing? How can we help?” The coaches and Scrum Masters work together to create coaching plans for solving those problems. I meet with the Scrum Masters on my team and find out what challenges they are experiencing and ideas they have for how I can help. I’m bringing more value to them, myself, and my company because I’m approaching challenges from this different perspective.
I am very grateful to the other SMPLs who have been amazing friends and collaborators: Gabe Weitkemper, Wendy Bennett, Tolun Ozarslan, and Rob Flynn. I’m grateful to Rebekah Wallace for being a friend who could talk me off the ledge. Most importantly, I’m thankful to my husband and our six children for being my safety and support so that I could chase my dreams.
Thank you especially to Johanna Rothman, both for shepherding me during this process and for her past leadership and mentorship.
Appelo, Jurgen. Management 3.0. Pearson Education, 2011.
Laloux, Frederic. Reinventing Organizations. Nelson Parker, 2014.
Ancora Amparo. “Caitlin Walker: What’s Clean Feedback?” https://www.youtube.com/watch?v=vHfRzQxXPls
Jennifer Davis Coaching, https://www.jenniferdaviscoaching.com/