Experience Report

Leadership and Agile: Are You ‘In Control’?

About this Publication

Leaders of organizations using lean-agile ways of working on the work floor need to adopt a new way of leading the organization. In this paper, I present a complete and reframed leadership system that allows leaders to gain a different form of being ‘in control’; a form that is compatible with a lean-agile way of working. I also describe experiences and lessons learned while implementing this leadership system.

1.      INTRODUCTION

For many years, the terms Lean, Agile and DevOps have been used to indicate a movement away from the traditional form of IT service management to a form that is more compatible with the speed of change in the current market conditions. These philosophies have proven to be successful in sufficient cases that increasing numbers of businesses are adopting these principles and applying them to their entire organization. This has led to the rise of the term ‘enterprise agility’.

Let me describe, in very broad strokes, the classic evolution that I have experienced on multiple occasions of IT organizations towards a new way of working:

The evolution usually starts with a single team, using a method such as Scrum. This is usually a ‘rogue’ operation, as it is not sanctioned, but rather tolerated (and sometimes encouraged) by leadership. In this case,  team leadership collaborates with the team to try out the new way of working. More senior leaders are often oblivious of what is going on. Since the team is highly motivated to work in a different way, a way that is more fun and which produces great results, the change is generally a success. Other teams follow the example of the first, often with similar results. At this point, leaders at different levels of the organization start to become enthusiastic about the potential of the new method. It is often mid-level leaders who see the potential first, as a result of their proximity to the teams and the desire to achieve successes with their teams. They see the enthusiasm in the teams; and enthusiasm often leads to better results. In some cases, even senior leaders get wind of what is going on and start to support the initiative.

The new way of working is ‘pitched’ to senior leadership, who see the change as highly desirable. Who wouldn’t? The pitch is that with the new method, IT can be more productive and more efficient, providing better quality at a similar or even lower cost. It sounds like a golden ticket to the company’s goals.

The ‘transformation’ picks up speed within IT. Unfortunately, the teams whose turn it is to take on the new methodology are less motivated than the first teams, since the later teams are cajoled into the new methodology rather than choosing for it themselves. The whole transformation starts to become a much more arduous affair. But the leaders have seen the potential and are eager to turn this into reality. Agile coaches are recruited to convert the more obstinate teams, transformation plans are drawn up, the way of working is standardized and documented, massive training programs are introduced, and goals are set for the adoption of the new way of working. All this results in some success, but there are many areas that are still not delivering on the promise.

This evolution towards more responsive and higher performing IT organizations is now well established; teams, even organizations as a whole, have been introduced to the new way of working. Unfortunately, the improvement in performance tails off and the promise of organization-wide agility is not achieved … this leads us to the next phase of the problem: leadership.

2.      Background

The trigger that eventually led to the journey that I describe in this paper was a management team meeting over five years ago. The scene takes place in the IT organization of a financial institution. The management team in question is responsible for large-scale back-office systems in which the financial exposure of the entire organization is calculated and reported. Jim is the head of the department; a department that was fairly traditionally organized with development teams, an operations team and a team with supporting roles, including, for example,  architects and agile coaches.

The conference room was stuffy. The meeting had been going on for almost two hours without a break. Each of the managers was giving an update of the status of their teams. With eight direct reports who all had to give their update, Jim’s meeting was always a lengthy affair.

The meeting followed a fairly predictable pattern. One by one, the managers hooked up their laptops to a beamer. A series of slides followed variously with fine print, others with neatly produced graphs. Jim always paid attention to each of the presentations, because he needed the information to report to the board. He needed to be able to answer all the detailed questions they could ask. He did, however, feel the energy drain as the meeting progressed. He also noticed that the other managers seemed to switch off after they had done their presentation. And there were very few questions from managers to one another. It was always Jim asking the questions.

The penultimate presentation was about to start. It was Eric, head of one of the product development areas. His area had been experimenting with agile ways of working for the last six months, in an attempt to speed up development. The problem was that the change in the way of working was causing interaction problems with the other departments and Jim had difficulty identifying how the new way of working had actually improved anything, despite Eric’s enthusiasm.

On top of that, Jim had recently received complaints from Eric’s primary customer, one of the company’s directors, Shirley, regarding the delivery of updates to the product. The meeting with the customer was still fresh in Jim’s mind; he still felt irritated about the pressure Shirley had applied. During the meeting, Shirley had openly wondered whether Eric was ‘in control’ of his own area. The irritation that Jim felt was the not-so-subtle implication that, if Eric was not ‘in control’, Jim was probably not entirely ‘in control’ either.

Eric started his presentation. He gave an overview of the progress of his agile transformation: the number of teams working with agile, the number of stories that were produced – something he referred to as ‘velocity’, the improvement in morale in his area and a number of other more or less interesting statistics.

Jim thought Eric was at the end of his presentation. Jim asked: “What about the disruption you had last week? And Shirley indicated that she was not happy with the delivery of updates that you appear to have promised?” Before waiting for the answers and before he knew he’d said it, Jim added: “Are you in control of your teams with this agile way of working?” He regretted that last question almost instantly, but it felt too late to take back his words. And, anyway, it was true: Jim wondered whether Eric was ‘in control’.

Jim noticed a change in the tension in the room. All of the other managers, who had been looking at their tablets or phones, turned their heads to Eric. Eric was quiet for a moment. The tension was palpable. Eric fidgeted with the presentation clicker as if he were looking for a new slide. He put the clicker down.

“We have had a couple of issues in the last few weeks. The incident that brought down our product was a problem with the data center. Our server crashed and the failover did not kick in. Maybe Steve [manager of the data center] can elaborate on that one.” He proceeded to talk about how some of the teams had not been able to deliver all of the stories in the last sprints because of the substantial amount of time they were spending on solving a variety of problems, including the recent disruptions. This led to a lower level of productivity than had been anticipated and a few of the important stories that had been planned had not yet been delivered.  He spoke for a couple of minutes explaining the situation and telling the other managers how his teams were still in the process of learning how to deliver in increments and in shorter cycles; he did not address the question on whether he was ‘in control’ or not.

3.      EXPERIENCES

This episode had a profound effect on me, not least because this situation occurred on a regular basis with different leadership teams in different organizations. As a consultant, I am privileged to be party to many difficult moments in organizations. Initially (10-15 years ago), these moments were primarily with operational teams, teams producing the value for the customers of IT organizations. More recently (the last 5-6 years), it has been with leadership teams at all levels of organizations. For definition purposes, when I refer to a leadership team, I mean a group of people who are responsible for but not directly involved with the delivery of customer value or specific expertise; delivery of value is done by operational teams, delivery of expertise by knowledge teams.

After years of working with operational teams, it slowly dawned on me that each team reached a ‘plateau’ in their ability to produce better and faster. We could not seem to help them to gain further improvements, at least not improvements that matched the initial performance growth that we had seen at the beginning. The key question was, of course: what is causing this tailing off of performance improvement?

At the same time, I started to experience more and more of the situations described above. Could there be a link? A recurring discussion with leaders of all levels centered on the difference between Lean, Agile and DevOps. There is still – over a decade on since these words became commonplace – confusion among leaders about these terms. After many interviews and discussions, it turns out that the difference is perceived based on the use of different ‘tools’. Taking a brief look at Agile, we see that Scrum is a set of prescriptive tools that solve common problems in software development environments. For example, a Product Owner is a solution to a problem: prioritization. Are there other ways to solve this problem? Certainly. Lean uses different tools than Agile, for example, Lean uses week and day boards, Agile uses Scrum and Kanban boards. In general, Lean tools have a more diagnostic nature, when compared to the prescriptive nature of Scrum. DevOps has a greater focus on automation tools; DevOps makes use of Lean and Agile. Throw these three terms (Lean, Agile, DevOps) into the mixing pot called ‘an organization’ and you have a recipe for confusion.

The solution for leaders to this confusion turned out to come from a lesser focus on tools and a greater focus on what we call ‘practices’. More about that later.

3.1      Problems

In my interactions with leaders and leadership teams within IT organizations, I started to ask more pointed questions about the relationship between the leaders and the new (‘agile’) way of working. The answers surprised me. On the outside, leaders were enthusiastic about becoming more agile. In private, or behind closed doors, leaders and leadership teams were more candid and skeptical.

There were three problems that were becoming apparent to leaders:

  • Where has my work gone? In the move to agile teams, we have delegated many tasks that were previously the domain of the leader to the teams. Planning the work, sharing the progress information, deciding who does what, measuring performance, giving feedback on performance, and so on. These were all tasks that filled the days of the leaders, but are no longer there; they are done by the teams themselves.
  • What is my job? The result of the introduction of agile into organizations has been an existential threat to many leaders. In many organizations, managerial layers have been removed “because the teams can manage themselves”. This has changed the nature and even the availability of work for mid-level leaders.
  • What is the path to leadership? Organizations used to have integral management, i.e. leaders were responsible for all aspects of their team/department/business unit, from the top of the organization all the way to the team leaders. Now, we have Product Owners (PO) for content, Scrum Masters (SM) for the ‘process’ and a people development role (PD). The team leader used to be responsible for all of that. Splitting the team leader role may not be a huge problem, except … how does someone move from a PO/SM/PD role into an integral management role? Do they have to do all three roles first before they move up or are we going to split the roles all the way up the hierarchy?

A complication I encountered was particularly related to the role of agile coaches. Many agile coaches encourage leaders to ‘let go’; let the team do their work without the leader getting in the way. This leads to frustrated leaders. A common response is “Stop telling me to ‘let go’; you [agile coach] don’t understand my responsibility. I can’t just let go. I have to manage this [organizational unit]. I am responsible for ensuring that the team works smoothly and delivers what is expected.” Here, again, the desire to be ‘in control’ is a driver for leaders to resist ‘letting go’. This is where leaders need to understand how they maintain their control, but in a way that is more compatible with the agile organization.

And this brings us to the heart of the problem: what does it actually mean to manage/lead in an agile organization? What is the role of the ‘hierarchy’ in an agile organization? What should people in this hierarchy do? Are there things they should do differently?

3.2      Tackling the Problem

I started to focus on the activities of leaders. I decided to dive into the history of management. Over a century ago, Henri Fayol formulated the activities of management. He used words that evolved to a list that we know well: Forecasting, Planning, Organizing, Commanding, Controlling and Coordinating. With the advent of Lean, Agile and DevOps, these words have acquired a rather sour taste: we don’t need so much commanding, controlling, coordinating any more.

In a corollary of the change of vocabulary on the work floor, I thought it would be a good idea to change the ‘verbal context’; if we want leaders to do something different, then let’s use different words to describe what they need to do. The first step was to define the new words that describe what leaders do in an organization with a lean-agile way of working. Working with colleagues and customers (in a number of iterations of a period of weeks), we came up with Visioning, Planning, Cascading, Monitoring, Developing and Organizing. These terms reflect better how leaders are expected to act in the ‘new world’.

The next step was to identify what leaders do within these activities that make leaders successful in the new environment. We refer to these parts of activities as ‘practices’. This led to months (even years) of observation, research, interviews and discussion. At this point, I started to engage with colleagues and customers; did they recognize the problem, too? The rest of the journey involved a wide range of other people. In the rest of this paper, I will use the term ‘we’ to recognize to the myriad of contributors who helped me to hone these ideas.

We started organizing sessions with leadership teams. One of the key questions we asked is: what is the value of leadership? There were two basic responses. The first was where the leader saw the end-customer as the customer of their leadership. The second was where leaders saw the team members as the customer of their leadership. In each meeting where this dichotomy of responses occurred, the leaders ended up discussing together how they should look at their leadership. Invariably, it led to agreement that the value of leadership is primarily focused on team members and peers. Having gained this common ground, we started to have discussions about how this value could be shown and put into action.

Having asked leaders what their value is, we asked them to take their calendars from the past few weeks and work out how much time they actually spent on activities that represent their value. This turned out to be a fairly confronting exercise. The first pass view of the calendars revealed often less than 50% of time was spent doing things that could be classified as ‘the value of leadership’.

The key lesson learned was that by being aware of the value of leadership, it is in fact quite easy to turn what appear to be unproductive meetings into value-driven meetings for leaders.  In large organizations, particularly mid-level managers will recognize that they sometimes need to be at meetings just to ‘show their face’, without having a significant contribution on each occasion. A strategy for these meetings is to look at the value of leadership and apply it in the meeting. An example of the value of leadership is ‘coaching and developing others’. In a ‘show-your-face’ meeting in which one of your peers or subordinates has an active role, the leader could focus on whether this person is successful in their interactions during the meeting, and give positive or constructive feedback after the meeting. In this way, the leader becomes actively engaged in the meeting.

Based on experiences of successful (and less successful) transformations, we started to build a picture of the elements that were important in each of the activities described earlier. We ended up with a rather long and unwieldy list of tools that leaders could use. These tools were extracted from common models and frameworks, for example Lean, Scrum, DevOps and SAFe, but also from ‘cool’ tools, like Earning Capacity Analysis or Dependency Mapping,  that were developed during transformations at specific organizations. Interestingly, there were a number of heated discussions on which was the best tool; a completely senseless activity. We settled on defining ‘practices’ within which tools can be selected.

As with every system, there are important (‘keystone’) practices and supporting practices. Take a car as an analogy, it is a system that allows you to drive from A to B. Important system elements include the engine, the wheels and the steering wheel. Supporting system elements are things like the air conditioning, the set heating or the navigation system. They help to make the experience much better, but if they are not present you can still get from A to B. In the ‘Results’ section of this paper, you will see the full model with its keystone and supporting practices.

Having built the model, we started enthusiastically showing it to leaders. It was met with almost unanimously positive initial responses from leaders with whom we shared the model. Key comments included that it was nice to have a model that showed an overview of everything that was necessary to make leading a Lean-Agile organization more successful. Also, leaders could see how the combination of practices would help them to still be ‘in control’. And the fact that there is a finite list of things to get sorted out gives leaders a goal to aim for. Even though, as with lean-agile teams, the journey is one of endless continuous improvement.

At the same time, there were some doubts. These doubts all had, in some way, to do with the time necessary to learn and adopt the new way of working. The number of things to get sorted out shows that leadership in a Lean-Agile environment is not a one-hour-a-day task that can be done as an afterthought. There is serious work to be done.

Having built a learning program around the model, we proceeded to work with leaders to build the new leadership system. A key lesson I learned here was to avoid associating the program with the word ‘training’. I came across very substantial resistance to training among leaders. It seems as though there is a taboo on leaders owning up to needing to learn something; the perceived omniscience of leaders (especially senior ones) is still a strong illusion in organizations.

3.3      Achievements

The result was the development of the Accelerative Leadership System, in which we identify the keystone practices are in blue; the supporting practices are recorded in the relevant activity block. (see figure 1.).  It is important to realize that the system works at all levels of leadership; it is primarily the scope that changes through the hierarchy.

Let’s take a brief tour of the model:

  • Visioning is the activity of defining the purpose of the organization and sharing it through change stories; stories that describe the journey to achieve a purpose. It is also about helping leaders to tell their story in numbers through key performance indicators. Lastly, leaders must determine the long-term priorities in strategic planning.
  • Planning is about translating the change stories into actionable work, both customer-oriented and internal. Through a long term-plan, leaders must guide the organization in describing the backlog of work and ensuring that the prioritized work fits the available capacity.
  • Cascading refers to the importance of sharing and disseminating information rapidly both horizontally and vertically throughout the organization, making use of visual management and synchronized meetings. Leaders develop their ‘obeya’ (leadership visual management) to ensure they have an overview of what is happening in the organization.
  • Monitoring practices focus on leaders gathering first-hand progress information and helping to solve the immediate problems that cause work to cease flowing through the organization. Leaders ‘go and see’ what is actually happening on the work floor or in mid-level leadership teams to better understand the challenges. They also encourage daily improvements (‘daily kaizen’).
  • Developing covers the activities that ensure people become more valuable to themselves and to the organization, primarily through working together in teams, solving challenging problems and finding ways to do work in a better way. Leaders also use delegation to help people to develop.
  • Organizing is about ensuring people are collected in the right teams to ensure work flows. Operational teams produce the value. Knowledge teams come in two sorts: 1) what we would previously refer to as ‘staff’ (e.g. HR, Finance, Legal) and ‘centers of expertise’ that develop a capability (e.g. architects, production engineers). Leadership teams are teams of leaders steering the organization; they may lead operational teams or knowledge teams, or a combination of both. Key to this activity is leadership’s understanding and active management of value streams. Dependencies between organizational units are often roadblocks to flow. Leaders seek these out and help to resolve them through organizing teams differently.

Figure 1. Accelerative Leadership System

[Note: the size of the blue areas does not indicate relative importance; all keystone practices carry the same weight. The model includes some indicative arrows to show connections between the various practices. The practices are much more connected to one another than is shown. However, visualizing all of the connections between the practices reduces the usability of the model]

The model works because all of the components (‘practices’) are connected and enhance each other. You cannot ‘cherry-pick’ this model, in the same way that you cannot cherry-pick the essential parts of a car; if you leave out the engine, you won’t get anywhere; if you leave out the doors, it won’t be safe, and so on. You can, however, choose what you want to work on first, you will eventually get round to all of the practices, anyway.  This is because of the systemic nature of the model, one practice carried out poorly will affect, even degrade, the entire system. One of the interesting by-products of the practices is that they encourage leaders to take a step back and admire how others contribute to the success of the organization; the practices encourage leaders to show respect (a core lean principle).

In the model, there are in fact only two practices in which the organization must choose a standard for the whole organization: Backlog Management and Problem-Solving. To ensure that work is planned and coordinated in an efficient and effective way, it is vital for the whole organization to use a common process / method.  A method like Program Increment planning is a good option. Lean-oriented organizations may work with Hoshin Kanri. And methods like Objectives and Key Results (OKRs) or Quarterly Business Review (QBR) work equally well. Any of these methods used consistently brings clarity and speed to the process of defining the goals, determining priority and resolving planning conflicts.

The same counts for problem-solving. If you want to solve an organization-wide problem, you need the people involved, who come from different parts of the organization, to get on with solving the problem as soon as possible. The way to do this is to have a common problem-solving method. The best example of this is Toyota, who have their own 8-step method known and, more importantly, used by everyone in the company.

There are many problem-solving methods, for example DMAIC, Kepner-Tregoe, PDCA or the Improvement Kata. I mostly recommend DMAIC because it is comprehensive and fairly intuitive, and easy for leaders to gain a high-level grasp of the intent of and steps in the method. A focused training program can get the organization up-and-running with the method. Unfortunately, successful embedding such a method in the organization’s DNA means that leaders must also use the method (or a slimmed down version of it) themselves (for leadership or organizational problems) and teach/encourage others to use it. This is where the effort tends to stall. The experience I have of leaders using a structured method to solve their problems is quite distressing: more than anyone, leaders have a tendency to ‘shoot from the hip’ without rigorous analysis and understanding of the problem. This is a common example of how leaders often fail to ‘walk the talk’, thereby shooting their own transformation effort in the foot.

One recurring question was: “Where is the decision-making process and the fact that I spend my life coordinating things?” These aspects are embedded in each of the activities and practices. They cannot be distinguished from the goal that you are trying to achieve with each of the activities.

Obviously, the key results are found in the change in behavior of leaders. In many cases, we have seen leaders make huge steps in their ability to lead their teams. One heartening story was a mid-level leader who was seen as uninspiring and a bit dull. He responded well to the model and worked hard at four key things over a period of about six months. He spent many hours working on his change story for the achievement of the purpose of the organizational unit, he took a prominent role in organizing and steering the planning process, he worked hard to ensure that all visual management was kept up-to-date, and he made gemba walks to go and see the work floor a permanent fixture on his calendar. This combination of activities substantially warmed the leaders and teams in his department. Was it easy? No, it took a lot of dedication, coaching and practice. Did it get easier as time went by? Emphatically, yes. And his environment started to see him as fun, inspirational and accessible.  These activities took place over a period of about six months, but the effects became visible within weeks. The most important challenge for the leader was to not fall back into ‘old’ behavior despite the fact that this was much more comfortable for the leader and his environment.

The main conclusion is that it is a long journey to change the leadership system of an organization. The key challenge is experiencing that the practices ensure that the leader is ‘in control’ in a different way.  But letting go of the old way and trusting that the new way will help is, to some extent, a leap of faith; a calculated and low risk leap of faith, but still a leap of faith.

4.      What We Learned

We must start by recognizing that changing leadership practices and behaviors is much more difficult than it seems. It is not that leaders do not understand the rationale behind the change, nor do they lack the understanding of what is required of them in terms of behaviors. These are things that can be explained. The key issue is actually making the move to practice and adopt the new behaviors. The difficulty stems from the fact that leaders are faced with a political choice.

A political choice is one in which there is substantial uncertainty and difference of opinion. Unfortunately, this difference of opinion is not always voiced. It is not always clear who is for and who is against. Solving a political problem is a challenge. The leadership hierarchies of organizations are places in which power and influence are major drivers, especially if making the right move can increase a leader’s hold on both. This means that the timing of a ‘move’ can be critical.

As a leader, when should you voice support for agility in the organization? Very quickly, because there is no rational argument against it. But when should you start to work in an agile way yourself? That is a more difficult question to answer. For a start, you need to get the timing right. Too early and you can be perceived as a threat; too late and you risk being seen as ‘behind the times’. Either way, adopting new leadership practices is a tense affair.

And it’s not just middle management that feels the pain. It goes all the way to the top; CEOs are accountable to stakeholders such as shareholders and supervisory boards, who often have more traditional views of how the organization should be run. So even CEOs are caught in a political environment in which changing behaviors is not always easy. As long as there are influential people (C-level or powerful staff departments like finance or legal) who continue to favor the traditional ‘controls’ in organizations, it will always be difficult for leaders to gain the full benefits of the new ways of working.

We have however learned that if leadership teams choose to make the move together, the chances of success increase substantially.

  • First, having a clear but limited set of practices helps to make things manageable;
  • Second, there is safety in numbers; working together with peers to do something new makes it less intimidating and politically ‘dangerous’;
  • Third, you have people with whom you can share your experiences; the others will make mistakes as well and you can have a laugh about them together.

This – continuing – journey has brought me both compassion and frustration regarding leaders. My frustration has very much to do with the limited desire to actually learn something new about leadership. It seems that, certainly, leadership veterans believe they know everything they need to know.

At the same time, I have substantial compassion for the current situation of leaders. We are in one of the largest upheavals of the last century: the world is moving from the ‘traditional’ mass production way of working to a lean-agile way of working, essentially this means changing the paradigm from inside-out (“we produce, you buy”) to outside-in (“you tell us what value you are looking for, we find out how to produce it”).  This is a journey of exploration and discovery in which no-one really knows what the new status quo will look like, or even whether there will be a new status quo. Leaders need to find a new control equilibrium rather than maintain control as they did in the mass production world.

Finally, it turns out that if you, as a leader, combine the use of the system described and carry out the practices showing respect for people, the organization will give you more control than you ever thought you could have.

5.      Acknowledgements

This experience paper is the result of many years of interactions with leaders in organizations that were in the process of adopting a lean-agile way of working. Each leadership team with which I interacted has in some way contributed to the development or validation of the model, and I am extremely grateful for the opportunities afforded me. I hope that the benefits were mutual. I would also like to thank the colleagues who challenged every inch of the model and improved it as a result. Lastly, I would like to thank Susan Burk for the excellent shepherding to get this paper in the right shape.

Add to Bookmarks Remove Bookmark
Add to Bookmarks Remove from Bookmarks
Add to Bookmarks Remove from Bookmarks
XP 2022

Your Bookmarks

No favorites to display. You must have cookies enabled to add bookmarks.

Have a comment? Join the conversation

Related Agile Experience Reports

Discover the many benefits of membership

Your membership enables Agile Alliance to offer a wealth of first-rate resources, present renowned international events, support global community groups, and more — all geared toward helping Agile practitioners reach their full potential and deliver innovative, Agile solutions.

Not yet a member? Sign up now