RESOURCES

Data Driven Portfolio Planning: Implementing the Rally ORCA tool

About this Publication

This session is for those involved in road map, resource, and budget planning and will appeal to marketing, PMO, and IT/engineering communities who desire to increase the predictability of their delivery success. Join us as we take you through a crash course of our journey in driving change all the way up to our executive levels by offering simple and transparent views of the business demand and engineering capacity and enabling easy what-if analysis. These what-if planning scenarios enabled us to show realPtime results of business value that could be delivered by increasing/decreasing demand and capacity. We took the emotion out of the difficult investment decisions and created a renewed focus on the highest value initiatives.

 1. INTRODUCTION

Dee Miller: After nearly 10 years of being part of and leading many agile pilot initiatives and transformations in IT organizations, I wanted the opportunity to be part of something bigger. While I had made significant progress partnering with the business organization as well as IT in earlier transformations, there were always organizational impediments that still stood in the way – how do we help build the portfolio of work based on business priorities while also understanding the resource demand and capacity required in an agile world, how do we evaluate teams versus just individuals, and how do we stop starting up and shutting down productive teams according to the way finance releases funding? It was at this point in my career that I came across the Scaled Agile Framework (SAFe) that offered more of an enterprise model for agile transformation that went far beyond some of those I had been a part of that had centered mainly on the IT organization.

I joined Elekta a little over a year ago. When I joined, the SAFe transformation had already been in motion for about 9 months. I am helping to pioneer new ways of thinking about how we get to market earlier and limiting work in progress (WIP) at the portfolio level to ensure we deliver on the most important commitments. I am guiding the organization to use estimation processes with enough accuracy to be “directionally good enough” and improving both the quality and timeline predictability of our solutions to customers. I head up the Agile Office and my co-­‐presenter Micah Schwanitz is a Senior Program Manager and is spinning up the BI portion of our organization that focuses on metrics.

Micah Schwanitz: I have been working for Elekta for five years now and come from the traditional heavy waterfall SDLC process of software development. I was here at Elekta working in the PMO as a program manager when our engineering organization introduced scaled agile and we experienced a lot of change very quickly. Having a natural fascination with metrics and statistics along with my project/program management background, I have frequently been involved with annual budgeting exercises to attempt to understand what an organization can get done in a finite amount of time. I was an early believer that defining and documenting everything to the most granular detail would eventually lead to a predictable and achievable plan. This all changed with the introduction of scaled agile and the data driven capacity analysis that I have been part of here at Elekta. Instead of exhaustive exercises that were never complete enough and ultimately never accurate, we are now learning to tolerate incomplete data and land on a roughly right or “directionally good enough” point that still allows us to plan out scenarios and saves a huge amount of time. What would have taken a quarter at some of my previous companies now takes a few weeks and actually feels like it has more reliability due to the cross functional nature of the process we are implementing. The momentum of the transformation grows with each success and this journey is leading Elekta to better software agility.

2. BACKGROUND

Elekta provides clinical solutions for treating cancer and brain disorders. As part of the medical device industry, we work in a high-­‐assurance environment regulated by the FDA. Elekta was founded in 1972 and has grown to 3,400 employees with offices around the globe. Most of the growth has been through acquisitions so there has been a need to refine our business focus and processes so that we don’t have legacy silos and we are delivering integrated products – it is crucial that our business areas work closely together to ensure the software systems and hardware we create come together as one solution.

When I arrived at Elekta a little over a year ago, the company was trying to get 4x the amount of work done in a fiscal year that they had capacity for, but the middle management in both business and engineering organizations didn’t have any data to prove it and senior leadership continued to push toward the desired roadmap. As a result, quality, commitment and morale were suffering. I joined the Portfolio Management Office (PMO) which was in very poor shape due to years of legacy command and control project management style. As a result, they had been pushed out of most activities during the transformation as a result. We started to rebuild the PMO with education about servant leadership. However, we really earned some respect back when we used our data driven portfolio planning approach to convince leadership to begin making some key changes.

3. THE PROBLEMS WE FACED….

  • The product roadmap was defined as “what was desired” or “what sales had already put contracts in place for” and it became the basis for judging the success of engineering delivery in lieu of any other information.
  • There was too much work in progress without a realistic picture of the demand versus capacity available. Understaffing made work take longer and expectations were not met. This led to an environment of disappointment and distrust.
  • New work was continuously approved and taken on in lieu of understanding the impact to the rest of the work in progress.
  • Teams struggled with constant feedback of missing deadlines, but felt like they had no input to the deadlines that had been set.
  • Everything was the highest priority.
  • The sales team was frustrated, customers were frustrated, and relationships between the business organization and engineering were strained.

4. WHAT DID WE WANT TO ACHIEVE?

  • Understand the demand and highest business priorities on the roadmap and remove non-­‐MVP (not required to produce a minimally viable product) work
  • Understand the real available capacity and align that capacity against the highest business priorities
  • Limit work in progress
  • Create predictability in delivery
  • Show transparency of progress against the roadmap/plan
  • Build trust between engineering and the business organizations
  • Create a more rewarding work environment for employees

5. WHAT WE DID

Late last year when the annual budgeting cycle came around, we took the opportunity to implement the data driven portfolio management concepts outlined below. All of this work occurred very quickly over a 2-­‐3 month period.

Step 1: Defining the Portfolio Backlog

Our business partners already had the concept of a product roadmap. However, it was still changing on a regular basis and the timeline wasn’t based on much input from engineering. Senior leadership within the business had already created investment themes – categories of products/services which they designated as Strategic Areas and to which they allocated specific budgetary amounts. We worked with the product managers of each strategic area to connect the roadmap to epics in Rally so we could have a clear portfolio-­‐ level backlog.

Step 2: Prioritizing the Portfolio Backlog

Within the strategic areas, we asked the product managers to prioritize the epics in rank order according to business value. The business struggled with this prioritization as it seemed as though everything was required and was of equal priority with nothing left to be removed. Creating mature, well developed business cases was not something that happened previously in the organization, but now finance was instituting the first round of this. The business now had to establish the return on investment and the time horizon in which it would be achieved which began to influence priorities. As prioritization continued, this caused the legacy silos/kingdoms issue to come up. Product managers felt like they had to protect their territory and the capacity that was previously considered “theirs” and not fungible.

Step 3: Quantifying Demand

We pulled together a cross-­‐functional team to size the portfolio backlog of epics that included Product Managers, Engineering SMEs, System Architects and System Owners. We defined and explained a process of relative estimation. First we reviewed some epics and picked a few that were similar in size. Then we relatively sized everything as bigger or smaller than that baseline set on a t-­‐shirt sizing scale of XS, S, M, L, XL. After we had a large enough sample size, we estimated how much bigger or smaller each size was – so for example a “small” seemed to feel half the size as a “medium”, etc. Then for each t-­‐shirt size we identified, based on some knowledge of historical data and subject matter expertise, we estimated how many sprints it would take an ideally sized scrum team to execute on the epic according to strict definition of done and quality standards. An ideal team was defined as 5 developers, 2 quality engineers, 1 product owner and a scrum master.

Step 4: Quantifying Supply (Resource Capacity)

We shifted the mindset of the organization to think in terms of teams, instead of individuals. It wouldn’t make sense to bring on more engineers for example if we didn’t have appropriate quality engineer capacity. Using the SAFe model, all of the teams were assigned to a single agile release train (ART). The concept of an ART is that all the teams on a single train are aligned with a common value stream or product so that everyone is working toward the same goal. Capacity was essentially fixed by the implementation of the ART concept because although we could hire new teams, it was accepted that the budget to do this was limited and the learning curve to bring them up to speed was long. In general the ARTs had a fixed set of teams and therefore fixed capacity. Having fixed capacity and a fixed quality standard meant that we needed to focus on feeding an appropriate amount of scope into the ARTs that matched their capacity. To determine capacity for our annual planning cycle, we had to determine a way to estimate the work a team could accomplish in a year without compromising quality. All of our teams are not the same size so we developed a formula for adjusting their capacity up or down according to their team size when not ideally sized. In our calculations, 1 point of capacity (supply) was equal to one sprint worth of work performed to our quality standard (defined in our definition of done). We determined how many sprints each team would have available in a year and reserved an appropriate amount to account for vacations and exploration/innovation type work that would require some capacity beyond the product delivery work we already knew about.

Step 5: Capacity/Demand Analysis, Creating Multiple Growth Scenarios

We used the Rally ORCA beta (now Rally Portfolio Scenario Planning tool) to create a planning time horizon over 3 years. We incorporated the backlog (demand), created the teams and then built scenarios based on assumed growth for our business area (decreased growth, no growth, moderate growth, high growth). These scenarios would later have a big impact explaining to the decision makers at our headquarters in Sweden how many products we could deliver based on the budget they granted our organization. We considered most of our developers fungible resources, but we did have to account for a few specialty areas where resources had radically different skill sets that could not be easily redirected to other work. We performed demand versus capacity cut line analysis. Each scenario had the same prioritized backlog, but based on the resource capacity enabled by the different growth scenarios there were different outcomes of what work could be accomplished or “landed above the cut line”.

6. HOW DID MANAGEMENT RESPOND?

When we compiled all the demand and capacity the first time and provided the results to leadership the results were two-­‐fold:

1. We discovered that we were working on some things we don’t need and some of the things we are working on only require a smaller portion of planned work in order to achieve the MVP (Minimum Viable Product -­‐ minimum scope required to have a product we can gain value from).

2. Leadership was blown away that we were trying to plan somewhere around 4X the amount of work we could accomplish in a fiscal year. Yet we all acknowledged that this had been the gut feel of teams and managers in both business and engineering organizations for some time.

This was a stressful time period. People were listening to the message, but they didn’t like it and it took time to digest and swallow. The processes and assumptions we used to do the analysis were picked apart for a while until it was agreed they were sound. After a few rounds of refinement (reprioritizing, de-­‐scoping, identifying missing portfolio level items, moving capacity to higher priorities, incorporating assumptions around resource growth scenarios), senior leadership used the data as the basis for submitting their Long Term Plan to our headquarters to request funding. They were able to show executive management different scenarios of what products would and would not be ready based on the resourcing they granted. We are anxiously awaiting the final results of the budget right now. The PMO has been asked to continue to develop and institutionalize this kind of discipline within the organization and one year after feeling broken, the PMO is excited, providing value, and growing!

7. PROGRESS WE ARE PROUD OF

We have been able to provide data to senior management that did not take a huge investment in resource capacity and is directionally correct or what we call “roughly right” – good enough to enable high level roadmap planning decisions. As a result of these efforts, our leadership recognized we had too much WIP and is beginning to make decisions on what we will, and more importantly, what we will not work on. Senior leadership is also differentiating between priorities and communicating the highest priorities throughout the organization. This exercise helped to show that the business and engineering organizations really have the same dilemma -­‐ we both had more work than we could get done and ended up blaming each other instead of teaming up to paint the picture of what we could realistically get done so we could all be successful. As all of this change happens, the old silos/kingdoms are beginning to weaken because we are finding that all capacity is required to deliver on the highest priorities.

8. STRUGGLES ALONG THE WAY

  • We struggled with truly understanding our value streams – our view of how to organize our Agile Release Trains (ARTs) changed several times as we gained better understanding of what was required to release fully integrated products.
  • We also struggle with multiple time zones and the logistics required to communicate affects how we set up the ARTs.
  • The legacy of historical silos/kingdoms has led to territorialism that can make resource capacity seem less fungible than it truly is. This leads to working on pet projects versus the highest business priorities, but we have made significant progress on this.
  • While the concept of limiting work in progress (WIP) is now well understood, actually taking the action and cancelling/reducing work has been more challenging.
  • Understanding the difference between architectural runway that needs to be planned for versus architecture that can emerge has been a pain point for us and affected our ability to estimate work.

9. WHAT NEXT?

  • Intake Management – we are beginning to implement a process to facilitate the intake of new work into the portfolio between annual portfolio planning cycles. This process will ensure that we have capacity available for the new work, that the priority of the new work is clear, and that decisions around impacts/tradeoffs with other work slated in the roadmap are made.
  • Epic vs. Feature – we intend to give the organization a better definition of epics and features and the differences between them so we have a clear portfolio backlog. Today we still struggle with being able to roll up progress against our annual plan in a consumable way.
  • Apply the same principles to Release Planning -­‐ we will also begin using the Rally Portfolio Scenario Planning tool at a lower level, the program level, by incorporating feature estimation into our processes so that we can do better release planning.
  • Inspect and Adapt -­‐ we will establish a quarterly review to look at all levels of estimates (team, program, and portfolio) over time and understand our predictability and improve.

10. ACKNOWLEDGEMENTS

Special thanks to our boss, Garth Andrews, who has been an incredible change agent who led the Elekta transformation. Garth’s ability to unravel the silos, encourage cross functional teams, and drive every part of the organization to positive outcomes for the company has created incredible career opportunities for many of us. Thank you to our executives Todd Powell, Richard Stark, Dan Fristedt and Michelle Jemus who partnered with us on the data driven portfolio management process and listened carefully to the results even when it was not good news. Thank you to Catherine Connor and the Rally Portfolio Scenario planning team for letting us incorporate features into the ORCA Beta tool as they developed it. Thank you to Nannette Brown, Experience Report shepherd, for her guidance on our first time presenting at Agile 2015. Most importantly, thank you for great colleagues across the globe who came together as a cross-­‐functional team to break down all the work on the roadmap and contribute their expertise to the estimates.

REFERENCES

Leffingwell, Dean. Scaling Software Agility: Best Practices for Large Enterprises. Upper Saddle River, NJ: Addison-­‐ Wesley, 2007

About the Author

No bio currently available.

No bio currently available.