When the Business Wants Waterfall: Implementing Agile in a Phase-Based Environment

 Abstract

Corporations often put processes in place that assume a waterfall approach to projects.  This can put software teams in a bind.  This paper covers how Halliburton implemented an Agile-based process for software development while still allowing software projects to comply with the corporate standard waterfall process framework.  It covers the specifics of how Agile was mapped into the existing waterfall process, the adjustments to the framework required in order to allow agility, the challenges encountered and how they were resolved, and the value received by the corporation by adopting this approach.

 

1.      Situation

Halliburton is a large oilfield services corporation with more than 80,000 employees, and operations in more than 80 nations.  Halliburton offers services throughout the entire oil field lifecycle, from exploration to drilling, to production, well completion, and finally plug-and-abandonment.  To support those services, Halliburton maintains extensive Research and Development capabilities, used to design and build a variety of new technologies, such as drilling tools, drill bits, chemicals, software, and more.

In 2010, Halliburton decided that the company needed a more rigorous process for the development of new technologies, and implemented a process called LIFECYCLE.

Figure 1. LIFECYCLE

There were a number of results Halliburton wanted to achieve with the process:

• Clearly defined business cases for new investments
• Early inclusion of functions that were sometimes forgotten until later in the delivery process, such as Supply Chain and Manufacturing
• Better and more formalized executive oversight via a system of Stage Gate reviews
• Improved Risk Management
• Improved rollout of new technology to the field

However, this process was designed with the construction of drilling tools in mind, where prototypes can cost millions of dollars and take months to produce, and so detailed upfront planning is valuable.  It was less well suited to the flexibility and changing requirements of software.  The software groups, especially Landmark Software and Services, a Halliburton business line, were concerned that such a tool focused approach would negatively impact its effectiveness.  In response, the Halliburton PMO, which owned the LIFECYCLE process, commissioned an effort to produce an interpretation of the LIFECYCLE process that would work for software.  This effort was sponsored jointly by the Sr. Director of the PMO and the Vice President of Landmark R&D.  I was a Chief Project Manager of one of the Landmark business lines at the time, and was selected to lead this effort.  I was given a budget and a mandate:

Figure 2. Mandate

1.1       IMPLEMENTATION

1.1.1        Conduct Kickoff

I started by setting a high level schedule, with the expectation that each step in the schedule would take about 3 months.

• Assessment – to build the team, contact each business line with a technical software team to build awareness and understand the status of software in that business line.
• Build – to construct the process and supporting material.
• Verification – to conduct pilots and tune material.
• Rollout – to roll out the process, material and appropriate training to each technical software group.

Since my prior experience was entirely within Landmark, the Director of the PMO also recruited a Product Manager from one of the services business lines of Halliburton to join the team, and IT contributed a representative.  Together, we brought in supporting consultants, to build a complete project team.  This gave us a team structure of:

Figure 3.  LIFECYCLE for Software Implementation Team

In addition, we eventually engaged various consultants on a part-time basis, including a Software Development Professor from a local university who was an industry expert in Agile, an Agile Product Management consultant, and two Agile coaches who already had existing engagements with Halliburton.

I was passionate about the software process and Agile in particular, and one of my primary goals with the new process was to make sure that it didn’t interfere with the software teams’ ability to do Agile development.  I knew Landmark was committed to Agile, but I expected the software teams in other Halliburton business lines to be following either a waterfall or a low-process ad-hoc approach.  The reality turned out to be different, and my first hint of this came from my new Product Manager teammate.  As I was sounding him out about his willingness to embed a tolerance from Agile into the process, he started sounding me out on the same subject.  It turned out that he was also a passionate supporter of Agile, and his business line had adopted Scrum.

My second hint that things might be different was when I conducted my first interview with the leads of one of the company’s  software teams, and the first thing they said, defensively and suspiciously, was ‘we do Agile.’  As it turned out, every software team we talked to except one was either doing Agile in some form, or intending to start adopting Agile.  Each team also thought they were the only team doing so, and that Agile adoption would be disapproved at the corporate level.  The single team that wasn’t doing Agile was small and looking for guidance.  At that point, we realized that we could not only build a tolerance for Agile into the process, we could actually build Agile into the process explicitly.

We had more consistency of our Agile process than we had any right to expect, due to the presence of two Agile coaches who had been interacting sporadically with several of the Halliburton software teams.  We quickly engaged those consultants as part of our team.  Some of the software teams were much farther along the Agile road than others, but most shared some basic Agile precepts.  The preferred Agile methodology was Scrum.  Most teams used an iteration length of 2 weeks, with 3 weeks coming in a weak second, and a few other iteration lengths in few places.  Most (but not all) software people believed that Agile meant that they should loosely define Epics at the beginning of a release and flesh out the details and child Stories shortly before the Sprints in which they were to be developed and delivered.  They also believed that Stories should be well‑defined and of the format “as a {role}, I want to do {an action} so that I can {obtain some kind of benefit}’, with well-defined acceptance criteria, but that was a stretch goal for most teams. Most teams tested functionality as soon as it was developed, but still needed some amount of time after “feature complete” to finish stabilizing the release.  Halliburton technical products in general are thick-client rather than web applications, and there are usually only one or two major releases in a year, so teams deliver end-of-iteration functionality and demos to the business, and strive to make end-of-iteration drops as close to production quality as possible.

1.1.2 Construct LIFECYCLE for Software Process

In order to insert Agile into the waterfall LIFECYCLE process, we made four key changes:

• Key stages run in parallel – Design/Build/Qualify (when coding of new functionality is done), Verification (when testing is done), and Validation (when validation with the customer is done) are expected and even encouraged to run in parallel rather than sequentially. This permits iterations where potentially shippable functionality can be delivered at the end of each iteration, because testing and customer feedback is enabled within the iteration.
• Stage Gates can be merged – we clarified that Stage Gates are driven by the end of a stage rather than the beginning. If multiple Stages are completed at the same time, or near the same time, the Stage Gates should be merged into one Gate meeting. This allows the shortening or elimination of separate hardening or beta testing Stages, for those teams of sufficient Agile capabilities to achieve that.
• Just-in-Time Planning – the process specifies that a high-level, sized, prioritized backlog is in place by the end of the planning stage, but that details are not fleshed out until necessary. This enabled the core Agile concept of just‑in‑time planning.
• Most Scope is Not Locked – most scope is defined as flexible, so that Product Owners have room to make Agile trade-offs.

In addition, we changed out some of the operational areas (rows) to make them applicable to software:

• Removed Supply Chain & Manufacturing
• Added Build and Installation Management
• Added Product Rollout
• Combined Compliance and Intellectual Property

We also made some of the task boxes conditional, to allow for projects that are part of larger software or tool programs.

This gave us two process diagrams, one a software version of the main LIFECYCLE diagram, and the other an expansion of the core part of the process to illustrate agility.  For the second, we targeted the case where small hardening and beta testing periods might still be required.

Figure 4. LIFECYCLE for Software Process

Figure 5.  LIFECYCLE for Software Agile Box

The adjustments complied with the mandate to retain the same Stages and Gates, but enabled the insertion of the ‘Agile Box’ into the process, where the team, including Product Owner, Scrum Master and Delivery Team, were able to practice Scrum for the actual execution of the project.  In addition, the Definition Stage, or Iteration 0, was lightened up substantially, to switch from a heavy waterfall planning phase to a just-in-time planning approach suitable for Agile.

One other adjustment came from the PMO, which made the change partway through the development of the software process and made the LIFECYCLE process easier for many teams, including many software teams.  LIFECYCLE was divided into levels, driven by project spend and project risk.  ‘LIFECYCLE Full’ required rigorous compliance with all materials and gates, while ‘LIFECYCLE Moderate’ required that only Gate 2 (Business Case Approval) and Gate 5 (Launch) be held formally in front of executives. ‘LIFECYCLE Core’ required only Gate 2 (Business Case Approval).  Since most software projects fell into ‘LIFECYCLE Moderate’, that substantially cut the number of Gates that were required.

1.1.3 Develop Supporting Material

After the basic process was defined, we produced a variety of supporting materials:

• Quick Reference Guide – This document detailed the activities that would generally need to be done during each Stage, as well as items to consider, what to present at a Gate Review, and differences in the process as applied to software projects.
• Roles & Responsibilities Guide – The team was asked to use the existing Landmark roles as the standard practice for software teams, which were Project Manager, Product Manager, Software Developer, Software Tester, Software Architect, Release Management (Configuration Management), Documentation, Organizational Change Management (software rollout), and User Experience. This document gave a description of each role, plus the timing of what the person would expect to be doing in each Stage.  The Product Owner role was given its own description under the Product Manager heading, and the Scrum Master role was given its own description under the Project Manager heading.  To create this document, we leveraged Halliburton experts in the various roles, along with the experience of our consultants. All the documents, but the Roles and Responsibilities Guide in particular, were intended to be supporting material rather than mandatory process directives.  This document provided material describing all the functions that needed to be covered, and the software teams then tuned them to be specific to their teams, both on an ongoing and an as-needed basis.
• Stage Gate Guides – a set of templates for each Gate Review, covering all the areas generally required, with helpful notes, and guidance as to which role usually would construct each slide for the review.
• Deliverables List – a list of each deliverable referenced, expected content, usual format, due dates, availability of templates, whether it was mandatory or optional, and a RACI (Responsible, Accountable, Consulted, Informed) chart. For instance, a Risk Log was mandatory and was to be done using a standard spreadsheet template (provided by LIFECYCLE); the Project Manager drives it and everyone else on the team helps, and it should be updated at least before every gate.  On the other hand, a Purchase Plan was optional and was often no more than a few words in a web part on the project SharePoint site; it is  usually created by the Project Manager with some help from an Architect, and it should be pretty much finalized by the time feature development is complete.  The only mandatory deliverables are a Business Case, a Risk Log, Stage Gate presentations (all three mandated by LIFECYCLE as a whole), Releasable Software, and a few corporate compliance deliverables such as Export Compliance paperwork and a security audit.
• Manager’s Gate Questions – a cheat-sheet for non-software executives about good questions to ask in a software Gate Review
• Product Management Support Materials – additional materials specifically targeted to Product Managers or Product Owners, including guidance on roadmapping, how to translate business drivers into dollars for business case purposes, Market Requirements Document (MRD) template, and others.

Again, none of the supporting material was mandatory, and was intended as helpful tools for those using the process.

1.2       OBSERVATIONS

Executives had little trouble with the idea of parallel Stages; to them it was just a change in timing of how the same functions were performed. Merging of the Gates was popular at all levels because it reduced the number of meetings.  The concept of Just‑in-Time Planning posed more challenges, as executives in the business lines were more familiar with tool projects, where a prototype tool might cost millions of dollars and take six months to produce, and extensive planning was therefore critical.  We couched Just-In-Time Planning as ‘LEAN for Software’. In most cases, software teams were able to adopt this approach because the executives understood that the details of tactical planning are best handled within each team.   Flexible Scope was also a challenge area because of the difference between software and tool projects; the executives demanded that teams commit to deliverables.  We handled that by recommending that teams identify scope as Mandatory, Important or Nice to Have, and promise official change paperwork if any Mandatory scope item is to be changed.  This worked in some business lines, but is still a challenge area in others. In practice, most scope flexibility happens at the detail level and doesn’t result in the complete elimination of major deliverables, and so doesn’t happen at a level where executives are engaged.

One of the concepts we leveraged was the idea that software development is not a spectrum with Waterfall at one end and Agile at the other.  Instead, we adopted a triangular model where Agile and Waterfall are two end-points, both highly rigorous, and the third point is low rigor ad-hoc development.  This helped us avoid setting up our Agile process in opposition to the corporate Waterfall process, and allowed us to emphasize the idea that our Agile version of LIFECYCLE would still be adding rigor to less rigorous operations.

Figure 6.  Triangular process relationship

After the overall process diagrams, the Roles & Responsibilities Guide and the Gate Guides got the most use.  Hiring managers especially liked the Roles & Responsibilities Guide and used it for position justifications and job descriptions.  The Gate Guides were leveraged for almost all software gates. The Gate Guides were so popular that the main LIFECYCLE team eventually created Gate Templates for all gates and the software gate material was integrated into that.  The LIFECYCLE team made use of the Gate Templates mandatory.

Of all the supporting material, the Deliverables List generated the most hesitation among our team.  The Deliverables List was a spreadsheet listing every item that was mentioned anywhere in the process, with considerable descriptive material.  In fact, we did not include such a document on our plan, as we were concerned that it would make the process too heavy, and only generated it much later in the project.  When we produced it, we embedded within the document a substantial amount of emphasis on the simplicity of the various deliverables in an attempt to discourage the generation of reams of unread paperwork. It was also one of our more challenging documents to produce and took multiple hours of working sessions with a small team of representative users to get right.  It turned out to be a quite popular document when it came out.

We ran a three month pilot program once we had produced the majority of the supporting material. Doing so turned out to be critical.  We got a lot of good feedback on the details, plus some major items, including some input from a couple of the business lines which forced us to do a major overhaul to the process itself to align more closely with LIFECYCLE.  Once that input was absorbed, however, the overall process and document structure stayed stable throughout the rollout process, with no further major adjustments required.

2.      Rollout

Once the process was defined and validated, the next challenge was to roll it out to the various technical software groups, in such a way that the teams both understood the process and adopted it.  Our executives had declared the process to be mandatory, but we did not want to depend on that to drive adoption, and expected the teams to see significantly better results from the process if they saw value in it.  As we worked with the software teams, we encountered a number of challenges.

2.1       Challenge: Business lines Didn’t Want the Process

As we rolled the process out, we worked with both the software teams and the management and executives responsible for those teams.  In most cases, the business line leadership recognized the mandate for adoption, but did not initially see the value to the process for their particular business line.  The pushback came in two forms.  Some business line executives, usually in business lines with small software teams, didn’t understand why software needed a different process from other projects.  Other business line executives and managers, usually in business lines with larger software teams, felt they were doing fine with software and saw no need to change.

In both cases, the business line leadership had valid points.  In those business lines with small software teams, the executives spent their time and attention on the main business of the business line, and for some small team to demand special process and exceptions was a distraction and a headache.  For business lines with large software teams, the executives focused their attention on software already and those teams were generally capable and successful.

2.1.1           What we did

We addressed the concern differently depending on whether the software team was a large or small operation.  For the executives with small teams, software was mostly a black box to them, and they didn’t have the time to become experts in managing the unique issues specific to software teams.  They needed their software teams to deliver with a minimum of fuss and distraction.  For these executives, we made several points:

• ‘It won’t look so different to you’ – we highlighted the similarities with the original process, and emphasized that this process was a way for their software team to comply with the same Stages and Gates as the other teams in their business line.
• Executive Sponsorship – for these executives, the fact that we had executive sponsorship was key. Even though they might not see the value, they weren’t inclined to spend the effort required to fight the corporate mandate.
• Peer Support – these executives knew they were not experts in software, and so they looked to their peers with larger software teams for guidance. When the executives with the larger teams (especially Landmark) embraced the process, the executives with smaller teams trusted that judgment.
• Value – we pointed out value that they and their teams would receive from the process, such as better opportunities to get support from other software teams in different business lines, guidance for their teams on how to conduct Gates, and better visibility for the executives into what their software teams were doing (with a minimum of software-specific jargon).
• Explaining the difference – when all else failed, we tried to explain how software was different. This was the least effective tool at our disposal, as executives hear all sorts of folks trying to explain how they’re unique, and they had no reason to give us special credence.

For the larger software teams, we took a completely different approach.  These executives understood software because it was a significant part of their business: any changes in software process imposed risk to a successful operation.  The larger software teams were all on the road to implementing Scrum already, and compliance with the corporate LIFECYCLE process was already mandatory, so process issues were mostly around bridging the gap between the two.  For these executives, we took the following approach:

• Due diligence – we started by emphasizing the success of the large software team, and by respecting the knowledge and expertise of everyone involved. We also pointed out the value of what our team might learn from the large team’s success.  After that, we focused on doing our due diligence investigations around software process in the business line.
• Corporate alignment and synergy – we pointed out that while the business line was doing a good job individually, there was value to be gained by alignment around a common process across business lines. The software teams all knew that process compliance was mandatory, so they knew that the other teams they interacted with would also be adopting the process, and they saw value to being able to work with them more easily.
• Value of software gates – some of the larger software teams were already in the early stages of leveraging the existing LIFECYCLE process to conduct gates. If the software team was conducting gates already, we highlighted the ease and convenience of working with a gate process that was tuned to software.  If the software team wasn’t conducting gates, we highlighted the additional visibility the executive would gain via the gates.

2.1.2           What happened

In general, we had better success with the business lines with larger software teams.  They understood software, understood the value of corporate synergy, and were willing to invest the effort into improving their ability to do software.  We usually found a few minor items when we did our due diligence, which we would address as part of the process rollout, and we picked up a few nice best practices from these teams which we then shared with the other business lines (a couple of business lines did an exceptionally nice job with Usability, for instance, and sharing of experiences with Automated Testing across business lines proved especially helpful).  One thing we did as part of the rollout effort with all the teams was a brainstorming session with the software team leaders, where they came up with ways that adopting this process would be helpful to them in particular.  Every single team, after thinking about it, managed to come up with a good list of reasons why they’d see value from adopting the process (official corporate recognition of Agile, synergies across business lines, opportunities to get in front of their executives with Gate presentations, recognition of the value of understaffed roles such as Product Owner, Support, or Usability, etc.).  The big business lines turned into our strongest supporters, both at the executive level and at the software team level.

With the business lines with smaller software teams, the executives gave the process cautious acceptance and compliance, which was enough to allow us to move forward.

2.1.3           Reflections

Understanding of the unique motivations of these different executives was key.  For an executive whose software spend is only a tiny fraction of their total budget, it is very legitimate that they don’t want to spend much time and attention on software.  It was critical that we addressed their particular needs and showed them how we could make their lives easier.  For the big operations, it was also critical that we respected their expertise, and also how critical software was to their business.

Asking the teams to come up with their own reasons why the process was helpful made a big difference and generated a lot of buy-in.  Especially for the big software teams, once the software team leadership was bought in, the executives were more inclined to trust us.  In all cases, though, the support of the software team leadership was critical.  I doubt we would have had a successful implementation in a business line if we hadn’t gotten the support of that team’s leadership.

2.2        Challenge: Change Resistance Among Teams

Any change to a process or organization encounters resistance merely because it’s a change, and the LIFECYCLE for Software effort was no exception.  In our case, we expected this resistance and planned for it explicitly.  We were fortunate in that our team was given a budget with discretion as to how we spent it.  This gave us the liberty to invest in formal change management, which proved to be critical to our success.

2.2.1           What we did

When we started our initiative, we expected to face challenges in four areas:

• Awareness
• Checkbox adoption
• Agile resistance among some teams
• General change resistance

Because none of the members of the team were knowledgeable in Organizational Change Management, we brought in an outside consultant.  This person was an expert in Organizational Change Management, as well as Business & IT Strategy and Business Process Design.  He provided extensive guidance for handling the organizational change and was a huge help to the project.  Driven largely by his guidance, we took several steps:

• Formal interviews with the leadership of each business line, to find out their particular issues and concerns.
• Online surveys of the software people in each business line, to find out their receptivity to change, and also any issues concerning them.
• Assessment of the level of Agile adoption in each software team.
• Regular monthly newsletter sent by e-mail, updating all software team members across all business lines of news of our effort. We also included other related information we thought they might find useful, such as general Agile information from one of our coaches, or links to good articles available online.
• High quality SharePoint site, with easy navigation, good user interfaces, and a large variety of useful information, well publicized and easily found.
• Monthly meeting of the Software Discipline Managers from each business line, as an informal steering committee.
• A formal rollout process customized to each business line, including input from managers and software team leadership, and informational presentations to executives and to the full teams.
• Regular reviews and updates to the PMO.

In general, we emphasized communication at every opportunity.   I knew from previous experience that an effective change management strategy was key to our success, and I also knew that this was an area that projects traditionally skimped on, to their detriment.  We went to the other extreme.

2.2.2           What happened

The intense focus on Organizational Change Management was quite successful.  The direct results fell into a few categories:

• Awareness – as the project progressed and the cumulative impact of all the various communication efforts piled up, the people within the organization who were impacted by the process became more and more familiar with the effort. Early surveys drew a lot of responses from people who had never heard of our team or the LIFECYCLE for Software effort, but by time we got to the later business lines in the rollout process, knowledge was fairly widespread.  People felt that they were being kept informed, and we didn’t have to deal with anyone who complained about being left out.
• Buy-in – we made a point of consulting with people at all levels of the organization, and incorporated that feedback in the process and in the rollouts for individual business lines. We also made a point of propagating the knowledge of best practices from one business line to the others.  In general, the people affected felt they had a voice in the process and so were willing to engage and contribute to the success of it.
• Adoption – in the end, the business lines adopted the process, in general with quite good enthusiasm. They’d had a voice in it, their specific concerns had been addressed, and they saw the value they could draw from it.

2.2.3           Reflections

The investment in Organizational Change Management was one of the most successful things we did.  There were a few people within the company who initially doubted the usefulness of such an effort, but by the end of the project, the consensus was that the project had been quite successful, and several executives specifically credited our investment in Change Management.

The other important thing to note is the value of all the feedback received.   During the initial pilot program, we got feedback that drove us to substantially overhaul the base process, which improved it significantly.  Later, during the rollout process, we produced an additional major process document, the Deliverables List.

Another benefit surfaced as an unintended side effect of the Change Management effort.  In general, business lines and software teams can be isolated from the other business lines.  This can lead to situations where each software team is learning the same lessons in isolation.  Because of the extensive contact our team had with the various software teams, we were able to act as a conduit of information and best practices between the software teams in different business lines.  Communication around Agile practice was the most common, but the most successful of these interchanges was the Software Discipline Managers meeting, made up of the managers of each business line’s software team and set up as a communication vehicle for the LIFECYCLE for Software initiative.  On numerous occasions, the Discipline Managers used part of the meeting to work with each other regarding issues unrelated to the LIFECYCLE for Software process.  When the LIFECYCLE for Software effort came to a close, the Discipline Managers decided to keep that meeting going, just so they would have the opportunity to continue to work with each other.

On the down side, rollout took substantially longer than we expected.  We had originally planned to roll the process out to all the business lines in three months.  Instead, it took us nine.  The sheer effort involved in meeting with each software team, addressing their particular customizations, interviewing executives, surveying team members, presenting results, training leads, training project managers, and making presentations to the teams as a whole, took far longer than we imagined, especially for the larger teams, and for the teams that weren’t located in Houston.  Rollout involved multiple trips to Dallas and Denver, and even trips to overseas sites.

In all, the investment in Organizational Change Management was very much worth it.  Except for Landmark, which sells software commercially, software is not a core business of Halliburton, and software teams can be a secondary focus for a business line.  The software teams came to see us as allies, people who understood their issues and constraints and were giving them tools to make their lives better.  As such, the teams adopted the process wholeheartedly.

2.3       Challenge: Skills and Staffing Issues

The LIFECYCLE process, including the LIFECYCLE for Software process, was built under the assumption that the teams implementing the process were already staffed with the appropriate people with the appropriate skills.  That wasn’t always the case, and there were a few pain points that bubbled up consistently:

• Product Management and Product Owner Gaps – coverage of this role was light, and Product Managers and Owners tended to be stronger technically than in formal market and business analysis. Often, developers or managers were handling requirements gathering, and strategic business considerations could often be overlooked.
• Role Coverage – teams tended to be heavily weighted with developers, maybe a few testers, and few people doing any other roles.
• Gate Presentation issues – software teams and projects hadn’t historically been exposed at the senior executive level, and were not necessarily prepared for that initial exposure.

2.3.1           What we did

We expected to see an issue with Product Management.  The gap there was well recognized within Halliburton and was one of the drivers behind the creation of the original LIFECYCLE process.  LIFECYCLE helped us here some, as it was already illustrating the need for more and better Product Management throughout Halliburton.  However, the Agile role of Product Owner puts additional demands beyond the Product Manager role, and most business lines didn’t initially see the business value of that additional investment.

We attempted to help by providing material specific to Product Owners.  We brought in a consultant expert in Product Management, Pragmatic Marketing©, and Agile, and he provided documents such as guidance on roadmapping, an overview of the role of the Product Owner, a template Market Requirement Document (MRD), a guide to revenue stories including how to calculate expected revenue for software for services, and substantial other material.  He helped us with the Product Manager and Product Owner role descriptions, and then he provided multiple training classes to the people acting as Product Managers or Owners in each business line.

For the Role Coverage issue, we worked with our Agile experts to produce recommendations for coverage for each of the roles, both minimum and optimal.  We also did a role mapping exercise with each team.  This was necessary not only to sort out coverage, but because each business line used a slightly different set of titles and responsibilities.  For a small team, the role map might look something like Figure 7.  Note that one person is covering multiple roles.  Also note that a different person is covering Product Manager and Product Owner.  Many of the business lines split this role, with Product Management in the business organization and Product Owner in the technical organization.  Note also that a different person is doing Project Management than is doing Scrum Master.  This split was less common, but this team did it because the Technical Application Manager managed the development team, and this avoided the scenario where the Scrum Master was also the boss of the team members.

Figure 7.  Small team role mapping diagram

The role mapping for a larger team might look something like Figure 8.  Note, for instance, that multiple people with different titles are covering the Project Manager role, and that several different roles contribute to Organizational Change Management.

  Figure 8.  Large team role mapping diagram

We attempted to help the teams with their Gate presentations by several approaches.  One of the core deliverables from our team was a set of Gate Templates for use with the Gates, and these were well appreciated by the teams.  We took the opportunity to tune the templates based on the early experiences, to address some of the early issues.  We also shared learnings across business lines, so later adopters could avoid some of the early challenges.  In addition, we provided mentoring for some teams that wanted it, and spread examples of successful Gate decks.

2.3.2           What happened

Strengthening Product Management is an ongoing effort across all Halliburton and goes beyond software.  Considerable progress has been made in both number of Product Managers and in training and expertise, but a full solution will take time.  Software teams are handling the gap in Product Owners internally by staffing that role themselves.  At this point, all except the smallest software teams have at least some Product Owners in place, and a cross-business line consensus is emerging around the role.  The software teams are staffing the Product Owner role because these are the organizations that feel the need and understand what the role should do.  That makes them a better fit to initiate the role, and as the role becomes well understood, we expect it to shift organizationally to the business side.

Because the Product Owner and Product Manager roles are most commonly found in different organizations, the split documented in the LIFECYCLE for Software material turned out to be quite helpful in communicating boundaries, and we didn’t have any trouble with pushback from existing Product Managers.  The evolution of a fully staffed and functional Product Owner role is still a work in progress, but because that evolution is generally in the hands of software managers who understand the need, the prognosis is looking good.

Right now, software team construction is changing all over Halliburton and we’re moving to a much more mature model than a focus exclusively on development.  The small team in the role mapping figure 7, for instance, has since hired testers.  Almost all teams have made at least some progress around enabling and using automated testing, which was a gap area in many business lines.  A couple of business lines have made good use of Usability experts for several years, but now other business lines are also adding that expertise.  Many business lines are making substantial improvements in Product Owner coverage and skills.  Much of this would likely have still happened without our team’s involvement, but we were able to provide tools and information for the software managers to use in taking these steps.  Many managers used the Roles & Responsibilities Guide when providing job descriptions, for instance.

Some of the early Gate Reviews did turn out to be pretty painful.  The most common mis-step was around business cases.  Several early presentations went light in that area, either because the team didn’t have the business side expertise, or the Product Manager didn’t have the time to invest in putting it together.  That always turned out badly.  The biggest thing the executives wanted to know in a Gate Review was ‘why are we spending our money on this’, and if the Product Owner or Product Manager didn’t have a good answer for that question, the review could become unproductive.  Another mistake teams made was to go into too much technical depth.  Executives, especially non-software executives, didn’t want to see pages of architectural diagrams.  The Gate Templates warned about both of these pitfalls, but a text warning in the notes section of a slide template didn’t turn out to be sufficient.  By the later stages of the process rollout, I made sure to spend some time with the software leads entertaining them with horror stories of bad Gate Reviews.  That worked.

The teams actually came to appreciate Gate Reviews quite a lot once they’d had a little practice.  Because software is often secondary in Halliburton business lines, it can be hard for software teams to get the attention of their executives.  Gate Reviews gave them a vehicle to get their executives’ attention, and a format for having the conversation in a language the executives understood.  This produced some good results for a number of teams, such as one team that was included in the business line’s strategic planning for the first time, one project that managed to get additional funding because of additional requirements that surfaced in a Gate Review, and multiple occasions where issues were exposed before the project started work that would have caused problems later otherwise.

The Gate Templates turned out to be a very useful tool, and even though their use was optional, all the teams used them.  In fact, the PMO had originally been reluctant to supply templates for the overall LIFECYCLE process because they didn’t want presenters slip into a checkbox mentality, but once they saw the success of the software templates, they decided to supply templates for overall LIFECYCLE.  They used some of the best parts of the software templates, provided an additional level of professionalism, and we ended up integrating the software templates into the LIFECYCLE templates, for a step up in the quality of the Gate decks.

2.3.3           Reflections

Non-software people tend to think that the way to invest in software is to add more developers.  Sometimes, they understand that testers are a good idea, too.  Pretty much every other role, including Scrum Master and Product Owner, looks like overhead to them, which can make the staffing of a good distribution of roles on the team a challenge.  The software managers used a lot of techniques to get their teams organized the way they wanted, and the information from the other business lines was key.  They pointed out successes from other business lines to their own executives.  They leveraged LIFECYCLE for Software material, industry best practice information, and information from Agile experts.  They shifted existing people to operate out of their job descriptions.  They hired contractors.  It’s a long process, but each time they make progress and then show some success, they are in the position to take another step forward.

Getting good Product Owners in place in particular is critical, and it’s a hard problem to solve, but there has been a lot of progress in the last couple of years.  I expect to see this area continue to improve.  Gate Reviews have added a lot of value to the teams, mostly in getting visibility for those teams within their own business lines, and the software teams have adopted them enthusiastically.  Most of the software projects and programs do two Gates, one before execution starts (Gate 2) and one at the time of launch (Gate 5), and are able to avoid the others, either because they have merged the Gates or because their spend is small enough to allow them to skip those Gates.

3.      Overall Reflections

It’s possible to consider the chance that an initiative will be adopted as a formula:

Success of the Adoption = Quality of the Product being Adopted x Quality of Rollout

It’s easy to focus on the quality of the product, and we did, because it’s obvious that if you don’t deliver a quality product, few people will adopt it.  This approach alone seldom works in real life, though:  just because you build it doesn’t mean that people will come.  Our team’s investment in the Organizational Change Management was a huge part of why teams were so willing to adopt the process, and key to our success.

A process initiative like this is a very political effort, and needs to be managed that way.  Executive sponsorship is crucial, and we needed buy-in at all levels of the organization.  Because our executives had declared that compliance was mandatory, that gave us the opportunity to be the “good cop”, and make ourselves allies of the teams we supported.  A mandate is good in theory, but if the individual organizations wanted to fight us, they could have adopted far less than they did.  If we had come through the software teams attempting to wield the hammer, we would have been far less successful than we were.

One area that turned out to be key was our alignment with the larger LIFECYCLE initiative.  Originally, we portrayed ourselves as a mostly independent initiative, and that led to some conflicts early on.  We addressed that by making some adjustments to the process, but also by changing our message.  We said that we were not a separate process, and that we were just providing the translation of LIFECYCLE, applicable specifically to Software, because the translation wasn’t intuitive.  That positioned us as a contributor to the corporate LIFECYCLE initiative, and brought us the strong support of the PMO.

The connection to LIFECYCLE also helped the software teams.  To a great extent, the details of software were hidden, and software teams were given a tool to communicate with executives in the same structure and in the same language that the other groups were using.  When software teams went in front of their executives for their Gate Reviews, the majority of their slides looked exactly the same as every other group, with only a few slides different because they were doing software, and this smoothed the interaction considerably for everyone involved.

Most of the technical software at Halliburton is developed by the software teams within the business lines, but some of it is supported by software infrastructure supplied by the IT department. IT was not covered as part of the LIFECYCLE for Software initiative, as they were a different part of the organization.  However, as soon as the technical software teams adopted the process, they started asking their IT dependencies to also adopt it.  The technical software teams wanted to see Gates from IT.  They also wanted IT to adopt Agile.  IT had its own extensive processes, but they were responsive to their clients, and so I ended up working with IT to align the LIFECYCLE for Software process with the IT processes.  That was a complicated endeavor, but the end result was that the software sections of IT are now following this process as well.

Our initiative never had the mandate to implement Agile within the organization.  We never had the mandate to even enable Agile within our process.  We were able to enable it because the ground level support for Agile already existed within the individual software teams.  All our team did was make Agile part of the approved process for Software.  That corporate approval, though, was quite valuable to the software teams, and there is a lot of energy behind Agile adoption across the technical software groups right now.

4.      Next Steps

The LIFECYCLE for Software effort is now complete and the team members have moved on to other roles.  The software process is now owned by the PMO, and the various business line Program Directors are responsible for ensuring that the LIFECYCLE process, including the software portion of it, is followed within their business lines. In general, the software teams are among the most successful groups at complying with the process, and so are not the primary focuses of attention.  In the meantime, the teams are leveraging their increased access to their executives, and supporting each other across business lines to upgrade their Agile capabilities.

There is still a lot of variance in level of Agile practice by business line.  The smallest teams struggle the most, and the greatest advances are being made in the mid-size teams.  We have a couple of teams that are very strong with Agile, and I expect to see several more in the next few years.  The advances being made in the Product Owner role, and all the things a Product Owner does, are the most significant, though there are further improvements coming across the board; automated testing is seeing widespread improvements as well.  Product rollout, or commercialization, is a pain area we’re just starting to address.

Agile is becoming sufficiently embedded in Halliburton that people sometimes mention adopting Agile beyond software.  Once Agile becomes widely successful, I suspect we may even see that happen.

Software as a whole is becoming more and more of a focus at Halliburton, as executives see it as critical to our long term success.  The LIFECYCLE for Software initiative was one of many efforts to continuously improve our capabilities.  There will be more, and hopefully we’ll be in a good place to make those improvements based on the Agile foundations being laid now.

5.      Acknowledgements

I would like to thank all the team members who contributed to the LIFECYCLE for Software team, including Mark Savery (Halliburton Cementing Product Owner), as well as John Wilmore and Mukul Agrawal (Halliburton IT). Thanks to our Agile experts Dr. Venkat Subramaniam (www.agiledeveloper.com), Rich Miranov (www.miranov.com), Robbie Mac Iver (www.robbiemaciver.com), and Simon Orrell (www.snowdolphin.com).  Special thanks for all their dedicated effort to Melody Lozano (Organizational Change Management) and Dr. Eric Threatt (www.transformationsciences.com).

Thanks to Todd Little for the idea and triangular diagram of the Ad-Hoc, Waterfall, and Agile relationship (figure 6).

Thanks to my shepherd Sue Burk (www.topfivetoseven.com) for her help editing this paper.  It would have been a much worse paper without your help.