Are you in agony because your boss just came and told your Agile team it will need to work with the Waterfall group on the next big company project? Is your current project a car wreck because Agile and Waterfall teams aren’t gelling well at all? Are you a QA Agile team member that’s out of tricks to move your Agile/Waterfall project down the road? Or are you in an organization that’s slowly morphing into Agile and you are searching for some low hanging fruit you can implement to help the transition? If you are road tripping your way through the questions above then take a drive through my experience with the Arbitrations project. It will deliver not only some history of how our Agile life began, but also how I tackled the tough challenges of this project. These challenges spanned from dealing with the old guard attitude of the Waterfall teams all the way to challenging Agile stereotypes in my organization. In the end, you’ll have a nice set of must have what to do’s and what not to do’s delivered in a retrospective format you can leverage in your current or future project and drive that needle towards success.
1. INTRODUCTION
This experience report centers around what many organizations across the globe have or will find themselves in. It’s a mash up of Agile and Waterfall groups that have to work closely and intimately together to deliver a must have project to the business. The Arbitrations project was where I experienced this head on collision of worlds. Not really anything good comes out of a head on collision but this is a different situation. The lessons I learned and creative process of grappling with the challenges encountered will surely find value to those that may find themselves in a project like this or are currently entrenched in.
Now, as I drive down this road many things will be witnessed. First, we’ll go into a little bit about who am I and what in the world is Manheim. Then, I’ll deliver the details and context of the Arbitrations project and at the end of the presentation a retrospective will be conducted on what to do and what not to do should folks find themselves in a situation like the Arbitrations project. One quick point of clarification, throughout this story I’ll refer to my team as Agile QA or my practices as Agile QA. This is just to make a distinction between my QA team and its practices and the Waterfall QA team.
1.1 Audience
Who exactly might the audience be for this presentation? If your bread and butter isn’t QA don’t worry. A mixed audience of people that are filling various roles on a project team would find value. Project team software delivery methodology doesn’t matter either. Skill level on the Agile scale or where the individual’s organization is currently with the Agile evolution is not a problem either.
2. My Street Credentials
Alright, who am I. Life in QA grabbed my attention about 16 years ago after serving some time as a developer and part time business analyst. I have been with Manheim for the last four years now. My role in the organization is QA lead over a team of functional testers and automation engineers which all exist offshore. The application I’m responsible for is a Microsoft CRM tool known as Dynamics. Its function is to manage all of our customer data. The larger team I’m a part of consists of BA’s, developers, product owners and an iteration manager (like a scrum master). We have been following our flavor of Agile for six years so we are pretty established. Our Agile life sprang forth when the online side of our business came into being. Consultants were brought in to lay the Agile rebar and ground work for how we would deliver software. They focused on installing the iteration manager, developers and business analysts. Interestingly they didn’t bring in QA. As QA teams were formed they simply had to create their own cog in the machine.
With no directives we had to figure out how to work with and shape the forces beside us. QA is always pinched and gets slammed by the forces upstream from them. Being vigilante but not paranoid is a virtue. This would prove very valuable later on down the road as the Arbitrations project came along.
3. What is Manheim?
Alright, by now I’m sure the question of what is Manheim is looming around in the air. Manheim is a wholesale auto auction business that falls under Cox Enterprises. It brings in about 3 billion a year and is the largest auto auction in the world. Manheim’s organization is a house divided because on one side you have brick and mortar auctions and the technology systems that exist to support it. They follow the Waterfall methodology for delivering software.
Fig. 1 One of our physical auctions
Fig. 2 Dealers bidding on cars in the auction lane
However, in the other side of the house is the customer facing web applications and supporting technologies that allow our customers to purchase vehicles online as well. These folks are Agile practitioners and evangelists and it is where I live.
Fig. 3 Picture of our main online application dealers use to access the online vehicle auction system
The Agile landscape that lies on this side of Manheim is one where we are all drinking the kool-aid!
In some companies and organizations you may have one or two disciplines of a project team following Agile. For us it goes from the product owners, BA’s and QA to the developers and the iteration manager.
4. Agile Landscape
Here is how the landscape of our iteration looks like. Prior to an iteration beginning, QA participates completely in the team joint estimation sessions. This is where BA, development and QA sit in the same room and groom through the story cards just prior to the upcoming three week iteration. As the cards are assessed and given a point estimation value by each team, QA leverages this opportunity. We point out risks, develop high level test cases and identify integration points with other applications and teams. Quick footnote, if cards are too large to fit in an iteration we break them up into smaller stories that fit into the iteration’s capacity.
As the selected cards move into the iteration for BA and development work to begin, my automation team jumps in. They work alongside the BA’s and development to automate relatively small and manageable functional tests for the cards that can be completed in the BA signoff time frame. This hit team keeps up a constant dialogue with development and the BA’s highlighting gaps or other pitfalls we may find later in the project. The value here is that problems found early are easier and cheaper to fix than if found later. The automated tests created are run after each build and are eventually added to our regression suite. During this phase the QA functional testers are also creating their tests from the cards as well as a small group of smoke tests which I’ll detail later. These tests augment those created in the estimation session.
Once the cards pass the BA team’s simple but effective battery of tests, called BA Acceptance Criteria, the code is confirmed to deliver the story’s basic functionality. It then moves into the QA phase. Upon the developers deploying the build to the QA environment the developers are responsible for running the smoke tests which were mentioned earlier. After a green light is given for the smoke test execution, my offshore QA team launches the first salvo of automated regression tests and functional testing commences. Throughout the QA phase we run multiple cycles of automated regression. If the team runs into any issues (testing blockers, etc.), we spotlight them in our daily standups.
As we close out the iteration, QA creates a release risk assessment document that details what the risk level of the release is to production. The risk assessment is delivered to the business giving them all the information they need to delay the release or go forward. This document is a bill of health for the release pulled from any area that could create risk. For example, were the planned number of testing cycles cut short because the BA’s were late in signing off on any cards? Were defects primarily because of configuration issues or simply because the code was intricate and complex? This method has proven very effective in taking QA out of the proverbial thumbs up or thumbs down for releases.
5. Arbitration Project Nexus
Alright, now we have a little history and understanding of what Manheim is and how we fit in, let’s move into the meat of this presentation and first tackle what an arbitration is. In the auto auction business there is a process known as arbitrations. An arbitration is when a dealer buys a car, but there is something wrong with it.
Either it was misrepresented somehow by the seller or the buyer simply believes he didn’t get what he was expecting. The vehicle along with the buyer and seller go through Manheim’s arbitration process to rectify the situation.
At the beginning of the project the current state of the arbitrations process was not in a good state of health. The process of going through arbitrations was painful for our customers as well as our employees. It was built using technology that was outdated at this point and couldn’t scale to the increased demand of our business. There was zero analytics around the system so getting any kind of data on performance, etc. was impossible.
5.1 Project Goals
The project’s goal was to bring the two groups together that owned the pieces of the arbitrations life cycle, my Agile group and the Waterfall group, and make it better for our customers. This entailed ripping out parts that were manual in nature and replace them with some slick automation. Analytics and reporting would be planted in areas that were once blind to it. Another goal was to put some rebar in its technical foundation so it was more scalable and a lot less fragile. We were going to join forces and make an uncomfortable process better and less stressful for our customers. What could go wrong, right?
5.2 Project Challenges
The challenge of this project came in many forms. For starters, this was going to be an intimate project between two different groups that had different methodologies. For it to be a success the left and right hands had to know what each other was doing. If folks pulled away and become hermits back at their desks, we were doomed. Another challenge was the arbitrations work flow was rife with undocumented dependencies across its legacy systems and there were no experts to help light the way across the labyrinth. This was a “butterfly flaps its wings scenario.” Any code being removed or changed was high risk. This meant QA had more load and pressure to create regression tests of systems and areas unmapped. Next, members of my team were offshore and the Waterfall guys weren’t familiar with how to handle the challenges this brought. The project was also going to be integrating with legacy applications. In fact, the AS 400 platform that would be impacted had a copyright date of 1997.
Fig. 4 Arbitration screen in AS 400
The challenges also spread in from the culture of the Waterfall side. There was a strong sense of old guard. They viewed us as the young new kids on the block driving their shiny Agile car down a street in their neighborhood. Manheim has been around for almost 60 years so I couldn’t really blame them.
5.3 Project Timeline
As one can see the challenges abounded in this project for sure. However, as the old saying goes look at your challenges as opportunities. I had plenty of these through the course of the Arbitrations project. There were many things my QA team did right, and things we could have done differently. Before revealing the What Went Well list I want to layout the project timeline. This will help anchor some of my What Went Well items. This is how it looked from the 10k foot scheduling perspective. The project was about six months in length with the first two months dedicated to our teams researching and scratching together requirements. On the third month into the project, my team had already begun working and turning out cards to be tested. The Waterfall side was still revising their requirements and getting various bureaucratic approvals; they have tons of gates and signoffs that happen before even making a key stroke to develop. In month four of the project, my team had already released functionality into production. It was dark but ready to be called upon once the adjoining pieces were released. Also in month four, with just two months before the project deadline, the Waterfall QA side was just beginning to kick the tires on their code.
6. Retrospective
6.1 What Went Well
Alright now we have the basic schedule topography so let’s discuss the What Went Well. For starters the business stake holders bought into the idea of having an Agile group take a major stake in the project. These folks were from the physical auctions and had been accustomed to working with the Waterfall folks for a long time. They had been proselytized with the Waterfall view of how projects should be done. It was a step of faith to work with us and their excitement was reassuring.
The next What Went Well is our Agile process brought all its team members together early to write and review requirements. This really wasn’t a departure of our normal Agile process, but squared against Waterfall and how they approached things this characteristic enabled us to get a jump start in many ways. For example, writing test cases began early. This was valuable because it mitigated risks and avoided pitfalls that were waiting to ambush us down the road. The Waterfall side being linear put them at a disadvantage. They weren’t going to let anyone look at the requirements until they were done and signed off by some committee. This of course increased the risk that gaps and pitfalls wouldn’t be found till later down the road when more people like QA were looking at them. One other plus of this Agile characteristic is I was able to begin tackling the problem of integrating our test environment with the Waterfall QA environment. This became no small task but pouncing on this early really helped me save more time which was already at a premium in this project.
The third item in the What Went Well column is definitely something that wasn’t orthodox to my QA Agile process. As I mentioned before, the Waterfall side was operating in a very linear fashion. They weren’t going to engage their QA lead until the requirements were signed off and no testing could start until development was complete and turned over. This project had a tight schedule and to offset the negative impacts of the linear Waterfall approach, I made an unorthodox move to help the QA lead on the Waterfall side. The lead didn’t have her QA resources completely brought in yet so I culled out some of my team to fill out her team. This had several advantages. For example, my team already knew a great deal about the Arbitrations project and the changes being made because they had been so involved with the early card requirement reviews and estimation sessions. Also, they were Agile and this would create an Agile ‘inception’ on the Waterfall QA side helping to lean some of the practices on that side hopefully in my direction. Did this move cost me some in the area of capacity? Yes it did. However, the payoffs turned out to be worth it.
The fourth item under What Went Well was similar to the previous point in that it wasn’t a move typically found in our normal QA Agile process. As my QA team was beginning to get cards in and analyze them, I was successful in making the business case to bring in the Waterfall QA lead about two weeks prior to their original arrival date. You see, the Waterfall leadership didn’t see a need to bring the QA lead in until their requirements were almost signed off on. I saw this as a disadvantage. The value I saw in bringing them in early was to get them involved in our stand ups, card estimation sessions and to begin knowledge transfer on our application and where its integration points were with the other arbitration components she would be managing the testing for. I also wanted to begin making this person an Agile sympathizer. I wanted them to latch onto our transparency, open flow of communication and see our efforts and work approach as a force multiplier instead of some shoot from the hip reckless organization. I did try to even go one step further and sit her in our team room to strengthen the collaboration. The Waterfall powers that be nixed that for some reason.
The final What Went Well bullet I’d like to mention was volunteering to take the vocational lead role on the project for QA. This was another unconventional move because typically you have a project manager that is over the effort and just manages each group independently. In this case each QA lead would have reported to the Project Manager. However, with the QA effort of the project rolling up under one lead it put an Agile person in this critical position. It also allowed me to lessen the silo mentality of the Waterfall side and give me a spidey sense of problems that were coming up on the horizon as a manifestation of the silo affect.
6.2 What Didn’t Go So Well
Now, in any good retrospective you learn a lot from What Went Well but probably more is learned from the stuff that didn’t go so well. Here are the items that fill the What To Improve column. The first item topping the list is our teams didn’t have a good huddle at the project genesis. I feel some of the cultural challenges mentioned at the beginning really persisted through the project. Some of this could have been mitigated if both teams had met early on and shared what might be weaknesses or strengths about each groups DNA. We could have done it in a lunch and learn format to promote a relaxed atmosphere and take the edge off of some people’s perspectives about Agile and Waterfall. For example, Waterfall brought an advantage to the table that wasn’t clear in the beginning. Agile in general avoids documentation. Well in Manheim our release approval process demands documentation and proof that tests have been run and passed. Our Agile ways wouldn’t have supported the level of detail that was needed for approval. As a result of my team being unaware of this requirement we would have been snagged by the requirement down the road.
Not enough transparency around each QA teams’ process is the second item I would call out as something to do better next time. This is somewhat related to the previous point, but gets to the brass tacks of the heartburn I faced dealing with the Waterfall group. Case in point was the schedule had a tight timeline and it is common with our QA Agile process that we begin light functional testing while the code is still in the process of being developed. I assumed we could do this and when we began turning the lights on for the Waterfall pieces I introduced this idea to the two Waterfall development leads and there was resistance. I managed to sit down with one and win him over after explaining what would be done and how we would not impact his velocity. The other lead just flatly denied my request indicating it just wasn’t the way they did things. I believe that if I’d covered my QA strategy and pitched it to them before the schedule became tighter and more tense they both could have been won over and won over earlier.
The final item in this punch list is the need for calculating key meetings with the Waterfall team separate from our larger project team meetings. I’m defining a key meeting as one that is timely and keeps the communication flowing. Our approach in this project was to let the Waterfall team drive the frequency of the meetings and how they were carried out. This wound up biting us a little bit. The symptoms were gaps and gotcha’s began to sprout up in the meetings when the project team was assembled. Several of these items (mostly on the Waterfall side) could have been given attention earlier. Why were they not given attention earlier? Attempts would be made to raise things which had the tingling of an issue. However, the Waterfall culture and play book had them in the zone on development. They couldn’t be bothered with anything and felt it could just be discussed in the larger project team meetings. Time would go by without these potential issues being investigated. At the larger team meetings is when the Bose noise cancelling headphones were removed and people opened up to discussions about these potential issues. Sure enough they’d be tagged as a gap or something would need to be changed.
My calculation for these key meetings would look like this. With the key players on the project, create a plan to establish simple and quick stand up type meetings for every other day. The location would have to be close and convenient for the Waterfall guys to inconvenience them the least. This solution is something that would be appropriate to bring up in the ‘huddle.’ These quick hit type meetings would have been successful because they were located conveniently for the Waterfall guys, have the frequency to catch things early, but not too often where it slowed their progress and they would keep the communication juices flowing to dig into possible issues.
7. Head on collision but Rich Experience
The Arbitrations project for both the Waterfall and Agile side brought many eye opening lessons to the table and was a fantastic growing experience. Was it a head on collision of ideas and methodologies, absolutely! However, if there were just a few things to absorb from my experience or an elevator speech, I’d say it was the following: 1. Different methodologies can attract and win 2. Take lead positions to influence 3. Be transparent with your processes- people fear what they don’t understand.
ACKNOWLEDGMENTS
I’d like to thank my Team Manager Jonathan Matas for putting me up to the challenge of writing this paper and telling my story. I would also like to thank Troy McLain for working so diligently and spending a painful amount of time with me during this process. Finally I’d like to extend a tremendous amount of gratitude to Joseph Yoder for his substantial time investment in working with me. Tons of great help that I couldn’t have gotten anywhere else…thanks Joseph.
