Infusing an Agile Requirements Backlog in a Large Department of Defense (DoD) Program

Unlike commercial Agile practitioners, the Defense sector has more restrictions when restructuring milestones for an Agile approach. However, by leveraging techniques from Agile software development into the Systems Engineering domain, it is possible to overcome those constraints and bring the benefits of Agile to one of the largest, most needed areas of development. This report documents the practical application of Agile techniques to create a requirements backlog for a large DoD project, within DoD constraints, and before the team even knew about the existence of Agile. The report also highlights the characteristics of Agile Systems Engineering learned from the experience, the organization of the teams, and the MBSE technical storyboarding and allocation approaches that resulted in the acceleration of the program’s Systems Requirements Review milestone by 70%. The report presents key considerations for scaling the techniques used during the experience for a large adoption of Agile in DoD.

1.  INTRODUCTION

The Defense sector presents unique challenges for the adoption of Agile techniques. Defense programs are typically characterized by the presence of multiple and diverse sets of stakeholders, each with different requirements for performance, funding, and schedule. Often, end-users are co-located with the systems overseas and are not easy to reach to gather their feedback. In many instances, DoD has separate “Oversight” and “Development” organizations, which adds levels of bureaucracy, slowing down communications throughout the program’s lifecycle.

Today, most DoD programs are implementing some type of Agile software development methodology to accelerate their deliverables. However, these programs may be overlooking the benefits of adopting Agile beyond the software development domain.

This experience report dives into the practical implementation of Agile in the Systems Engineering (SE) domain to generate a requirements backlog within the constraints of the DoD acquisition lifecycle. The product developed was a complex system-of-systems comprised of both hardware and software components. This experience report dates to 2009, thus preceding the 2015 release of the DoD 5000.02 instruction which provides guidance for iterative and incremental releases of capabilities within the DoD acquisition lifecycle. Moreover, the techniques and accomplishments reported herein were achieved without the team being aware of the existence of Agile techniques, or their potential use in SE.

This experience therefore presents an authentic validation of Agile SE. The approach led to a 70% acceleration of the DoD Systems Requirements Review (SRR) milestone. It validates that shifting from using Agile for rapid product delivery into an end-to-end asset does in fact enhance team productivity, processes, and effectiveness, as well as promotes frequent collaboration and communication.

The experience report elaborates on the application of Agile techniques used by Mr. Smith’s team to accelerate the SRR milestone of a major DoD acquisition program, and provides insights about how Ms. Castro’s DoD organization will leverage this approach to support a large adoption of Agile in DoD.

2. Background

Mr. Smith is a Systems Engineering Partner at WRAYN, LLC. He has been utilizing systems engineering tools and methods his entire career. He has worked as a Systems Engineer, a Program Manager, a University Instructor, a Field Engineer and entrepreneur. His 30 year SE career has spanned systems development, test, and working for tool vendors. As a long-time entrepreneur, he has developed re-use methodologies and deployed Agile techniques in seemingly inflexible organizations. Currently, he is providing Systems Engineering to convert piloted helicopters to autonomous flight. His company’s primary focus is supplying experience-based on-line and live-learning in Agile, MBSE, and Re-Use. While he has worked on a variety of systems including Military Vehicles, Rotary and Fixed-wing aircraft, Spacecraft, Training Systems, Medical Devices and IT, he says working on amusement park rides was the most fun.

Ms. Castro is a Lead Systems Engineer working at DoD. In her 14 years working there, she has been the Lead Systems Engineer of a major DoD program, a product owner and team lead of a scrum software development team delivering capabilities for a major DoD program, and a subject matter expert in her organization on the topic of Agile systems engineering. She is currently championing efforts for a large adoption of Agile at her organization.

Throughout the years, the authors have worked on large multi-year DoD programs and experienced customer dissatisfaction for two reasons: (1) The delivered systems typically do not meet expectations since system requirements negotiated early in the program’s lifecycle are never revisited or updated to assess changing customer needs, and; (2) The SE phase has been lengthy, often with detailed design starting prior to completion of SE’s system specification. Traditionally, SE follows a sequential waterfall lifecycle, where once a lifecycle phase has been completed it is not revisited again during the program’s life. In many instances, SE and software development teams are separate work units with minimal interaction with each other, leading to a disconnect between implementation and the system’s requirements that were identified early in the program. This often results in the rework of SE products. Even worse, SE artifacts and processes can be perceived as being done only to comply with policy, and not as an activity that delivers value to the entire lifecycle of the program.

In today’s dynamic environment where technology is constantly changing, requirements are not expected to be static during the lifecycle of the program; and there is a demand for flexible SE processes and program milestones to incorporate new requirements identified through customer and end-user feedback.

This experience, performed during the infancy of formalized Agile SE concepts, superbly aligns with and validates what have become currently-accepted Agile SE techniques. We offer up alternatives where differences do exist.

3. Deploying an iterative systems engineering approach in a large DoD program

3.1  Scope of effort

Modernize major Army vehicle system-of-record: A complex System of Systems

In the 2000s, the US Army initiated a large program to modernize one of its combat vehicle systems. The program consisted of a complex system-of-systems comprising a family of variant vehicles built on common architectures. For this program, both hardware and software upgrades were planned. The upgrade included new system software, processors, sensors, controls, displays and communications. This report discusses the period of time leading to the System Requirements Review (SRR).

Mr. Smiths’s portion of the project was to be executed in three stages:

Stage 1:   Preparing for the analysis effort during November and December;

Stage 2:   Performing the actual work of analyzing over 50 system capabilities, from January through May, and;

Stage 3:   Successfully completing a SRR in June.

The second stage compressed what normally would have been 18 months of analysis into a mere 5 months, a seemingly impossible 70% reduction of the schedule. This type of reduction had never been performed by the organization before. None the less, it had to be done: the SRR is a high-stakes review in Defense work, usually tied to progress payments from the customer. Failure was not an option. Without the benefit of any specific methodology to fall back on, the team was forced to come up with a new approach to achieve this objective.

During the DoD SRR Milestone, the system developer (the “Prime Contractor”, or “Prime”) demonstrates understanding and mastery of the system requirements to the Customer (the U.S. Army in this case), and allocates those requirements to the implementation teams. It is important to deliver the work products required; otherwise the results could be rejected.

The system functional requirements describe what the system must do to satisfy its mission objectives. “Allocation” of those system requirements involves deriving a set of subsystem requirements which together collaborate to satisfy the system requirements. The requirement backlogs are provided to individual teams for implementation.

This experience report focuses on the work performed to generate the system functional requirements, and the subsystem functional and interface requirements. The “requirements backlog” was the set of “allocated” subsystem requirements received by each implementation team.

3.2  Stage 1: Preparation, Planning, Coordination and Training: November and December

In September, Smith’s company submitted a competitive proposal which included deploying an “Engineering SWAT team” concept previously used at the Prime. The SWAT approach brought in a specialized team with tool skills for documenting, leadership skills for facilitating, and domain expertise for focus, to rapidly accelerate technical progress. Previous beneficiaries at the Prime had described Smith’s Engineering SWAT team as “…doing the impossible…”. In late October, the Prime awarded the subcontract to lead the SRR effort to his company.

With no methodological basis to build from, the first task was to determine key success factors, and devise a workable approach to complete this work. November and December were consumed with planning, and building the team.

The Approach: Planning for success

Planning Success Factor 1: Eliminating the Review Cycle. One primary factor that could thwart the effort was stakeholder buy-in. On DoD programs then and now, the review/signoff process typically involves having many people review and comment on a completed work product such as specification, after a small team has invested significant time developing it. This time-killing process involves creating a work product, eliciting comments, negotiating updates to the work product, then fighting for (or cajoling!) sign off from all of the stakeholders. Worse, multiple iterations and additional negotiations are usually required as people learn of the changes others have made. The review/sign off alone would cause the schedule to unravel.

The only way to avoid this dire fate was to obtain concurrence as part of the work cycle. This required including all stakeholders during the analysis. Therefore, Smith decided to develop the analyses with all stakeholders together in the room. When disagreements surfaced, the stakeholders themselves would directly negotiate the resolution, collaboratively resolving any differences. Once the analysis satisfied all the players, concurrence was simply a byproduct, not an additional step. In the end, the effort was successful, proving this theory correct. During November and December, the Prime coordinated and obtained commitments from all the stakeholders to participate in the analysis meetings starting in January. These stakeholders included the customer, the development teams who would be receiving the requirements backlogs, and the testing teams. The Prime also coordinated to have the customer invite system users, in this case soldiers, to the analysis meetings.

Once the decision was made to perform the analyses in team settings, some simple math determined how many would be required. In the 20 weeks between January and May, three teams each with 17 analyses totaling more than 50, would need to be completed, at a rate of about one full analysis every 5 to 6 days. That meant that all stakeholders would need to lend up to 3 people to the effort: one for each team, for 1 to 2 hours per day for nearly the entire 5 month period. Smith selected his three team leaders largely based on four qualities: leadership skills, communications skills, facilitation skills, and tools skills. Each team leader was provided a list of system capabilities to analyze and prioritize. The capabilities were prioritized based on availability of subject matter experts and complexity.

The considerable coordination effort cannot be understated. The Prime obtained buy-in on our approach from the managers and customer. Skeptical managers had to be persuaded to commit significant resources to the effort. Because the imperative of meeting the short schedule was so critical, and the consequence of failure so grim; managers agreed to take the risk on this completely new approach.

Planning Success Factor 2: Model Based Systems Engineering (MBSE). A SE Model would source and warehouse the requirements. At the time, the model-based systems engineering (MBSE) concept was gaining wider acceptance. The tool used by the Customer was Rhapsody, at the time owned by Telelogic. In November and December, Smith procured copies of the tool, and trained the three team leads to act as tool jockeys as well as leaders. The tool framework was also structured by the Prime’s manager to support the model integration that would be required as multiple teams built their models.

Planning Success Factor 3: Easy to Understand Storyboards. The final critical factor was ensuring people could easily understand the analyses. To be successful, only easy-to-understand analyses would do; understanding overly complicated diagrams or lengthy pages of dry text would sideline the effort. Therefore, Smith chose to use a technique unknown within this military-vehicle community called, “Storyboards”. This method involves creating images on individual sheets representing scenes in a sequence of events. In a functional analysis, this would allow functional sequences (at the system level) to be quickly drafted, reorganized and visualized. This technique bridged the divide between the soldier’s operational world and the engineer’s development world, thus becoming one of the key pillars of success for the effort, and was enthusiastically received by all participants. In addition, storyboards greatly reduced discussion because, as it turns out, a picture really is worth a thousand words!

3.3  Stage 2: Analyzing over 50 system capabilities to create the requirements backlog: January through May

By January the work was ready to begin. The infrastructure was in place, the teams and customer were aligned, and critically, months of scheduled meetings were established.

We began by analyzing a relatively simple capability to ease into the “daily meetings”. Smith and the Prime’s manager led the first assessment. Since this was the first time working this method, all three team leaders attended the first assessment to ensure consistency in our approach. The first participants were the customer, the Prime’s development teams, and the SWAT team leaders/tool jockeys.

Iteration 1 – System Level Capability Analysis using Activity Diagrams. SysML “Activity Diagrams” were chosen as the primary artifact for capturing the system functional analyses. As the team progressed forward, the customers would describe their vision of the capability. The leader quickly documented the dialog in Rhapsody using Activity Diagrams. As the functional sequence was described, the activity diagrams evolved. Sequential and simultaneous behaviors were captured. An incredible number of notes capturing the thoughts, concerns and experiences of the participants found their way on the page. At this stage, “pretty” pictures were not a priority: simply getting the wealth of information into the model was the primary objective.

Building Effective Teams. A key leadership factor was a strong knowledge of SE. It was crucial that the SWAT team be able to quickly operate the tool to summarize diverse perspectives into words and pictures in the model. The SWAT team members all came from Systems backgrounds, which leant itself to valuable categorization of information. Organizing the deluge of data in real time was a key skill. Recognizing the difference between functional, interface or constraint requirements meant more focused discussion. Understanding the system’s levels of abstraction to properly assign information reassured the participants that key points wouldn’t be lost, even if they weren’t immediately pertinent. Interpersonal skills were also key. As with any group endeavor, some people had a tendency to monopolize the conversation, or drive the focus on narrow aspects of the effort. It became clear that the SWAT teams would also be “facilitators”. It was critical that all voices were heard and that work progressed apace. Selecting the SWAT team for their leadership skills turned out to be a fortuitous decision. Key skills in this area included active listening, articulating back to people who are passionate about their points that they were heard.

Daily Rhythm: Morning Meetings. The meeting mechanics were straightforward. The meetings were held every morning for 1 to 2 hours. The facilitator coordinated the meeting, starting with a specific topic to be addressed by the stakeholders. Often the discussion would explore tangential topics, which would be run to ground. Throughout the entire effort, the leaders would capture the inputs, thoughts and insights of the team members in the model as quickly as possible. Where possible, specific verbiage for function (i.e. activity) names, descriptions and requirements were hashed out in the meeting until all participants agreed.

Daily Rhythm: Preparing for Tomorrow. After the meetings, the majority of the team would return to work at their “day jobs,” attending to whatever tasks they had on their agendas. For the SWAT team leaders, the remainder of the day was spent organizing the flurry of notes that had been taken at the meeting. This primarily consisted of cleaning up the diagram content, ensuring semantically correct models, and integrating the model elements created that day with previous models. For example, normalizing “send message” and “transmit message” with the correct parameters. In addition, when concurrence on verbiage for the requirements was lacking, the leaders would apply their SE skills and craft well-written requirements. Completing this clean-up task often took the SWAT leaders all day and well into the evening.

The results were impressive, however. The net effect was holding to the three days of System Analysis, and three days of Allocation analysis, per capability.

Customer and End-User Engagement. From the beginning, we obtained customer buy-in and reasonable participation. This ensured that we incorporated the input from the Government engineering civil servants managing the contract. While not participating daily, the U.S. Army Customer participated enough to feel very comfortable with our system’s operational analysis. Our early request for Active-Duty soldiers who knew the system resulted in one very productive visit. However, their participation was restricted. To fill this gap, we turned to the Prime’s workforce itself. Dozens, if not hundreds, of Reserve soldiers worked at the plant, many with recent deployment experience working with our system. We elicited those with recent combat experience, in using, maintaining and supporting our system as crew, logisticians, and mechanics. We enthusiastically engaged these users, who participated in our daily meetings and added tremendous value to our analyses.

Early in the effort, we discovered that graphics could be attached to the model activities, taking full advantage of the Storyboard concept. Smith added two part-time artists to the meetings to produce the story artwork. By understanding the behavior, they developed rough, but descriptive sketches representing everything from concept drawings of new equipment, to depictions of soldiers operating the equipment which provided a very-understandable flow.

Figure 1 displays a close-up, sanitized example of one of the models showing how the graphics helped describe the system operation. The red box shows the context of this segment in the broader model. The broader model shows the high data density developed during the effort. Each yellow “note” is tied to activities or data flows, and represents a requirement derived from the Activity Diagram, operational capabilities, design data or architectural constraints. As the anticipated issues arose, and the inevitable engineering changes occurred, they were folded seamlessly into the diagrams.

The requirements were aggregated into a system specification, with the models included as context.

Figure 1. Storyboard Example and Data Density

 

Iteration 2: Allocation of Subsystems Requirement Backlogs. The first iteration consisted of a system-level black-box assessment. It ensured all capabilities of the system, as a whole, were fully analyzed and system-level requirements were generated and validated with all stakeholders.

The second iteration had a white-box subsystem-level focus. Starting with the requirements for each system action, we analyzed the internal system behavior for that action to identify the functions each subsystem team had to implement. Additionally, we defined the subsystem interfaces required to support that behavior. It was in this iteration, that we actually generated the requirements backlogs, which we infused into the organization using our new SE approach.

The hardware/software subsystem architecture receiving the allocated behavior was defined previously and owned by the implementation teams. Therefore, by defining the functions and interfaces for each subsystem, we were able to hand-off a requirements backlog to each implementation team, with the backlogs directly traceable to system behavior.

Had we not inherited the architecture, a similar approach could have been used to develop the systems architecture. In future publications, the authors anticipate describing how to apply Agile SE techniques to develop the system architecture.

In light of the white-box approach, the Iteration 2 tasks were slightly different. The tasks involved taking each activity from the system models, and breaking them down into a series of steps performed by the system using a sequence diagram. The players for this activity included members of the implementation teams.

We used the same team-approach, with the facilitator/tool jockey, the daily meetings, the “cleanup” and model integration during the day. But this iteration involved decomposing each action on the system activity diagram separately. We generated sets of sequence diagrams from each Activity Diagram.

As depicted in Figure 2, sequence diagrams show blocks at the top, each representing a hardware or software element of the system. The vertical line dropping from each block or “lifeline[1]” represents the life of that subsystem over time, which flows down the page. Interactions between lifelines represent subsystem interfaces. Reflexive interactions (to/from itself) are behaviors that the subsystem performs.

During the Iteration 2 meetings, we negotiated the interfaces between the subsystems, and identified the reflexive iteration behavior each subsystem needed to perform. We wrote requirements for both the interfaces and the subsystem behaviors, and related them to each operation as shown in Figure 2. These requirements then became the requirements backlogs for each implementation team.

The negotiation process between the teams exhibited some interesting dynamics. It became clear during Iteration 2 that some needed behaviors were overlooked during the bidding process. One benefit of this process was that all needed functions were unmistakably obvious. But on several occasions, the cost of implementing specific functions was more than a team could afford. Watching the teams hash out various allocations was a revelation. Typically, omissions of this type don’t surface until well into design, often remaining hidden until they reveal themselves during integration or test, when the cost impact is the most damaging. Another unforeseen benefit was its flexibility in handling dynamic requirements changes.

Figure 3 depicts the iterative process our team used. At the time, we did not know that this was an Agile process very similar to SCRUM! Figure 3 shows how the 50+ system analyses were divided among the three teams. Each team held daily meetings, broken into two, 3-day iterations; the first iteration produced the system functions, while the second iteration allocated to subsystem behaviors and requirements backlogs.

 

Figure 2. Requirements Allocated to Each Team (sanitized)

Figure 3. The Iterative Process Used

 

The pace of the entire effort was grueling. The team members were thrust together with varying inter-personal skill-sets. Some people thrived. Tempers definitely flared on multiple occasions whenever opinions clashed. There were even a few people who felt unsuited to the intense pace of the effort, and chose to leave. They were replaced with individuals better equipped to participate. This self-adjustment resulted in teams that operated efficiently and focused on delivering value.

3.4  Stage 3: Successful Completion of the SRR- June

By the end of May, the teams were wrapping up their work. The requirements were exported from the model and placed in a document for delivery. The document was approved by the stakeholders, which took only the time needed to route the document. As planned, no additional review cycle was needed. Preparation began on the SRR material. The decision was made to present the actual Activity Diagrams for each of the 50 system analyses to the Government. The Prime decided to have Mr. Smith’s team actually present the material, in a humbling departure from common practice. It is highly unusual for a Prime to invite their subcontractor to present at reviews of this type. This fact however, validates an important, often under-looked benefit of Agile: creation of the “badgeless” team as organizational lines fall aside with this type of multi-disciplinary team.

The review went very well. To ensure the audience remained cognizant of the “big picture”, we created “zoom out” slides with identified zones which could be referenced as we zoomed into the details.

The Government customer typically invites other agencies to these reviews. These other Government participants often ask the Prime probing questions from a fresh perspective. An unintended consequence to our Agile-like approach was that our Government customer answered many of these questions. Their participation in the process itself brought a high degree of knowledge to and ownership in the requirements that reduced the program risks, and validated our approach in an unexpected fashion.

4.  What We Learned

Among the many lessons learned from this experience, the following are the most important:

Storyboards. We were successful in engaging the Customer in this process. By clearly conveying the nuances of the system operation relative to every stakeholder, the Storyboards were absolutely critical in drawing out their input, and ultimately, their approval.

Change Management. Requirements changes were easily accommodated using this approach. This led to a much more thorough specification, and a more complete design activity.

Information Density. We benefited from two types of information density. First, we performed the work in only 30% of the typical time. The team addressed issues as they arose, making decisions or quick follow-ups after one day of ‘homework’. Problems never lingered, and were quickly resolved. Instead of a dozen emails over the course of days, our approach resolved most issues within 20 minutes. Second, the density of the information on the model pages was incredibly high as shown in Figures 1 and 2. These work products included: Analysis, Requirements, Traceability and Allocation, all without having to thumb through documents.

Self-Organization. Our self-organizing team successfully adjusted to team member’s needs as warranted.

Maximum Benefits, Minimum Change. The most important validation is this: this system WORKS! We successfully accomplished the objective, while fitting within the DoD milestones. The importance of this cannot be understated. While many people, correctly, point out that even greater productivity gains can be made by changing the DoD acquisition lifecycle, the reality is that an undertaking of this magnitude will take more than a change to 5000.02, as important as that is. It takes a cultural change: a change in the perspectives of dozens, if not hundreds of stakeholders. This process validates that gaining Agile benefits can come without changing the acquisition lifecycle. These benefits can be enjoyed today, while providing a vision of the even greater productivity which awaits the organization tomorrow.

MBSE. Show me, don’t trust me. This process quickly brought the benefits of MBSE and Agile to organizations. This effort shows a path that can deploy MBSE with a minimum investment. As the benefits become readily apparent through the models themselves, more organizations can invest in their own SWAT teams, spreading the methods throughout the organization, secure in the knowledge that results will follow.

Relating to Agile. After successfully concluding the SRR, we ran across the software Agile Manifesto, learned of the many Agile software techniques, particularly SCRUM, sprints and backlogs. As we mentally mapped our approach to Agile, it became clear we had independently ‘discovered’, and validated “Agile SE”.

Agile SE Work Products. A key lesson learned is that Agile SE does differ from Agile Software development. Agile SE is characterized by the creation of SE work products. Unlike Agile software development, which releases completed software versions at the end of each sprint, Agile SE won’t update ‘the whole system’. Instead, Agile SE uses Agile techniques to release SE work products such as specifications, system architectures, trade studies or verification plans/procedures.

Validating Agile SE. As we have mentioned, this experience occurred absent any awareness of Agile, but used many of today’s formalized Agile SE approaches and techniques. In addition, the effort fit neatly into pre-existing DoD milestones. Therefore, this example can be very useful in convincing your organization to adopt an Agile SE approach. Why? Because it provides independent validation that Agile WORKS, and is LOW RISK!

APPLYING THE LESSONS: We seek to replicate the success of our approach at Ms. Castro’s large DoD organization. To that end, we will leverage many of the techniques proven during our foray into the Agile workplace, and modify some of our approaches to accommodate the wider deployment. Table 1 categorizes the techniques used during this experience that we consider a validation of the Agile SE concept. The table also identifies how the techniques can be adapted to an enterprise deployment of Agile SE.

5. Conclusions

We found the techniques discussed in this experience report instrumental to transition the traditional SE process to Agile SE while supporting a large DoD acquisition program. The techniques provided tangible results with the following benefits:

• The process made it easier for the stakeholders and leaders to identify the strengths and weaknesses of the system to be developed;
• The teams were able to stay focused on the high value tasks and shared a common vision of what needed to be done and why;
• The SE teams were frequently improving their internal processes as they gained more experience and knowledge about the system to be developed and their own processes;
• The SE products were kept current and up to date;
• Acquisition program milestones and technical reviews could be achieved in a shorter timeframe when compared to the traditional waterfall DoD lifecycle.
• The process helped bridge the gap between the software development and SE teams since they had to frequently collaborate and exchange information, in this new approach.

6. Acknowledgements

Mr. Smith would like to acknowledge and thank his friend and manager at the Prime during this effort, Jozsef Bedocs. In addition to performing the necessary contractual and organizational work, it was Mr. Bedocs who built the Rhapsody MBSE framework which enabled the daily model integration.  Ms. Castro would like to thank her organization for providing an independent review of this experience report.

While both the authors are previously published, we are primarily products of the “school of hard knocks”-real life deployments in high-stakes environments. While very gratifying, our academic publication deficiencies have left our raw authoring skills somewhat lacking. That has made the shepherding provided by Mary Ann Lapham an even greater advantage. In addition to suggesting much-needed changes in the report’s flow and clarity, Ms. Lapham’s incredible optimism and her breadth of experience and knowledge in Agile has made our regular editing sessions a highlight of each week! Thank you so very much!

Copyright 2017 is held by the authors.

[1] Lifeline- Standard SysML term