In a heavily regulated environment, such as the Federal government, comprehensive documentation provides traceability and is viewed as a necessary and critical artifact of the standard software development life cycle. However, the government’s approach to documentation often clashes with one of the major tenets of the Agile Manifesto, which lauds “working software over comprehensive documentation.” While the government is increasing its support of agile Information Technology (IT) development approaches, documentation requirements set forth by Federal IT governance continue to pose challenges for agile teams in the Federal space. In this document, we will explore the challenges presented by Federal documentation requirements through a case study of the Food and Drug Administration (FDA) Emergency Operations Network Incident Management System (EON IMS) Agile Team. We will then identify solutions and best practices used by the EON IMS Agile Team that can help other agile teams in heavily regulated environments overcome these obstacles.
In 2010, the White House Office of Management and Budget (OMB) released its “25 Point Implementation Plan to Reform Federal Information Technology Management.” Within the plan, the OMB stated that “despite spending more than $600 billion on information technology over the past decade, the Federal government has achieved little of the productivity improvements that private industry has realized from IT” (Kundra, 2010). These failed IT investments often lead to heightened scrutiny of Federal IT spending by watchdog organizations, lawmakers, and the American public.
In an effort to meet demands for transparency, accountability, and traceability, the government emphasizes comprehensive documentation as a best practice for IT portfolio management. When implemented effectively, documentation is a useful tool for managing a project’s progress and ensures continuity of operations during transition periods (e.g., when a new contractor takes over a project). Documentation can also help Federal IT projects provide evidence of their compliance with Federal IT governance (i.e., Federal laws, policies, and government/industry standards) and project-‐level requirements, including agency-‐specific software development life cycles. As a result of its potential to provide these positive gains and increase the government’s return on investment, documentation is viewed as a critical IT project deliverable, almost as important as the software itself.
However, as the Agile Manifesto suggests, documentation does not inherently guarantee the production of high-‐quality software, or even functional software. In fact, documentation does not inherently ensure continuity of operations or compliance either. As a result, the gap between the government’s need for comprehensive documentation and the agile team’s focus on working software creates an overarching challenge for agile teams in the Federal IT space. These teams must strike the proper balance between delivering software in alignment with the team’s agile values and satisfying documentation requirements enforced by Federal IT stakeholders.
2. CASE STUDY: FDA EON IMS PROGRAM
To frame our discussion of this challenge, we will examine a case study of the team that supports the Food and Drug Administration (FDA) Emergency Operations Network Incident Management System (EON IMS) Program. The EON IMS Agile Team uses an agile methodology based on Scrum that is tailored to Federal software development life cycles. In Exhibit 1, the roles of the EON IMS Agile Team are mapped to traditional Scrum team roles.
The responsibilities typically executed by a Scrum Master are split between the team’s Operational Program Manager and Functional Program Manager. Unlike a traditional Scrum Master, the Program Managers have managerial authority over the team members involved in development activities and lead the project monitoring and control functions. Similar to the traditional Scrum Master, the Program Managers ensure that the team is successful in accomplishing its work, and they are responsible for removing any obstacles. For the EON IMS Agile Team, the equivalent of the Scrum team’s Product Owner is the FDA Contracting Officer’s Representative (COR). Unlike the traditional Product Owner, the FDA COR is not the final decision making authority for the program. As the liaison among the program’s many different stakeholder groups, the FDA COR communicates the final decision to rest of the EON IMS Agile Team from the relevant stakeholder group(s). The EON IMS Development Team is similar to Scrum’s development team. The EON IMS Development Team performs the work necessary to achieve the customer’s objectives. One variation between the EON IMS Development Team and a traditional Scrum development team is the inclusion of a Technical Writer, who leads the writing and editing efforts for most program documentation.
The EON IMS Agile Team supports the system at the center of the EON IMS Program. The EON IMS is the FDA’s system for capturing the large volume of scientific and regulatory data related to product problems and adverse events for FDA-‐regulated human and animal food. Since the program’s initiation in 2006, the number of active system users has increased by 900%, and the program has expanded to five additional Centers and Offices within the FDA. Currently, six FDA stakeholder groups actively seek system enhancements and new development work to ensure that the EON IMS continues to meet the FDA’s mission objectives.
The EON IMS Program adheres to the FDA Enterprise Performance Life Cycle (EPLC) framework. The FDA originally developed the framework based on waterfall development principles and with the assumption of a “big bang delivery.” The framework has 10 phases, each of which ends with a quality gate, or Stage Gate Review (SGR). SGRs require deliverables to be submitted for review to the project’s Critical Partners. The Critical Partners are government stakeholders that represent the key functional areas involved in the project. The Critical Partners then review the team’s progress and evaluate the work based on a defined set of exit criteria. If the Critical Partners uncover any issues, they send the deliverable(s) back for revision. As a result, failure to pass the SGR can be a significant source of risk for FDA programs. Exhibit 2 illustrates the 10 SGRs involved in the EPLC framework.
As part of FDA’s approach to enterprise IT portfolio management, FDA projects must adhere to legislative policies, regulatory requirements, and industry IT standards in addition to the Enterprise Performance Life Cycle SGR requirements. Therefore, FDA projects are required to create documentation as evidence of compliance with Enterprise Architecture, Security, Acquisition Management, Finance, Budget, Human Resources, Section 508, Capital Planning and Investment Control, Records Management, and Performance policies and standards. Representatives for these functional areas are included as Critical Partners and evaluate documentation as part of the EPLC framework.
The EON IMS Agile Team was not immune to the risks posed by Federal IT governance requirements and agency-‐specific standards, such as the SGRs of the EPLC framework. Throughout this section, we will explore the following specific challenges faced by the EON IMS Agile Team:
⦁ Accounting for the SGRs within the program schedule
⦁ Managing multiple stakeholder groups during document review cycles
⦁ Planning for ad hoc documentation
⦁ Creating lean documentation
One of the most significant challenges encountered by the team involved accounting for the SGRs within the team’s agile program schedule. The FDA schedules SGRs without accounting for individual program/project schedules. Therefore, the EON IMS Agile Team had to properly account for the SGRs when creating their overall program schedule. The team needed to consider the amount of time to be spent on the documentation associated with these SGRs, each of which has its own unique set of requisite documentation. To ensure optimal utilization of technical resources and avoid missing deadlines, the team needed to plan properly for team members to complete SGR documentation without negatively impacting the sprint life cycles and the overall program timeline.
Another significant challenge for the EON IMS Agile Team was managing the program’s multiple stakeholder groups during document review cycles. According to the EPLC framework, project teams must submit working versions of the SGR documentation to the Critical Partners for their review and approval prior to the SGR. For waterfall projects using the EPLC framework, the process of obtaining approval and finalizing documentation can take as long as four weeks—an impractical timeframe for agile teams. As a result, the EON IMS Program Managers and the FDA COR needed to set expectations to manage schedule risks effectively. Since the document reviews could be lengthy and were dependent on stakeholder availability, the team needed to ensure that stakeholders were aware of program deadlines and committed to meeting those deadlines. Additionally, the team had to set aside time in the program schedule to prepare for comments from peripheral stakeholders that may involve significant effort to address.
Beyond the challenges associated with the SGR process, the team had to plan properly for ad hoc documentation. The EPLC framework requires ad hoc documentation that provides evidence of the project’s compliance with Enterprise Architecture, security, acquisition, and other policies and standards. For example,
the team is regularly subject to FDA Security audits and assessments that often require the creation or revision of documentation unrelated to the SGRs. These audits can be troublesome for agile teams in the Federal IT space that do not plan appropriately within the program schedule.
Lastly, the team faced challenges related to creating lean documentation. The team had to produce documentation that both met the requirements of the EPLC and did not detract from the team’s ability to produce working software. Most importantly, the EON IMS Agile Team needed to ensure that documents were approved once submitted to the FDA COR and Critical Partners to avoid rework. Possible reasons for non-‐ approval could include noncompliance with the EPLC template, incomplete sections or missing information, or numerous grammatical and/or mechanical errors. An added challenge was that the team typically had to use the tools, templates, and resources required by the EPLC framework. Therefore, the team was unable to use more agile-‐friendly documentation tools, such as wikis, that could ease the process of updating documentation.
To address the challenges detailed in the previous section, the EON IMS Agile Team worked with the FDA COR to tailor the EPLC framework to better align to the team’s agile methodology. The EPLC framework was tailored by completing the Project Process Agreement (PPA), an EPLC deliverable that formally defines the framework that will be used by the project. The PPA allows the FDA “to preserve a consistent and repeatable project management methodology while recognizing in a deliberate manner when certain elements of the framework are not applicable or not cost effective for a particular project” (Department of Health and Human Services, 2012). Exhibit 3 illustrates the framework established through the EON IMS PPA.
After completing the PPA, the EON IMS Agile Team addressed the challenge of accounting for the SGRs within the program schedule. The team created a comprehensive and agile program schedule by combining SGRs to establish a unified documentation review cycle and used sprint planning sessions and sprint customer evaluation reviews to obtain the Critical Partners’ authorization to proceed. For example, as illustrated in Exhibit 4, the team combined the Design, Development, and Test Stage Gate Reviews to avoid any disruptions to the sprint life cycle. Then, the team could obtain authorization to proceed to implementation and
deployment through the sprint customer evaluation review. During these meetings, the team provided a demonstration of the working software, and the Critical Partners were presented with the appropriate documentation. Based on the working software, the Critical Partners provided their authorization to proceed. Then, they agreed to provide any feedback on documentation by an agreed upon date that provided the team with enough time to address any documentation revisions prior to deployment.
The comprehensive program schedule allowed the team to avoid version control issues that may have resulted from multiple documentation cycles. For example, if the team had not tailored the EPLC requirements, then the team would have needed to deliver finalized storyboards during sprint planning to satisfy the Requirements SGR. However, the delivered storyboards could become out of sync with the master storyboards by the end of the sprint. Therefore, the team would have needed to update all of the “finalized” documentation again, resulting in unnecessary rework. Then, the second round of finalized documentation could have also led to documentation version numbers being out of sync with the software version numbers—a headache for any team’s configuration management process.
According to the new comprehensive program schedule, the team was able to draft feature documentation throughout the sprint without pausing for SGRs. Critical Partners provided an official go/no go decision during both sprint planning and sprint customer evaluation reviews to satisfy EPLC requirements. Similarly, design, development, and test documentation was developed throughout the sprint and formally delivered during the sprint customer evaluation reviews. To properly plan for the work associated with updating these documents throughout the sprint, team members estimated their time needed (in hours) for documentation on a story basis. For example, when development team members provided estimates for features, they included the time it would take to update the necessary documentation with the information for that feature. Then, the Program Managers included documentation within the schedule and assigned a document owner. The document owner(s) then worked with the Technical Writer to create lean documentation using the solutions and best practices described in this document. This approach ensured that team members self-‐manage their workload and did not overcommit their time in a given sprint.
After addressing issues related to planning for the SGRs, the team was able to focus on managing the program’s multiple stakeholder groups. The EON IMS team regularly communicated information to stakeholders in meetings required as part of the government’s standard process (e.g., Integrated Project Team Meetings and Weekly Status Meetings). Additionally, the team always met with the FDA COR before communicating with the larger stakeholder groups. After obtaining the FDA COR’s buy-‐in, the team could then ensure that the COR was the primary liaison for communicating program information to the Critical Partners, which helped to secure FDA stakeholder buy-‐in. The team continued to invite all stakeholders to daily stand ups, sprint planning meetings, and customer evaluation reviews, so they remained up-‐to-‐date on all program information.
To address ad hoc documentation, such as FDA Security documentation, the team accounted for the documents in advance using the program schedule. The team included each new document as a future sprint activity, which allowed the team to iteratively approach the documentation alongside regularly occurring sprint activities. These documents were added to the schedule as soon as possible, so the team could plan accordingly. When these documents (or any program document) involved significant effort to address, the team implemented a pair-‐writing technique. The team’s Technical Writer periodically met with the documents author(s) (e.g., Development Lead, Test Lead, etc.), and the team members reviewed each section of the document to identify the content that should be added or updated. As an outcome of the pair-‐writing process, the team members had a rough draft of the material that needed to be included in that section, which then allowed the Technical Writer to draft the document section by section. The Technical Writer then submitted the document to the document owners for iterative reviews throughout the sprint. This practice alleviated pressure from the document owners, who were typically other members of the development team who needed to focus on development and testing.
To address the challenge of creating lean documentation, the team implemented best practices for creating documentation that both receives customer approval and does not prevent the team from performing high-‐ quality sprint work. These best practices were developed from lessons learned captured during the team’s sprint retrospectives. The team has found that the application of the following best practices helps to ensure that all documents meet their intended purpose in a concise, targeted manner.
⦁ Reference to the source of information rather than duplicating/re-‐writing (i.e., single-‐source information)
⦁ Document the as-‐is state rather than the to-‐be state
⦁ Document in real time during conversations with the internal team or stakeholders (i.e., document while developing, document during feature gathering sessions, etc.)
⦁ Assign document owners
⦁ Utilize a good technical writer or a team of technical writers
⦁ Use pair-‐writing for lengthy documentation or new documentation
⦁ Make informed decisions about adding to a document versus creating multiple documents
In summary, the EON IMS Agile Team was able to overcome the challenges presented by documentation requirements in the Federal IT space. Specifically, the team overcame the challenges detailed in this paper through the implementation of the following solutions:
⦁ To address the challenge of accounting for the SGRs, the team tailored the standard process with approval from the FDA COR to align the EPLC framework with the team’s agile methodology. The team could then prepare for SGRs in the program schedule and combine SGRs to prevent any disruptions to the sprint life cycle.
⦁ To address the challenge of managing multiple stakeholder groups during document review cycles, the team used standard communication tools and project meetings to manage stakeholder expectations and keep them informed of the program’s status. The team also worked closely with the FDA COR to socialize decisions before they were formally documented, which increased the likelihood of obtaining stakeholder buy-‐in.
⦁ To address the challenge of planning for ad hoc documentation, the team included ad hoc documents as items in the program schedule and assigned them to future sprints. This approach allowed the team to reduce the risk of overcommitting time for sprints that aligned with the due date for ad hoc documentation.
⦁ To address the challenge of creating lean documentation, the team developed best practices based on lessons learned shared during the team’s sprint retrospectives and applied the best practices for creating lean documentation to their approach.
Through the implementation of these solutions, the EON IMS Agile Team established documentation as a product of the sprint and realized many critical benefits. The team was able to improve its defect removal efficiency, increase the likelihood of customer acceptance, expand the program to additional stakeholder groups, and increase overall program funding while demonstrating transparency, accountability, and
traceability. The EON IMS Agile Team also decreased the time spent on SGRs and reduced the time lapse between sequential deliveries of requirements, design, development, and test documentation. Today, these solutions and their consequent benefits continue to be critical to the sustained success of the EON IMS Agile Team.
Although the government is increasing its support of agile development methodologies, documentation requirements are likely to remain a primary challenge for agile teams in the Federal IT space. However, using the knowledge gained through this case study, agile teams in the Federal IT space and other heavily regulated IT environments are poised to successfully reduce the burden of documentation requirements while continuing to add value for their customers and focusing on what matters most: delivering high-‐quality, customer-‐driven solutions.
Department of Health and Human Services (2012, July 18). Enterprise Performance Life Cycle Framework Overview Document. Retrieved from http://www.hhs.gov/ocio/eplc-‐lifecycle-‐framework.pdf.
Kundra, V. (2010, December 9). 25 Point Implementation Plan to Reform Federal Information Technology Management. Retrieved from https://www.dhs.gov/sites/default/files/publications/digital-‐strategy/25-‐point-‐implementation-‐plan-‐to-‐reform-‐federal-‐it.pdf.