Since our original Experience Report at Agile2016 on applying agile methods to clinical decision support software configuration in electronic health records (EHRs), we’ve expanded our use of agile to larger projects, more teams, and additional healthcare domains. Productivity, defect rates, and clinical user satisfaction all measurably improved. Employing user stories with acceptance criteria and agile modeling methods continued to benefit project scoping and solution design, even with increased range and size of projects. With larger projects, story-splitting techniques took on greater importance. Challenges encountered on larger EHR-based projects included prioritizing across a healthcare organization’s multiple product owners’ backlogs for the single EHR, and coordinating work when multiple development teams work together on a single project. Approaches to these challenges are described, along with potential future directions for expanding analytics for more data-driven iterative improvements in EHR feature reliability and usability, and for gauging their benefit to patient care.
1. BACKGROUND: Brief Summary of Our Initial Experience Report (Agile2016)
Organizations increasingly derive benefit from applying agile principles and practices to endeavors outside pure software development. At Agile2016, we presented an Experience Report on “Agile Clinical Decision Support Development” (Kannan 2016). In that report, we described common issues with electronic health record (EHR)-based clinical decision support (CDS) tools, and how applying agile methods aided us in addressing them.
As background, healthcare transitioned relatively late from traditional paper medical charts (with their characteristically illegible physician scrawling) to digital records. A few pioneers adopting EHRs demonstrated substantial improvements in quality of care and efficiency. Being able to provide electronic clinical decision support to physicians dealing with immense amounts of medical data yielded major benefits over paper records. Examples of CDS enabled by EHRs include automation of medication interaction checking, and suggestion of best practice evaluations and treatments for patients with specific conditions such as diabetes or high blood pressure. Financial incentives accelerated the transition to digital. In 2009, the bipartisan passage of the HITECH Act funded substantial incentives for physician practices and hospitals to adopt EHRs and show their “meaningful use” in the care of patients. Over the ensuing years, a marked digitization of Americans’ medical records transpired—between 2008 and 2015, the percentage of non-federal acute care hospitals with EHR systems increased from 13% to 88%. A similar increase occurred among office-based physician practices, with 87% on EHRs by 2015. The legislation required EHR software vendors to pass technical certification tests to promote interoperability. Not all EHRs achieved this certification and the EHR market consolidated predominantly on a small number of software vendors. Additionally, most organizations previously on home-grown EHR software migrated to vendor-supplied EHR platforms.
Though adopting vendor-supplied EHR software means healthcare organizations no longer need to do direct software development of the EHR code base, some EHR platforms support extensive local configuration. One prime area for EHR configuration proves to be developing various CDS tools to provide all parties in the healthcare delivery cycle (patient, office assistants, nurses, physicians) relevant information and “nudges” tailored to a specific patient to help accomplish best practice care.
Yet, as described in our initial experience report, these configured CDS tools suffered from several issues, including “alert fatigue” by clinicians, higher than expected defect rates (over-firing or under-firing in practice compared with expected behavior), as well as lengthy development times.
Our 2016 story centered on an EHR-based Specialty Patient Registry project we embarked on during 2015, which included the need for creation of numerous CDS tools. In the 2 years prior to 2015 we had built only 2 EHR registries used in production, each taking over 1 year to bring live. In contrast, during 2015 by employing agile methods including agile project management and agile model-driven development exclusively on the Specialty Registry project, we were able to build over 50 EHR Registries in one year, with 111 CDS tools in the EHR. UT Southwestern’s work applying agile principles effectively to this Specialty Registry project was selected as Healthcare Informatics’ 2016 national Innovator Award (1st-place) in 2016, and cited as a key contributor to UT Southwestern’s receiving the Healthcare Information Management Systems Society (HIMSS) Davies Award of Excellence in 2017 [Hagland 2016; Kannan 2017].
2. OVERALL UPDATE ON WHAT’S BEEN HAPPENING SINCE 2016
Since our presentation at Agile2016, we’ve continued to grow our experience with agile development of CDS and other EHR features:
The number of our specialty registries now is 105, with over 100,000 patients followed actively in one or more registries.
The adoption of agile methods across the entire EHR team led to the following tangible results:
- A 75% increase in our EHR teams’ productivity, as measured by the number of completed features actually introduced live in production (due in large part to a decrease in abandoned work, which previously had been a productivity- and morale-sapping bane of our work). Agile methods proving particularly helpful at reducing abandoned work included (a) active, contemporaneous engagement with the feature requestor to maintain and assure continued high interest in using the feature under development, and (b) writing user stories to better capture who would actually use the feature and the value they would derive from it.
- A simultaneous 17% absolute reduction in defects reported (even though moving many more features into production). Agile methods proving helpful in reducing customer reported defects included a) co-writing user stories with progressively elaborated acceptance criteria, to help make sure we were “building the right thing” [Kannan 2019], and test-driven development to help ensure we were “building the thing right” [Basit 2018].
- A reduction in the time a newly submitted enhancement request remains in the backlog, from 9-12 months previously down to 6 weeks. This came as a natural consequence of the previous two bullets (improvement in effective productivity, and decrease in defects which in turn decreased the amount of development team bandwidth spent on re-work).
- A marked decrease in need for after-hours work by EHR staff to meet deadlines.
- A higher-than-national-average score of physician satisfaction with our EHR configuration.
Internally, interest grew among Information Resources teams and managers as well as our internal customers for more teams to adopt agile methods for managing enhancement requests. Our Revenue Cycle, Training, and Health Information Management teams adopted agile methods, and our ancillary systems teams (Laboratory, Radiology, Pharmacy) are in the process of doing so. Agile champions from our initial teams’ adoption led education sessions for other teams and served as internal consultants. The remaining teams in our Department include our technical services teams that support the EHR and other system infrastructure and hardware, whose work differs in important ways from other teams, and are evaluating a DevOps approach. Agile techniques adopted broadly across all groups include use of user stories with acceptance criteria for lightweight requirements, the same 2-week iteration cycle which aided in cross-team collaborative projects, and the same agile project management tool. Teams adapted other agile practices to their own environment and work, including nature and timing of their scrum meetings, and their method of communicating with customers about enhancement prioritization and user story status.
Our EHR vendor (Epic Systems, Verona WI) recognized the benefits agile methods have yielded for us, and promoted adoption by their other customers (several of whom had simultaneously been adopting agile methods as well). Epic arranged a customer webinar for us to describe our agile experience, and over 100 other customers joined. Other customers independently began adopting agile methods, and it’s been gratifying to see continued growth in the number of healthcare organizations presenting on applications of agile methods at Epic’s annual national user group and expert group meetings, from 1 in 2014, to 4 in 2016 and 12 in 2018.
Epic itself recently migrated from their historically approximately 18-month large release cycle to a quarterly small release cycle sharing many agile goals: getting value to end-users early and often, reducing risk, and getting earlier feedback from customers to co-evolve and co-develop useful features. Other healthcare software developers actively embrace agile development principles as well, perhaps notably in the rapidly-growing fields of population health and healthcare analytics.
3. WHAT WORKED WELL
User stories serve us very well, by quickly getting to a shared understanding of the “who”, “what”, and “why” of a new feature request. We now use these ubiquitously, even on large projects which end up being split into multiple component user stories [Kannan 2019]. An example large user story (an “epic” in agile terminology) for a Prostate Cancer Registry project is shown in Figure 1.
Figure 1: Prostate Cancer Registry — Overall User Story and Acceptance Criteria
Agile model-driven development has assumed greater importance on larger, cross-team initiatives. Understandable models remove ambiguity, enable peer review of designs, and promote shared understanding among multiple teams and stakeholders in ways we’ve not been able to match purely with written documents. While we continue to use all the tools and model types in our original Experience Report, the following see use on most projects: Use Case Diagrams for displaying the scope of a project on a single page, and Decision Trees (or Decision Tables) for specifying clinical decision support logic unambiguously. The following diagrams also see frequent use: UML Activity Diagrams (swimlane workflow diagrams) for business process modeling, object diagrams for brainstorming and peer-reviewing solution designs. Additional diagram types we’ve adopted not described in our original report include State Diagrams for diagramming patient journeys through a given condition [Willett 2018] and Feature Breakdown Structure diagrams (Work Breakdown Structure) composed of User Stories, as a communication tool with stakeholders and management to depict project scope. We couple Feature Breakdown Structure diagrams in conjunction with Use Case Diagrams (depicting overall project scope) to provide a mental model for switching from traditional project management scoping methods to agile project management. Both diagrams prove useful during story splitting of a large initiative into component stories providing incremental value and small enough to be scheduled into an iteration. Figures 2 and 3 show a Use Case Diagram and a Feature Breakdown Structure diagram for the overall Prostate Cancer Registry user story above.
We’ve found use case diagrams—depicting user stories as use cases—to provide multiple benefits: (a) as a visual aid during story splitting, (b) emphasizing the value that smaller stories provide to various roles/personas, (c) prioritizing among multiple component stories, and (d) providing a shared “table of contents” and naming of user stories for consistent reference during subsequent discussions.
For analyzing behavior of our clinical decision support alerts (best practice advisories), we employed a dimensional model (Kimball style “star schema” data model) for monitoring use and user responses to all best practice advisories. Being able to interactively explore CDS tool behavior in our business intelligence tool (PowerBI, Microsoft), greatly facilitated:
- discovering unintended behavior (under- or over-firing),
- recognizing burdensome alerts,
- making decisions about which alerts to retire, and
- analyzing user responses as feedback for iteratively evolving user interfaces.
4.1 Managing iteration-based work on cross-team projects involving non-agile teams:
The mindset of time-boxed work and production-ready features delivered at end-iteration clashed with traditional waterfall methods used by teams not yet adopting agile. Coordinating work in these project situations proved more complex than when all involved teams were following an agile approach and on the same 2-week iteration cycle. We work to mitigate this by looking for ways to split user stories so that (when possible) the features being worked on by agile teams don’t have dependencies on the teams not yet on agile. “Stubbing out” work by other teams can also be employed where practical.
Figure 3: Prostate Cancer Registry—Feature Breakdown Structure (ca=cancer, pt=patient, qnr=questionnaire, dx = diagnosis, Epic=our EHR software, SmartForm= custom EHR documentation form; MyChart=our EHR’s patient portal software)
4.2 Competing priorities among differing stakeholders:
Within a single small workgroup, we often just had one or two main internal customers wanting to prioritize features (user stories) for delivery. As we scaled, a challenge became evident that although we have a single enterprise EHR, we have multiple governance groups representing different stakeholders, each with different priorities. Examples of stakeholders providing priorities include: Inpatient Operations, Inpatient Quality, Ambulatory Operations, Ambulatory Quality, Revenue Cycle, Imaging, Laboratory, Pharmacy, and so on. Because of the ubiquity of the EHR, many if not most performance improvement projects desired by these stakeholder groups include an EHR development/configuration component. In addition to these top-down governance initiatives, we also continued to receive numerous enhancement requests generated bottom-up from folks in the trenches who recognize a potential time-saving or safety-enhancing configuration change opportunity. We work to address this by maintaining separate product backlogs for different governance customer groups. A certain level of EHR developer bandwidth is roughly allocated to each customer group. Requests from all groups come to a single EHR governance structure, which helps ensure internal technical consistency of change initiatives. The highest level of the EHR governance structure includes executive-level representation when tradeoffs between competing priorities among different stakeholder groups need to be addressed.
4.3 Tooling, especially integration with our change management system:
We use commercially-available agile project management software for managing user stories, iterations, and backlogs, but use a separate commercial system for incident reporting, configuration management, and change management. This can lead to double entry of EHR-based feature information into both systems, which has been the main dis-satisfier for staff. We currently work around this by having managers do the bulk of the double data entry (for user story entry) on behalf of their teams. But this is non-value-added work. A more effective configuration would integrate our Agile Project Management and Configuration/Change Management capabilities (either as a single vendor solution or an interfaced one), and we are actively pursuing our options to do so.
We are still on the journey to having all teams, including technical infrastructure teams, adopt agile methods, and expect to continue to learn more as those teams see how these principles best apply to their work. For instance, technical infrastructure teams may interact less directly with customers in terms of enhancement requests, though frequently provide crucial capabilities such as interface development to achieve customer-desired feature behavior. These teams also have “keep the lights on” type of work, including hardware refreshes, vendor-supplied dictionary update loads (e.g. medication database updates for the EHR), often on a scale that appears to well exceed our typical two-week iteration. We are learning if and how best to represent this work as user stories and the most applicable strategies for splitting such stories to fit into iterations. Because some types of work involve daily updates, a DevOps approach may prove more appropriate for some technical infrastructure teams than the Scrum-based method employed currently by our EHR application teams.
We are also still early on the journey towards automated test-driven development, and automated regression testing. We’ve published an article on feasibility with open source software (FitNesse), but believe there’s substantial potential for enhanced system reliability and safety by much broader adoption. The broader adoption we envision would include both test-driven development (TDD) of new features in our development environment, and progressive expansion of automated regression testing suites in our Test and Production environments. In addition, we currently run our tests on a time-delayed SQL extract from our EHR’s real-time database (a hierarchical database optimized for transactional speed). Moving to real-time testing directly on the EHR’s database in Development and to a real-time shadow copy of Production would advance both our TDD and regression testing, respectively.
5. SUMMARY OF LESSONS LEARNED
5.1 Team Collaboration:
We have a mix of small projects accomplished within a single team and larger projects that cross teams. The latter category particularly benefits from a shared iteration schedule across teams. Regression testing in our Test environment at end-iteration also encompasses changes from all teams. Our clinical customers express change fatigue from too-frequent alterations in their EHR user experience. Accordingly, coordination of iteration schedules and release schedules across teams proves beneficial for clinical user release notes and education.
We employ a multi-level EHR governance structure, attempting to keep EHR configuration decisions as close to the organizational units that will benefit (or be affected) as possible. The docking of governance groups within a single overall Health System EHR Governance structure provides a coordination and review mechanism to help ensure design integrity across the application, and the ability to implement Health System level initiatives. Proposed changes or initiatives that span multiple organizational units (e.g. ambulatory clinics as well as the emergency department) are brought to a higher level of governance that spans all affected areas.
Healthcare organizations contain notoriously byzantine, complex organizational structures, and in practice multiple stakeholder groups serve as product owners, each with their own prioritized backlog of desired EHR features. We found consistent use of user stories with acceptance criteria a particularly lightweight and effective way of achieving shared understanding of the who, what, and why of each request, for integrated prioritization.
5.3 Shared Architectural Modeling and Design:
We find user stories consistently beneficial for capturing the “who”, “what”, and “why” of a proposed feature. For many initiatives, a high-level User Story is written for the whole initiative, along with acceptance criteria and proposed measures of success. However, accomplishing the whole story frequently necessitates splitting into many smaller user stories (features) schedulable into single iterations. Providing a single-page overview of these smaller stories helps with prioritizing and scheduling into iterations for early delivery of value. We’ve found Use Case Diagrams invaluable as a graphical table-of-contents of the user story features to be delivered, and which persona/role will interact with each. Before adopting use case diagrams, in projects involving more than one CDS tool we frequently found ourselves mixing up which tool’s objectives or design was being discussed.
A Feature Breakdown Structure diagram also can depict the same exact set of more detailed features, but stresses the whole/part relationships among them, rather than the persona/role relationships. We’ve found both helpful, and as above, have found the combination useful in communicating with stakeholders more familiar initially with traditional project management and Work Breakdown Structures, as a transition tool to Use Case Diagrams and User Stories for iterative, incremental delivery.
5.4 Challenges and Benefits of Shared Tooling:
As above, we have multiple stakeholder groups contributing feature requests to governance over our single EHR platform, and multiple teams that work across various stakeholder groups’ projects. Accordingly, we’ve sought a common, shared agile project management (APM) platform that can provide visibility into the story backlog and progress relevant to specific customer subgroups, as well as a view for each team of their own work for the current and coming iterations. We migrated from paper User Story cards on a wall to an electronic APM tool early on to begin meeting this need, though we are still on a limited version of the platform for budgetary reasons, with a shared account for each team rather than individual accounts. Currently our APM software and change management software are not integrated, leading to double data entry and synchronization work when changes are made. Though originally the barriers to our adopting agile methods were conceptual, as an organization we now believe in agile’s proven ability to deliver value reliably and repeatedly. At present, accomplishing expansion of shared APM software visibility to all users (EHR feature builders as well as stakeholders) and integrating APM with change management pose some of our remaining obstacles to more widespread adoption of agile methods.
6. WHERE TO FROM HERE – FUTURE DIRECTIONS
6.1 Expand use of Test-Driven Development
We aspire to much broader application of Test-Driven Development and automated regression testing, which we believe can lead to a measurable advance in quality of CDS tools within a production EHR. We envision increasing breadth in two ways: (1) to more types of CDS tools, and (2) to test coverage of a higher percentage of CDS tools delivered to production. To accomplish the latter will require some streamlining of doing TDD (reduce the “overhead” cost of doing TDD) as well as embracing the substantial advantages of TDD even if it adds some development time up-front.
6.2 A/B Testing of Clinical Decision Support
Our initial experience with A/B testing of CDS looks promising [Wang 2019]. A primary obstacle to realizing the potential benefit of automated CDS tools remains clinician reticence to engage with a CDS “nudge”, even when fully tested and meeting its initial acceptance criteria. Outside of healthcare, consumer software frequently iteratively increases in user acceptance (even enjoyment) and effectiveness through serial A/B testing of configuration options. We believe A/B testing’s benefits could similarly extend to configuration of EHR CDS nudges to clinicians, as well as other EHR features. We hope, along with others, to rapidly explore the potential of this method, while respecting the regulations guiding human subjects research.
6.3 Do Randomized Trials of Evolving CDS
For wide adoption in the medical community, scientific proof of the advantages of a new method works best. Accordingly, we would like to conduct randomized controlled trials of advanced agile methods vs. the status quo for developing clinical decision support. For instance, randomly assign CDS tools to be developed using test-driven development vs. the current method which is traditional testing-after-development. Similarly, randomly assign CDS tools to be evolved after delivery to a production EHR via A/B Testing vs. being evolved via status quo methods (typically based on spontaneous user feedback and expert opinion on changes to be made.)
6.4 Broaden CDS Analytics
We’d like to expand our star-schema dimensional data modeling for reporting on utilization of other common CDS tools (order sets, order questions, patient-facing CDS), not just best practice advisories.
Ideally, we’d like to design and implement a common reporting/analytic framework for measuring benefit of various CDS tools (the “so that” clause of the User Story) to help analyze for all tools the important question: are we achieving the desired end(s)? One possible grain for such a dimensional data model: one row per measurable benefit per CDS tool per time period. The facts (measures) of the dimensional data model could then include:
- a binary (0 vs. 1) value for achieving the target measure in that time period
- actual numeric value of the measure in that time period
- desired target numeric value for that time period
- descriptive value of the measure for that time period (e.g. “not at all met”, “partially met”, “fully met”)
Dimensions of this data model could include: (a) Date range (time period) and associated date attributes; (b) CDS tool and associated attributes (CDS tool type, for instance), and (c) Measure name and associated attributes (e.g. process vs. outcome measure type; others).
Agile methods scale well to larger healthcare initiatives involving electronic health record feature configuration. Challenges at scale include balancing multiple stakeholder groups’ priorities, and distributing iterative work effectively across multiple collaborating teams. Lightweight governance benefits from adoption of user stories and acceptance criteria for rapid shared understanding and value prioritization. Story-splitting across collaborating teams benefits from agile modeling of both project scope and solution design. Data-driven evolution of features offers promise for further gains in EHR feature effectiveness.
We extend our profound thanks to Sue Burk for her guidance as our shepherd in writing this Experience Report as well as our initial report for Agile2016. Sue’s provided us invaluable insights and encouragement, and been instrumental in shaping this report. We also express our deep appreciation to Josh Youngblood for championing the introduction of agile methods across the EHR team at UT Southwestern, and for providing quantitative data on the benefits of doing so; to Mujeeb Basit, Richard Medford, and Angela Carrington for sharing their insights and enthusiasm when co-presenting with us on adopting agile; to Vinod Nair, Scott Minnerly, and Jimmie Glorioso for their leadership extending agile techniques to additional healthcare domains; to Krystal Baldwin and Ki Lai for generously sharing with us their knowledge and experience applying agile methods at other organizations; to Dennis Pfeifer and Adam Wright for introducing us to DevOps concepts; and to Kathryn Flores and Mark Rauschuber for their insightful vision of the organizational benefits of expanding adoption of agile methods and unwavering support. We also are indebted to our teammates at UT Southwestern, whose collective drive for excellence continually inspires us, and whose willingness to share lessons learned helps us all improve.
Basit MA, Baldwin KL, Kannan V, Flahaven EL, Parks CJ, Ott JM, Willett DL. Agile Acceptance Test-Driven Development of Clinical Decision Support Advisories: Feasibility of Using Open Source Software. JMIR Med Inform. 2018;6(2):e23. http://doi.org/10.2196/medinform.9679.
Hagland M. The 2016 Healthcare Informatics Innovator Awards: First-Place Winning Team—UT Southwestern Medical Center: Healthcare Informatics; 2016 The UT Southwestern Medical Center’s Ambulatory Quality Outcomes Project is turbocharging quality improvement in MD practice. Available from: https://www.healthcare-informatics.com/article/2016-healthcare-informatics-innovator-awards-first-place-winning-team-ut-southwestern.
Kannan V, Willett DL. Agile clinical decision support development. Agile Alliance Experience Reports [Internet]. 2016. Available from: https://www.agilealliance.org/resources/experience-reports/agile-clinical-decision-support-developments/.
Kannan V, Fish JS, Mutz JM, Carrington AR, Lai K, Davis LS, Youngblood JE, Rauschuber MR, Flores KA, Sara EJ, Bhat DG, Willett DL. Rapid development of specialty population registries and quality measures from electronic health record data: an agile framework. Methods Inf Med. 2017;56(Open):e74-e83. https://doi.org/10.3414/ME16-02-0031.
Kannan V, et al. Rapid Development of Specialty Population Registries and Quality Measures from Electronic Health Record Data: Supplementary Material [Internet]. Methods Inf Med.2017;56(Open): online supplement. Available from: https://www.thieme-connect.de/media/10.1055-s-00035037/2017S01/supmat/10-3414-me16-02-0031-supp.pdf.
Kannan V, Basit MA, Bajaj P, Carrington AR, Donahue IB, Flahaven EL, Medford R, Melaku T, Moran BA, Saldana LE, Willett DL, Youngblood JE, Toomay SM. User Stories as Lightweight Requirements for Agile Clinical Decision Support Development. J Am Med Inform Assoc. 2019, doi: https://doi.org/10.1093/jamia/ocz123.
For those interested in more information on the following two references, please feel free to contact the authors:
Wang K, Burton ME, Basit MA, Kannan V, Willett DL. Rapid-Cycle Serial A/B Testing for Improving Clinical Decision Support Effectiveness. AMIA 2019 Informatics Summit; March 27, 2019; San Francisco, CA: American Medical Informatics Association; 2019. pp. 891-892.
Willett DL, Pandey A, Ifejika NL, Kannan V, Berry JD, Basit MA. State Diagrams for Automating Disease “Risk Pyramid” Data Collection and Tailored Clinical Decision Support. Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics; Washington, DC. p. 551-2. https://doi.org/10.1145/3233547.3233660.