Of the various software delivery processes under the agile umbrella, scrum became the most popular and is the most common implementation of agile in Corporate America. I like scrum. I like its simplicity. I think that scrum’s popularity is due to the fact that you can get quick business gains because it is quick to adopt 1. It is exciting to be able to watch a software system grow and change direction at will. This is in fact why agile exists and why it is called – agile. Managers like this concept and sign up for Scrum heartily for this benefit.
Technical Debt
There is an anti-pattern that I have noticed as a consultant working with Scrum implementations however. While management are happy with their agile process and seeing new functionality being added every sprint, typically they are not budgeting for any maintenance.
In the early days this is easy to get away with, particularly on a new codebase or green field project. However, unless you are cleaning up after yourself continuously (particularly if you are cutting corners in the sprint) and fighting to keep the code clean, you will be incurring technical debt. It is subtle in the early days and you will not notice it, but it starts to add up.
The reason that we call it debt is that you need to pay it back at some point. The longer you wait to pay it, the more interest it has accrued and compounded. Not only are you not paying the compounding interest back, you are continuing to add to the debt by adding complexity onto complexity and sloppy code onto sloppy code increasing the already compounded debt. The longer you leave the debt unpaid, the more effort and time it takes to fix the mess. This is what agile looks like without any discipline around cleaning up the code during each sprint and being vigilant to improve the architecture as it evolves. This is agile done wrong – which I see all too often in scrum implementations, and has been called “flaccid scrum” 2. To dig yourself out of this debt, it is going to cost you – a lot.
In short, the scrum Achilles heel (or blind spot) is this:
On a long term project / large product, scrum (done poorly) creates technical debt. Too much technical debt left unpaid results in a Legacy System.
Legacy Systems and Cost of Ownership
I was working with an agile client that had implemented scrum as their software delivery process. They were on a second version of their product. That is to say; they had re-written the first system and were very proud of their new second system which was now using more modern languages, technologies and frameworks. They retell the stories of the horror of working in their legacy system. Stories like:
- It took weeks to deploy
- The code was hard to understand and only a handful of developers knew how to work on it
- It was extremely fragile and they were always breaking the system when trying to add a new feature or even fix a bug
adding any new functionality took a long time - The system was bug-ridden
- Bugs took a long time to find and remove
The bullet points above describe a legacy system. Any of it sound familiar? Alas, it is not an uncommon situation.
These are some questions that I asked in order to provoke thought and insight into how they managed to get into that mess in the first place:
- Did you have senior architects and senior developers when you created the original system?
- Did they intentionally architect a legacy system? (I’m being facetious on purpose.)
- Now you have a new system, what are you doing differently to ensure this doesn’t happen again?
Ouch!
Then I point out that every day that I sync my laptop to their codebase, I am pulling in hundreds of new files. That is just not sustainable! They are on track to hit the same technical debt wall that they had in their original product and create a Version 2 Legacy system.
If you carry on sprint after sprint delivering business value while disregarding code quality and architecture, you will end up with a legacy system.
In a legacy system, the rate of adding new features decreases and the cost of adding features increases. You will not necessarily notice a drop in velocity however as the size of the stories starts to increase (to account for the complexity of working in a legacy system). Velocity will stay the same but the rate of adding new features decreases. (Which is one reason you should not use velocity as an indication of team’s performance.) You will also notice the bug count going up and much more time spent on fixing bugs. At this point a business knows that things are going slow but it can’t work out why and starts to lean harder on the engineering department to produce more, which only results in them cutting more corners, writing more bad code and compounding the mess.
Once a company realizes that they have dug themselves into a technical debt pit they scramble to get out. A common (anti) pattern I see is creating technical debt stories (or even sprints!), refactoring stories and technical stories. If you find yourself doing this you need to change your development process as stories, by definition, are supposed to come with business value. More information below on how to change your development process.
You pay for quality one way or another. You either pay for it as you go (by building quality into your process) and build a system with a low maintenance cost that you can continue to build on, or you pay the price toward the end of the project (where it now costs more) and have high maintenance costs. The option that makes most economic sense is to pay your debt as you go.
The solution to technical debt – Refactoring (Mercilessly)
The only one tool to stop the natural code entropy toward technical debt is – Merciless Refactoring.
Code Refactoring Definition : Code refactoring is the process of restructuring existing computer code – changing the factoring – without changing its external behavior.
Each sprint you have to build SLACK into the sprint, so that you can MERCILESSLY REFACTOR the code, evolving the architecture (EMERGENT ARCHITECTURE) and keeping the code clean (CODE CRAFTSMANSHIP) to enable the business to be able to change direction on a moment’s notice and keep the cost of change low.
Refactoring is a scary word, particularly to conventional QA specialists and to management who struggle to see what value refactoring can bring. Which is often why it is not only avoided but actively discouraged. In fact I agree that you should be scared of refactoring. Without other supporting disciplines/practices, refactoring is very dangerous!
What are these other disciplines that support refactoring? Well there are eleven of them and when you package them up with refactoring, you have – eXtreme Programming (XP). The practices all work together to support each other. If you are going to implement XP, don’t mess with Grandma’s Recipe (well at least until you are proficient) by trying to do just part of it. (This is why I dislike it when I hear “Engineering Practices” or “Technical Practices” used in place of “XP”. It only encourages the misuse of Grandma’s recipe.)
Extreme Programming is a suite of twelve disciplines, designed to work together to ensure code quality and enable refactoring, which in turn enables emergent design/architecture, which enables having a delivery model of changing requirements (agile).
The goals of refactoring are:
- Remove dead code
- Reduce complexity (often reduces lines of code)
- Remove duplication (including tests)
- Make code easier to read and understand (to support collective code ownership and ease of future changes)
- Change an underlying architecture to a better one as the system emerges (emergent design/architecture)
- Change an underlying architecture to match a change in requirements
Because refactoring returns no new functionality, management often struggle to see value in the practice.
The main value of refactoring is that you are not accruing technical debt and can therefore continue to add features and alter your system ad infinitum (with few to almost no bugs to boot).
This goes part way to addresses the agile principle behind the manifesto of sustainable pace.
Sustainable Development
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely – Principle behind the Agile Manifesto
Scrum has no answer as to how to achieve the principle of sustainable development where XP does. Consider these graphs below of two long term projects. One delivered with Scrum and the other XP:
This is a graphic illustration of the sustainable nature that XP will give to your development process over scrum. The state toward the end of the graph in scrum is a legacy system. In fact XP helps fill in many more agile principle gaps that scrum leaves unanswered. Read on.
Scrum is not enough
Scrum is a framework and its implementation is left up to you. Much of the implementation of today’s corporate scrum has come from XP. This includes: Stories, Story Points, Velocity, Continuous Integration and Release Planning.
When you take all the Principles Behind the Agile Manifesto and map them to Scrum and XP practices, it becomes pretty clear as to why you should be supplementing scrum with XP; supporting my claim of – Scrum is not enough.
| Agile Principle | Scrum | XP |
|---|---|---|
| Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. | Sprint Review – Demo | Small Releases
On-site customer Continuous Integration |
| Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage. | Sprint Review
Sprint Planning |
Refactoring
Emergent Architecture On-site customer Planning Game Feedback (XP Value) |
| Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. | Iterative Development | Iterative Development
Continuous Integration Small Releases Simple Design |
| Business people and developers must work together daily throughout the project. | On-site customer
Stories Feedback (XP Value) |
|
| Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. |
Co-located team
Play to win Collective Code Ownership |
|
| The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. | Scrum (daily standup) | On-site customer
Co-located team Pair Programming Feedback (XP Value) |
| Working software is the primary measure of progress. | Sprint review – Demo | Continuous Integration
On-site customer (signoff) Small releases Agile Testing (including TDD) Clean Code / Craftsmanship Simple design |
| Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. |
Refactoring
40 hour week (no overtime) Clean Code / Craftsmanship Collective Code Ownership Simple Design Emergent Architecture No Bugs Agile Testing |
|
| Continuous attention to technical excellence and good design enhances agility. | Simple Design
Emergent Architecture Clean Code / Craftsmanship Refactoring |
|
| Simplicity–the art of maximizing the amount of work not done–is essential. | Small Releases
Simple Design YAGNI (Similar to MVP) |
|
| The best architectures, requirements, and designs emerge from self-organizing teams. | Emergent Architecture
Co-located team Pair programming On-site customer Simple design Refactoring Play to win YAGNI |
|
| At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. | Sprint Review – Retrospective | Retrospective
Continuous Improvement |
| Other Aspects of Development (not included in the agile principles) |
||
| Code Quality | Agile Testing
Coding Standards Pair Programming Simple Design Clean Code / Craftsmanship No Bugs Continuous Integration |
|
| Iterative development | Sprint Planning
Sprint Review |
Planning Game
Last week’s weather Velocity Emergent Architecture |
| Agile Requirements | Product Backlog Increments
Product Owner |
Stories
YAGNI Simple Design Small Releases On-site customer |
Scrum + XP = A More Complete Agile Process
It is recommended that Scrum be complimented with XP 3, and it has been recognized that there are significant speed (5-10x) and quality advantages 4 in doing this.
So if it is recommended, why are there so few Scrum implementations that do this? According to the State of Agile 2015 survey, out of the total of scrum agile implementations reported, only 10% are doing Scrum + XP! (Of the entire population sample, only 1% are doing just XP.)
Scrum’s Weaknesses
Scrum on its own fails to address:
- Removing technical debt
- Maintaining a sustainable pace
- Code quality
- Emergent architecture
- Small releases
XP fills in these gaps, and more. To be successful in iterative development (e.g. Scrum), your developers need new skills and understanding.
Software developers moving from waterfall to an agile delivery environment need training in agile development practices and ways of thinking in order to deliver software incrementally and architect a system while requirements are changing.
Most Scrum transformations/adoptions I have seen have focused only on changing business processes without spending any resources on training the developers and setting them up for success (or leave it late in transformation by which time bad development habits have already set in). Developers need new skills and understanding to deliver in an agile environment. XP training and coaching can provide these skills. This includes:
- Incremental delivery
- Incremental and emergent architecture
- Emergent design
- Merciless Refactoring
- Agile Testing (The Agile Testing Pyramid)
- Continuous Integration
- YAGNI and Simplicity
- Simple Design
- Choosing technologies that support testing and refactoring
Summary
If you carry on sprint after sprint delivering business value while disregarding code quality and architecture, you will end up with a legacy system (i.e. crippled with technical debt) with high cost of ownership and many bugs.
Code suffers naturally from entropy. A software system tends toward technical debt and legacy (if left unchecked). The ONLY way to fight this entropy is code refactoring.
To refactor code requires supporting practices e.g. agile testing. There are at least eleven other of these practices/disciplines and the sum of all these practices are packaged together as eXtreme Programming (XP).
XP gives developers skills in delivering and architecting a system for changing requirements – which Scrum does not.
In a long term software project, you need to supplement scrum with XP.
Side Note – I came to these conclusion independently, but in writing this blog and researching references, I discovered that some of the content isn’t particularly new. The thoughts of Bob Martin, Martin Fowler, Michael Feathers and other thought leaders, in many cases, match my own observations. I find that encouraging.
Agile Terms and Lingo
XP – eXtreme Programming
MVP – Minimal Viable Product
YAGNI – “You Ain’t Gonna Need It”. Similar thought process in XP to Minimal Viable Product (MVP) in Lean Startup
TDD – Test Driven Development
Agile Testing – The suite of testing practices exercised by XP developers that follow the Agile Testing Pyramid paradigm
Refactoring – Code refactoring is the process of restructuring existing computer code – changing the factoring – without changing its external behavior
Technical Debt – A backlog of improvements to the code and/or architecture that will help make it more maintainable, easier to read, and robust
Legacy System – A computer system that has a large amount of technical debt
Incremental Delivery – The process evolving a software product by delivering working sub-portions (increments) of a system (typically on a cadence). Each increment typically builds further functionality on the previous, growing the complexity of the system incrementally.
References
1 – Agile Principles and Values, by Jeff Sutherland
“the Scrum framework for software development was designed to get a team started in two or three days, whereas engineering practices often take many months to implement. Therefore, it left the question of when (and whether) to implement specific practices up to each team. Scrum co-creators Jeff Sutherland and Ken Schwaber recommend that Scrum teams get started immediately and create a list of impediments and a process improvement plan. As engineering practices are identified as impediments, teams should look to XP practices as a way to improve. The best teams run Scrum supplemented with XP practices. Scrum helps XP to scale, and XP helps Scrum to work well.”
2 – Flaccid Scrum, by Martin Fowler
“What’s happened is that they haven’t paid enough attention to the internal quality of their software. If you make that mistake you’ll soon find your productivity dragged down because it’s much harder to add new features than you’d like. You’ve taken on a crippling Technical Debt and your scrum has gone weak at the knees.”
3 – Scrum and XP: Better Together, by Mike Cohn
4 – The Origins of Scrum, by Jeff Sutherland
“Few implementations of Scrum achieve the hyperproductive state for which Scrum was designed (5-10 times normal performance). Those that do all implement variations on XP engineering practices”
More from other Authors
The Art of Agile – James Shore
My favourite book and blog on XP right now.
The Land that Scrum Forgot – Bob Martin
Ask me Anything session – Jeff Sutherland
“Ken Schwaber convinced me that Scrum would be easier to adopt and work anywhere if we didn’t mandate specific software practices”
XP is the Mac of Agile – Jonathan Rasmusson
Working Effectively with Legacy Code – Michael Feathers
A great example of why developers need to refactor – Manuel Bernhardt
“It is one thing to get to a reasonable functionality in a short time, and another one entirely to build a piece of software that can be maintained or worked with in a team over an extended period of time. It is all too tempting to leave working code as-is, without refining it or improving it in a timely fashion. And before you know it, someone else has picked it up, or you yourself have picked it up again, copy-pasted it (god forbid!) or increased its complexity where there would have been a simpler solution would you have given it some more thought.”
Code Refactoring – Wikipedia
