RESOURCES

One Bug per Month

About this Publication

I believe great programmers can create unique software in a way it was never done before. This is a story about me joining a team and how we came a long way from a traditional development process to an agile one. We will contemplate on the events and decisions that led from chaos to order and from strict rules to a happy workplace. A place where a complete enterprise storage solution is made with almost no bugs found in production, thanks to a team ever open to change and continuous learning.

1. INTRODUCTION

Summer 2010. I was searching for a new job after the company I worked for before went bankrupt because of the financial crises. Luck and opportunity made it so that Syneto was hiring. I knew almost nothing about the company, but my interviews went well. I immediately observed that they were not the usual corporate company with their rigid procedures. They asked me about my past professional experience. They gave me decent programming tests on their own workstations, not theoretical exams on paper as the big companies do. Basically I worked alongside them for a couple of hours. Afterwards, I figured out that the company seemed to be doing well enough to avoid bankruptcy, and I thought I might find a friendly crowd here. Little did I know what was waiting for me.

My name is Patkós Csaba; I am a software developer at Syneto where I have worked for more than four years. In this time I have witnessed an extraordinary evolution in how we develop software. I’ve progressed several steps in my software career too.

Syneto is aiming to become the European enterprise storage solution provider. It is a company most known by its Storage OS operating system. I am one of the members of a small team responsible for the development of Storage OS.

In the past, the company went through several periods of rough time. It initially developed network security related projects. These projects were developed using a waterfall model that proved to be deficient, leading to 2-3 months delays in releases and tens of critical bugs found in production.

This paper will present the changes we went through both as a team, a company, and me personally. We took two journeys that changed our way of thinking and working, a technical one and an organizational one. In this paper I will cover the latter and talk about how we learned to manage our tasks and releases efficiently.

These organizational changes include adopting Scrum and Lean, using Kanban and Scrum boards, doing meetings and reflecting on actions. I will dive into great detail about how these and some more techniques were introduced in the company and how the team and I personally felt about the changes.

I will uncover how these essential changes, alongside with a larger set of practices, led to a more stable product. As we evolved, fewer and fewer critical bugs were found in production. We started with a bug count of tens or even hundreds of critical issues found in production. Recently we have reached a point were there are only a couple of critical bugs found in production, and we are able to release a bugfix in a matter of hours.

2. BACKGROUND

When I started at Syneto, I was assigned to a legacy project. During school and at my previous jobs I had used only a Waterfall development process. I was taught that we should analyze a lot, plan for all possible future events, execute, test, and deliver. I was told these were steps I should carefully evaluate, estimate, and draw Gantt charts for. The charts should be respected no matter what. That old legacy Syneto project was also made following this mindset. It had a lot of problems, but, nonetheless, it followed a process I was able to recognize and I felt somewhat comfortable with.

After the first few months of working at Syneto I realized that there were some problems I initially didn’t know about. Always late releases and fast running out of budget required a new vision, a new way of thinking and working. Both the company and the dev team had much to learn on the technical part and even more on the social side. We were somewhat lost, having little direction at the beginning and there were no other companies in our town, or even in the country, who could provide a solid example about what should we change to achieve a better process. I also discovered that there was a long history leading to this situation.

Between 2000 and 2005 the old company had a chaotic development process. They just did whatever and whenever any client wanted. While this may sound agile, it was not. There was no vision about the product. There was no organized task selections or prioritization. Sometimes the manager handed out features written on sticky notes to developers, but it was not clear to anyone why the feature should be implemented. There was little documentation, like in agile, but it was not enough. There was nothing agile about our process. We did not try to respond to change; we just tried to implement everything all the time. We did not think about customer collaboration, we just followed every request they made like slaves. We did not think about individuals and interactions, we just programmed whatever we were handed to. We didn’t really care about working software, we only wanted to release features. Then Waterfall came.

By mid-2008 the development process got more organized. The company went from chaos to a Waterfall that was so bad it probably introduced more bugs and delays than the chaos before it. In the same time the company’s founder pledged money for about one and a half years and there were six months long release cycles which were constantly delayed.

The situation got so desperate that people started to make radical decisions. A good example of such a decision came from the project manager. In an attempt to improve development speed and product reliability, when a new feature was required for our legacy project, the UTM – Unified Threat Management – appliance, the project manager decided to step in.

After the feature requirements were carefully written, the technical documents were next on the list. The manager decided it was a nice architectural challenge for him. He retreated into a separate office, isolated himself from the rest of the company and team. After two days of sweat and hard thinking he came up with this nice, big architecture with several classes, design patterns, and concerns carefully analyzed. He went to the team with a happy and proud attitude.

“- Look what I did! Here is the architecture for the notification system. It is nice, clean, built using design patterns.
Just a handful of classes, well separated, with carefully crafted dependencies!” But the team’s response was shockingly unexpected.

“- We implemented the feature one and a half days ago as a one-page script. Works just right.”

People started to realize things were going wrong, so they devised a plan to implement the “perfect waterfall”. Documents were written about how to write other documents. Then they planned everything, including detailed class specifications, with pieces of code and algorithms. Functionalities and test scenarios were also detailed in lengthy documents. The process concentrated so much on implementation details, that anybody, after reading such a document, should have been able to implement the solution. The details were amazing.

This led to nine month long release cycles and even longer delays with more bugs.

Then management tried to enforce new techniques like unit testing, but met great resistance from the employees and had little or no success. Disaster was knocking at the door. Solutions had to be found otherwise money would run out and the company would fail.

In the second half of 2009, Flavius, the manager I mentioned above, read Kent Beck’s, Extreme Programming Explained [Beck] and Andrew Hunt’s, The Pragmatic Programmer [Hunt]. These books made him believe there is a better way to write software. He started to passionately share his beliefs with the team and some of my colleagues got curious.

On the other hand, change is hard. And it comes slowly. In 2010, when I arrived, my colleagues had just begun studying unit testing from Gerard Meszaros’ great book, xUnit Test Patterns [Meszaros]. I got hooked on the idea of automated testing immediately. Before this moment in my career, I had not even heard about programs testing programs. I was impressed both by technical part and social part. We created a collective learning environment where each programmer presented a chapter from the book each week. People seemed to feel good about what was happening and I felt good I ended up with them.

These first humble steps opened up several new possibilities. We embarked on two extraordinary journeys. One organizational journey that taught us to accept change and organize our work differently, and another technical one that enabled us to adopt a clean architecture. In the upcoming chapters we will talk about the former.

3. OUR JOURNEY

3.1 Organizational Challenges

Any team can be defined by the practices they use in everyday work. Finding and adopting these practices in order to define a more optimized process is one of the greatest challenges of agile. We at Syneto believe that the only way to

find what is best for you is to adopt a new technique and do it for six months. At the end of that period we have enough data to analyze and then decide what to keep and what to drop from our process.

We did this several times throughout our journey and we continue experimenting and gathering data that we analyze to find better ways to write software. Here are the most relevant techniques we tried and the conclusions we have drawn from experimenting with them.

3.1.1 Discovering Scrum

One morning my manager, Flavius, came in about one hour late. This was very unusual since he was always one hour early, every day. But there was a reason for all of this. We didn’t know what, yet, but there was.

Then he sent Bogdan, a colleague, to buy some things. We were already immersed in coding, so we didn’t give it much attention.

A few minutes later Bogdan came back with a few sheets of flipchart paper, a mountain of sticky notes in various colors, and a bunch of markers. Something was happening. People started to raise their heads from the screens and look around curiously.

But programming was still more interesting and our jobs had to be done. We went back to our source code. I am not sure how much time passed before Flavius interrupted us. There was something new right in the middle of the front wall.

It was a planning board of some sort, the reason for Flavius’ being late. We got curious. He explained to us that he learned about better ways to organize our tasks and that we would start using that board to plan our actions.

There would be a “Release Backlog” column, where he would put all the things that must be done for the next release. Then we would talk and decide what we need to do in the current sprint, and drag those tasks to the “Sprint Backlog”. Then each of us would take a task and put it in “Development” until it is done, when the sticky note should end up in “Done”. It sounded pretty easy and straightforward.

I expected this board to help us significantly by inspiring order in chaos. I expected less upfront design and more action. I expected change.

Figure 1 - Initial Board
Figure 1 - Initial Board

Surprisingly the effects were not what I expected. The board helped only a little to better organize our tasks. We still planned and designed upfront almost all the time and were implementing tasks without prioritization. Yes, we tested a few things here-and-there, but our actions were based more on gut feeling than well-defined rules.

But what this initial board did to us was something else. It became an amazing way to visualize our work, to see the big picture, to see what must be done and what progress was made. This made us feel differently about our process. Every time we entered the office, we saw the board. It was in such a central place that it could not be ignored. And it fueled contradictory feelings. On one hand we got frustrated and sometimes panicked when there was a slow progress. On the other hand, we started to believe in change. We started to believe that what we were doing was headed in the right direction. We started to believe we were writing software in a more organized way and that it would lead to better stability and more predictable releases.

Soon after the board was up we started to invest ourselves not just technically but also emotionally. We started to care about our work, about or project. One day Flavius came with other rules for our evolving development procedures. We needed to hold meetings. We started with retrospectives. He told us nothing formally. He just gathered us from time to time, and we reflected on what we did. Then we did this every week and he introduced us to the concept of “Sprint”. We not only did a retrospective of what happened in the past 5 days, we also planned what we wanted and needed to do in the next week. Our sprints were short, one week. We learned many things, like discipline and prioritization, but there was another thing that I found more interesting. We learned to communicate. I remember I was quite frightened at the beginning. Talking for the first time in front of the whole team induced a certain amount of anxiety. Fortunately we all learned to overcome it.

Then we started to have meetings every morning and so Flavius taught us to the concept of “Daily stand-ups”. These seemed to be quite interesting at the beginning. However they became one of the things we gave up later on. The problem was that they became very monotonous. We pretty much knew who did what, the planning board was communicating enough. We also talked with each other enough to know what each of us is about to do.

Then we did retrospectives after each release and a planning meeting before the next sprint. This practice greatly enhanced our capability to analyze our work and our environment. The meetings were organized. Flavius came with various techniques to encourage collaboration. The most effective one was to draw three columns on the whiteboard: what was good, what went wrong, and what actions should we take.
Each person was asked to contribute a minimum of three ideas to each list. Usually after someone provided the first few ideas, the discussions got going naturally. This technique of good-bad-actions is one we still do. And at the next release retrospective we take the previous “actions” list and check if we did everything we proposed to do.

We experimented with the order of the columns. We tried bad things first, then action, then good things. While this had the advantage of dissecting bad things in more detail, when we ended up talking about the good things at the end of the meeting everyone was in a bad mood, bored or sleepy. We also tried bad-good-actions, but this had the drawback that when we started talking about actions, the problems were not fresh in our minds any more.

Figure 2 - Retrospective
Figure 2 - Retrospective

We found that starting with the good things will increase morale at the beginning of the meeting, and will engage attendees faster in the conversation. When you are already in a good mood, talking about bad things can be done with a more rational attitude, which leads to easier ways to find the actions that can alleviate the bad things that were identified.
The actions we need to take being the last column had unexpected benefits. Mary Lynn Manns and Linda Rising identified in their book, Fearless Change: Patterns for Introducing New Ideas [Manns], a social pattern called The Right Time. This pattern tells us that the most memorable meetings are the ones held at the end of the day. Because after a meeting is finished our brain continues to process the information, we will best remember the things discussed at the end of it. We did not know about this pattern back then, but in retrospect I can only agree that this pattern is effective.

3.1.2 The Sinking Ship Approach

Another retrospective experiment we did a few times was the sinking ship approach. Our project is represented as the ship itself. There are various forces pulling it down, trying to sink it. There are things pushing it forward and finally there are forces lifting it upwards. It turned out to be a very powerful metaphor to visualize our activities.

Figure 3 - The Sinking Ship
Figure 3 - The Sinking Ship

Each participant had to write at least three reasons for each direction. They wrote bad things sinking our ship on red sticky notes. These notes were meant to be placed in the depths of the furious waters.

We mentioned the good things we done on green sticky notes destined for the bright sky.

Then we thought about the actions we need to take and wrote them on yellow sticky notes. These were then placed at the back of the ship so that they can push our efforts forward.

This approach had a few interesting effects. People were forced to think for themselves and come up with ideas. Yes, forced, and some of the ideas were not that good, but they were their own. They were ideas of individuals and reflected their own point of view about what happened.

The silence and thinking allowed for anonymity also. All sticky notes were gathered after 10-15 minutes and mixed up. If you really wanted, you could recognize someone’s handwriting, but nobody really cared.

Then all the notes were put on the table in no particular order but respecting the placements explained above. Some observations and ideas were unique, but most of them repeated at least twice. Everyone wrote the most important ideas, from his own point of view. This offered a very effective way of visualizing the problems we had in our past sprint or release.

We even made the experience of these meetings more pleasant by providing some snacks and drinks so peoples’ subconscious minds would relax a little bit.

After using this practice for a while we managed to increase the courage of people to express their ideas. Then we reverted to the simple written board with three columns. This time however, more people collaborated from the beginning and the general feeling of the retrospectives became more relaxed and focused on solutions.

Maybe now that we have added quite a few new colleagues, we will start using the sinking ship approach again for a while.

3.1.3 Our First Board Update

Sometime in mid-2011 we updated the board a bit and we started to use it in a much more advanced way (see Figure 1). Here are a few key aspects that improved our development:

  • The Development column was broken down in three steps: Design, Development, and Test.
    • Design was introduced to separate the definition of APIs and UIs from the development part. We needed this partly because it forced us to think about some problems upfront and partly because there may have been different persons implementing the UI than writing the business logic backend.
    • Development represented the actual coding part of the story. This included TDD and any automated testing, as well as refactoring. Not tracking refactoring and writing tests as separate actions made us think about dev-test-refactor as a single process and estimate accordingly.
    • Testing is a little mystery column. Many people ask me why do we have it and what it is good for considering all the above process. Well, automated tests are great and we have great overall coverage (more than 80%), but we still have bugs and mistakes that just defy any automation. So after each story is completed by the developer(s), he or she puts it in Testing. Then someone else, someone who did not contribute at all to the task must take and test it by hand. That person is solely interested in the functionality itself and not how it was implemented. This is a great way to catch little stupid bugs as well as spreading knowledge about the product.
  • Each step had a “definition of done”. Only when a task fulfilled all the required aspects of that particular column was it allowed to be moved to the next one. These are the notes on the bottom of each column in the image. We used a “column exit criteria”, meaning that a story was allowed to exit the current column only when the definitions for that column were done. For example, we were not allowed to start working on a story if its acceptance criteria were not defined. Or, if a story was in the design phase, we could not start developing it before the API and UI mockups were ready. In the same manner, tasks in development had to be coded, refactored, unit tested, end-to-end tested, and acceptance tested before they could be sent to testing. Of course, not all conditions were applicable to all type of stories. Some had UI design but no API, others could be unit and end-to-end tested but there were no need to refactor anything. But we always tried to apply as much as we could to each story we needed to do.
  • Tasks had to be categorized. We identified that we had three different things we did throughout our development: “stories”, “bugs”, and “technical tasks”. Sometimes we used subtasks also, but we tried to break stories in such small pieces that you could not or need not break them down more. Each type had a different color we respected religiously. Red was for bugs, yellow for stories, and blue for technical tasks.
  • We introduced another section in the Backlog for the technical tasks. These were infrastructure and build related things that could not be associated with the product directly.
  • Probably the most important change though, was to take responsibility for our actions. Each sticky note had a smaller sticky note on it with the name of the person who chose to implement / do the task. By not being assigned to tasks by somebody else, and willingly saying, “I believe I can do this task”, we started to take responsibility. We put our names on tasks, or we took shared responsibility with another colleague by implementing it using pair programming, another useful agile technique we adopted.

Even though figuring out things on our own as we went along may not have been the easiest way to learn something so different from waterfall, it was a process I liked very much. I was actually waiting and expecting to be introduced to something new almost every week. It was certainly not the fastest way to learn Scrum, but it was probably the best way to develop an emotional attachment to each change. It made us feel good when we changed and bad when we stagnated. And we continued to change in ways we could not imagine back then.

3.1.4 Understanding Lean

One of the difficulties we faced with Scrum was how to allocate our work. It was unclear who should take a story and when to do it. We had synchronization issues between team members and we were searching for ways to optimize our person-to-person interactions.
So we turned to Lean and we tried to adopt each of the seven main lean principles: Eliminate Waste, Amplify Learning, See the whole, Build integrity in, Empower the team, Deliver as fast as possible, Decide as late as possible. Each principle improved our process a little bit, but I will only cover the two that had the most significant impact.

3.1.4.1 Eliminate Waste

The first step in eliminating waste is to identify waste. The waste we already knew about was dead-times, periods of time when one person waits for another to finish some task. Our actions were too sequential. At some point in our journey we had two people dedicated to UI and 3-4 dedicated to backend. Based on our Scrum process and on our board, we needed to respect the column exit criteria, which led to our process having inherent delays and inefficiencies.

The product owner and scrum master had to define the acceptance criteria. Then the two guys from UI did the initial design and API definitions. This was followed by the backend developers’ actions. Finally, when the task was in testing some other developer had to take over and do some manual testing.

This seems a surprisingly simple and logical process. Responsibilities are clearly defined. Everyone knows what to do for each task.
But it is an equally wasteful process. There is no way to define your stories so that each developer has something to do all the time. So we ended up asking ourselves questions like, “What should the backend developers do at the beginning of the sprint when all stories are in the hands of the UI guys?” The same issue was also true at the end of the sprint when all that had to be done was just backend programming. There were a lot of intermediary stages also when we were waiting on each other.

How could we eliminate wasted time?

The best thing for us was to do even more pair programming and do it cross-team. Instead of the two UI guys working in a single pair on a single task and the 4 backend developers watching YouTube, we decided to pair in domains we are not familiar with. So, two stories were taken into UI design. For each story a UI developer and a backend developer collaborated. After the UI design was ready, and by design I mean sketches and mockups not the implemented UI, the story went on to development. Most of the time, two backend developers would pair on creating the business logic part of the story while a single UI developer was enough to develop the visual part. The three developers worked closely together on the task at hand. Meanwhile, the other 3 team members could apply a similar approach to another story.

This kind of pairing was very efficient for about 80% of our stories and it led to a greater cross-domain knowledge. In time this enabled each of us to contribute to any story both at the UI and backend level. I am one of the backend developers, and I will never be able to create the amazing UI designs the dedicated designers can, but I am now capable of intervening and fixing bugs or implementing small UI only features. Which leads us to another time wasting scenario.

In about 20% of the cases, the wasted time was of a different type. A programmer knew he would need to work on a task in about 2-3 hours, but there was nothing he could do until then. This was a good time for small fixes and manual testing. On a sustained development pace, most of the time there was something to test, and these small dead- times were perfect for that. But even when that was not possible, there were always small bugs to fix. There is no perfect product, so you can always take your time and fix a small misalignment on the UI or optimize a for loop on the backend. But to do so you must be familiar with all parts of the projects. And yes, I, as a backend developer, am doing small UI fixes several times a week. They are a good dead-time fillers and a welcome distraction from complicated business logic.

While lost time was our biggest identified issue, we also found several other types of waste and we eliminated them.

Here is a short list:

  • Comments in source code – any comment is text written to explain what the code does. You have to read both the comment and the code to understand the whole. This is a wasteful situation and the comment is legacy as soon as it is committed to the version control system. We transformed all comments we could find into well named extracted private methods. The result? On average we now have 3 lines of comments for every 97 lines of actual code (not counting empty lines). Our average lines of code per method are 2.5 (yes, two point five) and we have only a few longer than 20 lines. Surprisingly the average public methods per class were reduced to 6 also. Yes we have more classes, but they are less complicated and we basically have almost no comments in our code. When you constantly think about how to write code that will never require a comment, you find yourself concentrating on naming and extracting and encapsulating a lot.
  • Task bureaucracy – Is it a bug? Is it a feature? Is it a technical task? Sometimes we spent hours debating if a task was a story, feature, technical task, or a bug. We needed to end this. So we adopted a policy: the person who writes it and adds it to the backlog is responsible for deciding what it is. Nobody will dispute the nature of the task. This turned out to be very effective. We have had no such discussion for a couple of years, and things get done at the same velocity.
  • Recurring bugs – One bug is a mistake. The same bug happening again is a waste. Our policy is that every time a bug is discovered we write a test (unit or end-to-end or just simple UI test) that is automated and that will be run after each commit in our continuous integration system. This is a practical way to build up a set of regression tests over time that requires very little overhead. And the gain is huge for the product’s image. We managed to transform a perception of “a new release means more bugs” into a perception that “a new release means fewer bugs”. Four years ago our clients were afraid of updating; today they are asking even for betas because they know that our product is getting better each release and the risk of breaking something critical is very low.

3.1.4.2 Amplify Learning

It’s not enough to want to do something better. You must know how to make it happen. To do it better you must become better, so the whole company started to emphasize on learning and evolving. Some of this learning was already part of our life. We collectively read and shared with each other the new things we learned. We expanded this to short meetings where one of us would present something he or she experimented with.

One such experiment was the change of the Prototype JavaScript library into jQuery. The experimentation of a single colleague led to the whole team accepting the technology. Eventually we decided to move all the code to jQuery. This is a long-time process, we only move the parts we need to touch in adding new features, but it goes well. And it all started by a single experiment.

Books, books, books. The company basically buys all the books we want to read. Until recently we bought paper books, but not so long ago we started to migrate to electronic books. Most of us own a Kindle and each employee has access to the company’s Amazon account. We are planning to by a couple of Kindles that will always stay in the office for those of us who do not like to read from the computer screen.

Courses and Conferences are another great way to improve our knowledge. Syneto encourages these activities by allowing employees to go to conferences on the company’s expense. Syneto pays for travel, accommodation, food, and conference tickets, plus we get our usual pay for the days we are not at work. One such course, Rebecca Wirfs- Brock’s Agile Architecture, made us rethink our planning board. Which leads us to our next topic about Kanban.

3.1.5 ImageApplying Kanban

Figure 4 - Kanban Board
Figure 4 - Kanban Board

A kanban board is like a funnel. You can have an unlimited number of tasks in the backlog and in the done columns, but you are severely limited for the rest. This allows the reduction of work in progress and a better management of the resources.

Our updated planning board adds these features to the old board we’ve already mentioned. The notable differences are:

  • There is no single “Working On” column. Instead we have “Design Spikes”, “Development”, and “Testing”
  • Design spikes replaced the old “Design” column. We realized that UI and API design were not the essence of our regular development
    activity. Instead we learned about the concept of design spikes that allows you to think about more than just API and UI. It encourages thinking ahead a little bit more than the usual agile dogma. It encourages you to think about databases, technologies involved, and problems that can be foreseen. We use this column especially for tasks that we are not sure how to solve. We put a task here and we start thinking about the problems that may arise. We invest only a limited amount of time and effort in this one, and we discipline ourselves to not get carried away on exploring possible solutions in great detail. This column can be identified as a place to publicly show off what you are experimenting with. The jQuery migration mentioned above started as an experiment for a task sitting in that column.
  • Each of the three columns of the development cycle has a limit set on it. We decided on these values based on the size of the team and the complexity of the tasks we usually have. Design spike is limited to 2 while development and testing to 3. These numbers represent the maximum tasks that may reside in the respective column, and they can be considered an “entry criteria”. The exit criteria we had let a task to be moved into the next column. Entry criteria restrict what tasks can be put on a column. If one finished developing a task and wants to move it to testing for example, he or she cannot proceed if there are already three tasks in testing. He or she must stop and test something so that a free spot will be available for the finished task still residing in development. This will focus efforts on testing. Limits force our resources to be allocated where they are needed most. But this is not the only effect. The task is still in development, and if there are no more free spots there, other team members cannot take issues from the backlog and put them in development. They, in turn, must take care of some tasks in testing so that the congestion is removed and the waiting tasks in development can progress to testing. So not only the person directly affected is redirected to testing, but the rules have effects on other programmers also who are not directly interested in the testing column.
  • We dropped the exit criteria. Not because we didn’t care about them any more, but because we became accustomed to think about them without needing visual reinforcements. We adopted this change naturally. Our subconscious mind told us we could drop the labels. In retrospect this was a perfect example of team culture as defined by Clayton Christensen in The Innovator’s Dilemma [Christensen]. We started with a rule, enforced by a set of publicly displayed criteria. When the criteria became common knowledge, we dropped them. Today all those things are natural part of our existence. We do not need to think about them, we just know them. It is part of our culture. We do naturally do so and we don’t question why.

3.1.6 Getting Digital

But the content of the board is just one side of the coin. The way we present the board to the team can also be very important. At some point between the first and second version of our board we tried to use Jira’s planning board.

Jira is a digital planning platform. It has support for Scrum, Lean, planning, sprints, various types of issues and so on. It even has a plugin that makes it easier to use by providing a board-like interface with easy to use drag-n-drop features.
Our first attempt to switch to a digital planning board ended in a total failure. While the board is cool at first sight, it is very hidden. Each user needs to explicitly open a web browser and go to the planning board’s web page to see the board. Compared to the always on the wall, always visible board, this is a huge step back. Users, and programmers especially, are notoriously lazy. If the board is not staring at them, they will not go and look at it. It’s just too much effort.

So we went back to the classic board with sticky notes, paper, and pen. And we kept using it up until recently when the paperboard had a problem. Our backlog started to grow too much. The sticky notes could not fit on the board any more. And while we could extend the board, the problem of searching and ordering the backlog stories became an even bigger problem. Duplicates started to emerge and backlog grooming took hours.

We needed a change, again. So we went digital again. This time I was the person responsible with planning and task management. So I was very curious if one and a half years extra experience would matter in handling the visibility problem. Did the team become aware of the problem? Was awareness strong enough to make users keep the board page always open and follow it consciously? The answer was no. We changed in many ways as professionals, but we retained our same level of laziness.

After the first two releases, the effects were obvious. We got more disorganized. People started asking around about who does what instead of looking at the board. Others started to work and did not assign the issues to themselves, or did not move them to the appropriate columns.

This led to some identifiable problems:

  • We spent four weeks doing integration and last minute fixes. This was about twice as much as usual.
  • We found about three times more last minute problems than before. Each had to be resolved before release. This led to an unforeseen delay of the release date.
  • The number of bugs rose. We had five new bugs found in the first 24 hours after the release of our last version. Compared to the usual 2-3 bugs per release, this can be considered a doubling of the bugs.

We identified that the main reason for this was the lack of visibility. So we bought a smart TV and set it up on the wall to show the board and the progress of the project.

ImageWe expect this step to increase visibility and encourage people to use Jira more.

On the positive side, the digital issue management gave us a set of benefits that already started to pay off:

Image_010
Figure 5 - Digital Board on a TV
  • Easier backlog management. Searching, ordering, updating issues became much easier.
  • Graphics and statistics can help us show current progress. This is important to raise awareness about problems in a timely manner so we can take actions before it’s too late. Now that graphs and progress are always visible, we are talking more frequently about the problems we identify and we take actions immediately.

3.2 Where Do We Go Now?

In the past year we hired four new people. In the same time period two colleagues left. We are in a situation where 30% of us were hired in the past twelve months. We have a cultural issue.

As Clayton Christensen explains in The Innovator’s Dilemma, company culture doesn’t just suddenly appear. It needs time to form, and everything starts with procedures and rules.

About five years ago, we started to define our procedures and rules. We put them on paper or on a board. We read them from books and respected them step-by-step. We needed to do new things but we didn’t know exactly how. All these procedures were public and we did our tasks by reading and respecting them.

As time passed and we started to assimilate all the knowledge about procedures, they started to disappear as publicly displayed lists of steps. There were fewer and fewer of them. Remember the example about the drop of exit criteria from the planning board? We still thought about the processes actively. We still stopped for a few seconds from time to time to recall the rules and decide the next steps to take. And we managed quite well.

In the past one and a half year, most of our procedures disappeared. They morphed into our culture. They shaped us and the way we act reflexively, without thinking. Our subconscious minds took over most of our processes. We can now dedicate our conscious mind force to produce and enhance our product and not to think about how do we do that. While this is great to a certain extent, it has a problem we just started to realize. When there are no procedures, no documents to read and no rules to follow, how do we integrate new people in our team? How do we teach them our culture?

I was a little shocked when I realized how many things we just do without thinking about details. When the newcomers asked me for documentation I had nothing to offer. Even simple requests like, “What should I learn next?” can be confusing when there are no visible procedures. After they constantly bugged me with small simple questions, and requests for directions, I realized something is wrong.

To my pleasant surprise I wasn’t alone. Others observed the problem, too. We discussed the situation at our next meeting and we defined our new direction.

I am proud to start a new journey in our evolutionary process that will analyze and dissect our culture back into written procedures. These procedures will represent the way we work and act in a written public form. Anyone will be required to respect them, and change them when needed.

It just happens that this paper is written exactly when we realized this issue. I consider this a first step in documenting the way we work and I think about introducing it as our first reading about procedures for newcomers.

4. WHAT WE LEARNED

Version 2.1 of StorageOS, in 2011, had 33 bugs fixed compared to the previous version 2.0. We were then at a 60-75 days release cycle. This happened when we started to speed up our release cycles and better organize our tasks. We were already improving considerably. Unfortunately we have no records of bugs before the beginning of 2011. They were only kept on sticky notes on the board.

Version 2.15, the last one before the reintroduction of Jira as a planning board, had a total of four bugs found in production, with an additional few found by us while developing the next version. This was probably the most bug free version we ever released.

At the moment of writing this document, our latest released StorageOS version is 2.16. It was released 112 days after 2.15. There were a total of four critical bugs and 14 smaller issues, a significant step back. We expect this to improve after the introduction of the planning board exposed on the TV.

Counting these bugs is not very precise. Still, the trend is obvious and the effects are visible. Our adoption of agile management and technical practices helped a great deal in improving our product. We incorporated all these in our culture, in our reflexes. As I like to say, the old team became naturally agile, culturally mature.

But we must not stop. Transferring our culture to all the new team members and the ones that will come in the future will be our next great challenge. Writing this document was a first step in that direction. Until then, here are a few stories my colleagues experienced first hand.

Vadim Comănescu:

“I was lucky enough to be exposed very early in my career to all the pioneers of the agile movement. This has shaped my way of thinking about software development. I often wonder how my journey would have been if instead of learning it right from the beginning, I would have struggled to get rid of bad habits and practices.

The techniques and the continues improvements we have applied over the last few years in the end not only made me a better professional but have shaped my character as a human being. And in my opinion, that’s what matters the most. Pair programming every day, talking in front of the team, sharing ideas made me more calm, more empathic and a way better teacher.

The one motto that remained very well engrained in my mind, like a brain tattoo, is that ‘the only constant thing is change’. And you need to embrace change as being the driving factor toward progress and evolution. Thanks everyone at Syneto for such a wonderful trip and for the years that are to come.”

Dan Vâtca:

“Being one of Syneto’s founders I have witnessed our team embrace continuous improvement. It did not matter what the status quo was, we always tried to improve and be better than yesterday. And looking back over the years I can see how a long stream of small improvements brought us a long way.”

Robert Călin:

“When things change it is a sign that there is will to improve, improve on a personal and professional level. At the beginning there was disorder and confusion, but as time passed we learned to communicate better and improve on our code. We learned how to better organize ourselves, how to test and write tests and how valuable is user feedback.

For me, it is important to create an easy to understand and use product, to provide a great user experience.”

Roberto Județ:

“I was lucky to be born after the fall of communism, but the practices and mentalities did not disappear overnight. I was intrigued when I learned in college about work methods like Agile and it is really awesome to work in a place where the employees are seen as real persons, not robots.”

Romeo Lazăr:

“Being the last to arrive to team Syneto, I feel like I belong here although it’s been only 46 days since then. My greatest joy is that I can help with many things for Syneto’s future. I’m glad that I can grow with the Syneto team by settings of standards in terms of Storage. Young team spirit is felt in every corner of our office, although this sprit must be adjusted from time to time.”

Dragoș Chioran:

“As I’m sitting at my desk and forcing my brain cells to come up with a few exquisite words, my calendar strikes four glorious years since I first sat at a Syneto desk. Although time is a purely human invention, four years is still an impressive achievement in my book. It has been four incredible years punctuated by peaks of ecstasy brought on by successful projects as well as chasms of sorrow birthed from inevitable failures.

It is human to err, but doing so is unpleasant; and yet, you have to live and learn. This is what really impressed me with Syneto as I began sitting at this desk: the fact that this commercial entity, unlike any other I was part of, allowed me to be wrong and ultimately, to learn from my mistakes. There are few companies out there that recognize your essentially human nature and allow to be wrong so that, in time, you can start being right.

To be fair, it is not the same desk I sat down at four years ago. It’s a brand new one located in our new offices. Evolution, agility and the ability to adapt are also qualities that I’ve found in Syneto. It’s a truly rare thing when over the years you become truly part of something; the strong foundation that holds the weight of the whole organization.”

5. ACKNOWLEDGEMENTS

I wish to express my gratitude to all the people at Syneto who made this experience possible.

Thank you Flavius Ștef for hiring me and believing in me from the beginning. Thank you for discovering agile and guiding us on our first steps.

It was a real eye opener for many of us. Finally, thanks for your feedback on this paper. It really made a difference.

Thank you Vadim Comănescu for being my mentor. Thank you for teaching me and being an example to follow.

Thank you for your effort on our project and your pioneering in technical practices.

Thank you Dan Vâtca for offering a work place where learning and personal improvement is a key aspect. Thank you for understanding and permitting of the writing of this paper in my work hours.

Thanks to all my colleagues for the learning experiences related to teamwork, communication, and collaboration. Thank you Rebecca Wirfs-Brock for shepherding this paper. You really know how to give constructive feedback.

This paper would have been quite different without your insights, questions, and edits.

REFERENCES

Beck, Kent and Andres, Cynthia “Extreme Programming Explained: Embrace Change” Addison-Wesley, 2nd edition, 2004
Hunt, Andrew and Thomas, David “The Pragmatic Programmer: From Journeyman to Master” Addison-Wesley, 1st edition, 1999 Meszaros, Gerard “xUnit Test Patterns: Refactoring Test Code” Addison-Wesley, 1st edition, 2007 Manns, Mary Lynn and Rising, Linda “Fearless Change: Patterns for Introducing New Ideas” Addison-Wesley Professional, 2nd edition, 2005 Christensen, Clayton M “The Innovator’s Dilemma: The Revolutionary Book That Will Change the Way You Do Business” Harper Business, Reprint edition, 2011

About the Author

No bio currently available.