About this Publication
In this experience report, I’m going to describe how I adapted Agile, what worked and what didn’t. I applied agile to my Ph.D., which is part of the Global Data Justice project at the Tilburg Institute of Law, Technology and Society in the Netherlands. I also collaborated on several other research projects, including leading the “AI & Cities” report for UN-Habitat and MILA -Quebec AI Institute, as well as working with the Global Observatory on Urban AI.
1. INTRODUCTION
I’m a social scientist and a PhD candidate. During my Ph.D., I had a burnout. Afterward, I discovered Agile, and it’s awesome. Here’s how I adapted agile to my context in order to write my Ph.D. and do social research.
My experience with academia is one of a culture of overwork, criticism, and a chaotic management environment. While this is based on my own observations, I also was a member of the Ph.D. council, where I gathered that it’s a reflection of systemic issues in academia and in the Dutch environment.
When I started my Ph.D. journey, I was instructed to make a Gant chart spanning four years and multiple sheets of A4 to show how my plan would lead me to success. As you can imagine, it didn’t quite work out like that.
At the age of 29 after 2 years of my PhD, I ran straight into a full-blown medical burnout. Partly due to personal reasons, and partly due to the environment I was in. I was really fortunate to have a great support network from my supervisors and my direct colleagues. I don’t want to imagine what it would have been like if it wasn’t for them. My experience is a reflection of the structure and institutional culture, not the people that I worked with. Mental health issues are common in academia, if little talked about. Unfortunately, I was not alone. Five of my direct colleagues also burnt out around the same time as I did. Across the Netherlands, 50% of Ph.D. candidates don’t finish their Ph.D. within their allotted time.
Then, I met my partner Nikitas, who, as an agile enthusiast, introduced me to the agile mindset. I still remember, we were sitting in a ramen restaurant over dinner, and the way he described agile at the time was that it’s a perspective that recognizes that you are human, not a machine. “What does that mean?” I asked. “Well, when you are tired, you rest.” The pause as this filtered into my brain was almost palpable. This idea was mind-blowing. What about the plan? Shouldn’t you push through it no matter how difficult it is? However will you get anything done otherwise?
As I approached and fell into burnout, the challenge was not that I could not work. It was that I was not able to stop working. It never felt safe to stop. It was never enough, so I worked in a constantly frazzled state, working until my body collapsed. And then, my prefrontal cortex went into hibernation and I really could not work anymore. I had memory lapses, difficulty speaking, back pain so bad I couldn’t walk, and it was difficult to concentrate enough to write sentences. As I recovered, the skills I had to learn were how to start working, then acknowledge when the task or time slot was complete, in order to be able to stop working and create a healthy, well-rounded life.
Agile, as I discovered it through Nikitas, was based on facing the facts of the situation. Recognizing what works and what doesn’t, and then having the flexibility to adapt. That flexibility felt very new and different compared to the rigid plans that I had been making for myself, plans I had been failing to live up to. The flexibility felt like there was room for me, and that felt safe. Agile, and scrum (which felt like the same thing at the time) were a framework of structure and focus, creating a corner of safety in a chaotic environment.
My colleagues don’t really know about Agile. And those who do tend not to have a very favourable opinion. I work in the field of technology regulation, and in my working world, agile is often seen as an alien, and sometimes evil, construct. First, agile becomes conflated with the “move fast and break things approach” of ‘big tech’. Second, the modularity of agile software makes it difficult to identify a single actor who is responsible for a product, because a product is based on a network of service relationships. Existing legal privacy norms struggle to respond to this mode of production (Gurses and van Hoboken 2017).
In practice, my academic environment had little to no understanding of Agile. For instance, when my graduate support monitor asked me what I needed to complete my PhD and I told them an agile coach would be ideal, they looked really confused before their professional composure kicked in and they told me they didn’t know what that was.
In the summer of 2019, Nikitas shared with me the Sutherlands’ little red book Scrum. I read it one sitting on a long bus ride in Greece. My back ached, but I wasn’t going to let this go.
So what did I do? I got very excited and jumped headfirst into doing Scrum. I became my own product owner, scrum master, and developer. I created 2-week sprints, a sprint backlog, a product backlog, retrospectives, and reviews. I tried to implement the whole scrum framework all at once. As was to be expected, it wasn’t that straightforward.
2. Context
A Ph.D. is not a typical agile product. If the Ph.D. is the “product,” I am the only one working on it. It is an open-ended, creative project. The Ph.D. is finished when the supervising professors agree it is, regardless of whether the deadline has passed. There are no clear requirements, nor a clear definition of done. Value is hard to define, and there is no equivalent to stakeholder feedback. There are no users for a PhD, and hardly any readers. Feedback is irregular. depending on the availability of professors, who are notoriously busy and overworked. The closest thing to an end client is the evaluation commission, which provides no input until the final pass/fail evaluation.
3. How I developed and adapted scrum
There were many failures. It was a huge learning curve. Though I was working alone, I was in close daily discussions with Nikitas, who helped me think through how to adopt Agile and iterated with me based on his experience with software engineering.
3.1 Establishing sprints and sprint goals
The first sprint I spent more time tinkering with how to build a Kanban and where my backlog should be than getting work done. I quickly found that retrospectives were important to do weekly. Especially in the early stages of applying Scrum because this is where I was building my system, learning how to adapt scrum and see what worked or not.
Erratic holidays and conferences very quickly messed up my neatly planned calendar. My first sprints were two sprints per month. However, coming back after the Christmas holiday with three days remaining of a sprint felt very unnatural. Two weeks is a good default sprint length, because there is a natural rhythm of high and low energy, and it gives me the space to move through that towards valuable increments. However, sprint lengths have to be flexible. For example, just before a mid-week holiday, I might have a 4-day sprint instead of two weeks. The typical sprint remains two weeks long.
It took at least a year for me to set achievable sprint goals. My first sprint goals were crushingly over-ambitious – smash that chapter in two weeks! Write four stories! Finish that paper AND the blog post! I had little idea of how much I could actually get done, and when I failed to reach the goal – every two weeks! – I was confronted with a lot of disappointment. I was thinking big goals, rather than smaller increments, and had little idea of how to actually break down those large goals into smaller achievable chunks. During sprint planning, I didn’t spend a lot of time really thinking through what the goal was and how I was going to get there. As a result, at that time sprint goals were large pieces of writing that weren’t actually realistic to complete. Sprint goals felt aspirational more than practical.
Looking back at my early Kanban boards, the tasks are a laundry list of disconnected to-do items, not clearly connected to a sprint goal. For example, after three months of doing sprints and applying scrum fastidiously, one of my sprint goals was two quite large but specific pieces of writing. One for my PhD, and another for a collaborative report. When I look at that sprint, the sprint was more a way of keeping track of things, rather than focusing on value. For instance, of the 32 tasks that I had put in my sprint backlog: 6 were meetings; 5 were administrative tasks; 4 were reviewing papers for other people; 2 were preparing for upcoming workshops, and 2 were my ‘sprint review’ and ‘sprint planning’ tasks. Only 13 tasks – not even half! – were reading and writing tasks that worked towards my sprint goal. Of those 13, I completed only one. What is clear now, in hindsight, is that the tasks most clearly producing value for my Ph.D. (e.g. “write vignette B and evaluate if that’s a viable option”) were actually quite large and complex, whereas the administrative tasks were mind-numbing but straightforward.
To remedy this, I started making smaller sprint goals. I liked these because small achievements gave me a sense of progress and hope. For example, instead of ‘work on chapter 3’, it became ‘draft section 3.2a and send to supervisors’. Getting the tasks ‘right-sized’ became an important skill.
Still, sprint goals were sometimes difficult to meet because a lot of my tasks are open-ended and can’t really be sized beforehand. In my writing, it is common that it is only in the doing of the task that I discover what the task actually is. As a result, the amount and level of detail in tasks usually increase throughout the course of the sprint. For example, with a literature review of a particular topic, it is difficult to estimate in advance just how broad the literature review is, or how deep the rabbit hole of specific concepts goes.
In November 2021, I attended a Scrum Master training. Out of a group of 20 participants, I was the only person not working in software. This was where I really learned about making valuable, functional sprint goals. I was very excited by the scrum values, and the training showed me a lot of things I hadn’t been doing. For example, it was after that training that I built a functioning product backlog. Before then, I had thought that a product backlog was only for coordination between teams.
In order to stop my sprint backlog from growing endlessly large, I needed to make my tasks functional. That means tasks need to accomplish something related to the goal. In my Kanban board, the name of the task needs to start with a verb (e.g. ‘write x page’, ‘read x’), because that makes it clear what the action is. Then, tasks must have a purpose that brings me closer to the sprint goal. My supervisor gave me great advice to this end; “read with a question in mind”. While reading for pleasure is beautiful and necessary, it is important to be aware when that is what I am doing, rather than thinking that this is directly bringing me closer to the sprint goal.
By doing this exercise, it showed me which tasks were tangible and that I could actually implement. The remaining tasks were either one of two things. Either, the tasks were something that could develop into a functional task in the future, I just didn’t have enough information about it yet. In that case, it went on the product backlog. The other option is that the task was more a placeholder for an idea, for something that could possibly develop in future, for this project, or even for another project entirely. These ideas are usually inspired, but irrelevant to the current product vision. In that case, I keep these ideas in a separate brainstorming backlog, in order to safekeep them, because these could be valuable for future research or collaborations. This separation between functional tasks, potential ideas, and future endeavors enables me to have a much more focused backlog.
3.2 Exploring the scrum team
The entire agile approach was based on making teams faster, and better – so early on in my agile journey I tried to build a team. The different scrum roles seemed useful and important, so I wanted to articulate a version of these in my environment. I tried to make my Ph.D. supervisors my product owners, and I wanted to share sprint goals with them. This had two responses. The first was indifference; nobody accepted my invitation to view the Kanban boards and backlogs. They were just too busy. The second, after I’d pressed a bit, was a concern: sprinting sounds like a terrible idea when the product is more like a marathon. It was not clear how each sprint goal would build up to a written chapter in my Ph.D.
Throughout 2020, I discovered that the product owner-team dynamic was not the nature of the relationship with my supervisors. I had thought they were my managers, who had the responsibility to oversee what I was doing. Over time, I figured out that it was my own responsibility to direct the work. Specifically, supervisors were too far removed from my daily workings to be able to provide guidance on the next steps; rather, they could give big picture overview for the next months. I couldn’t wait for feedback. I had to move forward, even if a meeting was canceled because of the Internet connection or confused calendars, or Covid19. The result is that I came to realize that I am a team of one, but I have a network of support that I can call on for dialogue and feedback. My position is as an individual, within a network of others.
3.3 Revising sprint reviews for feedback and motivation
In 2020, I stopped telling my supervisors about my sprints. I stopped setting unrealistic expectations. I could at any moment tell them what I was working on, but I was not dogmatic about saying exactly what I had to move towards that week. I kept my head down and stopped trying to make a team out of a solitary endeavor. Working in sprints became a personal management practice.
Instead, I decoupled the function of sprint reviews from receiving feedback. Receiving feedback and sprint reviews are separate activities.
I get feedback in two ways. First, in an email response. My primary increment of value is always written text, in Word documents. The vast majority of the time, my sprint goals are about a particular piece of writing, and finishing that writing is the achievement. I will often get feedback in an email, with general thoughts in response to what’s in the document and what is missing. The second way is to have a face-to-face meeting on video call, with one, two, or three supervisors. These are usually about a particular text, and I prefer to have their written feedback before the meeting, so I have a chance to go over it before we meet. I strategically choose to schedule my supervisors for meetings as close to the end of my sprints as I could. With their busy schedules, that doesn’t always work. In some sprints there was no feedback at all. The feedback then is also a chance to discuss what my supervisors see as important next steps. After a supervision meeting, it is a task to take all the notes and suggestions they made and input them into the product backlog.
On the other hand, sprint reviews are solitary moments of reflection and celebration to keep a sense of pace and motivation. In the first year, when my sprint goals were too big, I would often hit a ‘slump’ because I felt like I was making no progress. I realized that what works well for motivation is small wins. The momentum of progress was important to keep rolling and to do that, I make a point of celebrating what is produced of value in the sprint. No matter the length of the sprint, it produces value. I use the sprint review to sustain that motivation. To do that, I created a template for sprint reviews to reflect on what I accomplished, what I missed, and how I am going to celebrate.
In late 2021, inspired by my Scrum Master training, I tried to calculate velocity to stay motivated by progress. That didn’t work very well for me. I assigned each task a 1,3 or 5 and then added them all up. I had to do this manually, and I reached the high 90s. This was incredibly tedious, and without graphs or forecasting, it didn’t add value. I quickly found that a lot of my tasks were super small, and the tasks that were a 5 were actually much larger. After a few months, I plateaued around the same velocity, and I decided the grunt work of calculating velocity was not worth the effort. So I stopped.
3.4 What Scrum leaves out
There are a few things that feel like they accumulate until they explode, and I did not find a method in the Scrum and Kanban framework to address them. Therefore, to care of these tasks, I added fixed calendar slots for product backlog refinement, administration, and ‘socials’. If product backlog refinement isn’t scheduled, I found I easily get lost in the mire of what I’m working on in the week. I also created an administration backlog, prioritized in order of value and urgency. Every Wednesday morning I go through it for an hour. Most things can wait a couple of days till this time slot.
The one thing that didn’t feel like it was taken care of was nurturing social relations, which as my own product owner is important yet isn’t quite a ‘task’. This is important, but it’s not a priority to do every day. So, I created an hour of batch responding to emails and messages to build and maintain a social network. This eliminates the feeling that I have to respond to all emails all the time.
Overall, having a couple of short, scheduled moments for these issues creates a sense of cadence and rhythm. Because I have them scheduled, I know that non-urgent issues can get addressed at those scheduled calendar slots. As a result, I do not have to address everything, all at once, all the time, every day. This creates a feeling of trust and ultimately more ease.
3.5 Agile advocacy
As I started writing chapters and producing value, I was on a roll. I became an advocate for Agile. Surrounded by stressed-out Ph.D. candidates in the midst of a Covid19 pandemic, I joined the Ph.D. Council, which is the interface between the students and the faculty board. In this virtual space, I invited the council to experiment with agile approaches in a team: identifying value, trying something out, and evaluating the results, retrospectives, and transparency for inspection. It was an incredibly invigorating experience.
I presented a peer workshop on ‘agile productivity for PhDs’. I shared why I was using Agile, why it was different, and what my adaptation of Scrum looked like. In that workshop, people got really excited by the feeling of trust. That despite a chaotic research environment, creating a framework of values and practices, like adapting scrum, could enable space to focus on what matters. In that way, you can trust that your space for creative work is protected, regardless of what is going on around you.
It was particularly revolutionary for my peers to think about how making a plan at the beginning when you have the least information is doomed to fail. They liked the sense of cadence with clear task management. For example, when I demonstrated the simple act of moving a task across a kanban board to ‘done’, the sense of relief was palpable. When I shared that a sprint review was not only an opportunity for reflection but for celebration, the question came:” But what if my supervisors give me bad feedback?” I felt like this for a long time, so I understand where this question came from.
In a culture of criticism and precarity, feedback, especially from direct supervisors, can be very difficult to hear. When people are stressed out and really identify with their work, negative feedback can be received and understood as a reflection of one’s self. If there is no space to acknowledge and celebrate the completion of an increment, all of the mental energy focuses on what needs to be done next because the current state isn’t good enough. That stress perpetuates a negative spiral that reinforces a sense of insecurity. Instead, taking the time to celebrate wins, and framing feedback as an opportunity for growth for the next sprint, creates a space for joy in the present. This process is one of the ways that the structure of scrum values creates a space of psychological safety and enables motivation to continue to create good work.
3.6 Research collaborations
Over the last three years, I’ve engaged in several research collaborations separate from my Ph.D. In each of these, I suggested doing retrospectives. Usually, the idea was warmly welcomed, tried once, and then very often squeezed out, and left until the end of the project, if done at all. Often, we planned a retrospective, but another meeting got canceled and the time that we had set aside for the retrospective was instead used as a buffer, so the other agenda items spilled into that time slot and the retro didn’t happen. Retrospectives aren’t prioritized, yet.
A lot of energy in research projects is spent figuring out the process of how to work together and when to set up meetings. Simply scheduling people to meet at the same time can be a nightmare. In general, there are no teams, just loose networks of individuals that coalesce for temporary projects. The norm is to have too many projects at a time. Given that context, it is not very realistic to form optimally focused teams. Instead, I’ve learnt how to improve the process, and I’ve used some ideas from Agile to do so.
For example, in 2022 I was part of a large research project led by 6 expert professors. However, once it was possible to schedule a common time slot, the meeting had to cover too much. The one-hour meetings covered product vision, conceptual discussions, and scheduling. Consequently, the product vision was not clarified, and after a while, the project was progressing, but going in several different directions at once. In response, I stepped up to take the role of a product owner, focusing on value, asking for stakeholder validation, shielding researchers from the stakeholders, and prioritizing tasks with a helicopter view of the product vision. In the end, we pulled it off. The experience showed me that there was no product role from the start and unclear expectations about what leadership should look like. In my field, we don’t have a framework for teamwork. While there are leadership roles such as Principal Investigator or Scientific Leader, in my experience, there is no common understanding of what that means. As a result, there is a lot of variability in leadership.
Ultimately, when I found agile, I thought I could save everybody. If only everybody did this, then things would work so much better, and everybody would be a lot less stressed! Of course, the reality is, people won’t change unless they want to. You can’t make them, and you can’t take responsibility for how everybody else chooses to live. Stress is addictive; it’s partly why I burnt out. For me, anxiety has a tendency to feel like I’m busy with something important, but actually prevents me from taking care of the things that really provide value. Stress will be there, but shifting out of panic mode requires a willingness and an intention that not everybody is interested in. I am. I’m clear on the type of environment I want to work in, and agile methods help me to create a structure and a space that is ultimately calm and creative. So, what I can do on projects, is to invite my colleagues and collaborators along for the journey to shift out of panic mode, and exercise some leadership in gently introducing a more agile mindset and useful practices.
3.7 Prioritizing sprint goals in a chaotic environment
Prioritizing my project requires ruthless focus. In research, there is always more than one project going on at the same time. Sometimes, I have goals for three different projects per sprint. It is not realistic to carve out space to work on only one project per sprint.
Originally, I had one product backlog per project. Predictably, I felt very quickly overwhelmed, because I brought too many tasks into the sprint backlog and little progress was made on any one project. In late 2022, after participating in an Agile Alliance webinar describing a similar issue, I consolidated my product backlog. Now, I have one product backlog which combines my current projects, which is prioritized, and refined every week. This forces me to ask difficult questions to prioritize between projects.
I try to focus on one project at a time, to minimize context switching. For example, if in one sprint I had a goal for my Ph.D. and for the report with MILA, I would try to work on my Ph.D. Monday and Tuesday, and work on the report Wednesday and Thursday.
Throughout 2022, mid-way through my agile journey so far, I came to understand that I was the product owner for my Ph.D. and most of my research collaborations. That meant taking responsibility for projects, explicitly focusing on value, refining priorities, and also navigating stakeholder relationships.
When I realized that I didn’t know how to do that, in January 2023 I completed a Product Owner certification. That showed me how to work on the product vision and better articulate a product goal. These skills are incredibly transferable to research.
Now, reprioritizing goals and project priorities happen each sprint. While the product vision and goal of my Ph.D. remain the same, I am much more able to adapt the way that I get there. In the beginning, my understanding of the project was very linear, I conceived of it as building a road one brick at a time. Now it’s about sculpting as I go and as the project evolves. Re-prioritization allows that evolution to happen. As a result, I work with a flow-based system with Kanban, but I do prioritize my tasks. My goal is not to stick to the plan but to create work of high value, which means adapting to the situation, accomplishing high-value work, and celebrating it.
Keeping the focus on my work is a delicate balance between creating autonomy and managing relationships with others. Others’ urgency breaks my flow sometimes. I regularly receive requests that were unforeseeable, that do not have a deadline, but feel urgent. This means it is very easy to respond to all the requests, and not make any progress on my sprint goal(s). To keep the priorities for my own work, when ‘urgent’ requests come in, I create a buffer. I respond with a question: “When do you need it by?”. The vast majority of the time, those seemingly urgent but not-so-important requests can wait a day or two until my ‘admin slot’.
Autonomy to prioritize my project is about claiming time and space, which creates a feeling of safety. Safety is important because it lets me shift out of the state where it feels like everything is urgent all the time. That’s a very stressed-out place to be. Claiming space is an active process. It does not happen automatically; you cannot just be reactive. It requires taking responsibility for the prioritization of your goals.
4. Takeaways
I am now four years into my agile journey. Adapting agile in a non-tech, chaotic environment, while playing multiple roles as the product owner, scrum master, and developer all at once, was not always easy. And yet, this experience has enabled me to understand the essence of agile and scrum values My approach to work has been radically transformed for the better. I am more calm and more creative, because my work evolves, all the time. I will bring this forward into all the work that I do. Ultimately, working with Agile also made me question the kind of research that I want to do because, at the end of my PhD, I want to work on projects with a clearer focus on providing value.
During the writing of this experience report, I have become even more aware of the impact that this agile journey has had. When I share this story with colleagues, it often challenges their views of what agile is. For instance, one colleague of mine was familiar with agile because it was imposed on them by their previous leadership as something performative, that needs to happen because this is how money is made efficiently. They were genuinely surprised and intrigued to see that this was something that I found incredibly valuable and that adapting agile values is so much more than just standing up for fifteen minutes at the beginning of the day.
My experience with Agile is connected to my recovery from burnout. Since the beginning, Agile felt very safe. I found agile through Nikitas, not because it was imposed upon me by corporate overloads. While it is not in the scrum guide (yet?), one of the key components of agile for me has been psychological safety.
Agile allows me to create my work from a place of trust. Scrum provides a structure, a cadence, and a set of principles that create space and tools to focus on value in a complex environment. That enables me first to deal with the chaos, to center myself, and then produce high-value increments. When agile is implemented successfully, it allows a safe ground for people to thrive instead of burrowing further into burnout.
In this way, I see agile as part of the larger shift in the culture of work that we are collectively experiencing. There is so much potential to support people who are brilliant and yet smothered by unsupportive working environments using an agile mindset and Agile-inspired practices. We are genuinely excited to see what happens next.
.
5. Acknowledgements
My agile journey was only possible because I have a strong support network. I would like to thank my supervisor, Prof. Dr. Linnet Taylor, prof. dr. Ronald Leenes, and dr. Merlijn van Hulst, for their support and for trusting me when I showed them that my agile stuff is working. My mentor, Jolanda Botke, for encouraging me to trust myself. Ioulia Ventouratou, for her invaluable mirroring and clarity of spirit. Ron Langens, for the guidance and good cheer. Ralph Jocham of effective agile for his energizing scrum training. The Agile Alliance, for creating inspiring and hopeful spaces. My shepherd, Rebecca Wirfs-Brock, whose dialogue and accountability made this report possible. And finally to Nikitas, my catalyst, unofficial agile coach, and my partner, who was there every step of the way.
REFERENCES
Gürses, Seda, and Joris Van Hoboken. “Privacy After the Agile Turn.” The Cambridge Handbook of Consumer Privacy, Selinger et al., Ed, 2017.
Sutherland, Jeff, and J. J. Sutherland. Scrum: The Art of Doing Twice the Work in Half the Time. Illustrated edition. New York: Currency, 2014.
About the Author
Experience Report
Author:
