In this Q&A followup to OnAgile2017, Jeff Patton explains his views on discovery and why you should design like you're right and test like you're wrong.
Following is a lightly edited transcript of his conversation with Jason Little, Chair of OnAgile2017.
Hi Everybody, I'm Jason Little the chair for OnAgile2017. Our theme was Back to Basics and we did all of our talks based on getting back to the core of how agile came to be and put more of a focus towards the customer because it seems we've lost our way a little bit and stumbled into trying to get different processes and methods right as opposed to focusing on the intent of agile which was to build awesome products for customers.
So here I'm talking with Jeff Patton who delivered a session at OnAgile. He wasn't able to make the QA, so we captured the questions that came in through the platform and we're just going to go through those questions and see if Jeff has some interesting answers for us.
Welcome Jeff and thanks for taking the time to do this.
I'm glad to be here now. I recorded the talk in my office before. Now you're looking at my office with my messy white board behind me and it's light out there so hopefully you see me okay.
Is It Advisable for the Triad to Share an Office?
Yeah everything is clear. Awesome. All right so let's get down to it. The first question was Is it advisable for the triad to share an office?
One of the concepts I talked about in the talk is triad.
Triad is the term that Atlassian uses for this core team. The core team is the the lead engineer, product manager and usually for consumer facing stuff UX Person.
Now the other weird thing I'm going to riff on this just a second. I think I might have used CarMax as an example of a company that I have worked with a lot. CarMax does a lot of things where they're iterating most software and operational policy. On their core teams they have an operational person also. So I might have joked in the talk that there's one company I've worked with they refer to it as a “team of three”.
And the weird thing about it is sometimes there were two people on their team of three sometimes there were four people on a team of three. And that core team were closest together to steer what's going on with that product.
The question was: “is it important for that triad to share an office?” Yeah it's sort of important that they're working pretty closely together that they communicate as fluidly as possible.
If they're sitting together that helps a lot. If they're not sitting together that's a disadvantage.
I work with teams that have the core team that doesn't share an office. They talk to each other via Slack many times a day and they communicate with each other at least once a day face to face.
Sitting together allows for a lot of natural organic communication. A lot of people complained about Agile development is having way too many meetings and a lot of the meetings go away if you just sit together and that's it.
Your alternative is to more formalize the communication if you're not sitting together. So the short answer is sit together if you can.
It sounds like as long as you have all the necessary concerns represented if it ends up a team of three was a team of four or a team of five, you just need the right people in the room.
Yeah you need the right people in the room and we're trying to make decisions.
I feel like agile has lost its way a bit. We focused so much on building software and we got the team together but we all thought the team was the developers and the testers but we assume the customer or the product owner was going to tell us what to do. We didn't have to worry about that but now we’re trying to fix things to say no, we do have to worry about that.
I frame that triad concept as people in the room that can talk about what's valuable, usable and feasible and it depends on what we're building. We want the smartest heads in the room to help make valuable usable feasible decisions. And we want the team in on those decisions. So yeah those are the people we want in the room.
I was looking at the question that somebody had sent in and they had continued on that, “In January management plans to move the dev team to a different office and locate all the product owners into another office.”
Now resist that if you can.
I worked with one organization who had had that separation where product people were in one office and developers were in one office and those product people had scheduled at least one day a week and several hours a day to sit with the team. There was always a desk for them in the development team area and they always had time where they came at least an hour a day every day and at least a day per week all day long to sit with the team. So even if you do get separated, look at a way to formalize being together.
Sometimes I think that there's a bit of an assumption that management is making this change, therefore we have to follow suit with whatever it is. I get this a lot in the change space and to me it's just go stay with the team. I mean you're not breaking any corporate rules by not sitting at a desk or spending two or three days of the week there. But do what you need to do to get it done.
I don't want to take this too far sideways. But you work with the change world a lot. And one of the things I see a lot is people thinking that “leadership is doing this to me.” And I find a lot of situations where leadership is doing their best. They think they're doing what they should be doing to help things go better.
And sometimes all they need to be told is that “this isn't helping. It's not helping things go better.” They need to be given advice on what's going to help.
You shouldn't need to ask permission to do your job right.
So yes sometimes just sitting where you need to sit, sometimes taking the lead yourself telling leadership where you need to be able to sit to do your job well. Sometimes that's in order.
Yeah it's a good point. Because especially with moving, it could just be the underlying reason is to save cost. They're moving because of some other reason where the intent is not to make us not collaborate it's just we need to do this for now.
It's totally different when you need more office space. We can't fit everybody in this one location we need to split it up and they're coming up with what seemed like a very logical way to split things up. The people making space plans don't understand that they're hurting the process.
Everything you build is a Bet. Are you willing to bet your lunch, or your 401k, that people will use your product?
Awesome. So the next one I guess wasn't really much of a question but “the 401k joke was pretty funny.” I don't actually remember what the joke was.
One of the things we talked about, and it actually leads into the next question pretty well, is that everything we build is a bet.
If you're using traditional scrum or agile we start every sprint with a bet that we can get this many stories done in this two week sprint and they'll be high quality. At the end of the sprint we’ll show what we did and we talk about our velocity. Did we get as much done as we bet?
But I also talked about the real bets we're making are those outcome and impact bets. Outcome meaning people try and use your product, they like using it, and it has the necessary business impact - it helps your business.
Those are the bigger bets.
The 401k joke came in as I ask people to articulate their hypotheses. We bet if we build this that people will use it and we'll get this benefit. And I ask people to say “what would you be willing to bet?”. I built a scale starting from lunch and going all the way up to your 401k.
I think the joke is some people's 401k’s are worth less than their cars are based upon lunch. But I put 401K up at the top of the scale.
Ask the next question because what I just said kind of leads into the next question.
How much research is really valid? And do proto personas just harden our assumptions and prejudices?
There were actually three questions wrapped up into one. I'll just read it as it came in.
Question on big research upfront followed by big design upfront.
How much research is really valid - meaning when is the research too little to yield valid data or is it okay for it to be a starting point and find out more later.
There is people who push against proto persona saying that we just harden our assumptions and prejudices.
May want to change the picture so that I can draw something really quick.
Research is a weird twisty word I think in contemporary process.
Well there was the original notion that we had might have an idea for something to build and we might start by doing some research. In that research we learn about the people and the problems that we're trying to solve.
We'd synthesize those people and problems into things like persona's or maps about how they did things. Then we would use those to go to ideate to come up with solutions. Then we would go back out and do this different kind of research to test our solutions.
Actually in a lot of cases we test for usability.
In some cases we don't even test. Sometimes we just ship it and build it.
But when we ideate and have design here. What they're talking about is a lot of the stuff that we did up here. This is the stuff that got labeled as big design up front or informal UX stuff. We talk about this is a research phase.
How things have changed is that we quickly turn our idea into a hypothesis. As we quickly turn it into a hypothesis, a bet, or a belief and that hypothesis is two sided.
The one side is about the problems that we're solving and specifically the people we're solving them for. This is where this term proto persona comes from because it's a persona hypothesis. It's a bet about who our people are.
And we also fashion the other side of the puzzle and that's a solution hypothesis.
Then we brainstorm about all of the risks where might I be wrong - both on the problem side and the solution side.
Then we figure out what is the smallest test we could do.
Then we get out and we may put something in front of somebody or we may talk to people and observe them and actually test.
Then we learn something and then we use what we learned to rebuild this hypothesis.
Now the question was about big research up front. We would never want to do big research up front.
In fact one of the ways we short circuit this big research phase is we quickly put together a hypothesis and we ask what are the scariest things.
The term proto persona comes from the idea that I can guess at who those users are but then we have to actively question those guesses.
There's a mantra that comes from people who do this work. That's design like you're right, but test like you're wrong. We talk about this as putting out black hats on. You black hat all the information that's in that photo persona.
The question asked “there's people who push against proto personas that will just harden our assumptions.” Well that's the testing like you're right. Your goals is to never harden those assumptions.
And your other goal here by moving very quickly to declaring and testing assumptions is to short circuit that big design up front phase. If we gather together all of our beliefs, filter this, and say what are our riskiest beliefs and how can we move forward as fast as we can to start testing those beliefs or testing those assumptions. Doing more targeted learning and we're moving as quickly as we can to start building things.
But this is where we also question the potentially shippable software things. Sometimes the things that we're building are prototypes - simple ways to test our assumptions about the solution; not potentially shippable software that turns out to be the big 401k sized bet.
Did I answer the question?
The short answer to that is: you always do as little research as you possibly can and you start validating your beliefs and you narrowly focus that research on the things that matter most.
And yeah people do push back against proto personas because we have a habit of designing like we're right and testing like we're right. We have a habit of trying to validate our assumptions and not asking, not black hatting, not trying to prove we're wrong.
No that's good. I can see a lot of influencing ideas from lean lean startup.
Where would you apply this?
Jason, I was talking about you just yesterday. In a back and forth e-mail somebody in Europe yesterday said you applied the same kind of testing sort of approach to process change.
It is just it's funny we talk about this as “lean startup” or “Lean UX” or product discovery sort of thinking.
It's not a product discovery thing. It's just a thing. Just testing our beliefs is always wise and I think that's one of the things that agile was supposed to have given us.
We sort of lost - or forgot - why we're working in shorter iterations. I think some people think okay we're just breaking down a big thing that we have to build into a lot of small pieces and we're just testing to see if we can get this thing built on time. There's a lot of other things we're testing.
I think some things that could get in the way would be how we budget for stuff.
So we've got a separate product discovery team that's funded a separate way from how the actual product team is funded, then I can imagine some of the question is, “well we only have X dollars to spend on this research phase. So how much is enough and how much do we do before we throw it over the wall to the delivery team?”
If the picture I just drew was the research phase…
If the research phase wasn't all just going out and learning about problems…
If the research phase was well framed the way real scientific research is, its testing assumptions. Its testing hypotheses.
If research always included testing our assumptions - whether they're assumptions embedded in a persona or that people really have the problems - the way you test those is by interviewing, observing, getting evidence that those problems exist.
And research also involves the testing part. This is all R & D (research and development). The interesting thing about software is software is mostly R & D.
If we're talking about building a new car or a new piece of hardware, or a new traditional product we'd ship and put on a shelf, we do all this research to perfect the design and then we mass produce it.
There's no mass production in software. We just copy this stuff and if it's cloud based it's just putting it out there. The manufacturing part is nearly free with software.
So software is almost all R&D and we budget for it.
The problem comes when we start trying to manage software like it's not R&D, like it's just construction.
Have you seen the ideo shopping cart video from the mid 90s?
A bazillion times.
I show that video pretty much to every client and we just look at stuff they were doing before "agile" was invented. Number one: go to the environment where your customers are using the product and know that you've got store managers and you've got clerks and you've got four different types of shoppers and just go watch them in their environment see how they use it.
You know the most I think I told the story after your session had been played at OnAgile was the best insight I got was I was developing mobile apps for a customer and we had gone through paper prototyping literally taking pictures emailing it back and forth. This was almost 10 years ago and then it got to the point where I went and sat at her desk and was hacking away little parts and they were using our content management platform as well.
So just sitting there and watching what her day was like actually got me insights into our other product - which was basically her day is just she's on like three phones she's got one on her foot and speakerphone and everything and just constantly bombarded with questions. She's never going to manage her own content.
So one of the five points I made was about empathy and about getting out and doing that.
I remember working with one team at a big company that has a lot of stock traders and stock portfolio managers and things like that. I can remember the developers that worked with the stock traders.
First off, traders and portfolio managers always work in big open rooms super noisy lots of monitors on desks and no, they don't work in offices. They really lean heavily on being able to communicate quickly with each other.
In this situation developers were working on one side of a table there was a low partition like just a nose height and across from them were the traders. They lived with the traders. And they had empathy built in.
Such a thing that's not pointed out enough in agile development.
What you describe - the empathy you got from being there - turns out to be critically important. It helps you solve problems a lot better. And what's interesting is we formalize that and call that research.
Before we understood what we were doing is research, is actually observing people as they work and deeply understanding their problems. Before we wrapped it with a little bit of formality, we just used to talk.
I was deathly afraid of research in the 90s when I was doing this stuff but I had no problem going out spending time with my customers. It wasn't later till someone told me, “well that's kind of research if you just be a little bit more rigorous about writing down what you saw.”
It would be better, but what you're doing is already research.
How do you get delivery team personnel engaged in the discovery?
Cool. All right, so let's move on to our next one.
How do you get devs QA's and other delivery team personnel engaged in the discovery was part one. And is it dependent on negotiating the urgency to go fast so they can peel away to be interested in discovery?
What's the second part? “Or is it dependent on negating the urgency to go fast?”
Well first off I like the way these questions start to feather together.
We just talked about as long as we understand discovery work and research work as doing routine things like going out and spending time with customers...
You just told the story about you sitting with somebody. Did you think about this as research? Did you think that his discovery work when you were doing it?
No it was just too annoying to deal with it through taking pictures and emailing so I thought I'll just go hang out there for the day.
What I heard you tell the story about was you learned more than you intended to learn by being there. I would call what you were doing is discovery. You were working with prototypes and that's discovery work and you were a developer. You know the first part of the question was How do you get QAs and other personnel engaged...
Now maybe I can enumerate things.
First off don't hide it. Keep it exposed. Make sure hypotheses are exposed. Make sure that the discovery work is exposed.
Make sure that the team knows we're going out and visiting customers today.
Make sure that they're invited.
Make sure that they're given lots of opportunity to participate.
And don't make it special make it routine. It's just the way we do things around here. We go out and visit customers and it's sort of important that we do.
I don't know if I'm answering that question. Well, how do you get them engaged...
Don't force them because that never works. Tell them it's happening and invite them and give them a job to do.
There's a company I'd worked with that had really simplified the process they used to take notes during interviews. As a consequence, they taught everybody how to take interviews and if you sat in an interview you were expected to take notes regardless of your role. And everybody was taught how to do it. So it made it super easy for testers and developers and almost any other role to be engaged and to feel productive during an interview. Things like that make it easy for those people to participate, make it visible.
Do you need to negate the urgency to go fast to get delivery team members interested in discovery?
The other part of that question was negating the urgency to go fast.
Remember I said design like you're right and test like you're wrong. The reason we do this discovery is not because it's a process point because we have to do it. It's because we're trying to de-risk what we're doing.
There's often an urgency to go fast building stuff that we haven't validated. It's oftentimes going fast the wrong direction. Yeah there's this urgency to go faster to build more stuff.
I really question the urgency a lot. When we start to honestly ask, “Would you bet your 401k that what we're building as fast as we can is going to work?” If everybody says "No I wouldn't". Then you start to really wonder why are we working so hard to try and build this stuff that we're uncertain of so fast.
What could we do to be more certain?
Everybody who works in a large organization has a lot of delivery pressure on them, a lot of pressure to go fast. One of the first things I always tried to do was use discovery work to kill ideas.
If I can use discovery work to validate what we're both building is worth building, one of the nasty things you'll start to find out is that what we're building is probably not worth building or not in the way that we were building.
And the minute you start using discovery work and realizing “oh my gosh I'm glad we did this - we just saved a lot of time that we would have spent building the wrong things fast” you start to realize this helps you start to wait. I think it'll get better.
Jason, you've been a developer before right? Did you ever adopt test driven development?
More Acceptance Test Driven Development, customer validation that type of approach or you know down to actually TDD. Yeah. Before I knew that was a thing.
Yes. And out of necessity because it was too hard. This is like going back into the mid 90s building stuff and functional programming languages. So there was no real frameworks or things out there to do it. I spent a lot of time just smoke testing stuff just to make sure, and I ended up adopting it out of necessity.
I was going to build that as a metaphor and I want to relate it back to what I was talking about with discovery work.
I've been a developer a lot. And when you're doing test driven stuff, when you're doing unit testing, when you're writing test scaffolding around all your code it seems like this is extra work. This really sucks. But then you start to get used to it. It's hard not to be such a heavy burden to do it.
As you do it more and more, pretty soon you've got a code base that's wrapped with this structure of tests. And every time you check code in all these test run. And the first time you check in code that you thought was working and it breaks something else somewhere else, you go “Oh my gosh! All that work I've been doing just paid for itself.” And that's when you feel the value of doing it.
Discovery worked for me exactly the same way. You do all this work and the first time you catch yourself and say, “Oh my gosh! I'm glad we did not build this thing because people wouldn't have liked it, wouldn't have used it. We wouldn't have gotten this benefit.”
You start to say, “Ok all this discovery work is paying for itself.”
When there is this super urgency to go fast, it's because we separate the team that builds things from the team that actually gets the benefit from it. If you're not accountable for the benefit you're getting from it then that urgency to go fast is there.
Basically, use discovery to save time.
It doesn't cost time. it saves time. If you're doing discovery work right, it's going to save you time and that makes it easier to prioritize the urgency.
For me discovery work is test driveness, but we're testing the outcome and impact not just the code.
How can a product manager be more inclusive?
Cool. Awesome let's go to the next question.
How can a product manager be more inclusive? When your title implies authority but you want to include the team members as equals what's the best way to overcome that?
That's a culture question for me.
How about you answer that one.
I don't think it implies authority at all.
I think it just implies that you are the link between the team and the customer. And there's perceptions around what that title and that responsibility sort of implies.
I think they are equals. Team members are equals because if team members have been working in that particular product for a period of time there... I have never worked in a team where there hasn't been one or two people who are developers that have very strong empathy towards the customer and they know their business domain very well.
So I've always kind of seen them as equal even when I was a product owner. It's invite them into the conversation and don't pretend that you have all the answers
Imbedded in this question is yes, your title does imply authority and there's a right way to have authority and a wrong way to have authority.
It starts to get into questions about what good leadership is.
A good leader isn't the one that figures out what everybody should do and tells them exactly what to do.
That's not a good leader.
A good leader is the one that involves everyone in deciding what to do. Make sure that everybody has shared understanding about what success looks like and where we're going, takes everyone's opinions into account, and makes decisions where the team feels like they own the decision making.
But that can go too far that way too.
A bad leader is one that turns it into a democracy where the team votes and you know if you've got four people on a team and two vote this way and two vote that way and we're at a deadlock and then we split the difference don't end up getting anywhere.
There's a subtle difference between between being very autocratic and telling everybody exactly what to do and being overly democratic and having everybody decide what to do.
Somewhere in between there is a leader that inspires people to want to follow.
A good product owner, a good product manager does have authority. It comes from experience. It comes from being able to balance a lot of different concerns at once. It comes from being a good leader, being able to inspire people to want to want to follow you.
How can you be more inclusive? The answer is embedded in the question. It's be more inclusive.
And yes there is authority, but authority doesn't mean deciding and telling people what to do. It means involving people. It means inclusively including them to make decisions.
Like you said it's a cultural thing. It's hard to articulate how to do that well in a process.
How many Sharpies do you have?
Yep and then we had how many many sharpies do you have on hand at any given moment.
Is that really a question?
I almost want to pick up my computer and walk you around.
I have hundreds at any given moment.
I teach classes and I supply Sharpie markers for the classes that I teach and I have a rule that I only use the Sharpies a couple times in a class so I always bring back bags of sharpies and so I can share my screen.
You see that... There's a picture for this is one of the Sharpies that I got another one. Another one and another one...
And these are just the used sharpies that I've got.
So look yeah the question of how many sharpies do I have - hundreds.
If anybody has a good use for slightly used sharpies...
I hate to use them in classes after they've been used a couple times.
But I don't know if that was a real question.
And how frequently did they go missing.
Oh I know, missing.
Actually, it's great when people take them.
Actually they go missing a lot when I teach in Europe and in Asia and places like that because they're a little bit more rare.
It seems to be size of company for me. The larger the company, the more likely they are to go missing because there's more controls in place for getting supplies. “A sharpie that works or a whiteboard marker that works I'm keeping this!”
That that was sort of all the questions that people asked on this. I expected that there might be more, and tougher questions.
Use Agile approaches to question your assumptions
So is there a question that you would have liked to have been asked or other important ones that you've heard from doing this talk?
You mentioned you did a few over in Australia.
Immediately after recording this talk, I had to head off on some overseas stuff and I gave the talk at a conference in Australia.
No similar questions...
The really big cultural shift that I see is starting to use the way we work to validate that we're doing the right things. Starting to question our beliefs and question our assumptions and starting to rethink if there was a way to rethink agile.
For me what's interesting is the way that I always used agile...
I started with extreme programming in 2000 and the term agile was going in 2001. But I came from a product world and I always thought the purpose of these short iterations was to see things working. Not just to show progress but to test my ideas very quickly.
I can remember building software for brick and mortar retailers and I needed feedback from them faster. I had a development team. One of the things that would slow us down was building software that was production ready that wrote into the big Oracle database we had to write to in order to have the scaling factors we needed. So what we would focus on building in early iterations were things that would write to an in-memory database and didn't have the scaling qualities it needed. It was never potentially shippable because we couldn't ship that stuff into production but it was working software in that we could use it in a very limited way to start to test ideas and start to iterate it.
And I always used agile that way and I think we've forgotten that.
I didn't explicitly think of them as hypotheses and assumptions and tests. I didn't explicitly make a decision along this continuum to not over invest in production quality software.
We didn't even call what we were building a prototype back then. We just thought it was the sensible thing to build. We as a team were accountable for the success of what we were doing and we had been burned before, so going straight to production software didn't make sense.
What I hear from people that are starting to work this way is the rediscovery of, or trying to sell their company on questioning their assumptions. Trying to sell their company and convince themselves that it's worth investing in experiment.
The big question I have is: how do I help people learn to start to experiment?
Jason how do you do this with process change? You ask people to start to try something and try something small and measure it instead of going big.
How do you convince them that that the goal is not to find the perfect process and roll it out to everybody, but the goal is to do something smart and do it.
It's usually to help them understand that testing a process or any type of organizational change and then trying to replicate it across rarely works because context is different from team to team. So it's more around overall organizational constraints, pros and cons, the same.
You mentioned feasibility usability lovability sort of the three things and the MVP and it's the same with you know helping companies transform to agile.
One organization I worked for, the director of HR said, “we could never use agile on this particular project.”=
I said, “well you could but why would you say that?”
And it basically came down to he said, “if I screw up payroll for 80,000 people the old way it's way better than if I tried something new.”
So it's not about finding the right way of doing it and rolling it out. It's more, in his position I wouldn't take that risk either. So let's have some empathy for context for different teams.
So in a way if he can't screw up a payroll for 80,000 people he can't afford to be wrong and he's hyper aware of it. Yeah. It calls for a small experiment. That's where you look if there is a problem here that we need to fix. And there's a risk of being wrong for 80,000 people.
How do I test my ideas? How do I be sure I'm right before I ship?
The tragedy is that he's saying, “I can't do this in an agile way.” And there's the real tragedy.
People don't see agile as a way to experiment or test ideas.
They see agile as a way to deliver incrementally and deliver little bits of crappy stuff until I get to something that actually is good.
In the product world, in much of the real world that we all live in we can't afford to be wrong. When I'm building software for large chain retailers we couldn't afford to be wrong when I had a customer that had hundreds of locations. I couldn't ship something crappy out to hundreds of locations. So we always found one small partner location. We always dealt with risk and dealt with small experiments. That just was the way we had to do it and that's what your HR person has to do.
We just don't have that conversation enough with software.
And again the tragedy is people don't see agile as a means to that end.
The weird thing is we had to invent or start to articulate that as lean start up and Lean UX. And talk about things like dual track development where one part is experimentation. The nasty part is we've pushed terms like “potentially shippable software” which in my head always reads “big bet.”
It's building like I'm right like this is the right thing to build. Yeah it does not make sense if we don't know. You don't know where we're going with that.
Have plenty of options, and ultimately let the people who have ownership of whatever that product or that system is, let them decide
And better give them the ownership and not just owning what they're building or owning the building of it but owning the outcome or the success of it. That's what changes everything I think.
Spend more time with the people who use your software
Yeah for sure.
Alright, any last parting words of advice for people that are trying to make a shift back towards more customer centric product development before we wrap up?
There's people I've known for years that I've kind of observed their careers. Their journeys as they've moved from just starting with product thinking and they've become really accomplished product people and often became product managers.
And I always ask them if there's one piece of advice you would give to your past self, if there's one piece of advice you give to a new product manager or somebody who's coming up. What will that be?
More than half the time I get the advice to spend more time with the people that use your software.
That changes everything. That's where you actually see it first.
You'll develop some pride in what you made or you develop a healthy sense of shame if you've made really bad stuff.
There's a dirty little secret here that teams that care about their customers outperform teams that don't know their customers. And if you were really worried about productivity, help your team care about their customers.
Spend more time with your customers.
Great That's a perfect wrap up.
So thanks for taking the time to to do the Q and A. And hope you enjoy the new year. We will hope you do too. Thank you.