Watch Dave Prior’s thought-provoking interview with Larry Maccherone, Director of Analytics and Research at AgileCraft. Larry is a thought leader in Lean, Agile, metrics, and visualization. His core expertise is drawing insights from data that enable people to make better decisions. Check out our Vimeo channel for more great content!
Transcript
Hi. This is Dave Prior. This is Agile2015. We are going to be doing video podcasts with speakers and thought leaders all week long. This is our very first one. Larry Maccherone, thank you for coming by.
Larry Maccherone: Thank you for having me again, Dave. It’s always a pleasure.
Dave: I have been really looking forward to this, because there’s not a lot of people I can talk to about the data stuff who can go as deeply as you can. But before we do that, you work at AgileCraft, right?
Larry: I do. I recently started there.
Dave: Would you like to talk a little bit about what the company does? Just to give it a little quick plug.
Larry: Sure. The reason that I love AgileCraft is because their strategy is very different from where I was – you are probably familiar with Rally before – in that AgileCraft doesn’t care what tool the team uses: you can use Jira, you can use Rally, you can use VersionOne, whatever, or you can use AgileCraft’s own team tool. But the primary value proposition of AgileCraft is that we sit on top of all of those. So you can do all your program- and portfolio-level planning and you can push down features and initiatives into the team level and you might have some teams using one tool, VersionOne for instance, and other teams using Jira, and yet the roll-up from those, since it came down from above, gets pushed back up. So you have status across a wide range and you can have unified performance metrics across a large organization with multiple tools at the team level.
Dave: So it’s just Agile tools, not traditional PM tools?
Larry: Well, since there’s four levels, the team being the lowest level, then program, portfolio, and then enterprise, we do cross into the portfolio-level planning type tools and even more strategy and enterprise-level sort of objectives made in that kind of thinking.
Dave: Before we get to your sessions, the segue I want to ask you about is what kind of guidance are you providing people now in terms of how you look at the projects across a portfolio to figure our strategically what’s working and what’s not? Is there a guiding rule about that that you guys are promoting?
Larry: Well, the message before products like AgileCraft were around was really pretty messy. You either had to make the team switch to all the same tool, which they hated.
Dave: Yes, like this one is 20% done and their velocity is 6.
Larry: Exactly. Otherwise you would have no unified view. You had to piece it together. I think the biggest thing we’re competing against, though, is spreadsheets. People have been using spreadsheets to do their planning and to track their progress and it doesn’t matter what tool is at the team level. The problem with spreadsheets is that it sometimes takes you weeks to gather the data from the team-level input and by the time you actually compile it and organize it and get it structured right, the reality has changed and so you are always a couple of weeks behind. This is instantaneously: you have all the status, all the way rolled up.
Dave: So you are giving two talks and I am looking at my notes so that I do not forget when they are. One of them is today.
Larry: One of them is today, at 2:00.
Dave: It’s about probabilistic – I have so much trouble with that word – forecasting for math models. What’s that session about?
Larry: Last year, I had the Impact talk and I had the SAFe Metrics talk and both of them were very metrics-oriented and then at the end of each of those, I sort of teased a little bit of content about how you can make decisions with a limited amount of data. We tend to think that if we don’t have a complete, well-vetted model, we may as well ignore it. But we don’t do that when it comes from our qualitative experience. We’ll take any evidence that is in our head and use it for the decision, so we should also be able to use the model. So that was very interesting to folks and they liked that, but I didn’t give them any concrete ways in order to do this.
Dave: Specific tools, you mean.
Larry: Specific tools and concepts and frameworks to do that. And this year I sort of developed a framework for doing this in a way and some examples that show people how to do this day-to-day, real-world things that they are experiencing every day.
Dave: So you are going to teach the Agile people about Monte Carlo?
Larry: I’m going to teach them about Monte Carlo, I’m going to teach them about other simpler even ways. The reason Monte Carlo is so troublesome…
Dave: … and scary.
Larry: The reason why it’s so scary is that the executives are not used to getting a probability curve when they ask the question, “When is my project going to be done?”. They want a date. So, there’s a mindset shift that has to occur both in the team delivering that probability curve – there’s some tooling and some technical knowledge that the team needs in order to be able to compose that probability prediction – but there’s also a mindset change at the executive and stakeholder level to understand that. So the talk today is really focused on that and I actually changed the title at the opening to “Probabilistic Decision-Making”. It’s listed in the conference as “Probabilistic Forecasting” and I show “Probabilistic Forecasting”, but then I put a line through it and say “Decision-Making” because there’s a lot about decision-making.
Dave: I know I have trouble with the whole accuracy versus precision argument, because it just seems like a semantic way of getting out of saying “this date”. But you’re going to teach people to use tools to have a specific way of answering that.
Larry: Exactly. So I deliver a probability curve to management – I’m a team member – and the pointy-headed manager says, “Wait a second, I asked for a date. Why didn’t you guys give me a date? You gave me this probability curve. I don’t know what to do with this.” “Well, how much risk can you bear?” “Oh, I’m the pointy-headed manager. I can bear no risk.” “Okay. We have that date for you. That’s the one way out here on the bell curve, on the tail of the bell curve”. “Well, I don’t like that date.” “Well, maybe we could bear some risk, but marketing is ramping up for this conference” and “Why don’t we go together to marketing and figure out the possible trade-offs”
Dave: So you are generating better conversations.
Larry: Exactly. That’s the whole point: to change the nature of the conversation from one that’s very contract negotiation oriented around dates to one that’s very collaborative about trade-offs associated with the value, that delivering at various moments in time will give you.
Dave: That is really cool to me because I was thinking, when we first started talking about it, that it was simply walking in and saying “Well, there’s math”. And half the time they would say, “Okay. There’s math. It’s in Excel so it has to be true” or whatever. But you’re using that not to get them to shut up, but to get them to go have different conversations with people and make better choices, more informed choices.
Larry: Exactly. So metrics, like you said – you just gave a great example there – sometimes tend to drive a wedge between management and the team and it sometimes shuts down conversation. This is an example of a way to use measurement to actually create a new conversation and make it be a healthy, collaborative one.
Dave: That is weird, but okay.
Larry: So come to the talk and you’ll see what I mean by it. Great examples for that one.
Dave: Healthy sounds strange. Your other talk which, if you haven’t changed the title, is “What? So what? Now what? How do you use data to influence others?” I don’t have a date and time on that.
Larry: That’s on Wednesday, at 3.45.
Dave: So what’s going to happen there?
Larry: Like I said, last year I sort of teased about making decisions. One of the things that people complained about that presentation last year or suggested for improvements was, “I know what the right thing to do is. How do I get the company to shift that way? The analysis – I believe in it, it’s sound, I’ve used all those techniques. But how do I get my stakeholders to agree with it? How do I move people off of square one? How do I overcome their cognitive biases against what I am suggesting?”
Dave: How do I get them to stop asking me for utilization reports?
Larry: Exactly. Let me give you an example that is from the sports world. Coaches rarely go for it on fourth down – fourth and one, fourth and two. It depends on where on the field they are, but it’s a very narrow band where they go for it on fourth down. Well, statistically however, it’s much more advantageous to go for it on fourth down much more often than the coaches generally do. You can do a value analysis that says that the likely points that would result from going for it on fourth down times the probability of achieving that, compared to the likely points that I am preventing from the other team by punting, you can actually come up with a system to calculate.
Dave: A sound reason for violating their gut.
Larry: Yes. And the reason they don’t go on fourth down, though, is fear. So fear is frequently the factor, so a low risk tolerance and fear is frequently the reason we make bad decisions and this is the reason coaches make bad decisions on fourth down – because if they don’t make it, everybody knows that it was a bad call. Now if you punt and the other team comes back and still scores a goal, well, you know, you’re not going to get blamed for making a single bad call.
Dave: It’s interesting though because it sounds similar to that Moneyball stuff.
Larry: It’s very similar to Moneyball stuff.
Dave: But you’re putting it into play on the field, as opposed to what players to recruit.
Larry: Exactly. So the “What? So what? Now what?” is trying to get you from metrics which are just the “What?” to metrics that say why that matters, the “So what?”, what is driving that. So don’t just say you have a 15% reduction in your coefficient of variance. Say that that correlates with you not being able to deliver on time and then not only that, but here is what you can do to improve it. You have to take it all the way to the “Now what?” – it is the action. Luke [Hohmann] was just saying in his talk right here that actionable is really all that matters. Any sort of help they give them, any sort of analysis that you do, quantitative or qualitative – it doesn’t matter, unless you take action and this talk is all about getting people to take action.
Dave: So you can give them all the data and all the numbers, but they still have this gut thing or their fear thing, or whatever emotional attachment they have to this stuff that they are used to seeing.
Larry: Cognitive bias.
Dave: Yes. I’m assuming it’s more than just the numbers that you need to make that shift. So what advice do you give people about – I can’t think of a better way to say it – how do you get over on them and get them to give in and say “Yes, this is the smarter decision”, even though it scares the crap out of them?
Larry: The first thing I do is I try to change the nature of the conversation to be probabilistic and I know that’s my first talk, but even in the “What? So what? Now what?” they’re related. But when you do that, then the fear just becomes quantified, you know. “I’m afraid that we are going to lose money here if it shows up in this part of the curve.” Well, let’s accept that risk, because the upside of doing that is so much more valuable. So you take people out of it. When people are disagreeing, when people are arguing, it’s about who is right. I try to change the nature of the conversation to be about decision making where what matters is what is right. So, arguing is about WHO is right, decision making is about WHAT is right. So we are just talking about alternatives.
Dave: And how you make an informed decision based on the numbers and stuff that are available. Cool. Awesome.
Larry: Exactly.
Dave: So is there anything here that you are looking forward to seeing? Anything at the conference that you are like, “I’ve got to go see that person!”
Larry: The holy grail of Agile metrics is this metric that nobody has ever been able to do very well that I sort of labeled comically the “build the right thing” metric. There is no “build the right thing” metric, but I have been talking about “build the right thing” metrics ever since I started there because the most value you can deliver is primarily driven by what you chose to deliver. So, there are folks now, there is a company named Pendo who is a sponsor of this conference, and they have a really great start on actually targeting that. They have tools that will look at user behavior and show you the ramp-up curve of that and then allow you to define what the escalation of sophistication is with tool usage, the stickiness of tool usage, and then you can sort of tweak the product to move people along that curve with guides that get them to using the deep sticky layers of the product, to get more value to them. So there are other talks like that – Todd Olson is giving a couple of talks about that here, so I’m really excited about that – build the right thing.
Dave: Agile is a different way of working with all that community collaboration and all that stuff and I love the fact that there is more math and science behind it now because it seems like we’re getting smarter about it. Do you find that people are ever resistant to that, like it seems sort of a not-Agile thing to do to start measuring all this stuff and to try to figure out what the value means? I mean it’s just value.
Larry: I used to get that a lot. When I gave my first talk in 2009 on Agile metrics, there were no other metrics talks. I was afraid I would get thrown out, you know – the baby with the bath water. So now, as Agile scales to larger organizations and larger projects, you just cannot manage without some quantitative feedback. There is no one with enough situation awareness of everything to be able to do it all qualitatively and so there’s much bigger acceptance of it. AgileCraft, the company I am with right now, is all targeting that: visualizations that help people make better decisions about their portfolio and program-level stuff.
Dave: So you have a talk today, on Monday, hopefully, we will have this up, at 2:00 and then on Thursday, you said.
Larry: Thursday, at 3.45. They are both upstairs in National Harbor.
Dave: And if people want to find you, what is the best way for them to do that?
Larry: Probably my Twitter handle – @LMaccherone – and I will be glad to respond.
Dave: Great. Thank you for coming by. It was great seeing you.
Larry: Great to see you again, Dave.
Dave: Good luck with your sessions.
Larry: Good luck for the rest of the conference preview.
Dave: Thank you!
