Experience Report

It’s all about Agility – Introducing Portfolio Analysis for Product Ideation

About this Publication

Over the past four years, Acuity Knowledge Partners has mastered the art of continuously delivering successful products as per customer requirement. Portfolio Analysis introduced at Acuity in 2017 has an important role in making this possible by selecting the right product ideas for development. In this report, we will walk you through why and how we embarked on this interesting transformation journey, how we went on to improve the portfolio analysis framework iteratively to make it more impactful for product ideation and ultimately for business outcomes.


Acuity Knowledge Partners is a leading provider of high-value research, analytics and business intelligence to the financial services sector. The company services top financial institutions and consulting companies through a variety of domain led service offerings. We provide service excellence in our offerings by leveraging our understanding of business domains and processes. Four years ago we saw an opportunity to supplement our services with full fledge products (our proprietary technology and automation solutions) to better satisfy our customer business requirements.

Our Agile transformation journey started in 2017 when the company realized that along with Domain led service offerings, we also needed to consider refining our product offerings by utilizing a combination of technology automation and our subject matter expertise. This not only involved building software tools to automate service deliverables but also develop some next gen products we can take to market to win new deals.

When we started this conversation with business units at Acuity, they had a multitude of great value propositions (business ideas) which they wanted developed as products. Our challenge was that with limited technology resources, we were not sure on the right product/product mix that would produce a good ROI.

This is our story of how we created a process for managing our portfolio using fast experiments to better forecast the right product mix across business verticals. This not only helped grow the existing business of Acuity but also played an important role in acquiring new clients.

2.     10 Pounds in a 5 pound sack

Our problem was simple; we had many more great business value propositions than we had the resources to build and realize those value propositions. Acuity has 10 business verticals and there were competing demands from all of them to build products. We had built applications earlier based on specific requirements from customers but we had never undertaken full-fledged software product development. Business teams did not have much experience on what it took to build a software product. They had a multitude of ideas for products based on market need but they were not sure on the right product mix. Also, there was a challenge comparing various products from across different business verticals to arrive at the right product mix.

In past we had developed bits and pieces of software but not at the scale that has the potential to digitally transform a company to having multiple teams work on portfolio of products. We had many questions that needed to be answered before we could undertake our product development journey. Will the products bring the right value to the customer? Are we selecting the right product mix? Will this product have desired ROI? There was no framework available in the company to answer these questions and scientifically test the hypotheses behind them. Therefore, the first challenge was related to prioritization of ideas that could help the business teams to arrive at the right product mix.

As a first step, a strategic team was formed in the organization, to bring the best engineering practices for software development into the organization. The team was named Strategic Initiative team and the suite of products developed by this team is referred to as Business Excellence and Automation Tools (BEAT) suite of solutions. Acuity management believed that Agile was the right approach because of its iterative approach to product development. Furthermore, Acuity management saw the benefit of releasing frequently and seeking regular customer feedback in order to ultimately deliver products as per customer’s expectations.  Acuity’s Chief Digital Officer (CDO) gave Vikash responsibility along with other Scrum Master(s) for introducing Agile to BEAT and the organization. While we have worked on Agile projects earlier, we had little experience of how to filter and prioritize the big business ideas that was coming from the multiple business verticals of Acuity.

Classical Scrum assumes an omniscient product owner and provides little portfolio management guidance.  To begin solving these issues, we held a series of brainstorming sessions amongst Scrum Masters (Vikash leads the Scrum Master Team and Rashmi is part of Scrum Master Team). We were always working under guidance of Acuity’s CDO. After a series of sessions, we arrived at a conclusion that we needed some sort of framework which must be robust enough to be applied to different product ideas using common evaluation parameters (with customer need being core of it). We named this framework as the BEAT Portfolio Analysis framework. Based on the principles of empiricism (i.e. accepting uncertainty and rigorously experimenting and discovery—rather than acting as if one can know and plan for everything upfront), we designed this framework.

Portfolio Analysis became an iterative approach to planning and guiding project processes. In Portfolio Analysis, each section or iteration is reviewed and critiqued by the management team at Acuity, which included representatives of the project’s various stakeholders. Insights gained from the critique of an iteration are used to determine what the next step should be for the product.

3.     OUR FIRST cut at a Porfolio Process

We wanted to build a framework for portfolio analysis with customer at core of it. We had a rough idea and we needed to fine tune the details. As part of Scum Masters team, we held a series of brainstorming sessions with other business stakeholders to come up with 1st iteration of Portfolio analysis framework. We arrived at four key dimensions for our portfolio analysis:

  • Financial Dimension: In what timeframe do we expect the value proposition to start generating sales?
  • Customer Dimension: How sure are we that customers will actually buy the value proposition from us?
  • Operational Dimension: How is value proposition impacting our current operations?
  • Execution Dimension: Do we have the capability to build and deploy the value proposition?

Once we were clear on key dimensions of our portfolio analysis, the next step was easy. We only needed to give it a structure to capture the necessary information across above dimensions. Below is high-level overview of our portfolio analysis framework:

  • The value proposition that business wanted developed. The value proposition is expressed in terms of a problem statement, perceived market size, SWOT analysis, what are the characteristics of a product that a business unit has requested, and validation KPIs/hypotheses. These KPIs are objectives, success criteria against which this hypothesis needs to be validated.
  • Portfolio heat sheet for each value proposition is plotted across the four dimensions such as cost saving vs. potential number of users, importance to current business vs. future business. This helped provide a comparative view of various value propositions for all Business Units.
  • Score for each value proposition across the four dimensions like financial, operational, customer dimensions etc. Each dimension has a set of parameters with scores and weights assigned to each as per their importance from company point of view. Based on these each value proposition across business units is assigned a score and ranked.
  • These ranked value proposition from BUs are presented to company management at beginning of each quarter for their final decision on portfolio mix. After the decision, the BEAT team started working on the development of these products.
  • Ongoing measurement and testing of the value proposition’s hypotheses against the operational objectives/KPIs—These hypotheses are regularly evaluated against various KPIs that were defined during portfolio analysis exercise to understand the performance of these.

We also designed a support system in each business Unit that can help the business leaders in developing their backlog of value propositions for quarterly portfolio analysis. A Digital Advisory team was introduced in each business unit, which consisted of Sales, Delivery team and business domain experts. The Idea was to bring all dimensions on the table while arriving at their value proposition (what customer wants, what industry is asking for and what are the new trends in the industry). Idea was to build a pipeline of business ideas with hypotheses, which can be churned by BEAT team regularly post Portfolio Analysis exercise.

4.     Validating the First Cut

Once we designed the portfolio analysis framework, the next step was to apply it across Acuity business units. We ran the framework through various business units at Acuity. We had joint sessions with Business Unit stakeholders in terms of coaching them on how to work with various sections of the portfolio analysis. Special emphasis was put on defining the KPIs and the underlying hypothesis.  This was an important measurement criteria for the product’s regular evaluation throughout its iterative release journey post portfolio analysis. Once Business Units submitted the portfolio analysis pack, the Scrum Master team worked with business unit team to fine tune the information. This was done to make sure that information was consistent across business units.

Once the list of selected candidate value propositions was finalized across business units, it was presented to Acuity Management for them to make the decision on the BEAT Portfolio for that quarter. Various parameters present in the portfolio analysis helped management arrive at an informed decision for the allocation of resources from BEAT to the selected value propositions (business ideas).

Once development has started, regular checks of KPIs as identified by Business Units for their value proposition were done to see if the results were being delivered as were initially projected as part of their hypothesis. This first cut at portfolio analysis has been run through business units for their value propositions each quarter since 2017.

5.     Lessons Learned from the First cut

During 2nd half of 2019, we realized that we were getting a number of value propositions from BUs and there was need to build in stringent checks and balances much early in the software development life cycle to eliminate chances of failure later. For one of the products that was developed under BEAT, we got the feedback from the customer that we should have developed a particular feature in the product to make it more useful for them. Our feedback cycle was coming too late in the product development cycle. We realized that we should be validating these value propositions from customer much earlier before we undertook a full-fledged product development on selected ideas. We also had couple of more experiences with BEAT products where we felt that validating their value proposition along with broad hypothesis much earlier would better help build products that are much closer to the customer requirement. Finally, there were demands from couple of business units for a PoC prior to full-fledged product development so that it can be validated in the market across multiple customers.

In response to these lessons learned, we refined our portfolio analysis process to run the portfolio analysis exercise twice. We still run the first stage of the portfolio analysis (Stage 1) to prioritize and pick value candidate propositions that have good prospects.  Then we run a small experiment in the form of a Proof of Concept prototype along with the user journey to further validate and refine the value proposition.  These PoC, Prototypes are showcased to various customers for their interim feedback. During this process, we have learned that we get some highly valuable insights from customers about these value propositions. After this second evaluation some value proposition are either refined or are eliminated completely.  Then we again run the portfolio analysis (Stage 2) using the updated value propositions from BUs.

We are still collecting data to determine the ultimate effectiveness of two-stage portfolio analysis process. However, given the additional checks and balances that have been introduced, we are confident that we are developing the products that are much closer to the market and customer requirements.

Some data we have collected suggests that value propositions are being further refined in terms of what needs to be built based on market feedback resulting from the PoC. One business unit had an idea about developing a new report around analysis of a customer. They had their user journey defined for this product. However, during the post experimentation phase, they found that they need to tweak the user journey slightly along with value proposition so that they release the most valuable feature 1st to the customers.

6.      Portfolio Analysis Impact on the organization

We have come a long way from when we started on our agile transformation journey in 2017. We have delivered number of successful products to customers.  Most of these products were analyzed using our Portfolio analysis lens before BEAT team picked these up for development. Below are some of key changes of this journey on the organization and on the software development cycle at Acuity:

  • We initially had waterfall methodology of software development. Business and organization realized the benefit of Agile for delivering incremental products in the hand of customer as per their requirement. Bringing this philosophy to Portfolio analysis played an important role in this transformation by applying this iterative approach to our portfolio analysis. We iterated by refining the hypothesis through frequent experimentation on the concept first with the customer and then taking on full fledge development of the product. This has ensured we are building our products with a focus on customer need. We also kept refining the hypothesis through frequent experimentation with every subsequent version release of product.
  • Initially business units wanted all of their product ideas developed. With our portfolio analysis framework in place, there is now a queuing system in place in the organization classified under different stages of product innovation pipeline (backlog, feasibility study -alpha, under development -beta, in production). This has ensured that business understands the timeline their product may take for development and this has helped business and sales team in effectively engaging with the customer.
  • Before portfolio analysis, we were building products without much experimentation and we often discovered redundancies or gaps in the product from customer perspective late in the product development cycle. With our portfolio analysis process, we have successfully adopted lean startup principles of frequent experimentation and validation of the product ideas and ensure a market ready product delivery. Focus has shifted from outputs to outcomes.
  • Portfolio analysis has played an important role in making BEAT more collaborative. Initially there was an “ask” from a business unit to fully develop their product ideas with a product development team working on very tight schedule. However, using our portfolio analysis process to prioritize product ideas the BEAT team is now accepting work based on the team’s capacity. The team now has enough time to think through product architecture and apply cutting-edge engineering practices while developing these products.
  • We have moved to building SaaS based products rather than developing internal tools meant only for productivity improvement. This is an outcome of frequent validation of product ideas with customer post each stage of portfolio analysis. Based on their interaction with our customers on product ideas, the business and sales teams realized there is huge revenue potential for Acuity in offering SaaS based solutions.

7.     What We Learned

When we started this journey in 2017, we were not sure how to do portfolio analysis. Though all of us wanted to give it a try and we knew that we may not succeed in the 1st step and there is always room for improvement. We used the agile formula of iterative improvement to learn from customer feedback. We did exactly that by learning from our experience, listening to other stakeholders at Acuity and improving the process as we went along on this journey of portfolio analysis. This has been a learning journey, which has helped us in not only developing as agile professionals but also building something, which meets the company requirement on this transformation journey.

One of the key learnings from this experience has been on how to effectively collaborate with the business and stay mutually focused on business outcomes. Initially we always had risk in terms of tech team building some product that may not be relevant from business perspective. With introduction of a refined portfolio analysis process and a new BEAT way of working, we now have close collaboration with business team. This synergy is all through the product development lifecycle, right from product ideation, defining the right hypothesis for frequent experimentation and ultimately building the right product. This has helped in delivering the right set of products under BEAT suite of solutions.

One of the BU stakeholders mentioned that this portfolio analysis is a time consuming process and involves a lot of time for data compilation. We explained the benefit of the process and requested two quarters of his time to highlight the impact of our portfolio analysis process. Once the company observed that products better matched to customer needs, he was convinced on the idea of portfolio analysis.  This had a positive impact on the business and company. The lesson for us was that we should look for a small wins. Consulting and coaching always helps to deliver the desired outcome in long run.

In another instance, there was a suggestion from a couple of BUs to improve the scoring matrix of portfolio analysis to make it more relevant to current business environment. We facilitated a workshop with the BU stakeholders to walk them through the process. We were also open to listening and accepting their ideas to improve the process. We incorporated some of their ideas such as assigning more weight to revenue parameters. It always helps to make all stakeholders part of journey. Acceptance becomes much easier if we incorporate their points of view and concerns in our process. Change becomes more of a pull rather than a push.

One of the stakeholders questioned the need for hypothesis and for refining the hypothesis and product features through frequent experimentation. We explained the benefit of taking frequent market feedbacks and how it will help in improving the outcome of the product. We explained that we may have an important feature in the product roadmap scheduled for later but customer may need that feature in the first release itself. For one of the products, a user management module was planned for later releases. Upon showcasing the initial releases to the customer, there was ask for user management by the customer. Accordingly, the decision was made to advance the user management module release for an upcoming version. We learned something important about customer needs through validation through our hypothesis driven development. Frequent experimentation is very important for a product to be successful in long run.

This transformation journey has been long one and we are still learning. We learnt that it is not easy to bring any kind of transformation in any organization. We faced lot of questions on the process, provided clarifications, and improved the process, multiple follow-ups with business while working on the portfolio analysis. We were always open to inspect and adapt as we proceeded on this journey. In the end, all of us together have embarked on a journey that has already started bringing positive change in the organization and its various businesses.

8.     Acknowledgements

Firstly, we would like to thank David Fellows, Chief Digital Officer at Acuity and our mentor & guide all throughout this Agile Transformation journey at Acuity. Thanks for having confidence in us, giving us opportunity to be part of this transformation journey and always guiding us with your valuable insights on various process improvements all through this journey.

Special thanks to our shepherd Steve Adolph for his tireless efforts in guiding us through this report. This experience report would not have been possible in this shape and form without his valuable inputs, keen insights, guidance and continuously challenging our thought process with right set of questions all throughout the process. Thanks, Steve, we could not have done it without you. It was great experience working with you on this report.

We will like to thank Acuity Management, BEAT Team, Scrum Master Team, Business Unit Owners and other business unit stakeholders at Acuity for their continuous support in making this portfolio analysis and transformation journey a success, thereby ensuring that Acuity is building and delivering right set of products to their customers.

Last, but certainly not least we will like to thank Agile Alliance and Rebecca Wirfs-Brock for giving us opportunity to share our experience.

Add to Bookmarks Remove Bookmark
Add to Bookmarks Remove from Bookmarks
Add to Bookmarks Remove from Bookmarks
XP 2020

Your Bookmarks

No favorites to display. You must have cookies enabled to add bookmarks.

Have a comment? Join the conversation

Related Agile Experience Reports

Discover the many benefits of membership

Your membership enables Agile Alliance to offer a wealth of first-rate resources, present renowned international events, support global community groups, and more — all geared toward helping Agile practitioners reach their full potential and deliver innovative, Agile solutions.

Not yet a member? Sign up now

Member Dues are Increasing on March 1, 2024
Member Dues are Increasing March 1, 2024

Renew your Membership
or Sign-up Now and Save!

Effective March 1, 2024, select membership levels will see a slight increase in dues, a change from our temporary reduction during the COVID-19 pandemic to support our community. Read more about the changes here.