Abstract: Yahoo Global Homepage is web’s leading homepage destination with over 200 million users and 3.09 billion page views. This paper provides an insight into how homepage achieved an optimized release process to deliver highest quality of production ready code in every 3 weeks.
HOMEPAGE EVOLUTION AND CHALLENGES
Homepage sprints began in mid 2008. Growing in complexity and with a sheer volume of 200 M users, ad-‐hoc change requests started pouring in from over 38 international markets. This resulted in product instability, long sprint cycles of 37 days, increased time to markets and dip in team morale. These were the primary challenges when I took on the program management ownership of Homepage.
Unlike other teams I had been on before, the Yahoo! Homepage team did have very fixed dates -‐ special events for coverage. This required the team to trade off scope and occasionally quality rather than schedule and resources, which were usually fairly rigid constraints. Due to this intense release schedule, the high demands of the product, and the many dependencies on other teams, we revisited our existing release process and challenges associated with it and come up with few optimizations. This paper examines how we optimized Homepage’s release process in a phased approach with high focus on better predictability, change management with minimal influx and higher code quality.
I. REVISIT PLANNING PHASE (FOCUS ON PRIORITIES AND EFFECTIVE PLANNING)
It used to take us 10 working days from features prioritization review to sprint planning.
We introduced the following changes:
- Earlier product backlogs are structured to have a single consistent stack ranking with a business justification and market impact.
- Pre-‐planning in smaller groups (agile feature groups) to unfold internal and external dependencies along with sizing and low level estimates
- Sprint planning meetings are held with an end goal in mind i.e. actual sprint commits which needs to be communicated to external global teams
- Paired feature teams (product, development and QE) to have better context and control on features right from the early cycles of development in order to minimize engineering costs
End result of these optimizations was a significant 70% cut down on time taken from feature prioritization review to sprint planning i.e. from 10 day to 3 days
II. ITERATIONS AND ITERATIONS TO FULLY BAKED FEATURES
Development cycles were entirely linear with a turn around time of 15 days and little early feedback from quality engineering. This was a major blocker to accommodate market requests
Daily check-‐ins on committed features
Automated daily builds for QE to verify and provide early feedback
Integrated daily performance checks to identify issues early in the cycle
Features demonstrated to owners for sign off to ensure there are no gaps in the expected behavior
End result was a significant 60% cut down on iterative cycles i.e. from 15 days to 9 days
III. INTRODUCED POST ITERATION STABILITY PHASES
Earlier development cycles lacked the dedicated stability phase. Given the complexity of homepage, we have created feature stability phase (4 days) and product stability phase (3 days) to ensure better product quality.
Features gaps are fixed on a priority basis to avoid huge backlog defects
Current feature defects and legacy defects are fixed on a priority basis
Framework to identify performance issues early in the cycle (performance checks runs on a daily basis)
Optimized QE regression cycles across OS, browsers, buckets
Future plans to cut down stability cycle by 40 -‐ 50% to be more agile.
IV. OUR BIG DAY – PRODUCTION DEPLOYMENT
Earlier Production deployment used to take us 3 days with silo’ed responsibilities.
We introduced shared management of production deployment across distributed teams using a follow-‐the-‐sun model.
Content Management System production deployment is followed by the back end production pushes a day later
Proposal to push CMS and backend pushes with a 12 hours lag
Flip for simple markets followed by global markets
Production deployment cut down by 50% i.e. from 72 hrs to 36 hrs
V. NEW INITIATIVES – INNOVATION AND BUG WARS!
We have introduced following new initiatives during our sprint stability cycles:
• Innovation day to empower engineers to contribute new product
ideas which goes through a defined process on a monthly basis with an expected outcome of mapping the selected ideas to sprint cycles.
• Bug wars on outstanding defects to have a better control on product
quality once in 3 sprints.
Earlier development models contained overlap across 3 sprints. We have eliminated overlapping significantly. With sprint 50 and beyond, we plan to further optimize our release to make it truly agile as part of Yahoo!'s 2011 goals set by the PLT