Reference Class Forecasting for Digital Agencies
If you’re involved in the delivery of client work and you haven’t read How Big Things Get Done by Prof. Bent Flyvbjerg and Dan Gardner, you should. Flyvbjerg distills his team’s takeaways from their database of over 16,000 big projects across multiple industries (including tech) and countries in an effort to understand how often projects deliver as promised.
“To create a successful project estimate, you must get the anchor right.”
One of the biggest takeaways is that the best way to estimate projects is to “anchor” to previous, similar projects. Flyvbjerg refers to this as “reference class forecasting” – a phrase derived from Daniel Kahneman and Amos Tversky.
Reference Class Forecast is based on the principle that few projects are actually unique. Sure, they’re special in many ways and may actually have unique features, but generally a project can be associated with a class of similar projects (e.g., the Sydney Opera House is distinctive, but it is one of many opera houses).
For a design or technology agency, your reference projects are the projects that your team has completed within a similar set of constraints (e.g., budget, timeline, scope, type of client stakeholders). So if you’re delivering a marketing website for $150k with a Shopify integration and a CMS component, you’d look to similar projects you’ve done in the past. This would allow you to provide a budget range for the client and a sense of what features might be able to be delivered within that range. This is reference class forecasting, and it differs greatly from how many agencies estimate: firing up a spreadsheet, listing all of the deliverables, estimating each one, and then adding some padding.
Reference Class Forecasting is far superior to conventional methods.
“The gap between a conventional forecast and one that uses Reference Class Forecasting varies by project type, but for over half the projects for which we have data, RCF is better by 30 percentage points or more. That’s on average. A 50 percent increase in accuracy is common. Improvements of more than 100 percent are not uncommon.”
Reference Class Forecasting is a competitive advantage for long-standing digital design and tech agencies. They have hundreds of projects to refer back to, if they’ve done a good job of collecting the data around project delivery:
Billed hours
Non-billed hours
Ideally, they have the data broken down by project phase:
Billed and non-billed hours in Discovery & Research
Billed and non-billed hours in Strategy & Planning
Billed and non-billed hours in Ideation & Concepting
Billed and non-billed hours in Design & Prototyping
etc.
Maybe they have this data broken out in terms of junior, mid-level, and senior hours (likely overkill for small and mid-sized agencies).
Young, frisky, upstart firms without a large portfolio to lean on – do not be dismayed. I think very few agencies have this data. So there is actually an opportunity for you to start collecting project-level delivery data and using that to inform your future estimates and SOWs.
It’s hard to overstate just how important this is for agencies that want to:
Reduce the risk of over-estimating; put forth the most competitive proposals possible
Reduce the risk of under-estimating; protect project profit margins by avoiding $0/hr work.
I’d be remiss if I didn’t comment on the impact AI may have here.
First, agencies that are using AI to streamline delivery are going to find their reference projects over-estimating the number of hours required for future work. My guess is that this will impact certain stages of work more than others (e.g., prototyping time could be greatly reduced whereas production-ready design may not be impacted as much).
Second, when all agencies are using all of the same AI tooling, what’s going to differentiate them is the quality of their data. Agencies need to start thinking about how they’re going to provide the best data to feed into these tools.
Again, AI has leveled the playing field for agencies that don’t have hundreds of projects under their belt to reference. Now is the time to put the right systems in place to start collecting project data. It’s just a matter of time before that data is being loaded into AI models that aid in your project estimations. ❦
Gathering project-level data requires a sound plan, buy-in from your team, and consistent follow-through.
If you’d like to get a jumpstart on this…
