1'500 tickets in your client's project management tool. A fixed deadline. And not enough resources. You know this will suck. But you create a project proposal, anyway. Your client knows that you will push the deadline back. But they agree anyway, because they don't trust any other process.
Why do clients operate this way? Why do they want a Big Design Up Front (BDUF)? And how do you get out of this treadmill?
TLDR: Use an iterative approach, hold your client's hand, and step by step, build up trust.
BDUF means to gather all requirements upfront and then model and design everything according to those requirements. We call these phases:
BDUF usually comes with fixed price projects. Fixed Price means: fixed time + fixed scope + fixed budget (the "iron triangle"). BDUF is a relic of waterfall project management
Waterfall relies on a sequence of pre-defined steps, where you must complete every previous step before starting the next phase. Typical phases are: Requirement Gathering and Documentation System Design, Implementation, Testing, Delivery/Deployment, and Maintenance.
Waterfall can work for small teams with predictable projects. But for larger teams and more complex projects waterfall results in failed projects, higher costs, and unused functionality.
BDUF builds on the assumption that we can gather all requirements up front. By this reasoning, we can create a plan that provides the most efficient way to reach the project goals. Another assumption of BDUF is that the gathered requirements form an effective product that solves the user's problems.
The reasons some customers demand BDUF are:
In complex projects, it is impossible (or at least incredibly expensive) to gather all requirements up front.
Note: when it is a small project, and you have ideally done similar projects before, waterfall (with BDUF) can be a pleasant process.
Without thorough user testing, you can't know what users need. You also won't know if the proposed solution covers all user needs optimally.
Humans are awful at estimating. 21% of waterfall projects fail, and 53% of projects go either over time, over budget or miss planned functionality (Chaos Study 2019). This shows that most estimates are wrong somewhere.
Correct estimates heavily depend on the team. Doing something with a team of experienced engineers is faster than with two junior developers right out of college. Teams also constantly change. Your team learns all the time. Some people leave, and additional people join.
Gathering all requirements and estimating precisely is a tremendous waste of time. A rough estimate about the price range of a project will take a few hours or days. Making the estimate more precise takes much longer: weeks to months (see: law of diminishing returns).
Stakeholders are most often not interested in a project 100% to spec but in the ROI of the project. The ROI of a project is a balance of engineering cost, time to market, and perceived value to the user.
Most of the time clients don't know what they need. And they can't articulate what they want.
Just like it is costly to gather all requirements up front, it is also incredibly expensive to fix all errors and flaws during design. "If the cost of planning is greater than the cost of fixing then time spent planning is wasted" - Wikipedia on BDUF. We also call this phenomenon analysis paralysis.
When you want to turn a waterfall project into an iterative project, these are your goals:
Start by finding out why your customer wants a BDUF project.
When you know the why, highlight the dangers of waterfall for their project:
Then, explain the advantages of iterative projects (you don't even need to call them "agile"):
Finally, propose them an alternative:
Offer them a smaller Proof of Concept (answering: "is it feasible?") or Minimum Loveable Product (answering: "is it valuable?") version based on what you know. Make it small enough that you can estimate it. In this smaller project, work iteratively and step-by-step show your client how agile software development is beneficial to them.
Your other alternative is to offer them a longer project with a defined vision and some high-level goals. Write the iterative process that you want to follow and how they can always prioritise what they find important and can directly influence the development through the iterative process. Write that they will get a predictable piece of working software after each iteration that they buy. Define what happens when you don't meet the iteration's goals. Help them find out what is valuable to prioritise. Contradict your client when you know a better way.
And over time, through good and working software, predictable results and instant feedback, the client will learn to trust you.
Hold them by the hand and step by step let them build trust with you and your iterative approach.
You will first build trust with your direct contact persons in the client's organisation. Then the light of your project will shine over to other departments. Depending on your client, this takes time. Sometimes only a few iterations, sometimes months or years. But it pays off. It is so incredibly rewarding to hear from your client that others in the organisation notice how well your project is going.
A colleague of mine said the best project she ever had was when the manager that hired her as a freelancer was not interested in the project itself. He just said: "this is our budget and here are the people who need a solution". She had time to understand the users and build a solution that optimally supported their workflow. After she finished the project, the manager in question got praise for such an outstanding success.
The answer to BDUF is "Just Enough Design Up Front" - Enough design to do your next few backlog items.