Chasing Waterfall(s): Fixed Price Projects in Modern IT
In a recent blog post “Why the *#^&! Do We Tolerate RFPs?”, Bob Morris entertained us with some great education around the RFP process. For consultancies who participate in these, the win rate is fairly low and as Bob states, a company’s policy requires RFPs. Outsiders will often lose to an insider (a vendor already doing business with the client) wasting valuable time and energy on proposals that had no chance. There are other times where a company will simply choose the lowest bidder. In almost all of these occasions, the product is defined loosely around a nebulous set of requirements defined in the RFP.
Reflecting (I do a lot of this in our 2020/2021 Covid world) back to when we were implementing Rational Unified Process (RUP), one of the authors of a book we used for training talked about a dinner conversation he was having with a long-term client. His client asked him how long it would take him to build an accounting system. His reply was “It will take me 5 years”. His client was taken aback and asked him why it would take that long. “Simple. I have no idea what you want beyond that you want an accounting system.” Sadly, many clients request a fixed price model that is based on a requirements not too far removed from the accounting system story.
On the engagement financial side of the house, there are 3 main categories.
Fixed Price
The vendor and client will often use a loose set of requirements, defined up front, to agree on both a timeline and a cost. The total project cost is often divided up to be paid on some interval until the total price is paid.
Time and Materials (T&M) with a Cap
The vendor and client agree on a rate card (billable rate broken down per type and often seniority of staff on a delivery team) and payment/work stops when a predefined amount is exhausted. Another phrase often used is “T&M Not to Exceed”.
Time and Materials (T&M)
In this scenario the vendor and client agree on a rate card (billable rate broken down per type and often seniority of staff on a delivery team) and the client has the ability to stop work at any time, or at any time following a certain calendar date. For budgeting purposes, vendors need an exception to bill above 40 hours.
Fixed Price is Bad for Everyone
Fixed Price projects change the focal point to one where the vendor is incentivized to minimally meet the loosely defined requirements in the contract. The client is equally incentivized to maximize what they get based on the contract. In short, neither party is fully focused on the deliverables but rather on the contract.
For engagements that are awarded to the lowest bidder (this happens more often than you think) the vendor is going to use change orders to make up for the lost margin as a result of the low bid.
Agile Waterfall is Still Waterfall
Companies that gravitate to Fixed Price are ultimately locking themselves into Waterfall. It doesn’t matter if the vendor breaks the project up into Sprints. At the end of the day, the scope is fixed. The vendor is required to fully flush out the requirements up front to manage their own risk. A change in requirements results in a change order and more cost. If the requirements represent the Minimum Viable Product (MVP), that in part is good however, agile teams need time to dial in their velocity. A team needs 6 sprints to dial in their velocity. In other words, it takes 6 sprints for a team to know how long a sized backlog will take to complete. Fixed Price is ultimately tied to a schedule. If a team can’t accurately define completion time until they’ve worked together for 6 sprints, how can a vendor estimate it accurately up front?
The Man Month and Story Points
Another challenge that results from Fixed Price is on the estimation side. Story points are not a representation of hours but a level of effort compared to other stories. They are also closely tied to the team. A 5 point story for one team may be a 3 point or 8 point story for another. Most estimates are broken down by role and on many occasions, an estimate may not include 100% of the time for particular roles. A Software and Systems Architect, for example, may only be 25% allocated to an engagement. This is in part to where and when architects are needed. Usually, they will be available for “Sprint Zero” effort where architecture is defined. The problem here though is that Story Points are tied to the Team, not to individuals. Velocity normalizes with a consistent team. As soon as a team composition changes, velocity must be recalculated.
How Did We Get Here?
Of all the engineering disciplines, Software Engineering is the youngest and unlike others our discipline continues to evolve. Somewhere along the lines, we borrowed from manufacturing to run our projects. In fact, the RFP process we use today has its roots dated back to 1880. This article discusses it in more detail. What’s interesting is that the intent behind an RFP doesn’t really appear to be tied directly to a Fixed Price engagement but rather vendor selection based on what services vendors offer.
If you’ve ever looked at the aerial shots of Microsoft’s data centers, you’ll notice that there’s a lot of concrete. Calculating yards of concrete needed for a pour is relatively simple. Trucks are built to carry a specific amount of concrete. When concrete is poured, forms are used to help the liquid state keep its shape until it’s set. For deeper pours, it’s also necessary to have someone run a tool called a concrete vibrator that helps the concrete settle its aggregate and dispel any trapped air. It’s hard and dirty work but there aren’t many variations other than types of concrete. The largest risk for a concrete job is weather. Most concrete doesn’t set well if it’s too cold and the rainy season can wreak havoc on a concrete finish. Overall though, estimating a job for “pour X yards of concrete” is pretty simple when compared to a set of requirements for a software solution. Hopefully you just learned something about concrete. Regardless, concrete pouring and software delivery is a comparison of apples and aardvarks. Why then do we try to use a manufacturing project management process to drive software delivery?
Managing Scope and Budget a “Better” Way
There’s one possible option that gives the client the opportunity to make sure they’ve selected the right vendor AND gives them a set of deliverables they could take elsewhere with a degree of clarity higher than the RFP would have provided.
Fail Fast
When we talk about project risk, we’re talking about things that may happen that can negatively impact time, scope and/or budget. The classic definition of a project failure usually revolves around the legs of the triangle at the end of a project being larger than they were when the project started. What if instead, we start with the smallest possible triangle and grow it until it’s the constrained goal size? What if we could fail in smaller increments?
Most of us love to find the shortcuts we can take in a project. The reasoning, while flawed, is based on the short-term belief that if we cut out things like user experience design and quality engineering, we are saving money. The reality is that cutting things out that allow us to fail faster (and cheaper) creates longer term negative outcomes across the duration of the project (e.g., lack of test automation).
Failing fast is really about failing predictably. You’ll find elements of “fail fast” in marketing. AB testing for example is an approach whereby users are presented with 2 or more options and measurements such as conversion and bounce rate are used to determine which option is most suitable for the most users. Typically, options for the test aren’t production quality. It isn’t until a winner has been decided that the “real” money is spent. Tools like Adobe Target perform really well in this area.
Use a Fixed Discovery Period to Define Backlog
Depending on the overall size of the engagement, a discovery period may range in size from a couple weeks up. You’ll need a good agile practitioner to conduct user story workshops and other activities. The deliverable should be a prioritized backlog. Based on an agreed upon timeline, the vendor and client both have complete transparency as to the total cost of Discovery.
The vendor and client can review at the end and the determination can be made to continue or to select a new vendor to execute on the backlog.
Dial in Velocity
As a next step, run 6 consecutive Sprints to allow the team to normalize (Forming, Storming, Norming, Performing) their velocity. Scrutinize both the deliverables and the team’s ability to deliver at this point. Size the entire prioritized backlog. This gives both the client and the vendor the opportunity to understand what the cost is likely to be. It’s also a great opportunity to really scrutinize the stories and determine if they are really adding business value. If they aren’t, put them at the bottom. Finally, welcome changing requirements BUT exchange lower priority scope for the equal number of story points. If the lower priority stories are truly needed then accept that there will now be additional sprints and additional cost.
One other aspect of this is the client has a clear picture of cost for a single sprint. If they are unwilling to share their budget numbers, once velocity is dialed in they can determine based upon how much they want to spend where to stop work.
The Final Checkpoints
At this point, both the vendor and the client should have a pretty good picture for what it’s going to take to get to the finish line. The prior milestones (Discovery and Dialing in Velocity) were predictable in terms of cost and schedule. All that remains is determining when “Done” is achieved based on what’s in the backlog. That stuff at the bottom may have seemed important at the beginning of the project but after further scrutiny, it may not be. Review after each sprint and ask the hard question. Are we done yet?
