03

Variability: exploit it, don't kill it

Manufacturing taught a generation of managers that variability is waste. In product development that instinct is wrong. We are paid to create information, and a result you could predict in advance carries none. The real questions are economic: when does variability create value (asymmetric payoffs), when is reducing it cheap (pooling, short horizons, repetition, reuse, buffers), and how can we cut the cost of variability without cutting the variability itself (fast feedback, operating in the linear range, moving it to where it is cheap)?

A sure thing teaches you nothing

V1V4
Development produces information, and the information in a test result is the entropy of its outcome. It is zero when you already know the answer and peaks when pass and fail are equally likely.

Information per test = entropy of its outcome

0.00.51.00%20%40%60%80%100%probability the test failsbits of informationmax at 50%1.00 bits
40 runs of this test
Mostly one color = mostly no surprise = little learned. A test that always passes confirms what you already believed.

Locating 1 bad candidate among 64

Each test checks a subset holding 50% of the remaining candidates (like a bisect step that is lopsided). Finding the culprit requires log₂64 = 6 bits; the number of tests needed is roughly 6 ÷ bits-per-test.

bits / test
1.000
tests (sim)
6.0
avg of 400 searches
tests (6 ÷ H)
6.0
vs 50/50
1.0×
tests needed
p = 50%
1.00 bit
p = 25%
0.81 bit
p = 10%
0.47 bit
p = 2%
0.14 bit

Try: drag to 2% (a “sure thing”) and watch the tests-needed count explode. Then try 98%: a test that almost always fails is equally uninformative.

Asymmetric payoffs: when more variance means more value

V2V3
Expected value is the payoff function integrated against the distribution of outcomes. Product bets look like call options: the downside is capped at what you invest, the upside is not. Under a convex payoff, a wider distribution adds more on the right than it removes on the left.

Payoff function × distribution of outcomes

outcome distribution (scaled)payoff
051015-10-50510technical / market outcomeeconomic payoffkink

Expected payoff as variability grows

expected payoffnet of variability costsymmetric payoff (same upside)
-2.00.02.00246outcome spread σexpected valuenowoptimum σ 4.3
E[payoff] at this σ
0.21
vs near-zero σ
+0.41
variability creates value
net of cost
0.03
optimum σ (net)
4.35

Try: with the option payoff, widen σ and watch expected value rise even though the average outcome never moves. Switch to “symmetric”: the curve goes flat. Switch to “concave”: variability now destroys value, which is why factories fight it. Raise the cost of variability to see a finite optimum (V3).

The factory's payoff function is concave: a part exactly to spec earns full value, any deviation is scrap or rework. That is why Six Sigma works there. Development payoffs are usually convex. Same mathematics, opposite conclusion. Don't import variance-reduction goals without checking the shape of your payoff.

Pooling and small experiments

V5V7
Variability is not always valuable; when it is not, the cheapest way to reduce it is statistical. The σ of a sum of N independent tasks grows as √N while the mean grows as N, so relative variation falls as 1/√N. The same arithmetic says many small experiments are less risky than one big one.

Short planning horizons

V6
Forecast error compounds: not only does the market move, the direction it moves in drifts too. Committing over a short horizon is the most effective variance reduction available, and it costs almost nothing.

14 possible futures vs. the plan made today

committed5010015020005101520months aheadwhat the market needs (index)
━ plan┅ ±1σ forecast error┅ ±2σ

Forecast error (1σ) at the commitment horizon

1 month
0.5
3 months ◂
1.9
6 months
4.8
12 months
12.7
24 months
35.0

Error grows like horizon1.5: the trend you extrapolate is itself wrong, and that error compounds. Doubling the horizon nearly triples the error (21.5 ≈ 2.8×).

error at 3 mo
1.9
vs 1-month plan
3.7×
24-month plan vs yours
18.7×
worse
horizon ratio
8.0×
error ratio 18.7×

Try: slide the horizon from 24 down to 3 months. A 8× shorter commitment cuts forecast error by ~19×. That is the case for short release cycles and rolling-wave plans.

Repetition, reuse and negative covariance

V8V9V10
Three more structural ways to reduce variability: do the same thing often, use what is already proven, and add a mechanism that moves against the disturbance.

Time to do the same job, by repetition (e.g. a release)

mean + 1σmeanmean − 1σ
0510152010203040repetition #days
σ on 1st
4.0 d
σ on 40th
0.8 d
spread reduction
5.3×
CV 1st → 40th
0.40 → 0.13

Repetition turns an event into a routine: checklists, scripts and muscle memory remove the causes of variation. A team that deploys daily has a predictable deploy; a team that deploys twice a year does not. Small batches (next chapter) give you the repetitions.

Buffers trade money for certainty

V11
A buffer does not reduce the variability of the work; it reduces the variability of the outcome you promise, and you pay for it every time. The right size is an economic decision: add buffer until one more day costs as much as the expected lateness it prevents.

Completion time (expected 20 days) vs the date you promise

0d10d20d30d40d50d60dexpectedpromise: day 25.0
red bars = outcomes that miss the promised date

Expected cost vs buffer size

totalbuffer (certain)lateness (expected)
$0$50k$100k$150k0102030buffer (days)expected costnowoptimum 6d
P(on time)
80%
expected cost
$58k
buffer $25k + late $33k
optimal buffer
6 d
P(late) = 17% = buffer$/late$
cost vs optimum
+$507

A buffer is paid for in full every time; it converts uncertain lateness into certain lateness. Try: raise the cost of being late and the optimal buffer grows; reduce task CV and the same reliability needs far less buffer (reducing variability and buffering are substitutes).

Reduce the consequence, not the variability

V12
Cost of variability = how far you drift × how long you stay there. You often cannot stop the drift, but you can shorten how long it goes unnoticed. Three projects below receive the identical random disturbances; only their feedback loops differ.

Three projects hit by identical disturbances · tick 0

fast feedbackcheck every 3loss0max |dev| 0.0slow feedbackcheck every 30loss0max |dev| 0.0no feedback until the endnever correctedloss0max |dev| 0.0
line = deviation from the right target (market need, spec, architecture). green band = tolerance. ticks on the bottom edge = feedback checks; a check outside tolerance resets the deviation to 0. Loss = deviation accumulated over time + 0.4 per check.
fast loss
0
0 spent on checks
slow loss
0
no feedback
0
same disturbances
yes
variability is identical; only its consequence differs

Press play. All three projects suffer exactly the same random drift: nobody reduced variability. Fast feedback cuts the loss by truncating each excursion before it grows. Try a 1-tick check interval: past a point, the cost of checking itself dominates.

Stay in the linear range

V13
On a nonlinear response curve the average outcome is worse than the outcome at the average. Queues near full utilization are the classic case in development.

Queue time multiplier 1/(1−ρ) under fluctuating load

load range01020300%20%40%60%80%100%utilization ρcycle time × service timeat average 5.0×average of outcomes 5.8×
at average load
5.0×
average actual
5.8×
penalty here
+16%
same swing at 50%
+2%
near-linear region

On a curved response, symmetric swings do asymmetric damage: the busy weeks cost far more than the quiet weeks save (Jensen's inequality). The fix is not only to reduce the swing but to move the operating point to where the curve is nearly straight, e.g. plan a development team at 70% load, not 95%.

Try: keep the swing at ±12 points and slide utilization from 50% to 90%. Same variability, very different consequence.

Iteration speed beats defect rate

V15
If each loop removes a fraction of the remaining problems, the time to reach a quality target is (number of iterations) × (cycle time). The two levers are not symmetric.

Time to reach the quality target = iterations × cycle time

cycle 10dcycle 5d (2× faster iteration)
01002003000%20%40%60%80%fraction of problems surviving each iterationdays to targetnowhalve defectshalve cycle
iterations now
5
50 days
halve defect rate
30 d
40% faster
halve cycle time
25 d
always 50% faster
better lever
iteration speed

Each iteration (build → test → fix) removes a fraction of the remaining problems, so iterations needed = log(target) ÷ log(survival). Halving cycle time always halves total time; halving an already-low defect rate saves only part of one iteration, and is usually far harder to achieve than a faster build or test loop.

Try: start at 20% surviving and compare the two levers. Then push survival to 60%: only for a really sloppy process does halving defects compete with halving the loop time.

Substitute and displace: put variability where it is cheap

V14V16
Variability that cannot be removed can still be moved: onto a cheaper variable (budget or minor scope instead of the launch date) or to a cheaper place in the process (the unstarted backlog instead of half-finished work).
expected slip to absorb: 1.60 wk (E[max(0,X)] = 0.4σ)

V14 · Which variable absorbs the slip?

slip schedule
$96k
7.5×
surge budget
$32k
2.5×
defer scope
$13k
cheapest

The variability does not disappear; you choose where it shows up. If a week of delay costs $60k and a week of contractors costs $20k, let the budget vary to keep the date fixed.

V16 · Where in the flow is the wait absorbed?

Unstarted backlog
$3k
In design
$24k
In integration
$72k
At launch
$144k

Same delay to market in every case; what differs is what sits idle while you wait. Air traffic control holds planes at the departure gate instead of letting them circle the destination. Hold variable work in the backlog (gate the start) rather than as half-done work.

cheapest lever
scope
saved vs slipping date
$83k
absorb in integration
$72k
vs absorbing at launch
2× cheaper
  • takeawayVariability is an economic quantity, not a defect. With asymmetric payoffs, more of it raises expected value; at a 50% failure rate, a test produces the most information.
  • takeawayWhen you do want less of it, use the cheap structural levers first: pool independent risks, commit over short horizons, run many small experiments, repeat, reuse, and size buffers economically.
  • takeawayUsually the best lever is the consequence: fast feedback truncates losses, a linear operating range keeps swings cheap, and remaining variability belongs in the cheapest variable and the cheapest stage.

All 16 principles of variability

book chapter 4
  1. V1
    The Principle of Beneficial Variability

    Variability can create economic value, so it should not be eliminated by reflex.

    A design project whose outcome is certain teaches you nothing; the value of development comes from uncertain experiments that might succeed. Manufacturing-style variance reduction (Six Sigma on every task) suppresses the very innovation you are paying for.

    ↑ see it in the visualization
  2. V2
    The Principle of Asymmetric Payoffs

    When upside gains are larger than downside losses, more variability raises expected value.

    A product bet usually loses at most its investment but can win many times that, like a call option. Under such a convex payoff, widening the spread of outcomes adds more to the upside tail than it costs on the downside.

    ↑ see it in the visualization
  3. V3
    The Principle of Optimum Variability

    The right amount of variability is an economic optimum, neither the minimum nor the maximum.

    Generating variability (radical designs, more risky experiments) has costs that grow faster than its benefits eventually. Find the point where the marginal value of extra spread equals its marginal cost instead of minimizing or maximizing it.

    ↑ see it in the visualization
  4. V4
    The Principle of Optimum Failure Rate

    Tests generate the most information when their chance of failure is about 50%.

    Information per test is the entropy of its outcome, which peaks at a 50/50 pass/fail probability. A test suite that always passes, or an experiment you are sure will work, produces almost no information; design experiments that genuinely might fail.

    ↑ see it in the visualization
  5. V5
    The Principle of Variability Pooling

    Combining many uncorrelated sources of variation reduces relative variation.

    The standard deviation of a sum of N independent tasks grows only with √N while the mean grows with N, so the coefficient of variation shrinks by √N. Hold one schedule reserve for the whole project instead of padding every task.

    ↑ see it in the visualization
  6. V6
    The Principle of Short-Term Forecasting

    Forecasts become dramatically more accurate as the time horizon shrinks.

    Errors compound: market needs and technology drift, and forecast error grows faster than linearly with horizon. Shorter release cycles and rolling plans let you commit only over horizons you can actually predict.

    ↑ see it in the visualization
  7. V7
    The Principle of Small Experiments

    Many small experiments produce less total variation than one big experiment of the same size.

    Splitting one large bet into many independent smaller ones lets their outcomes pool, so the total result is far more predictable. Ten two-week experiments are less risky than one twenty-week program, even at the same total spend.

    ↑ see it in the visualization
  8. V8
    The Repetition Principle

    Doing something repeatedly reduces its variation.

    Repetition builds routine, tooling and learning, so the spread of task times narrows. Frequent small releases make deployment a well-practiced, predictable operation instead of a twice-a-year adventure.

    ↑ see it in the visualization
  9. V9
    The Reuse Principle

    Reusing proven designs reduces variability.

    A reused component has known behavior, cost and schedule; only the new parts contribute large uncertainty. Reuse where novelty adds no value, and spend your variability budget where it creates differentiation.

    ↑ see it in the visualization
  10. V10
    The Principle of Negative Covariance

    You can reduce variance by introducing a counterbalancing effect that moves opposite to the disturbance.

    Variance of a sum includes twice the covariance, so a negatively correlated term cancels swings. Cross-trained engineers who move to whichever stream is behind, or scope that shrinks when the schedule slips, create negative covariance.

    ↑ see it in the visualization
  11. V11
    The Buffer Principle

    Buffers buy reduced variability of outcome, at the price of time or money.

    A schedule buffer turns an uncertain finish into a later but more certain one; you pay its cost every time, even when it is not needed. Size it economically: add buffer until the cost of one more day equals the expected lateness cost it avoids.

    ↑ see it in the visualization
  12. V12
    The Principle of Variability Consequence

    Reducing the consequences of variability is usually a better lever than reducing variability itself.

    The cost of variability is the product of how often deviations happen and how much they cost. Fast feedback that catches a drifting design early truncates the loss tail at a fraction of the cost of preventing every deviation.

    ↑ see it in the visualization
  13. V13
    The Nonlinearity Principle

    Operate in the linear range of system performance, where variation has small consequences.

    In a nonlinear region (a queue near 100% utilization, a structure near its limit) symmetric swings produce asymmetric damage, so average performance is worse than performance at the average. Back away from the knee of the curve.

    ↑ see it in the visualization
  14. V14
    The Principle of Variability Substitution

    Accept variability in something cheap to protect something expensive.

    When schedule slips are costly, let budget or low-value scope flex instead: paying for overtime or dropping a minor feature is cheaper than missing a market window with a high cost of delay.

    ↑ see it in the visualization
  15. V15
    The Principle of Iteration Speed

    Improving iteration speed usually pays more than improving the defect rate.

    Time to reach quality = iterations × cycle time. Halving cycle time always halves the total, while halving an already-low defect rate removes only a fraction of an iteration. Invest in fast builds and fast tests before perfect first passes.

    ↑ see it in the visualization
  16. V16
    The Principle of Variability Displacement

    Move variability to the stage of the process where it costs the least.

    Air traffic control holds flights on the ground, not in the air, because waiting at the gate is far cheaper than circling. In development, absorb variation in the unstarted backlog rather than as half-finished work or a slipped launch.

    ↑ see it in the visualization