How we cut pipeline time 60% with self-healing CI/CD

When we started instrumenting pipeline runs across our design-partner workspaces, the median build wasn't slow because of compute. It was slow because of retries — a flaky integration test failing on the first attempt, a dependency cache missing by one lockfile hash, a deploy step timing out against a cold container. None of these are hard problems individually. Collectively, they were adding 40-plus minutes to a pipeline that should have taken 12.

Treating flakiness as a first-class signal

Most CI systems treat a failing test as binary: red or green. We started treating it as a distribution. Every test run now writes its outcome, duration, and failure signature to a per-test history, and a test that fails intermittently against an otherwise-unchanged codebase gets automatically quarantined into a parallel "flaky" lane rather than blocking the pipeline outright. Engineers still see the failure — it shows up as a warning with the historical flake rate attached — but it no longer holds up a deploy that has nothing to do with it.

That alone doesn't fix flaky tests. What it does is stop a 4%-flaky integration test from costing the whole team 40 minutes of retries every single day, while surfacing exactly which tests are worth someone's time to actually fix.

Caching that understands your dependency graph

Generic layer caching gets you part of the way, but it breaks down the moment a monorepo has partial changes — a lockfile bump for one package shouldn't invalidate the build cache for twelve unrelated ones. Forge's cache keys are now scoped to the actual dependency subgraph touched by a change, computed from the workspace's build graph rather than a single hash of the whole lockfile. In practice this meant cache hit rate on partial-monorepo changes went from around 35% to 89%.

Auto-bisecting failing deploys

The last piece was the one engineers noticed fastest: when a deploy step fails against a commit range rather than a single commit — common after a batch merge — Forge now automatically bisects the range in a background pipeline and annotates the PR that actually introduced the regression, instead of leaving the on-call engineer to do it by hand at 2 a.m. This runs in parallel with the primary pipeline, so it costs nothing when everything's green.

The result

Across our cohort, median pipeline duration dropped from 19 minutes to roughly 7.5, and the number of pipelines that needed a manual retry fell by 71%. None of these changes required teams to rewrite their tests or their CI config — the self-healing behavior is on by default the moment a repo is connected to Forge.