Engineering

Taming flaky tests before they tame you

Marcus Lee, Principal EngineerJul 20266 min read

A flaky test is worse than no test. It cries wolf until the team learns to ignore the alarm — and then a real failure slips through on the one day it actually matters.

Flakiness almost always traces back to hidden nondeterminism: a race between the test and an async effect, a shared fixture that leaks state between runs, or a clock the test assumes stands still. Name the category before you reach for a retry, because a blind retry just hides the bug for a while longer.

We quarantine before we fix. A flaky test gets tagged, pulled out of the blocking suite, and tracked on a dashboard — so the pipeline stays green and honest while the root cause is hunted down. What you don't do is sprinkle `sleep()` and hope.

The durable fixes are boring: await the actual condition instead of a fixed delay, reset state between tests, freeze time and randomness, and mock the network at the boundary. Each flaky test fixed this way tends to kill a whole family of its siblings.

Measure flake rate like any other reliability metric. When a green build means the code works — every time — the whole team starts trusting CI again, and velocity quietly follows.

Let's build what's next.

Tell us about your product and we'll come back within one business day with a clear path forward — no obligation.

Headquartered in Hong Kong — engineering and delivery across Eastern Asia and Eastern Europe, with US partners as fractional CTOs.

Start a project

Tell us what you're building — no obligation.