Skip to main content

How we build

The dull work, written down.

Reliability is a set of artefacts somebody has to write. This is the order we work in, what each step produces, and the things we refuse to skip, so you can check our products against it instead of taking our word for it.

Three stages, and what each one produces

The method
01

Decide

Before we build anything we write down what it is for, the ways it can fail, and what we are leaving out. Most of the reliability work happens here. The duplicate post and the half-run migration are cheaper to design out than to debug.

What it produces

A decision record for each choice that matters.

02

Build

Small changes, one ticket per pull request, each reviewed and tested before it merges. Jobs are idempotent and retries are explicit, so a retried post is checked against what already went out. Deploys run from a script that runs the tests first and checks the release, and a release that fails its checks is rolled back to the previous version.

What it produces

A pull request with its tests, and a deploy we can undo with one command.

03

Run

We run what we ship. Its alerts go to whoever built it, and when something breaks we write up what the system allowed and what we changed.

What it produces

Alerts that reach us, and a written review after each incident.

What we write down

If it isn't written down, it's a habit.

Each of these belongs in the product's repository as a plain file. They are what lets us pick a product up months later and still know why it is the way it is.

Decision record — what we chose, what we rejected, and whyPer choice
Runbooks — how each failure shows up, and what to do firstPer failure
Privacy policy — what we hold, where, and for how longPer product
Incident review — what the system allowed, what we changedPer incident

Definition of done

Nothing launches until all six are true.

Diffuso's public launch is held to this list, and so is every product after it. Six things that are either done or not. Diffuso is working through it now, in private beta.

See our products
The runbook exists and someone has followed itRequired
Alerts reach a human who can actRequired
A restore from backup has been testedRequired
The data export works, end to endRequired
Every sub-processor is listedRequired
Rollback is one commandRequired

On blame

The person holding the paddle is the least interesting part of an incident.

Every incident review we write answers three questions: what the system allowed, what it made hard to see, and what we're changing so the same conditions produce a different result. Names appear only where someone deserves credit.

A team that expects blame hides its near-misses, and near-misses are the cheapest information you'll ever get about your own bin of beads.

Try the red bead experiment

The four we don't trade away

01

Reliability first

Uptime and correctness are features, and we plan them like features.

02

EU data storage

We store product data in EU regions. GDPR posture by default.

03

No lock-in

Open formats, exportable data, no hostage-taking.

04

Transparent

Plain pricing, and honest status and changelogs from public launch.

Questions about how we build?

Ask about any of the stages above, or about Diffuso.

office@redbeadstudio.com