Changing engines while the system stayed live

Some infrastructure is small in lines of code and enormous in consequence. Our request relay was one of those pieces: a narrow boundary that every query crossed before reaching the rest of the platform.

We wanted better isolation, predictable retries, and clearer ownership. Replacing it sounded local. In practice, it meant discovering years of assumptions encoded in callers, deployment tooling, metrics, and human habits.

Why replace the relay?

The old component handled routing, authentication context, request shaping, and several historical exceptions. It worked, but every feature added another conditional and every incident required knowledge held by too few people.

Three constraints shaped the project:

• Existing clients had to remain compatible.
• Traffic needed an immediate route back to the old path.
• We needed evidence from production before raising exposure.

Find the parallel boundary

Instead of translating the whole component, we identified the smallest stable contract: normalized request in, normalized response out. Both engines could live behind that boundary while the router decided which one received a request.

Spin up the new path

The first version accepted real requests but returned nothing to users. It executed in shadow mode, compared its normalized result with the old engine, and emitted a structured difference.

export async function shadow(request: CanonicalRequest) {
  const [stable, candidate] = await Promise.all([
    oldRelay.execute(request),
    quarryRelay.execute(request)
  ])

  compare(stable, candidate, {
    ignore: ["traceId", "timing"]
  })
}

The comparison layer ignored values that should differ, such as generated trace identifiers, while preserving ordering and semantic errors. That distinction prevented noisy dashboards from hiding real incompatibility.

What shadow traffic revealed

Most differences were not algorithmic. They came from defaults nobody had documented: empty arrays serialized as absent fields, timeouts inherited from the caller, and error messages that clients had quietly begun to parse.

Build an explicit traffic gate

Once comparisons were quiet, we allowed a small percentage of eligible requests to use the new path. Eligibility was separate from allocation: unsupported shapes never entered the experiment, regardless of percentage.

const eligible = supports(request) && !isPinned(request)

if (!eligible) return oldRelay
if (forceFallback()) return oldRelay

return bucket(request.accountId) < exposure
  ? quarryRelay
  : oldRelay

Allocation was stable by account so one customer did not bounce between behaviors. The emergency fallback lived outside the exposure configuration and could be activated without a deploy.

Treat errors as part of the contract

A migration can match successful responses and still fail users if errors change. We mapped each internal failure to the public categories callers already understood, then preserved safe context for debugging.

function toPublicError(cause: RelayFailure) {
  switch (cause.kind) {
    case "upstream_timeout":
      return unavailable({ retryable: true })
    case "invalid_shape":
      return badRequest({ field: cause.field })
  }
}

Raise exposure with evidence

We reviewed a small set of signals at every step: successful response equivalence, public error rate, tail latency, fallback frequency, and memory per request. Exposure increased only when all of them remained within the agreed range.

What we would repeat

The successful part of this migration was not the final switch. It was the ability to run both paths, describe their differences, and reverse each decision without drama.

For future replacements, we will define the stable contract first, invest in semantic comparison before traffic allocation, and treat the fallback path as a product with an owner and an expiry date.

Changing an engine mid-flight is possible when the aircraft has two engines for a while—and when the cockpit tells you clearly which one is doing the work.