There is a specific kind of regression that modern test suites are structurally unable to catch. It looks like this: a team upgrades a dependency, runs the full E2E suite, sees green across the board, and ships. Two weeks later, an internal integration breaks. A data sync script starts failing. A third-party admin extension returns unexpected fields. The API behavior changed — but the UI never showed it.

This isn't a hypothetical. It happens every time a backend adds, removes, or restructures a field that the frontend doesn't render. The browser never sees it, so the E2E test never asserts on it. The test passes. The drift ships.

What "API drift" actually means

API drift is a change in the shape, status, or semantics of an API response that occurs between two versions of an application — even when the user-facing behavior appears identical. It's not a bug in the traditional sense. The endpoint still returns 200. The UI still renders. But the response body is different, and anything downstream that parses that body is now working with an assumption that no longer holds.

Drift falls into a few categories:

  1. Response structure changes — new fields appear, existing fields are removed or renamed.
  2. Status code changes — an endpoint that returned 200 now returns 201, or vice versa.
  3. Endpoint additions or removals — a UI flow triggers a new API call that didn't exist in the baseline.
  4. Value changes — the same field returns a different shape of data.

None of these categories are inherently bugs. Some are intentional schema extensions. Some are side effects of dependency upgrades. The problem is that no one is watching.

A real example: Medusa 2.13.6 → 2.14.0

Medusa is an open-source e-commerce platform. Between versions 2.13.6 and 2.14.0, the admin UI continued to function normally — product categories could be created, listed, and edited. A standard E2E suite exercising the admin panel would pass on both versions.

But when BoxProbe ran the same UI flows and captured the browser-side API traffic, it found a concrete structural change:

POST /admin/product-categories now returns an external_id field that did not exist in 2.13.6. The UI doesn't render it. No E2E test catches it. But any integration that parses that response just encountered an unexpected new field.

This is real. The field was added intentionally — external_id: string | null in the ProductCategoryDTO, documented as "An external identifier for the product category, such as an ID from a third-party system." We confirmed it by diffing the source at packages/core/types/src/product/common.ts across both versions.

Why E2E tests can't catch this

End-to-end tests assert on what the user sees. They click a button, wait for a page to load, and check that an element is visible. It is structurally blind to API-level changes for three reasons:

  1. E2E tests don't inspect response bodies. The browser receives the full API response, but the test framework only sees the DOM the response produces.
  2. E2E assertions are UI-bound. You assert on .product-card being visible, not on the JSON payload that populated it.
  3. Nobody writes API assertions inside E2E flows. In practice, almost nobody does this — it's brittle, verbose, and couples your E2E suite to a contract that changes with every release.

The gap is structural, not a matter of test quality.

What a behavior diff report shows

BoxProbe runs a real UI scenario against two versions of an application — baseline and target. It drives the browser through the same user flow on both, captures every API request and response the browser makes, and pairs them by step.

{
  "step": "create-product-category",
  "endpoint": "POST /admin/product-categories",
  "diff": {
    "type": "response_structure",
    "path": "$.product_category.external_id",
    "baseline": null,
    "target": "null (field added)"
  }
}

Each change is tied to the user action that triggered it — not to a raw API endpoint in isolation, but to the specific step in the flow where the response diverged.

The three audiences who need this

Admin extensions, third-party integrations, data sync scripts, mobile clients, and report generators all parse API responses directly. When a field appears, disappears, or changes type, they break — often silently, often in production, often weeks after the deploy that caused it.

— The integration problem nobody tests for
  1. QA engineers who need release confidence beyond UI pass/fail.
  2. Engineering managers who approve version upgrades and need to know what actually changed.
  3. Integration developers — anyone building on top of an API they don't control.

Moving beyond pass/fail

E2E tests answer a binary question: does the user flow work? Behavior diffs answer a structural question: did the API surface change, and where?

Both questions matter. They are not substitutes for each other. A team that runs E2E tests and behavior diffs has coverage over two different failure modes — UI regressions and API drift — with two different tools optimized for each.

If your E2E suite is green but you still worry about what changed underneath, that worry is well-founded. The answer isn't more UI assertions. It's watching the API layer in the context of the flows that exercise it.