How a Persistence Fix Skipped Mastra's Output Processors After Provider Failure

Technical Replay | Open-source historical replay

For the concise evidence summary, view the Mastra output processor Regression Case File.

A model provider threw an error. Mastra still emitted the terminal error state, but a second failure followed. The framework did not run the final output-processor pass. Any configured logic waiting there, including moderation, transformation, structured-output, channel, or persistence handling, could be skipped.

This behavior appeared in @mastra/core 1.56.0. Version 1.55.0 ran the final processor after the same provider failure. Version 1.59.0 restored it.

We selected this known public regression and ran an independent targeted historical replay of a bounded 30-commit segment inside the 1.55.0 to 1.56.0 release window. Under Early’s benchmark controls, Regression Guard received the configured repository comparison. It did not receive the public issue, the later fix, the expected answer, or external web access. Early is not affiliated with Mastra.

Early identified the regression on the exact changed file. It traced the problem to a status guard around the shared output-processor pipeline and classified the finding as Regression. Published package behavior and Mastra’s later fix independently confirmed the mechanism.

The Failure Inside the Failure

The original provider error was not the regression. The provider had already failed, and the stream exposed that error. The added regression was the loss of final handling after the failure.

Mastra’s public issue #21292 described a concrete contract change. A configured processOutputResult handler ran in version 1.55.0 but stopped running from 1.56.0 when the model provider threw. The stream produced the same error evidence, yet consumers no longer received the final processing signal.

That distinction matters. Applications can deliberately place important work at the end of a stream. A provider error should not silently erase unrelated terminal handling unless the framework defines that behavior explicitly.

The Fix Had a Valid Goal

The regression came from PR #19716. Its goal was reasonable. When generation failed before producing assistant output, Mastra needed to avoid persisting an orphaned user input as if the turn had completed normally.

The change preserved the failed stream status and added a guard around runOutputProcessors. If the stream was failed, the shared output-processor pass would not run.

The introducing change placed the condition around the shared call:

if (self.#status !== 'failed' && self.#status !== 'canceled') {
  self.messageList = await self.processorRunner.runOutputProcessors(...)
}

That prevented the memory-specific write, but the guard lived above the individual processors. It did not only stop MessageHistory. It stopped every configured final processor attached to that lifecycle stage.

The Guard Was Too Broad

The architectural mistake was placement. A rule about whether one persistence processor should save an input-only failed turn became a rule about whether the entire processor pipeline should execute.

The persistence rule was enforced at the shared pipeline boundary instead of inside the processor that owned persistence.

The code change was small. Its behavioral reach was not. Custom processors could perform transformations, moderation, structured-output extraction, channel handling, or persistence work. The guard removed all of those opportunities on failed streams.

This is the kind of regression that a local reading can miss. The changed condition looks aligned with the immediate bug. The risk appears only when the reviewer follows the guarded call into every downstream consumer that depends on it.

What Early Found

The Early finding was titled “Output processors skipped when status failed/canceled.” It pointed to packages/core/src/stream/base/output.ts, the exact file changed by the introducing PR.

The finding explained that the terminal processor pass had previously run on finish, but the new status guard skipped it when the stream was failed or canceled. It identified the downstream processor and memory effects and proposed verifying the changed contract with a mutating processor and memory configuration.

Across the complete replay, independent review confirmed this target as one Regression, rejected two other Regression rows as false positives, and left one candidate unresolved. Ten findings were classified as Expected. The detailed accounting appears in Verification and Limits.

The actual Early finding from the September 2 targeted historical replay. Mobile uses a focused rendering so the evidence remains readable.

The product procedure covered failed and canceled states, which was broader than the benchmark’s exact trigger. The independent reproduction closed that gap for the narrow claim in this case: an unrecovered provider throw followed by the final processOutputResult pass.

The Three-Version Proof

The reproduction used exact published packages and a local model that always threw an HTTP 503 shaped error. It needed no provider credentials and made no provider request. The public issue includes the complete probe. Its essential mechanism was a throwing model plus a processor that recorded whether processOutputResult ran:

const throwingModel = {
  specificationVersion: "v2",
  provider: "probe",
  modelId: "throws-503",
  supportedUrls: {},
  async doStream() {
    const error = new Error("probe request failed with HTTP 503")
    error.statusCode = 503
    throw error
  },
}

let outputPassRan = false
const marker = {
  id: "probe-marker",
  processOutputResult: ({ messages }) => {
    outputPassRan = true
    return messages
  },
}

The probe attached marker as an output processor, drained fullStream, and reported outputPassRan. Readers can inspect and run the full credential-free reproduction in Mastra issue #21292.

Three executions per version produced the same result:

Each execution produced five stream chunks and one error chunk. Only the processor count changed.

@mastra/core	Processor calls	Result
1.55.0	1	Last good
1.56.0	0	First bad
1.59.0	1	First fixed

The provider failed in every version. Only the final output-processor invocation changed.

The public issue appeared 5 days and 13 hours after version 1.56.0 was released. Version 1.59.0 restored the behavior 10 days and 7 hours after the first affected release. The released window covered versions 1.56.0 through 1.58.0.

Why CI Stayed Green

The captured GitHub record for the introducing merge showed 105 check runs. Ninety-two succeeded and thirteen were skipped. None failed. These were check runs, not 105 behavioral tests. The pull request recorded no human approval, but that is context about the review path rather than evidence of why the regression escaped.

The useful lesson is not that automated checks are ineffective. The change included a test for the original memory problem. That test asserted that an input-only failed turn was not persisted. It did not also configure an independent processor and assert that the shared final pass still ran.

The missing protection was a cross-processor contract test. The test suite covered the local goal but not the neighboring behavior changed by the same guard.

What the Fix Changed

Mastra’s PR #21370 separated the two responsibilities.

Failed streams once again ran the shared output-processor pipeline with an error finish reason. The narrower persistence exception moved into MessageHistory, which could decide not to save an input-only failed turn while leaving unrelated processors intact.

The corrective change removed failed from the shared guard:

if (self.#status !== 'canceled') {
  self.messageList = await self.processorRunner.runOutputProcessors(...)
}

It then enforced the persistence rule inside MessageHistory, where the rule belonged:

if (result?.finishReason === 'error' && newOutput.length === 0) {
  return messageList
}

The fix also added the missing tests. One confirmed that a configured processor ran when the model threw. Additional memory tests distinguished an input-only failure from a failed turn that had already produced partial output.

The later fix is independent confirmation. It was not input to Early’s analysis.

Verification and Limits

The replay returned 14 findings. Early classified ten as Expected and four as Regression. Independent review confirmed one Regression finding as the known target, rejected two Regression rows as false positives, and excluded one unresolved candidate from confirmed defect counts.

This disclosure is part of the result. Early found one confirmed regression in this case, not four.

The reproduction proves that the final processor invocation ran in 1.55.0, disappeared in 1.56.0, and returned in 1.59.0 after an unrecovered provider throw. It does not establish behavior for cancellation, recoverable errors, onFinish, every processor implementation, or a real storage backend.

The replay occurred after the incident, and researchers deliberately selected a window containing a known regression. It does not establish a universal detection rate, claim that Early prevented the release, or show that every regression will be found.

The narrower conclusion is supported. In a targeted historical replay, Early identified the exact changed file, the broad guard, the skipped shared processor pass, and the resulting behavioral regression. Independent package execution and the upstream fix matched that finding.

Sources and Method

Primary public sources are the upstream issue, the introducing pull request, and the corrective pull request.

Human verification compared the stored Early finding with the release code, the public history, and a reproduction against published npm packages. The replay repository remains private and license-filtered. Runtime claims come from the published packages rather than a build of that private repository.