A Pydantic AI Regression Inside a 47-File Fix
A targeted historical replay traced incomplete streaming responses to one drain loop removed during a 47-file cleanup.
Regression Case File | Open-source historical replay
One removed drain loop in Pydantic AI v1.92.0 could leave an agent run with an incomplete or empty response. Tool calls could disappear. Token usage could read zero. With some providers, the caller received less than the model produced without an error explaining why.
The affected path was specific. A caller had to supply an event_stream_handler, then return from that handler before consuming the full stream. The default path without a handler still drained the stream correctly.
Pydantic AI is the Pydantic team’s Python framework for building AI agents. In a framework like this, the returned response is the record of what the agent did. Losing the end of that response changes the result of the run.
We later selected the release that introduced the regression and ran a targeted historical replay. Regression Guard analyzed the change from v1.91.0 to v1.92.0. It produced six findings, classified one as a Regression, and that finding described the real defect.
Early did not participate in the original release. Pydantic did not sponsor or participate in the replay, and this case does not present Pydantic as an Early customer.
The Failure
The public report, Pydantic AI issue #5769, documented incomplete and often empty results when a streaming handler did not consume every event. The report covered versions v1.92.0 through v1.105.0 and described the behavior as provider-agnostic.
The visible symptom varied. A real provider could return a truncated result with missing tool calls and under-reported usage. In our offline reproduction, the empty response triggered Pydantic AI’s retry logic and ended with UnexpectedModelBehavior after the output retries were exhausted.
Both outcomes came from the same state: the stream ended from the caller’s perspective before the complete ModelResponse had been assembled.
The Timeline
The culprit merged on May 8, 2026 and became the v1.92.0 tag commit. The release was published 21 minutes later. A user reported the problem on June 3, 26 days after release. The fix merged the next day and shipped in v1.106.0 on June 5.
That gives the case three different measurements, and each means something different:
- 26 days from release to the public report
- 28 days from release to the fixed release
- 14 affected releases, v1.92.0 through v1.105.0
The release timeline separates report timing, fixed-release timing, and the fourteen affected releases.
The Culprit Change
The change came from PR #5313, “Clean up streaming responses on cancellation”. It was a legitimate fix for cancellation behavior, spread across 47 files. The culprit commit passed all 78 of the project’s executed CI checks.
Inside that cleanup, one loop was removed from the normal completion path. Its job was to keep consuming events until the stream was drained. Removing it changed completion behavior when a supplied handler returned early.
The sequence was straightforward:
- The model produced a stream of events.
- The supplied handler consumed only part of that stream.
- The handler returned.
- The remaining events were no longer drained.
- The stored response remained incomplete.
- The partial response became the result used by the caller.
The behavior changed when a handler returned before consuming the full stream.
The same contributor later opened PR #5771. The fix restored draining after the handler returned and added 92 lines to tests/test_streaming.py. This is the useful part of the case. A change can solve the problem named in its pull request and still alter an adjacent behavioral contract.
How We Evaluated It
This was a targeted replay, not a blind trial. We knew which historical release contained the regression before selecting the comparison. Regression Guard did not receive that expected answer.
The run compared the v1.91.0 source tag with the v1.92.0 destination tag. Nothing in the project configuration mentioned streaming, event_stream_handler, usage accounting, or incomplete responses.
Per Early’s input scoping and access controls, the analysis could read only the repository contents between those tags and had no web, GitHub, retrieval, or other external access.
There is an important methodology detail. The repository copy contained later history on main, including the eventual fix. That history was outside the run’s base-to-target input scope. The claim that the fix was unavailable rests on Early’s documented input controls and is consistent with the run metadata and the finding, which cited only release-time code.
What Regression Guard Found
The comparison contained six commits. Regression Guard returned six findings:
| Verdict | Count | Review result |
|---|---|---|
| Regression | 1 | The real streaming defect |
| Expected | 5 | Classified as expected changes |
The Regression finding appeared first and was the only finding carrying a confidence score. There were no other Regression findings to sort through. We independently verified that its reasoning matched the later upstream evidence.
The finding identified the exact deleted drain loop, distinguished the normal path from the exception path, and traced a reachable caller through event_stream_handler. It also explained how an unconsumed stream could leave get() and usage() reading a truncated ModelResponse.
The user-visible finding was:
Regression Guard finding
“Removed forced drain truncates usage on early exit”
Regression | High severity | Confidence 8/10
The finding as it appeared in Early:
The actual Regression Guard finding from the Early product.
It did not identify the originating pull request or connect the result to the later public issue. Those links were added during human verification. We also did not independently verify every Expected classification, so the precision statement here is deliberately narrow: the run produced one Regression finding, and that one finding was a true positive.
Verification and Limits
Reproduction Results
Regression Guard’s finding included a verification procedure: run the agent once normally, run it again with a handler that stops after the first event, then compare the output and token usage.
We executed an offline version with Pydantic AI’s deterministic TestModel. It needed no API key and ran against the published PyPI packages.
| Version | Result | Observed behavior |
|---|---|---|
| v1.91.0 | Pass | Output and usage matched |
| v1.92.0 | Fail | Empty response triggered retry exhaustion |
| v1.106.0 | Pass | Output and usage matched |
The failing run followed the same handler and graph path named in the finding. The result also matched the later upstream report and the fix that restored stream draining.
This reproduction does not prove that the project’s checks were weak. It establishes a narrower fact. The culprit commit passed 78 executed CI checks, while a small test of the early-returning handler condition failed on the released package.
Honest Limits
The replay was conducted after the incident, and the release window was selected because it contained a known regression. That makes it a useful case study, not an unbiased estimate of detection performance across arbitrary releases.
Regression Guard’s finding also led with truncated usage accounting. The more important customer-visible symptom was an incomplete or empty response. Its mechanism was correct, but its headline undersold the impact. The finding also needed the explicit qualification that the default path without a handler still drained the stream.
The result supports one precise conclusion: in this targeted historical replay, Regression Guard identified the defective behavior from the selected release context, and its proposed verification failed on the affected version and passed before and after it. It does not establish that every regression will be detected or that Early prevented this release.
What This Case Shows
The original cleanup addressed a real cancellation problem. The regression sat beside that purpose, in the completion behavior that followed a handler’s early return. Review and CI evaluated the change without a test for that exact condition.
Regression Guard adds a different question to release review: which existing behaviors changed when the release candidate moved away from its production baseline? In this case, that question surfaced one removed loop inside a 47-file cleanup and gave the team a concrete experiment to run.
The strongest part of the result is not the label. It is the chain of evidence. The code-level finding, the offline reproduction, and the later upstream issue and fix all point to the same mechanism.
Reproduce the Failure and Verify the Finding
The public earlyai/test-pydantic-ai repository contains the release history, Early-generated investigation prompt, and exact offline reproduction used for this case.
With the repository, you can:
- Inspect the code and Git history between v1.91.0 and v1.92.0.
- Give Claude Code or Codex the supplied Early-generated prompt and require it to verify, refute, or correct the finding with file, line, commit, and tag evidence.
- Run the offline script with v1.91.0, v1.92.0, and v1.106.0 to reproduce the pass/fail/pass sequence without a model API key.
This independently verifies Early’s finding and reproduces the defect and fix. It does not rerun Regression Guard or recreate its proprietary detection process. Read the technical replay guide for setup instructions, expected results, and primary-source links.
Sources and Method
The primary public sources are issue #5769, the introducing release comparison, the introducing pull request, the fixing pull request, the fixed release comparison, and the follow-up cancellation-semantics tracker.
The historical replay ran on August 14, 2026. Human verification followed the product output and compared it with the repository history, the executed reproduction, and the later upstream record. Internal environment identifiers and non-public calibration details are intentionally excluded.