Skip to content

Deflake test_memo: gate on hydration and capture page diagnostics#6641

Closed
masenf wants to merge 1 commit into
mainfrom
claude/intelligent-bell-2sg4ti
Closed

Deflake test_memo: gate on hydration and capture page diagnostics#6641
masenf wants to merge 1 commit into
mainfrom
claude/intelligent-bell-2sg4ti

Conversation

@masenf

@masenf masenf commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

test_memo interacted with the page immediately after goto, racing the
websocket connect/hydrate startup; the typed input event could be lost,
leaving #memo-last-value empty through every rerun of an affected CI
job. Gate all memo tests on the standard #token hydration marker used
by the rest of the playwright suite before interacting.

Also add a tests_playwright conftest that records console messages,
page errors, and websocket frames per test and attaches them to failed
test reports, so any future flake of this kind is diagnosable directly
from CI job logs instead of reproducing blind.

https://claude.ai/code/session_01Xvaqxi4LJeWu2fo4Fk7qMD

test_memo interacted with the page immediately after goto, racing the
websocket connect/hydrate startup; the typed input event could be lost,
leaving #memo-last-value empty through every rerun of an affected CI
job. Gate all memo tests on the standard #token hydration marker used
by the rest of the playwright suite before interacting.

Also add a tests_playwright conftest that records console messages,
page errors, and websocket frames per test and attaches them to failed
test reports, so any future flake of this kind is diagnosable directly
from CI job logs instead of reproducing blind.

https://claude.ai/code/session_01Xvaqxi4LJeWu2fo4Fk7qMD
@masenf masenf requested a review from a team as a code owner June 10, 2026 00:57
@greptile-apps

greptile-apps Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a race condition in test_memo where tests interacted with the page before the WebSocket/hydrate cycle completed, causing typed input events to be silently dropped in CI. It also introduces a new conftest.py for the Playwright suite that records console messages, page errors, and WebSocket frames per test and attaches them to failed reports.

  • test_memo.py: adds a #token input bound to router.session.client_token and a load_page() helper that waits for it to be non-empty before any test interaction, using the same hydration-gate pattern as the rest of the Playwright suite.
  • conftest.py (new): autouse _page_diagnostics fixture hooks into Playwright's page events and a pytest_runtest_makereport hookimpl attaches the captured log to failed/rerun call reports.

Confidence Score: 4/5

Safe to merge; the hydration fix is correct and the diagnostic conftest is a net improvement, with one minor gap in the WebSocket frame logging.

The WebSocket frame callbacks pass the raw WebSocketFrame object into an f-string rather than its .payload attribute, so frame contents will appear as opaque object addresses in CI logs. This doesn't affect test correctness — the hydration fix is solid — but the diagnostic output that motivated the conftest will be less useful than intended whenever a WebSocket-related flake occurs.

tests/integration/tests_playwright/conftest.py — the ws frame logging callbacks on lines 44-45.

Important Files Changed

Filename Overview
tests/integration/tests_playwright/conftest.py New conftest adding autouse diagnostic capture for console/pageerror/websocket events; WebSocket frame callbacks log the raw frame object instead of frame.payload, producing opaque addresses in CI logs.
tests/integration/tests_playwright/test_memo.py Adds hydration guard via #token input and load_page() helper; all five tests updated consistently; no issues found.

Reviews (1): Last reviewed commit: "Deflake test_memo: gate on hydration and..." | Re-trigger Greptile

Comment on lines +44 to +45
ws.on("framesent", lambda frame: stamp(f"ws sent: {frame}"))
ws.on("framereceived", lambda frame: stamp(f"ws recv: {frame}"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The frame argument passed to framesent/framereceived callbacks is a Playwright WebSocketFrame object. It does not override __repr__ or __str__, so interpolating it directly with an f-string logs an opaque object address (e.g. <WebSocketFrame object at 0x7f…>) rather than the actual payload. Use frame.payload to capture the frame contents that make these diagnostics useful.

Suggested change
ws.on("framesent", lambda frame: stamp(f"ws sent: {frame}"))
ws.on("framereceived", lambda frame: stamp(f"ws recv: {frame}"))
ws.on("framesent", lambda frame: stamp(f"ws sent: {frame.payload}"))
ws.on("framereceived", lambda frame: stamp(f"ws recv: {frame.payload}}"))

@codspeed-hq

codspeed-hq Bot commented Jun 10, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 26 untouched benchmarks
⏩ 8 skipped benchmarks1


Comparing claude/intelligent-bell-2sg4ti (8476246) with main (fdb00db)

Open in CodSpeed

Footnotes

  1. 8 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@masenf

masenf commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator Author

this actually doesn't seem like it would fix the problem

@masenf masenf closed this Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants