feat(extension): Network Interceptor v1 — HAR streaming for BrowserStack LT#53
Open
nafees87n wants to merge 23 commits into
Open
feat(extension): Network Interceptor v1 — HAR streaming for BrowserStack LT#53nafees87n wants to merge 23 commits into
nafees87n wants to merge 23 commits into
Conversation
Add network interception recording via chrome.webRequest.onCompleted + onErrorOccurred. Captures all resource types (XHR, fetch, script, stylesheet, image, font, document) with metadata (URL, method, status, type, size). - NetworkRecordingService in service worker with start/stop lifecycle, tab-scoped event buffering, and max duration auto-stop - External messaging API (startNetworkRecording/stopNetworkRecording) via existing externally_connectable for BrowserStack domains - Chrome sidepanel UI with live request list, summary counters, text + method filters, recording timer - URL validation, error propagation, tab close cleanup with event recovery to sender tab - Sidepanel scoped to recorded tab only (disabled globally by default) - Guards for chrome.sidePanel API (Firefox/Safari compatibility) - Build setup: Rollup entry for sidepanel, manifest sidePanel permission - Test harness on localhost:3099 (dev builds only) Jira: RQ-2895 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the custom NetworkRecordingEvent schema with HAR Entry objects
(@types/har-format, already in repo). Network recording now produces the
same format Chrome DevTools exports, so LTS can parse it directly.
- New networkRecording/harBuilder.ts: webRequest details -> HAR Entry mapper
with mapResourceType (webRequest -> DevTools _resourceType enum). Emits
spec-complete entries (required request/response/content/timings/cache
fields) plus _resourceType / _request_id / _fromCache extension fields.
- Add a separate onBeforeSendHeaders listener + requestId correlation map to
populate request headers; consumed and cleared on completed/error. Cache
hits (no onBeforeSendHeaders) still produce an entry with empty headers.
- _request_id is an extension-assigned id, not the browser webRequest id.
- Move networkRecording.ts -> networkRecording/index.ts (directory module).
- Migrate sidepanel to read the HAR Entry shape, key off _request_id, and
count by _resourceType.
Stop still returns { success, events } in this PR (shape unchanged) so the
test harness keeps working; the summary-only contract lands in a later PR.
Jira: RQ-2895
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the streaming data channel for BrowserStack LT. LTS opens a long-lived
external port ("network-recording") and subscribes to a target tab; the
extension backfills the buffer from t=0, then streams each HAR entry live.
- initNetworkRecordingPort() via chrome.runtime.onConnectExternal, gated by
the existing externally_connectable allowlist. Registered in the SW index.
- subscriptions: Map<tabId, Set<Port>> — multiplexed (one LTS page can watch
several tabs from one port).
- subscribe: ack, synchronous backfill of existing entries (no await, so no
live entry can interleave -> no gap/dup), then register for live entries.
If the recording already ended, send complete immediately.
- deliverEntry now fans out to both the internal sidepanel and subscribed
ports; per-port send is try/caught so one dead port can't block others.
- complete { totalCount } is emitted to ports on stop and on tab close,
before the buffer is torn down. onDisconnect removes the port everywhere.
Stop still also returns { success, events } in this PR (changed to summary
in the next PR). LTS dedups on _request_id across reconnects.
Jira: RQ-2895
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nvelope
The stream is now the sole data channel to LTS. stopNetworkRecording is a
control command that ends the recording and returns a summary only — the HAR
entries are delivered over the port, not re-sent here.
- stopNetworkRecording returns { success, summary } where summary is
{ targetTabId, url, startTime, endTime, duration, totalCount }. No entries.
(Recording state now also tracks url for the summary.)
- External start/stop adopt the nested { action, payload } envelope:
start payload { url, config? }, stop payload { targetTabId }.
(getExtensionMetadata stays flat — no args.)
- Test harness updated: subscribes to the port after start, accumulates HAR
entries locally (dedup on _request_id), and on stop reads summary +
reconciles summary.totalCount against the streamed count.
Jira: RQ-2895
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Render the recording panel in Firefox's native sidebar (sidebar_action), the same React app the Chrome/Edge side panel uses. Firefox lacks chrome.sidePanel; this is the per-browser split. - manifest.firefox.json: add sidebar_action -> sidepanel/network-recording HTML. Never add side_panel / sidePanel to the Firefox manifest (Firefox rejects the permission). - networkRecording: abstract panel open/close per browser. Chrome/Edge use chrome.sidePanel (accessed via a dynamic `sidePanelApi` so the Firefox build lints clean — no UNSUPPORTED_API). Firefox uses browser.sidebarAction.open() (auto-open validated on FF 151, no gesture). Safari / other: no-op (capture + streaming still work). - Rollup already copies the panel HTML into every browser build. Verified: web-ext lint on the Firefox dist returns 0 errors and no sidePanel UNSUPPORTED_API; Chrome build keeps side_panel + sidePanel and no sidebar_action. Jira: RQ-2895 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ding An open port does not keep an MV3 service worker alive — only events/API calls reset the 30s idle timer. During idle gaps (user reading a page, no requests firing) the SW would die and drop the in-memory buffer. Add a two-part keepalive active only while a recording is running: - Prevention: a ~20s setInterval that calls chrome.runtime.getPlatformInfo() to reset the idle timer and keep the SW warm. - Backstop: a chrome.alarms tick at the 0.5min floor (sub-0.5 is clamped in packed builds). It survives SW death/re-wake and runs the max-duration auto-stop (so a quiet page still stops) plus a stale correlation-map sweep (60s TTL, so cache hits / aborts that never reach onCompleted don't leak). Moved the max-duration check out of onCompleted into the alarm tick. Started on first recording, cleared when the last one stops/closes. Adds the "alarms" permission to chrome, edge, and firefox manifests. Jira: RQ-2895 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- _request_id: use crypto.randomUUID() instead of a per-SW counter. A counter
resets to 0 on service-worker restart and would re-issue ids the LTS client
already saw, so its _request_id dedup would silently drop new entries.
- content size: parseContentLength returns -1 (HAR "unknown" sentinel) instead
of 0 when content-length is absent/unparseable; error entries use -1 too.
formatSize renders <0 as "—" so "unknown" is no longer shown as "0 B".
- maxDuration: restore a prompt inline auto-stop in onCompleted (busy page);
the alarm tick remains the backstop for a quiet page. Shared isOverMaxDuration.
- streaming: subscribing to a never-recorded tabId now returns { type: "error" }
so LTS can distinguish a bad targetTabId from an empty recording.
- remove dead senderTabId (write-only since the terminated-message removal).
Jira: RQ-2895
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the user stops a recording from the side panel, the stream consumer
(LTS) learns of the end via the port `complete` message but never receives
the summary — that only comes back in the stopNetworkRecording response,
which goes to the side panel, not LTS. Add a fetch-on-complete path so the
consumer can retrieve the summary regardless of who triggered the stop.
- New external message getNetworkRecordingSummary(targetTabId): returns the
summary while the recording is active (live snapshot) and for ~5 min after
it ends (retained), since the buffer/state are torn down on stop.
- stopNetworkRecording and tab-close now retain the summary before teardown.
- complete stays minimal { totalCount }; the contract decision holds — the
summary is fetched on demand, not pushed on the stream.
- Test harness: on `complete` it calls getNetworkRecordingSummary and renders
the summary in a dedicated panel (works whether stop came from the side
panel or the harness button; both go through the same complete path).
Jira: RQ-2895
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sidepanel feedback fixes: - Show the request host on the detail line (host · status · type · size) and the path on the primary line. Cross-host requests (CDN, third-party, subdomains) are now distinguishable, and bare relative paths are no longer ambiguous across page navigations. Host truncates first on a narrow sidebar so status/type/size stay visible; full URL on hover (title). - Auto-scroll only while the user is pinned to the bottom of the list; once they scroll up mid-stream, hold their position until they return to the bottom. - Method filter chips wrap instead of being clipped on a narrow sidebar. - Remove the "Sending live updates to BrowserStack Load Testing" footer text (kept the version, right-aligned). Jira: RQ-2895 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the summary retrieval consistent regardless of who stops the recording
(LTS or the side panel): both end the recording, the stream fires a pure
`complete` signal, and the consumer fetches the summary the same way.
- stopNetworkRecording returns { success } only — no inline summary. (When
the side panel stops, that response goes to the panel, not the stream
consumer, so returning it there was inconsistent.)
- `complete` is now a pure signal { type: "complete" } — no totalCount.
- getNetworkRecordingSummary errors while the recording is still active
("...is still active") so a half-finished summary is never mistaken for the
final one; returns the retained summary after stop; errors if unknown/expired.
- Harness: `complete` just logs the signal and fetches the summary; the
streamed-vs-totalCount reconcile moved into the summary panel.
Jira: RQ-2895
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
chrome.webRequest reports both XHR and fetch() as "xmlhttprequest", which mapResourceType maps to "xhr" — it never emits "fetch". Remove the dead "fetch" display-map entry and the unreachable fetch check in the XHR counter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ness Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ct auto-stop, optional maxDuration, stop banner - Return focus to the originating LTS tab on stop, with a cascade fallback: LTS tab -> its window -> a hardcoded LTS fallback URL (TODO before merge: replace with the real URL from the LTS team). Captures senderTabId + senderWindowId at start (re-adds the field removed when it was write-only). - Auto-stop a recording when its LTS port disconnects and no reconnect arrives within a 3s grace window. In v1 the port is the only data channel, so a recording is pointless once its consumer is gone; the grace tolerates the brief drop+reconnect the _request_id dedup was built for. - maxDuration is now optional with no default: when LTS omits it there is no time cap (recording runs until user stop / tab close / port disconnect). The port-disconnect auto-stop is the real leak-guard, so the 15-min default is redundant. isOverMaxDuration no-ops when no cap is set. - Side panel reflects why a recording ended via a new NETWORK_RECORDING_ENDED message + StopReason enum (user | max-duration | connection-lost | tab-closed): amber banner for max-duration, red for connection-lost; user/tab-closed show none. Panel now stays open in the stopped state (closePanel removed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nafees87n
commented
Jun 3, 2026
Contributor
Author
There was a problem hiding this comment.
FYI: temp setup. For testing without integration but we can have it there
Comment on lines
+53
to
+56
|
|
||
| if (manifestJson.externally_connectable?.matches) { | ||
| manifestJson.externally_connectable.matches.push("http://localhost:3099/*"); | ||
| } |
Contributor
Author
There was a problem hiding this comment.
To be revisit. Seems connected to testing setup
Block startNetworkRecording when the extension is disabled and return a structured error so LTS can surface "enable the Requestly extension" instead of opening a tab and streaming an empty HAR. Also auto-stop any active recording if the extension is toggled off mid-recording (the recorder's webRequest listeners are independent of the enabled flag), via a new "extension-disabled" StopReason that runs the normal teardown — LTS gets the complete signal + fetchable summary, the panel gets a banner reason. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 20s keepalive ping already keeps the MV3 SW warm for the whole recording (well under the 30s idle limit), so the alarm backstop is redundant — and in packed production builds its 0.5min period silently clamps to ~60s, making it a weaker max-duration guard than a timer. - max-duration: per-recording setTimeout (cleared on every teardown path), with the inline onCompleted check as the busy-page fast path. - correlation-map sweep: folded into the keepalive ping callback. - removed the "alarms" permission from chrome/edge/firefox manifests and the alarm listener. Accepted edge case documented: an OS sleep/wake SW kill can delay max-duration enforcement by a few seconds on a fully idle tab — no data loss (nothing records while asleep). Also adds a correlation-flow doc comment at onBeforeSendHeaders. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Opening the side panel inside the chrome.tabs.create callback (off the external LTS message path) broke after the extension-enabled guard added an await before tab creation — sidePanel.open()'s user-gesture window was already gone. Stop opening the panel eagerly there. Instead lean on the already-wired CLIENT_PAGE_LOADED path: the NETWORK_RECORDING tab flag is set on start, and handleNetworkRecordingOnClientPageLoad opens the panel once the new tab's page loads — the same decoupled pattern session-recording uses. Also: - openPanel now chains setOptions().then(open) so open() targets the registered per-tab path instead of racing it, and temporarily logs open() failures to the SW console to confirm the open path. - GET_EXTENSION_METADATA now returns isExtensionEnabled so LTS can check enabled state up front via the metadata call it already makes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reverts the panel-open changes that stopped the side panel from opening. The decoupled CLIENT_PAGE_LOADED path did not open the panel (a page-load event is not a user gesture), and the earlier extension-enabled guard's await before tabs.create had already pushed sidePanel.open() past the gesture window. Restores the pre-change synchronous path: startNetworkRecording is sync to chrome.tabs.create and calls openPanel(tab.id) eagerly in the callback, exactly as it worked before. Drops the isExtensionEnabled() start-guard entirely (LTS can still read enabled state from GET_EXTENSION_METADATA). Retained from this session: the chrome.alarms->setTimeout max-duration refactor, the mid-recording extension-disabled toggle auto-stop, and isExtensionEnabled in the metadata response. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…abled Re-adds the disabled-guard dropped earlier, but reads the enabled state from a synchronously-readable in-memory cache instead of awaiting storage. The previous guard's `await isExtensionEnabled()` before chrome.tabs.create pushed sidePanel.open() past its user-gesture window, so the panel stopped opening. Reading the cache keeps the path to openPanel fully synchronous. The cache (isExtensionEnabledCache) is seeded at init and kept fresh via the existing IS_EXTENSION_ENABLED onVariableChange listener — the same pattern clientHandler uses. Optimistic default (true) covers the brief window before the seed resolves. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The extension-enabled guard and mid-recording auto-stop both relied on onVariableChange(IS_EXTENSION_ENABLED), which filtered to ChangeType.MODIFIED only. IS_EXTENSION_ENABLED is lazily stored (getVariable defaults to true when the key is absent), so the FIRST time a user disables the extension the write is a CREATED change (no prior value) — the MODIFIED-only filter silently dropped it. Result: the cache stayed true and recordings started and continued while the extension was disabled. - onVariableChange now takes an optional changeTypes param (defaults to [MODIFIED], preserving existing callers). - The network-recording toggle listener opts into [MODIFIED, CREATED] so the first disable is caught — both the start-guard cache and the mid-recording auto-stop now react correctly. Also fixes a stale "alarm tick" comment left from the chrome.alarms removal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
returnFocusToSender exists to return the user to their LTS context after a recording they were watching ends (manual stop / max-duration / connection lost). An extension-disabled stop is different — it's a side effect of the user toggling the extension off, not a recording finishing — so moving their focus to another tab is surprising. The stopped-state banner already explains why. Skip the focus return for reason === "extension-disabled"; other reasons are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Return focus to the LTS context only when the user themselves stops the recording (clicked Stop). max-duration, connection-lost, and extension-disabled are background/system events, not actions on this recording — yanking the user's focus on top of the explanatory banner is surprising. Collapses the prior extension-disabled exclusion into the simpler positive rule (reason === "user"). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eaders (XHR/Fetch via web-sdk) (#56) * feat(extension): network recording v2 — request/response bodies + headers for XHR/Fetch via web-sdk Captures full request + response bodies and headers for XHR/Fetch by reusing the web-sdk Network interceptor (the module session recording uses), filling the HAR fields v1 left empty (request.postData, response.content.text). - SDK is the SOLE source for XHR/Fetch: webRequest is hard-suppressed for "xmlhttprequest" in onBeforeSendHeaders/onRequestCompleted/onRequestError, so there's exactly one source and no correlation. The v1 correlationMap is left intact for the non-xhr/fetch resource types that still come from webRequest. - New MAIN-world page script (networkBodyRecorder) calls the global Requestly.Network.intercept; the web-sdk UMD is injected per recorded tab (re-injected on webNavigation.onCommitted, single-tab scoped). overrideResponse is false (observe only — never alters the real request/response). - Size caps re-implemented in the page script (Network.intercept has no options; maxPayloadSize/ignoreMediaResponse live only on SessionRecorder): media-skip + per-body maxPayloadSize, surfaced as a _truncated extension field on the entry. - maxPayloadSize is configurable via startNetworkRecording config (default 100KB). - Stop gates the page callback off (never Network.clearInterceptors() — that would nuke other SDK consumers like session recording). - SDK-sourced entries flow through the same deliverEntry/stream path as v1, so LTS and the side panel consume them identically. No new stream message type. Known limitation: an XHR/Fetch the SDK can't capture (pre-injection race, no-cors/opaque, CSP-blocked injection) is dropped entirely (not backfilled from webRequest) — accepted trade for the no-correlation simplicity. Builds clean; message wiring verified static end-to-end. Live-browser recording not yet exercised. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(extension): reference the web-sdk global as bare `Requestly`, not window.Requestly The web-sdk UMD declares a top-level `var Requestly` (no explicit window attach). Referencing it bare — as the shipping sessionRecorderHelper.js does with `Requestly.SessionRecorder` — resolves the global binding regardless of how the injected file's scope reflects onto window; `window.Requestly` was a fragile stronger assumption. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(extension): restore eager side-panel open + drop isExtensionEnabled start guard (v2) The decoupled CLIENT_PAGE_LOADED panel-open path didn't open the side panel (a page-load event is not a user gesture), and the isExtensionEnabled await before tabs.create pushed sidePanel.open() past the gesture window. Restore the synchronous start path with eager openPanel(tab.id), and drop the start-guard. (Converges with base's 53d9d75; reconciled in the following merge.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a Network Interceptor to the browser extension that lets BrowserStack Load Testing (LTS) programmatically record all network activity on a target tab and stream it back in HAR 1.2 format. Includes a live UI panel (Chrome/Edge side panel, Firefox sidebar).
How it works
{ action, payload }envelope overonMessageExternal):startNetworkRecording { url, config?: { maxDuration? } }→ opens a target tab + the panel, begins capture, returns{ success, targetTabId }.stopNetworkRecording { targetTabId }→{ success }.getNetworkRecordingSummary { targetTabId }→{ success, summary }(the single, consistent way to fetch recording metadata after the stream completes).chrome.webRequest(onBeforeSendHeaders→onCompleted/onErrorOccurred) across all resource types, correlated into HAR entries with request + response headers.onConnectExternalport (network-recording). On subscribe:subscribedack → synchronous backfill of buffered entries → liveentrypushes →complete(pure signal). LTS dedups on the extension-assigned_request_id._-extension fields —_resourceType(DevTools enum),_request_id(extension-assigned UUID, not the browser-internal webRequest id),_fromCache.setInterval(prevents MV3 idle death during quiet pages) +chrome.alarmsbackstop running the maxDuration auto-stop and correlation-map sweep.side_panel(Chrome/Edge) vssidebar_action(Firefox);sidePanelis never added to the Firefox manifest (unsupported).Stop / complete / summary contract
The two stop paths (LTS-triggered and side-panel-triggered) are consistent:
stopreturns{ success }only.completeover the port is a pure end signal (no data).getNetworkRecordingSummaryis the one retrieval path for summary metadata, fetched aftercomplete; errors while the recording is still active.Scope
v1 (this PR): HAR wire format, request + response headers, streaming, summary, Firefox sidebar, SW keepalive.
Out of scope (v2): request/response bodies, advanced settings (disable cache / wipe SW),
extraHeadersfor Cookie capture,chrome.storage.sessioncrash resilience. Safari out of scope.Testing
mv3/test/network-recording-test.html(served viamv3/test/serve.jsonlocalhost:3099) simulates the LTS page — start/stop, port subscription, backfill + live stream, summary reconciliation. Thelocalhost:3099externally_connectableentry is dev-only (gated behind non-production build in rollup; never shipped to production).🤖 Generated with Claude Code