Skip to content

feat(telemetry): add streaming JSON export writer#960

Open
EhabY wants to merge 2 commits into
mainfrom
feat/issue-903-export-telemetry-json-writer
Open

feat(telemetry): add streaming JSON export writer#960
EhabY wants to merge 2 commits into
mainfrom
feat/issue-903-export-telemetry-json-writer

Conversation

@EhabY
Copy link
Copy Markdown
Collaborator

@EhabY EhabY commented May 17, 2026

Summary

  • Add writeJsonArrayExport that streams telemetry events to disk as a JSON array. Chunks are piped through stream/promises pipeline so memory stays flat regardless of maxTotalBytes (default 100 MB, no enforced ceiling).
  • Extract a shared src/util/fs.ts module hosting tempFilePath, renameWithRetry, and a new writeAtomically(path, write) helper that captures the temp-file + rename + best-effort cleanup pattern.
  • The new helper replaces CliCredentialManager.atomicWriteFile (16 lines -> 4) and backs the export writer. SSH config / support bundle / CLI manager keep their bespoke flows and just update imports.

Refs #903.

Stack: 2 / 4. Base: #959. Next: #961.

Why streaming

Export ranges include "All time", which reads everything on disk. With maxFileBytes and maxTotalBytes user-tunable and unbounded, a buffered writer would peak at ~2x the export size in memory and OOM at high caps. Streaming holds memory constant at ~12 lines of code vs ~10 for the buffered version.

Changes

  • src/telemetry/export/writers.ts: writeJsonArrayExport returns the event count; uses serializeTelemetryEvent from the shared wire format module so the output cannot drift from the on-disk shape.
  • src/util/fs.ts (new): tempFilePath, renameWithRetry, writeAtomically.
  • src/util.ts: drops the moved helpers.
  • src/core/cliCredentialManager.ts: atomicWriteFile now defers to writeAtomically.
  • test/unit/util/fs.test.ts (new): suite for the three helpers, including atomic-write success, partial-write rollback, and callback return values.
  • test/unit/telemetry/export/writers.test.ts: switched to memfs, shared createTelemetryEventFactory, and the shared serializeTelemetryEvent. Added a test that a mid-stream failure leaves the destination untouched.
  • test/unit/util.test.ts: moved-out blocks removed.
  • test/unit/core/supportBundleLogs.test.ts: vi.mock target retargeted to @/util/fs.

@EhabY EhabY self-assigned this May 17, 2026
@EhabY EhabY force-pushed the feat/issue-903-export-telemetry-json-writer branch from 199d2e7 to 3a29ac9 Compare May 17, 2026 19:42
@EhabY EhabY force-pushed the feat/issue-903-export-telemetry-core branch from 9d7d4ce to 3c02990 Compare May 17, 2026 19:42
@EhabY EhabY force-pushed the feat/issue-903-export-telemetry-json-writer branch from 3a29ac9 to eb776a7 Compare May 18, 2026 17:03
@EhabY EhabY force-pushed the feat/issue-903-export-telemetry-core branch from 3c02990 to 350d662 Compare May 18, 2026 17:03
@EhabY EhabY force-pushed the feat/issue-903-export-telemetry-json-writer branch from eb776a7 to ba095a9 Compare May 18, 2026 19:05
@EhabY EhabY force-pushed the feat/issue-903-export-telemetry-core branch from ca1bd07 to c0e517d Compare May 19, 2026 08:47
@EhabY EhabY force-pushed the feat/issue-903-export-telemetry-json-writer branch from ba095a9 to 8066586 Compare May 19, 2026 10:00
@EhabY EhabY changed the title feat(telemetry): add JSON export writer feat(telemetry): add streaming JSON export writer and shared atomic-write helper May 19, 2026
@EhabY EhabY changed the title feat(telemetry): add streaming JSON export writer and shared atomic-write helper feat(telemetry): add streaming JSON export writer May 19, 2026
@EhabY EhabY force-pushed the feat/issue-903-export-telemetry-json-writer branch from 3d77d32 to c9c3b74 Compare May 19, 2026 11:02
@EhabY EhabY force-pushed the feat/issue-903-export-telemetry-core branch 2 times, most recently from d550ffd to aca8405 Compare May 20, 2026 10:27
@EhabY EhabY force-pushed the feat/issue-903-export-telemetry-json-writer branch from c9c3b74 to 9c5a19d Compare May 20, 2026 10:28
@EhabY EhabY requested a review from mafredri May 20, 2026 10:32
Base automatically changed from feat/issue-903-export-telemetry-core to main May 20, 2026 13:48
EhabY added 2 commits May 20, 2026 16:49
Move `tempFilePath` and `renameWithRetry` from `src/util.ts` into a
focused `src/util/fs.ts`, alongside a new `writeAtomically(path, write)`
helper that captures the temp-file + rename + best-effort cleanup
pattern previously copy-pasted into the telemetry writer and
`CliCredentialManager.atomicWriteFile`. Both call sites now use the
shared helper; the SSH config / support bundle / CLI manager keep their
bespoke flows and just update imports.

Rewrite `writeJsonArrayExport` to stream chunks through
`Readable.from(...)` into `createWriteStream` via `stream/promises`
`pipeline`. Memory stays flat regardless of `maxTotalBytes`, which has
no enforced ceiling and can comfortably exceed the default 100 MB cap.

Tests for `tempFilePath` and `renameWithRetry` move into
`test/unit/util/fs.test.ts` alongside a new `writeAtomically` suite
covering success, partial-write rollback, and callback return values.
@EhabY EhabY force-pushed the feat/issue-903-export-telemetry-json-writer branch from 9c5a19d to 914025b Compare May 20, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant