Skip to content

feat!: Add per-execution runId, at-most-once tracking, and cross-process tracker resumption#249

Merged
jsonbailey merged 3 commits into
mainfrom
jb/aic-2207/update-ai-sdks-billing-spec
Jun 3, 2026
Merged

feat!: Add per-execution runId, at-most-once tracking, and cross-process tracker resumption#249
jsonbailey merged 3 commits into
mainfrom
jb/aic-2207/update-ai-sdks-billing-spec

Conversation

@jsonbailey
Copy link
Copy Markdown
Contributor

@jsonbailey jsonbailey commented Apr 15, 2026

BEGIN_COMMIT_OVERRIDE
feat!: Add per-execution runId on all AI track event payloads for billing isolation
feat!: Enforce at-most-once tracking — each metric type (duration, tokens, feedback, success/error, time-to-first-token) records once per tracker; duplicates are dropped with a warning
feat!: Add ResumptionToken on ILdAiConfigTracker and ILdAiClient.CreateTracker(resumptionToken, context) for cross-process tracker reconstruction with the original runId
feat: Add MetricSummary property on ILdAiConfigTracker summarizing recorded metrics
END_COMMIT_OVERRIDE

Summary

  • Per-execution runId: Every tracker gets a unique runId (UUID) included in all track event payloads, enabling per-execution billing isolation
  • At-most-once tracking: Each metric type (duration, tokens, feedback, success/error, time-to-first-token) can only be tracked once per tracker instance — duplicates are dropped with a log warning
  • CreateTracker() on LdAiConfig: Each call on a config (e.g. LdAiCompletionConfig) returns a new tracker with a fresh runId and independent tracking state. The returned tracker is always non-null, even when the config is disabled
  • ResumptionToken property on ILdAiConfigTracker: URL-safe Base64-encoded JSON containing { runId, configKey, variationKey, version } for cross-process tracker reconstruction
  • CreateTracker(string token, Context context) on ILdAiClient: Reconstructs a tracker from a resumption token with the original runId, enabling deferred feedback from a different process. Model and provider names are set to empty strings on reconstruction
  • MetricSummary property on ILdAiConfigTracker: Snapshot of metrics recorded on this tracker

Test plan

  • CreateTrackerReturnsTrackerWhenDisabled — factory returns a non-null tracker for disabled configs
  • CreateTrackerReturnsNewTrackerWithFreshRunId — each tracker gets a unique runId
  • CreateTrackerReturnsTrackerWithIndependentTrackingState — at-most-once state is independent per tracker
  • CreateTrackerCanBeCalledMultipleTimes — factory produces multiple independent trackers
  • ResumptionTokenContainsExpectedFields — token contains exactly runId, configKey, variationKey, version
  • ResumptionTokenIsUrlSafeBase64 — no +, /, or = characters
  • ResumptionTokenIsConsistentAcrossCalls — same token on repeated access
  • CreateTrackerFromResumptionTokenRoundTrips — full round-trip preserves runId
  • CreateTrackerFromResumptionTokenSetsEmptyModelAndProvider — reconstructed tracker has empty model/provider
  • CreateTrackerFromInvalidTokenThrows — malformed input throws ArgumentException
  • CreateTrackerFromNullTokenThrows — null input throws ArgumentNullException
  • CreateTrackerFromTokenMissingRunIdThrows — incomplete payload throws ArgumentException

🤖 Generated with Claude Code


Note

Medium Risk
Changes public tracking semantics (runId on events, duplicate metrics ignored) and default version behavior, which can affect downstream analytics and billing correlation; resumption token parsing is a new trust boundary for cross-process flows.

Overview
AI config trackers are now scoped to a single run with a UUID runId on every track payload, ResumptionToken / ILdAiClient.CreateTracker(token, context) for cross-process reuse of that run, and a MetricSummary snapshot of what was recorded.

LdAiConfigTracker is rebuilt around explicit fields (not the whole config): each CreateTracker() gets a new runId; duplicate Track* calls are dropped (with warnings) using atomic slots, including shared success/error; empty token usage does not consume the tokens slot. Track metadata adds runId, omits empty variationKey, and resumption tokens are URL-safe Base64 JSON without model/provider.

Smaller contract tweaks: caller defaults use version: 1; missing _ldMeta.version defaults to 1; ModelConfig stores parameters in ImmutableDictionary so downcasts cannot mutate maps.

Reviewed by Cursor Bugbot for commit f7488be. Bugbot is set up for automated code reviews on this repo. Configure here.

@jsonbailey jsonbailey changed the title feat!: Add per-execution runId and at-most-once event tracking feat!: Add per-execution runId, at-most-once tracking, and cross-process tracker resumption Apr 15, 2026
@jsonbailey jsonbailey force-pushed the jb/aic-2207/update-ai-sdks-billing-spec branch from fe3cd88 to 4dee48f Compare April 16, 2026 16:53
@jsonbailey jsonbailey marked this pull request as ready for review May 14, 2026 15:23
@jsonbailey jsonbailey requested a review from a team as a code owner May 14, 2026 15:23
Comment thread pkgs/sdk/server-ai/src/LdAiConfigTracker.cs Outdated
Comment thread pkgs/sdk/server-ai/src/Config/LdAiConfig.cs Outdated
Comment thread pkgs/sdk/server-ai/src/LdAiConfigTracker.cs Outdated
Comment thread pkgs/sdk/server-ai/src/Interfaces/ILdAiClient.cs
Comment thread pkgs/sdk/server-ai/src/Interfaces/ILdAiConfigTracker.cs Outdated
Comment thread pkgs/sdk/server-ai/src/Interfaces/ILdAiConfigTracker.cs Outdated
Comment thread pkgs/sdk/server-ai/src/LdAiConfigTracker.cs Outdated
Comment thread pkgs/sdk/server-ai/src/LdAiConfigTracker.cs Outdated
Comment thread pkgs/sdk/server-ai/src/Config/LdAiConfig.cs Outdated
Comment thread pkgs/sdk/server-ai/src/LdAiConfigTracker.cs
Comment thread pkgs/sdk/server-ai/src/LdAiConfigTracker.cs Outdated
Comment thread pkgs/sdk/server-ai/src/LdAiConfigTracker.cs
Comment thread pkgs/sdk/server-ai/src/Config/LdAiConfig.cs Outdated
Comment thread pkgs/sdk/server-ai/src/LdAiConfigTracker.cs Outdated
@kinyoklion
Copy link
Copy Markdown
Member

Tangentially — not introduced by this PR, but surfaced while reviewing it — ModelConfiguration.Parameters and .Custom are typed as IReadOnlyDictionary<> but the underlying instances are mutable Dictionary<>s that the builder hands off by reference. A downstream consumer can downcast through any IDictionary<>-compatible interface and mutate them, and writes are visible on config.Model.Parameters.

Failing test that pins the desired behavior (it goes green once the dictionaries are stored as a defensive copy):

// ModelConfiguration.Parameters is typed as IReadOnlyDictionary<>, so the
// public contract is "read-only". Even if a consumer downcasts to a
// mutable view, writes through that view must not be visible on
// config.Model.Parameters.
[Fact]
public void DowncastedMutationOfModelParametersMustNotAffectBuiltConfig()
{
    var config = LdAiConfig.New().SetModelParam("temperature", LdValue.Of(0.7)).Build();

    if (config.Model.Parameters is System.Collections.Generic.IDictionary<string, LdValue> mutable)
    {
        mutable["temperature"] = LdValue.Of(2.0);
    }

    Assert.Equal(0.7, config.Model.Parameters["temperature"].AsDouble);
}

Two options for the fix:

  1. ImmutableDictionary<string, LdValue> in ModelConfiguration's ctor (and copy the input into it). Type system enforces no mutation.
  2. Store the whole thing as LdValue instead — LdValue.ObjectFrom(...) already handles immutability and deep-copies primitive map data.

Happy to file as a separate issue if it's out of scope for this PR.

Comment thread pkgs/sdk/server-ai/src/LdAiConfigTracker.cs Outdated
Comment thread pkgs/sdk/server-ai/src/LdAiConfigTracker.cs
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0f3f7d5. Configure here.

Comment thread pkgs/sdk/server-ai/src/LdAiConfigTracker.cs
Comment thread pkgs/sdk/server-ai/src/LdAiConfigTracker.cs Outdated
@jsonbailey jsonbailey force-pushed the jb/aic-2207/update-ai-sdks-billing-spec branch from 0f3f7d5 to 6fdf30e Compare June 3, 2026 15:46
Implements the AI SDK billing-spec changes for the tracker:

- Per-execution runId (UUIDv4) is generated at tracker construction and
  included in every emitted event so LaunchDarkly can correlate all
  metrics emitted by a single run.

- At-most-once event tracking uses Interlocked.CompareExchange on
  StrongBox-wrapped nullable slots. Each metric (duration, time-to-
  first-token, tokens, feedback, success/error) is recorded exactly
  once per tracker even under concurrent calls. Subsequent calls log
  a skip warning that includes the track data and points to
  CreateTracker for a new run.

- Cross-process tracker resumption via the new ResumptionToken
  property and LdAiConfigTracker.FromResumptionToken static method.
  The token is a URL-safe Base64 JSON payload (runId, configKey,
  variationKey, version) emitted in canonical key order. The ILdAiClient
  interface also gains CreateTracker(resumptionToken, context) as the
  user-facing resumption entry point. Reconstructed trackers share the
  original runId so deferred-feedback flows correlate with the original.

- MetricSummary record exposes the current per-metric state via the
  new Summary property on the tracker.

- Validation hardening:
  - TrackFeedback throws ArgumentOutOfRangeException for invalid Feedback
  - TrackTokens validates Usage before consuming the at-most-once slot
  - FromResumptionToken throws ArgumentException for malformed or
    incomplete tokens; catches all non-OutOfMemoryException parse failures

- Wire format: empty variationKey is omitted from event track data and
  from the resumption token. Empty means "no variation served"
  (fallback / default), distinct from "served a variation with key ''".

- Tracker construction: single internal ctor takes individual fields.
  Trackers are produced via LdAiConfigBase.CreateTracker (server-side
  evaluation, wired by ConfigFactory) or FromResumptionToken
  (cross-process resumption).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@jsonbailey jsonbailey force-pushed the jb/aic-2207/update-ai-sdks-billing-spec branch from 6fdf30e to 4fd0cda Compare June 3, 2026 16:21
jsonbailey and others added 2 commits June 3, 2026 12:39
- Default _ldMeta.version to 1 (was 0) when the field is absent on the
  wire, matches python and js-core. Three sites updated for consistency:
  ConfigFactory.ParseMeta, ConfigFactory.BuildCompletionFromDefault, and
  LdAiConfigTracker.ResumptionPayload (so a resumption token without
  "version" deserializes to 1, not 0).

- Default _variationKey to "" in the tracker constructor so a tracker
  reconstructed from a resumption token without a variationKey holds
  the same value internally as one created from LdAiConfigBase (which
  already defaults variationKey ?? "").

- Drop unused `using System;` from LdAiCompletionConfigDefault.cs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ncast

ModelConfig.Parameters and .Custom are typed IReadOnlyDictionary<> but
were stored as the literal Dictionary<> the caller handed in. The
runtime type was mutable; a consumer downcasting to IDictionary<>
could mutate the live config.

Materialize into ImmutableDictionary<> in the ctor. The cast still
succeeds (preserving the IReadOnlyDictionary<> contract) but write
members throw NotSupportedException, matching the typed read-only
promise. Pre-existing issue surfaced during PR review.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@jsonbailey jsonbailey requested a review from kinyoklion June 3, 2026 17:57
@jsonbailey jsonbailey merged commit 44ff485 into main Jun 3, 2026
15 checks passed
@jsonbailey jsonbailey deleted the jb/aic-2207/update-ai-sdks-billing-spec branch June 3, 2026 22:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants