feat!: Add per-execution runId, at-most-once tracking, and cross-process tracker resumption#133
Draft
jsonbailey wants to merge 13 commits intomainfrom
Draft
feat!: Add per-execution runId, at-most-once tracking, and cross-process tracker resumption#133jsonbailey wants to merge 13 commits intomainfrom
jsonbailey wants to merge 13 commits intomainfrom
Conversation
jsonbailey
changed the title
- Each tracker now carries a runId (UUIDv4) included in all emitted events, scoping every metric to a single execution - At-most-once semantics: duplicate calls to track_duration, track_tokens, track_success/track_error, track_feedback, and track_time_to_first_token on the same tracker are dropped with a warning Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ess tracker resumption Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jsonbailey
force-pushed
the
jb/aic-2207/update-ai-sdks-billing-spec
branch
from
bdf7384 to
211ead4
Compare
…osure The run_id parameter on LDAIConfigTracker is now required (no default). UUID generation happens in the tracker_factory closure in client.py, keeping the tracker itself a plain data holder. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Break long tuple lines in client.py to stay under 120 char limit - Add required run_id parameter to LDAIConfigTracker calls in openai and langchain provider tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the redundant _tracked dict from LDAIConfigTracker. The summary already stores each metric with None as the unset sentinel, so the nil-check on summary properties serves as the at-most-once guard. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New order: ld_client, run_id, config_key, variation_key, version, model_name, provider_name, context, graph_key. All call sites converted to keyword arguments for resilience against future reorders. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…oken Reorder LDAIConfigTracker.__init__ to match updated spec: context now comes before model_name and provider_name. Also fix resumption_token to omit variationKey from the JSON when it is empty, and handle the absent key when reconstructing from a token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All six at-most-once guard warnings in tracker.py now log the track data dict (runId, configKey, etc.) to aid debugging duplicate-track scenarios. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the resumption token decoding logic from LDAIClient.create_tracker into a classmethod on LDAIConfigTracker per spec 1.1.20.2. The client method now delegates to LDAIConfigTracker.from_resumption_token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Match the resumption token behavior: only include variationKey in the track data dict when it has a non-empty value. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The create_tracker field on AIConfig is now always a callable that returns a working tracker, even when the config is disabled. The factory is always set to tracker_factory — callers use the enabled flag to decide whether to proceed, not the factory result. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BREAKING CHANGE: The `tracker` field has been removed from all config dataclasses (AICompletionConfig, AIJudgeConfig, AIAgentConfig). Users must now call `config.create_tracker()` to obtain a tracker instance. ManagedModel and ManagedAgent no longer accept a tracker constructor parameter — they call `create_tracker()` from the config on each invocation. The `__evaluate` return tuple no longer includes a pre-created tracker. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add graphKey to the resumption token following the spec key order: runId, configKey, variationKey (if set), version, graphKey (if set). The from_resumption_token classmethod now decodes and passes graphKey to the tracker constructor. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
runId: Every tracker now includes a uniquerunId(UUID) in all track event payloads, enabling billing isolation per executioncreate_tracker()factory on config objects:AICompletionConfig,AIAgentConfig, andAIJudgeConfignow carry an optionalcreate_trackercallable that returns a freshLDAIConfigTrackerwith a newrunIdeach time it's called. Set toNonewhen the config is disabled.ManagedModel.invoke(),ManagedAgent.run(), andJudge.evaluate()now callcreate_tracker()at the start of each invocation to get a fresh tracker, fixing the multi-turn tracking issue where at-most-once guards blocked metrics from second+ invocationsresumption_tokenproperty on tracker: URL-safe Base64-encoded (no padding) JSON string containing{runId, configKey, variationKey, version}for cross-process tracker reconstructionLDAIClient.create_tracker(token, context): Reconstructs a tracker from a resumption token for deferred feedback scenarios. Validates required fields and raisesValueErrorfor invalid tokens.Test plan
create_trackercallable; disabled config hasNonecreate_tracker()call returns a new tracker with a distinctrunIdManagedAgent.run()usescreate_tracker()when available, falls back to stored trackercreate_tracker(token, context)reconstructs tracker with originalrunIdand empty model/providerValueError🤖 Generated with Claude Code