Per-component / tag cardinality limits in client-side stats#11387
Draft
dougqh wants to merge 203 commits into
Draft
Per-component / tag cardinality limits in client-side stats#11387dougqh wants to merge 203 commits into
dougqh wants to merge 203 commits into
Conversation
ConflatingMetricsAggregator.publish does a handful of redundant operations on every span. None individually is large; together they show as ~2.5% on the existing JMH benchmark once the benchmark actually exercises span.kind. - dedup span.isTopLevel(): publish() reads it into a local, then shouldComputeMetric read it again. Pass the cached value in. - resolve spanKind to String once: master called toString() twice per span (once inside spanKindEligible, once at the getPeerTags call site) and used HashSet contains on a CharSequence (which routes through equals on String). Normalize to String up front and reuse. - lazy-allocate the peer-tag list: getPeerTags() always allocated an ArrayList sized to features.peerTags() even when the span had none of those tags set. Defer allocation until the first match; return Collections.emptyList() when none hit. MetricKey already treats null/empty peerTags as emptyList, so no behavior change. Drop the spanKindEligible helper — the HashSet.contains call inlines fine in shouldComputeMetric. Update the JMH benchmark to set span.kind=client on every span. Without it the filter path short-circuits before the peer-tag and toString work, so the wins above aren't measurable. With it: baseline 6.755 us/op (CI [6.560, 6.950], stdev 0.129) optimized 6.585 us/op (CI [6.536, 6.634], stdev 0.033) 2 forks x 5 iterations x 15s. ~2.5% mean improvement and much tighter variance fork-to-fork. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduce SpanKindFilter -- a tiny builder-built immutable filter whose state is an int bitmask indexed by the span.kind ordinals already cached on DDSpanContext. Each include* on the builder sets one bit (1 << ordinal); the runtime check is a single AND against (1 << span's ordinal). CoreSpan.isKind(SpanKindFilter) is the new entry point. DDSpan overrides it to do the bit-test directly against the cached ordinal -- no virtual call, no tag-map lookup. The two existing test-only CoreSpan impls (SimpleSpan and TraceGenerator.PojoSpan, the latter in two source sets) implement isKind by reading the span.kind tag and delegating to SpanKindFilter.matches(String), which converts via DDSpanContext.spanKindOrdinalOf and does the same AND. Refactor: DDSpanContext.setSpanKindOrdinal(String) now delegates to a new package-private static spanKindOrdinalOf(String) so the same string-to-ordinal mapping serves both the tag interceptor path and SpanKindFilter.matches. This is groundwork -- nothing in the codebase calls isKind yet. The next commit will replace the HashSet-based eligibility checks in ConflatingMetricsAggregator with SpanKindFilter instances. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the two ELIGIBLE_SPAN_KINDS_FOR_* HashSet<String> constants and the SPAN_KIND_INTERNAL.equals check with three SpanKindFilter instances: METRICS_ELIGIBLE_KINDS, PEER_AGGREGATION_KINDS, INTERNAL_KIND. Eligibility checks now go through span.isKind(filter), which on DDSpan is a volatile byte read against the already-cached span.kind ordinal plus a single bit-test. Also defer the span.kind tag read: previously read at the top of the publish loop and threaded through both shouldComputeMetric and the inner publish. isKind no longer needs the string, so the read can move down into the inner publish where it's still needed for the SPAN_KINDS cache key / MetricKey. Supporting changes: - DDSpanContext.spanKindOrdinalOf(String) is now public so non-DDSpan CoreSpan impls can compute the ordinal at tag-write time. - SpanKindFilter gains a public matches(byte) fast-path overload that callers with a pre-computed ordinal use directly. - SimpleSpan caches the ordinal in setTag(SPAN_KIND, ...), mirroring what TagInterceptor does for DDSpanContext, and its isKind now hits the byte fast path. Without this, the JMH benchmark (which uses SimpleSpan) would re-derive the ordinal on every isKind call and overstate the cost. Benchmark on the bench updated last commit (kind=client on every span, 4 forks x 5 iter x 15s): prior commit 6.585 ± 0.049 us/op this commit 6.903 ± 0.096 us/op The slight regression is a SimpleSpan-via-groovy-dispatch artifact -- the interface call to isKind through CoreSpan, then through SimpleSpan, then through SpanKindFilter.matches, doesn't fold as aggressively as a HashSet contains on a static field. In production DDSpan.isKind inlines to a context field read + ordinal byte read + bit-test, so the production path is faster than the prior HashSet approach. A DDSpan-based benchmark would show this; the existing SimpleSpan-based one doesn't. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The existing ConflatingMetricsAggregatorBenchmark uses SimpleSpan, a groovy mock. That's enough for measuring queue/CHM/MetricKey work, but it conceals the production cost of CoreSpan.isKind: SimpleSpan's isKind goes through groovy interface dispatch into SpanKindFilter.matches, while DDSpan.isKind inlines to a context byte-read + bit-test. This new benchmark uses real DDSpan instances created through a CoreTracer (with a NoopWriter so finishing doesn't reach the agent). Same shape as the SimpleSpan bench (64-span trace, span.kind=client, peer.hostname set). Numbers (2 forks x 5 iter x 15s): master: 6.428 +- 0.189 us/op (HashSet eligibility checks) this branch: 6.343 +- 0.115 us/op (SpanKindFilter bitmask) About 1.3% faster on the production path. The SimpleSpan benchmark in the same conditions shows a ~2.2% slowdown -- the mock's dispatch shape gives a misleading signal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make SpanKindFilter.kindMask and its constructor private now that DDSpan.isKind no longer needs direct field access -- it delegates to SpanKindFilter.matches(byte). The Builder.build() in the same outer class still constructs instances via the private constructor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the producer-side conflation pipeline with a thin per-span SpanSnapshot
posted to the existing aggregator thread. The aggregator now builds the
MetricKey, does the SERVICE_NAMES / SPAN_KINDS / PEER_TAGS_CACHE lookups, and
updates the AggregateMetric directly -- all off the producer's hot path.
What the producer does now, per span:
- filter (shouldComputeMetric, resource-ignored, longRunning)
- collect tag values into a SpanSnapshot (1 allocation per span)
- inbox.offer(snapshot) + return error flag for forceKeep
What moved off the producer:
- MetricKey construction and its hash computation
- SERVICE_NAMES.computeIfAbsent (UTF8 encoding of service name)
- SPAN_KINDS.computeIfAbsent (UTF8 encoding of span.kind)
- PEER_TAGS_CACHE lookups (peer-tag name+value UTF8 encoding)
- pending/keys ConcurrentHashMap operations
- Batch pooling, batch atomic ops, batch contributeTo
Removed entirely:
- Batch.java -- the conflation primitive is no longer needed; the
aggregator's existing LRUCache<MetricKey, AggregateMetric> IS the
conflation point now.
- pending ConcurrentHashMap<MetricKey, Batch>
- keys ConcurrentHashMap<MetricKey, MetricKey> (canonical dedup)
- batchPool MessagePassingQueue<Batch>
- The CommonKeyCleaner role of tracking keys.keySet() on LRU eviction --
AggregateExpiry now just reports drops to healthMetrics.
Added:
- SpanSnapshot: immutable value carrying the raw MetricKey inputs + a
tagAndDuration long (duration | ERROR_TAG | TOP_LEVEL_TAG).
- AggregateMetric.recordOneDuration(long tagAndDuration) -- the single-hit
equivalent of the existing recordDurations(int, AtomicLongArray).
- Peer-tag values flow through the snapshot as a flattened String[] of
[name0, value0, name1, value1, ...]; the aggregator encodes them through
PEER_TAGS_CACHE on its own thread.
Benchmark results (2 forks x 5 iter x 15s):
ConflatingMetricsAggregatorDDSpanBenchmark
prior commit 6.343 +- 0.115 us/op
this commit 2.506 +- 0.044 us/op (~60% faster)
ConflatingMetricsAggregatorBenchmark (SimpleSpan)
prior commit 6.585 +- 0.049 us/op
this commit 3.116 +- 0.032 us/op (~53% faster)
Caveat on the benchmark: without conflation, the producer pushes 1 inbox
item per span instead of ~1 per 64. At the benchmark's synthetic rate the
consumer can't keep up and inbox.offer silently drops. The numbers measure
producer publish() latency only; consumer throughput at realistic span rates
is a follow-up to validate. Tuning maxPending matters more in this design.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
With the per-span SpanSnapshot inbox path, the producer can lose snapshots when the bounded MPSC queue is full -- silently, since inbox.offer() returns a boolean we previously ignored. The conflating-Batch design used to absorb ~64x more producer pressure per inbox slot, so this is a new failure mode worth surfacing. Wire it through the existing HealthMetrics path: - HealthMetrics.onStatsInboxFull() (no-op default). - TracerHealthMetrics gets a statsInboxFull LongAdder and a new reason tag reason:inbox_full reported under the same stats.dropped_aggregates metric used for LRU evictions. Two LongAdders, two tagged time series. - ConflatingMetricsAggregator.publish increments the counter when inbox.offer(snapshot) returns false. This doesn't fix the drop -- tuning maxPending and/or building producer-side batching are the actual fixes. But it makes the failure visible in the same place ops already watches. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dougqh
added
type: enhancement
dougqh
commented
May 15, 2026
5 tasks
…nflating-metrics-background-work
Two general-purpose utilities used by the client-side stats aggregator work (PR #11382 and follow-ups), extracted into their own change so the metrics-specific PRs can build on a smaller, reviewable foundation. - Hashtable: a generic open-addressed-ish bucket table abstraction keyed by a 64-bit hash, with a public abstract Entry type so client code can subclass it for higher-arity keys. The metrics aggregator uses it to back its AggregateTable. - LongHashingUtils: chained 64-bit hash combiners with primitive overloads (boolean, short, int, long, Object). Used in place of varargs combiners to avoid Object[] allocation and boxing on the hot path. No callers within internal-api itself yet -- the metrics aggregator PR will introduce the first usages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Standalone classes for swapping the consumer-side LRUCache<MetricKey, AggregateMetric> with a multi-key Hashtable in the next commit. No call sites use them yet. - AggregateEntry extends Hashtable.Entry, holds the canonical MetricKey, the mutable AggregateMetric, and copies of the 13 raw SpanSnapshot fields for matches(). The 64-bit lookup hash is computed via chained LongHashingUtils.addToHash calls (no varargs, no boxing of short/boolean). - AggregateTable wraps a Hashtable.Entry[] from Hashtable.Support.create. findOrInsert(SpanSnapshot) walks the bucket comparing raw fields, falling back to MetricKeys.fromSnapshot on a true miss. On cap overrun, it scans for an entry with hitCount==0 and unlinks it; if none, it returns null and the caller drops the data point. - MetricKeys.fromSnapshot extracts the canonicalization logic (DDCache lookups + UTF8 encoding) from Aggregator.buildMetricKey, so the helper can be called from AggregateTable on miss. This also commits Hashtable and LongHashingUtils (added earlier, previously uncommitted) and lifts Hashtable.Entry / Hashtable.Support visibility so client code outside datadog.trace.util can build higher-arity tables -- the case the javadoc describes but the original visibility didn't actually support. Specifically: Entry is now public abstract with a protected ctor; keyHash, next(), and setNext() are public; Support's create / clear / bucketIndex / bucketIterator / mutatingBucketIterator methods are public. Tests: AggregateTableTest covers hit, miss, distinct-by-spanKind, peer-tag identity (including null vs non-null), cap overrun with stale victim, cap overrun with no victim (returns null), expungeStaleAggregates, forEach, clear, and that the canonical MetricKey is built at insert. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace LRUCache<MetricKey, AggregateMetric> with the AggregateTable added
in the prior commit. The hot path in Drainer.accept becomes:
AggregateMetric aggregate = aggregates.findOrInsert(snapshot);
if (aggregate != null) {
aggregate.recordOneDuration(snapshot.tagAndDuration);
dirty = true;
} else {
healthMetrics.onStatsAggregateDropped();
}
On the steady-state hit path the lookup is a 64-bit hash compute + bucket
walk + matches(snapshot) -- no MetricKey allocation, no SERVICE_NAMES /
SPAN_KINDS / PEER_TAGS_CACHE lookups. The canonical MetricKey is now built
once per unique key at insert time, in MetricKeys.fromSnapshot.
Behavioral change in the cap-overrun path
-----------------------------------------
The old LRUCache evicted least-recently-used: at cap, a new insert would
push out the oldest entry regardless of whether it was live or stale.
AggregateTable instead scans for a hitCount==0 entry to recycle, and drops
the new key if none exists. Practical impact: in the common case where
the table holds a stable set of recurring keys, an unrelated burst of new
keys is dropped (and reported via onStatsAggregateDropped) rather than
evicting the established keys. The existing test that asserted "service0
evicted in favor of service10" is updated to assert the new semantics.
The other cap-related test ("should not report dropped aggregate when
evicted entry was already flushed") still passes unchanged: after report()
clears all entries to hitCount=0, the next wave of inserts recycles them.
Threading fix
-------------
ConflatingMetricsAggregator.disable() used to call aggregator.clearAggregates()
and inbox.clear() directly from the Sink's IO event thread, racing with the
aggregator thread mid-write. The race was tolerable for LinkedHashMap; it
is not for AggregateTable (chain corruption can NPE or loop). disable()
now offers a ClearSignal to the inbox so the aggregator thread itself
performs the table clear and the inbox.clear(). Adds one SignalItem
subclass + one branch in Drainer.accept; preserves the single-writer
invariant for AggregateTable end-to-end.
Removed: LRUCache import, AggregateExpiry inner class, the static
buildMetricKey / materializePeerTags / encodePeerTag helpers (now in
MetricKeys).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
MetricKey existed for two reasons -- the prior LRUCache key role (now handled by AggregateTable's Hashtable.Entry mechanics) and as the labels argument to MetricWriter.add. The first is gone; the second is the only thing keeping MetricKey alive. Fold its UTF8-encoded label fields onto AggregateEntry, change MetricWriter.add to take AggregateEntry directly, and delete MetricKey + MetricKeys. What AggregateEntry now holds ----------------------------- - 10 UTF8BytesString label fields (resource, service, operationName, serviceSource, type, spanKind, httpMethod, httpEndpoint, grpcStatusCode, and a List<UTF8BytesString> peerTags for serialization). - 3 primitives (httpStatusCode, synthetic, traceRoot). - AggregateMetric (the value being accumulated). - The raw String[] peerTagPairs is retained alongside the encoded peerTags -- matches() compares it positionally against the snapshot's pairs; the encoded form is only consumed by the writer. matches(SpanSnapshot) compares the entry's UTF8 forms to the snapshot's raw String / CharSequence fields via content-equality (UTF8BytesString.toString() returns the underlying String in O(1)). This closes a latent bug in the prior raw-vs-raw matches(): if one snapshot delivered a tag value as String and a later snapshot delivered the same content as UTF8BytesString, the old Objects.equals would return false and the table would split into two entries. Content-equality matching collapses them into one. Consolidated caches ------------------- The static UTF8 caches that used to live partly on MetricKey (RESOURCE_CACHE, OPERATION_CACHE, SERVICE_SOURCE_CACHE, TYPE_CACHE, KIND_CACHE, HTTP_METHOD_CACHE, HTTP_ENDPOINT_CACHE, GRPC_STATUS_CODE_CACHE, SERVICE_CACHE) and partly on ConflatingMetricsAggregator (SERVICE_NAMES, SPAN_KINDS, PEER_TAGS_CACHE) are all now on AggregateEntry. The split was duplicating work -- SERVICE_NAMES and SERVICE_CACHE both cached service-name to UTF8BytesString. One cache per field now. API change: MetricWriter.add ---------------------------- Was: add(MetricKey key, AggregateMetric aggregate) Now: add(AggregateEntry entry) The aggregate lives on the entry. Single-arg. SerializingMetricWriter reads the same UTF8 fields off AggregateEntry that it previously read off MetricKey; the wire format is byte-identical. Test impact ----------- AggregateEntry.of(...) takes the same 13 positional args new MetricKey(...) took, so test diffs are mostly mechanical: new MetricKey(args) -> AggregateEntry.of(args) writer.add(key, _) -> writer.add(entry) ValidatingSink in SerializingMetricWriterTest now iterates List<AggregateEntry> directly. ConflatingMetricAggregatorTest's Spock matchers (~36 sites) rely on AggregateEntry.equals comparing the 13 label fields (not the aggregate) so the mock matches by labels regardless of the aggregate state at call time; post-invocation closures verify aggregate state. Benchmarks (2 forks x 5 iter x 15s) ----------------------------------- The change is consumer-thread only; producer publish() is unchanged. SimpleSpan bench: 3.123 +- 0.025 us/op (prior: 3.119 +- 0.018) DDSpan bench: 2.412 +- 0.022 us/op (prior: 2.463 +- 0.041) Both within noise -- the win is structural (one less class, one less allocation per miss, one fewer cache layer) rather than benchmarked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dougqh
force-pushed
the
dougqh/optimize-metric-key
branch
from
050a998 to
3738c85
Compare
Replaces the per-field DDCache layer inside AggregateEntry with the two new
cardinality handlers. Each per-field handler holds a small HashMap working
set; when its budget is exhausted, subsequent values collapse to a stable
"blocked_by_tracer" sentinel UTF8BytesString rather than growing without
bound. The handlers are reset on the aggregator thread at the end of each
report() cycle (10s default), so the cardinality budget refreshes per
reporting interval.
Caches replaced (limits preserved from the prior DDCache sizes):
RESOURCE_HANDLER 32
SERVICE_HANDLER 32
OPERATION_HANDLER 64
SERVICE_SOURCE_HANDLER 16
TYPE_HANDLER 8
SPAN_KIND_HANDLER 16
HTTP_METHOD_HANDLER 8
HTTP_ENDPOINT_HANDLER 32
GRPC_STATUS_CODE_HANDLER 32
PEER_TAG_HANDLERS per-tag-name TagCardinalityHandler, each 512
Two production-only changes to the handlers as the user wrote them:
- Fixed import: datadog.collections.tagmap6lazy.TagMap doesn't exist;
TagCardinalityHandler now imports datadog.trace.api.TagMap which has the
Entry API the handler uses.
- Added TagCardinalityHandler.register(String) overload so AggregateEntry's
peer-tag canonicalization doesn't have to allocate a TagMap.Entry per
call -- the snapshot already carries peer-tag values as a flattened
String[] {name, value, ...}.
AggregateEntry split into two construction paths:
- forSnapshot(snapshot, agg): the hot path; runs each field through the
appropriate handler.
- of(...): test-only factory; bypasses the handlers and creates UTF8
instances directly, so tests don't pollute static handler state. Content-
equality on the resulting entry still matches the production-built one.
Thread-safety: handlers are HashMap-backed and not safe for concurrent
access. Both forSnapshot and resetCardinalityHandlers must be called from
the aggregator thread. After the prior commits that moved MetricKey
construction to the aggregator thread, this is the only thread that
canonicalizes; the test factory path runs on test threads but doesn't
touch the handlers.
Reset semantics: clearing the handler's working set drops the {value ->
UTF8BytesString} mapping but doesn't invalidate existing AggregateEntry
fields -- those keep their UTF8BytesString references alive on their own.
Subsequent snapshots with the same content still resolve to the existing
entries via content-equality matches(). New values after reset get freshly
allocated UTF8BytesStrings via the handler.
Known limitation (not fixed here): hashOf(SpanSnapshot) hashes from the
raw snapshot fields, not from the post-handler canonical form. So when
cardinality is exceeded, multiple distinct raw values that collapse to
the "blocked_by_tracer" sentinel still produce distinct hashes and land
in different AggregateEntry buckets -- the wire payload will carry
multiple rows that all label as blocked. This is the same behavior the
prior DDCache-based design would have had at capacity. Collapsing those
into a single sentinel entry would require canonicalizing before hashing
and is a follow-up.
Tests: new CardinalityHandlerTest covers PropertyCardinalityHandler and
TagCardinalityHandler in isolation (hit/miss, over-limit blocking, reset
behavior, sentinel stability). Existing ConflatingMetricAggregatorTest /
SerializingMetricWriterTest / AggregateTableTest all pass unchanged
because the test factory bypasses handlers.
Benchmarks (2 forks x 5 iter x 15s) -- producer side unchanged because
the handlers live on the consumer thread:
SimpleSpan bench: 3.114 +- 0.045 us/op (prior: 3.123 +- 0.018)
DDSpan bench: 2.364 +- 0.113 us/op (prior: 2.412 +- 0.022)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The prior commit ran every snapshot through the cardinality handlers but still
hashed the raw snapshot fields. When a field exceeded its cardinality budget
the handlers collapsed many distinct values to a single "blocked_by_tracer"
sentinel, but the raw hashes were still all different -- so the blocked entries
fragmented across the AggregateTable. This commit makes hash + match work off
the canonical (post-handler) UTF8BytesString fields, so blocked values land in
the same bucket and merge into one entry.
How the lookup path changes
---------------------------
A new package-private AggregateEntry.Canonical scratch buffer:
- holds the 10 canonical UTF8BytesString refs, primitives, peerTags list,
and the precomputed keyHash;
- exposes populate(SpanSnapshot) which runs each field through the
appropriate handler and computes the long hash from the canonical refs;
- exposes matches(AggregateEntry) for content-equality lookup;
- exposes toEntry(AggregateMetric) which copies its refs into a fresh
AggregateEntry on miss.
AggregateTable holds one Canonical instance and reuses it per findOrInsert.
On a hit nothing is allocated -- the buffer's refs feed the bucket walk and
matches() directly. On a miss the refs are copied into the new entry and the
buffer is overwritten on the next call.
Hash function
-------------
hashOf now takes UTF8BytesString fields (plus primitives + peerTags list)
instead of raw CharSequence/String from the snapshot. UTF8BytesString.hashCode
returns the underlying String's hash, so:
- content-equal entries built via AggregateEntry.of(...) (test factory,
bypasses handlers) produce the same hash as entries built via
Canonical.toEntry(...) (production, via handlers);
- all values that collapsed to "blocked_by_tracer" share that sentinel
instance and therefore that hashCode -- they land in the same bucket and
merge into one entry.
Matches
-------
The SpanSnapshot-keyed matches() on AggregateEntry is gone. Lookup goes
through Canonical.matches(entry) which compares the buffer's UTF8 fields
against the entry's UTF8 fields via Objects.equals (content equality on
UTF8BytesString). This is needed because across handler resets the
UTF8BytesString instance referenced by an existing entry differs from the
freshly-issued instance for the same content -- content-equality lets the
existing entry survive resets.
The peerTagPairsRaw field on AggregateEntry was previously kept for matching
against snapshot.peerTagPairs (the flat String[]). Canonical.matches uses
List.equals on the encoded UTF8 peerTags directly, so peerTagPairsRaw is
dropped.
New test in AggregateTableTest -- cardinalityBlockedValuesCollapseIntoOneEntry
inserts 50 distinct services into a table whose SERVICE_HANDLER has a
cardinality limit of 32, and asserts the final size is 33 (the 32 in-budget
services plus a single collapsed "blocked_by_tracer" entry, not 50 separate
entries).
Benchmarks (2 forks x 5 iter x 15s) -- producer side unchanged:
SimpleSpan bench: 3.117 +- 0.026 us/op (prior: 3.114 +- 0.045)
DDSpan bench: 2.344 +- 0.114 us/op (prior: 2.364 +- 0.113)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…chema-indexed handlers
Replaces the producer's early {@code (name, value)}-pair encoding with a
schema-based design: peer-tag values are captured into a parallel String
array, and the consumer applies the matching {@link TagCardinalityHandler}
by index using a {@link PeerTagSchema}'s parallel name/handler arrays.
This removes the {@code Map<String, TagCardinalityHandler>} the prior commit
left in {@code AggregateEntry} -- handler lookup is now a single array
dereference instead of a hashmap probe.
PeerTagSchema
-------------
New package-private class that holds:
- {@code String[] names} -- peer-tag names in stable order
- {@code TagCardinalityHandler[] handlers} -- parallel to names
Two schemas exist: a static singleton {@code INTERNAL} for the internal-kind
{@code base.service} case, and a {@code CURRENT} schema for the peer-
aggregation kinds (client/producer/consumer) that lazily refreshes when
{@code features.peerTags()} returns a different set of names.
Each {@link SpanSnapshot} captures the schema reference it was built against
so producer and consumer agree on the indexing even if {@code CURRENT}
changes between capture and consumption.
A fast-path identity check (cached last input Set instance) keeps the
{@code currentSyncedTo} call cheap: when the producer hands in the same
Set instance as last time -- the steady-state case --
{@code currentSyncedTo} returns immediately without iterating names. The
{@code matches()} loop only runs when the Set instance changes, which in
production is rare (only on remote-config reconfiguration).
Snapshot shape
--------------
{@code SpanSnapshot.peerTagPairs} (a flat {@code [name0, value0, name1,
value1, ...]} array) is replaced by:
- {@code PeerTagSchema peerTagSchema} -- nullable; schema for the values
- {@code String[] peerTagValues} -- parallel to schema.names
The producer captures only values; the consumer constructs the encoded
{@code "name:value"} UTF8 forms via {@code schema.handler(i).register(value)}
on its own thread.
Consumer-side cleanups bundled in
---------------------------------
While here, also addresses the perf review items raised against the prior
commit:
- {@code hashOf}'s peer-tag loop is now indexed iteration; no more
iterator allocation per snapshot.
- {@code Canonical} now owns a reusable {@code peerTagsBuffer} ArrayList
that's cleared+refilled per {@code populate} call -- zero allocation
on the hit path. The buffer is copied into an immutable list only on
miss when the entry needs to own it long-term.
- {@code Canonical.matches} uses indexed list comparison; no iterator
alloc in {@code List.equals}.
- The {@code HashMap<String, TagCardinalityHandler> PEER_TAG_HANDLERS}
on {@code AggregateEntry} is gone, replaced by the {@link
PeerTagSchema}'s parallel array layout.
Benchmark (2 forks x 5 iter x 15s)
----------------------------------
SimpleSpan bench: 3.165 +- 0.032 us/op (prior: 3.117 +- 0.026)
DDSpan bench: 2.727 +- 0.018 us/op (prior: 2.344 +- 0.114)
Some producer-side regression from the per-snapshot schema sync (volatile
read + identity check). The fast-path identity comparison keeps it small;
hoisting the sync out of the per-snapshot loop is possible but would change
behavior in the edge case where {@code features.peerTags()} returns
different Sets within a single trace (covered by an existing test). Choosing
correctness over the marginal speedup.
Tests
-----
AggregateTableTest's snapshot builder is updated to construct a schema +
values via {@code PeerTagSchema.currentSyncedTo}, exercising the same code
path as production. Existing peer-tag test in {@code
ConflatingMetricAggregatorTest} still passes unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "Conflating" in the name dates from the prior design that used a Batch pool + pending map to conflate up to 64 hits per inbox slot. That mechanism is gone -- the producer now publishes one SpanSnapshot per span and the consumer's AggregateTable is the conflation point. The new name matches the existing protocol/metric terminology (HealthMetrics.onClientStat*, stats.flush_payloads, etc.). File renames: ConflatingMetricsAggregator.java -> ClientStatsAggregator.java ConflatingMetricAggregatorTest.groovy -> ClientStatsAggregatorTest.groovy ConflatingMetricsAggregatorBenchmark -> ClientStatsAggregatorBenchmark ConflatingMetricsAggregatorDDSpan* -> ClientStatsAggregatorDDSpan* Plus all symbol references in MetricsAggregatorFactory and the test fixtures that referenced the old class name. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three small follow-ups carried over from a /techdebt pass: - TracerHealthMetrics: previousCounts array was sized 51, but the prior commits added a 52nd reporter (statsInboxFull). Without this fix the new counter's report() call would throw ArrayIndexOutOfBoundsException; the Flush task swallows that exception, so the failure would be silent (statsInboxFull would just never make it to statsd). - Aggregator: removes the now-dead public clearAggregates() method. The ClearSignal route from ClientStatsAggregator.disable() supplanted it several commits ago; the method had no remaining callers. - TagCardinalityHandler: removes the unused register(TagMap.Entry) overload and its isValidType helper. The String-keyed overload covers all current callers (AggregateEntry's peer-tag canonicalization). - PeerTagSchema: spotless-driven javadoc reflow only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ClientStatsAggregator.publish was calling features.peerTags() + PeerTagSchema.currentSyncedTo for every span. Peer-tag configuration is stable for the duration of a single trace publish in production -- DDAgentFeaturesDiscovery returns the same Set instance until remote-config reconfiguration -- so the per-snapshot sync is wasted work. Move the sync to once per publish(trace) and pass the resolved schema to the inner publish(span, isTopLevel, peerAggSchema). INTERNAL-kind spans still use the static PeerTagSchema.INTERNAL regardless. Behavior boundary ----------------- Schema changes from features.peerTags() now take effect at the next publish(trace) call rather than mid-trace. Production-equivalent (a trace takes microseconds to milliseconds; remote-config refreshes are seconds apart), but a Spock test that used `>>> [...]` to mock different peerTags() returns on successive calls within one trace no longer makes sense in the new model. That test is rewritten to assert the production-relevant case: peer-tag NAMES are stable, peer-tag VALUES vary per span, distinct value combinations produce distinct aggregate buckets. Benchmark (2 forks x 5 iter x 15s) ---------------------------------- SimpleSpan bench: 3.133 +- 0.057 us/op (prior: 3.165 +- 0.032) DDSpan bench: 2.454 +- 0.082 us/op (prior: 2.727 +- 0.018) Recovers ~270 ns/op on the DDSpan bench -- most of the regression introduced by the per-snapshot lookup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
JFR profiling showed ~21% of producer CPU time spent in tag-map lookups during ClientStatsAggregator.publish. One of those lookups -- span.kind -- is redundant because DDSpanContext already caches the kind as a byte ordinal that resolves to a String via a small array. - Add CoreSpan.getSpanKindString() with a default that falls back to the tag map for non-DDSpan impls; DDSpan overrides to delegate to the context's cached resolution. - Hoist schema.names array out of the capturePeerTagValues loop. - Avoid an unnecessary toString() in isSynthetic by declaring SYNTHETICS_ORIGIN as String and using contentEquals. Benchmark (ClientStatsAggregatorDDSpanBenchmark): before: 2.410 us/op after: 1.995 us/op (~17% improvement) vs. master baseline (6.428 us/op): now ~3.2x faster. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the producer/consumer split, the canonical-key trick that makes cardinality-blocking actually save space, the once-per-trace peer-tag schema sync, the role of each file in datadog.trace.common.metrics, and the rationale behind the redesign from ConflatingMetricsAggregator. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dougqh
force-pushed
the
dougqh/control-tag-cardinality
branch
from
8020ec4 to
1221b2b
Compare
Both classes existed only to support tests against AggregateEntry -- one for positional-args fixture construction, the other for value- based equality matching. The split was artificial; folding them into a single AggregateEntryTestUtils removes a file and gives test sites one place to look for AggregateEntry test helpers. - Move `of(...)` into AggregateEntryTestUtils alongside the existing `equals(a, b)` / `hashCode(e)` helpers. - Delete AggregateEntryFixtures.java. - Rename 51 caller sites across ConflatingMetricAggregatorTest and SerializingMetricWriterTest. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… dougqh/control-tag-cardinality # Conflicts: # dd-trace-core/src/test/groovy/datadog/trace/common/metrics/ClientStatsAggregatorTest.groovy # dd-trace-core/src/test/java/datadog/trace/common/metrics/AggregateEntryFixtures.java # dd-trace-core/src/test/java/datadog/trace/common/metrics/AggregateEntryTestUtils.java
Two doc-only additions surfacing design context that reviewers would otherwise have to reconstruct: - AggregateEntry: name the "5 responsibilities concentrated on one object" tradeoff explicitly (UTF8 caches + label fields + raw peerTag arrays + encoded peerTag list + counter/histogram state). Prior MetricKey + AggregateMetric design allocated two objects per unique key on miss; folding them yields one. The class is wider as a result; that's the trade we chose. - AggregateEntry + AggregateTable: note that the single-writer invariant is convention-enforced -- the @SuppressFBWarnings documents the assumption but nothing checks the calling thread at runtime. Point to ClearSignal as the explicit mechanism for funneling cross-thread mutators back onto the aggregator thread. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… dougqh/control-tag-cardinality # Conflicts: # dd-trace-core/src/main/java/datadog/trace/common/metrics/AggregateEntry.java
On the miss path, AggregateTable.findOrInsert computed the snapshot hash for the lookup, then AggregateEntry.forSnapshot computed it again via the same hashOf(s) call to set keyHash on the new entry. Three reads per snapshot field on a miss (findOrInsert hashOf + forSnapshot hashOf + constructor canonicalize), with two of those also paying for the per-call Arrays.hashCode(peerTagValues). Pass the hash that findOrInsert already computed into forSnapshot instead. Two reads per field on miss, one Arrays.hashCode(peerTagValues) per miss. Kept a no-arg forSnapshot overload for test callers that don't have a precomputed hash on hand. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
- Drop unused PeerTagSchema.hashCode/equals + cachedHashCode field; the schema is never compared via Object.equals or used as a Set/Map key. hasSameTagsAs(Set<String>) is the only schema-equivalence primitive in use, and it's a name-content check that doesn't go through hashCode. - Canonical.populatePeerTags now skips null values directly instead of round-tripping them through handler.register only to filter EMPTY back out. Cheap win on sparse-null peer-tag arrays from capturePeerTagValues. - Canonical.matches reordered to compare highest-cardinality fields (resource, service, operationName) first for faster short-circuit on bucket-chain collisions; UTF8 fields use direct .equals (the EMPTY-sentinel invariant guarantees non-null on both sides) instead of Objects.equals + dead null branches. - PropertyCardinalityHandler/TagCardinalityHandler.register compute the mixed hash once and reuse the start index for both the current-cycle and prior-cycle probe paths. The probe helper is inlined since it's no longer shared. - ClientStatsAggregator.reconcilePeerTagSchema now flushes the outgoing schema's accumulated blockedCounts via resetCardinalityHandlers() before discarding it on a tag-set change, otherwise partial-cycle block telemetry would silently disappear. - Document the "one ClientStatsAggregator per JVM" invariant implied by AggregateEntry's static cardinality handlers and PeerTagSchema.INTERNAL, plus the load-bearing size guard in TagCardinalityHandler.isBlockedResult that prevents accidental sentinel materialization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…leton contract AggregateEntry.clear(): note that only per-cycle counters/histograms reset; the label fields (resource, service, ..., peerTagNames, peerTagValues) are the entry's bucket identity and persist across cycles so subsequent same-key snapshots reuse the entry. Stale entries get reaped by AggregateTable.expungeStaleAggregates. SignalItem: document the singleton fire-and-forget contract -- the inherited CompletableFuture is completed on first handling and never reset, so callers that want one-shot completion semantics (e.g. forceReport) must allocate a fresh instance instead of reusing the STOP/REPORT/CLEAR singletons. Pre-existing pattern on master (this PR added the CLEAR singleton following the same convention); doc just makes the contract explicit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…dinality # Conflicts: # dd-trace-core/src/main/java/datadog/trace/common/metrics/AggregateEntry.java # dd-trace-core/src/main/java/datadog/trace/common/metrics/AggregateTable.java
dougqh
commented
May 27, 2026
|
|
||
| private synchronized void discoverIfOutdated(final long maxElapsedMs) { | ||
| final long now = System.currentTimeMillis(); | ||
| final long elapsed = now - discoveryState.lastTimeDiscovered; |
Contributor
Author
There was a problem hiding this comment.
Claude, this change is unnecessary
…matches Optional fields on AggregateEntry (serviceSource, httpMethod, httpEndpoint, grpcStatusCode) carry UTF8BytesString.EMPTY when the snapshot had no value. SerializingMetricWriter (and its test) previously read this contract via reference-eq against EMPTY, leaking the storage choice into callers. Add hasServiceSource / hasHttpMethod / hasHttpEndpoint / hasGrpcStatusCode predicates on AggregateEntry; switch the four serializer sites and the mirroring four test sites to call them. The EMPTY-sentinel is now an internal implementation detail of AggregateEntry. Reorder AggregateEntry.hashOf so its field order mirrors Canonical.matches (UTF8 fields first, then peer-tag list, then primitives). The hash value itself is order-stable across all callers; this is purely so future readers can reason about lookup and equality in lockstep. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When peer-tag discovery returns a different tag set than the cached schema carries, ClientStatsAggregator.reconcilePeerTagSchema replaces the schema. Previously every per-tag TagCardinalityHandler was rebuilt from scratch, losing the prior-cycle UTF8 cache for tags that survived the rebuild -- so a persistent tag like peer.hostname re-allocated UTF8BytesStrings for every value on the cycle following the rebuild. Split PeerTagSchema.resetCardinalityHandlers into two operations: - resetCardinalityHandlers: full rotate (called at end-of-cycle on the cached schema). - flushBlockedCounts: telemetry-only flush (used by reconcile on the outgoing schema before discard, so partial-cycle counts still reach HealthMetrics without disturbing the handlers). Add a donor overload PeerTagSchema.of(names, state, healthMetrics, previous) that transfers TagCardinalityHandler instances by name for any tag present in both the previous and replacement schemas. Names absent from the previous schema get fresh handlers; names absent from the replacement schema are dropped along with the outgoing schema. reconcilePeerTagSchema now calls flushBlockedCounts then constructs the replacement via the donor overload. The end-of-cycle reset that runs immediately after reconcile rotates the (now-transferred) handlers in the normal way, so cycle N+1 sees cycle N's UTF8 values as priorValues for persisting tags. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ebuilds" This reverts commit 703c9e1.
Cardinality limiting alters the wire format under high cardinality -- overflow values get the "blocked_by_tracer" sentinel and collapse into one aggregate bucket. Customers with dashboards or alerts keyed on specific tag values would see the sentinel value appear unexpectedly if their workload exceeds the per-field budgets. Put the substitution behavior behind a config flag so the rollout is opt-in. With the flag off (the new default), the per-field handlers still act as bounded UTF8 reuse caches sized by their cardinality budget, but over-cap values get a freshly-allocated UTF8BytesString instead of the sentinel and flow to distinct aggregate buckets. The wire format is unchanged from master for any workload. With the flag on, current behavior (sentinel substitution + bucket collapse). Handler refactor: - PropertyCardinalityHandler / TagCardinalityHandler take a useBlockedSentinel constructor arg. On cap-exhaust the cache no longer claims a slot (since it's full), but prior-cycle reuse still runs so repeat over-cap values pay only the probe, not the allocation. - TagCardinalityHandler.isBlockedResult now reads cacheBlocked directly rather than calling blockedByTracer(), so query-time never forces the sentinel to materialize when limits are disabled. - Test-convenience single-arg constructors default useBlockedSentinel to true so existing tests of the limits-enabled mode don't churn; new tests cover the disabled mode. Wiring: - AggregateEntry exposes static final LIMITS_ENABLED read from Config.get() at class init, threaded through all PropertyCardinality Handlers and the PeerTagSchema-built TagCardinalityHandlers. Config: - GeneralConfig.TRACE_STATS_CARDINALITY_LIMITS_ENABLED (default false) - Config.isTraceStatsCardinalityLimitsEnabled() - Registered in metadata/supported-configurations.json Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three small doc additions calling out behavior that's correct in code but easy to miss on a cold read: - AggregateTable.evictOneStale: explain that this is no longer just a pathological-case backstop. With LIMITS_ENABLED=false (the new default), over-cap values flow to distinct buckets and maxAggregates becomes the load-bearing cardinality enforcement -- the cursor- resumed scan was added for this regime. - AggregateEntry.LIMITS_ENABLED: document the over-cap repeat tradeoff in disabled mode (over-cap values can't promote into the current cache so repeats re-allocate) and the class-init caveat (static final read of Config, frozen for the JVM at first class load -- tests needing to exercise the limits-on path through the static handlers must construct handlers directly with explicit useBlockedSentinel args). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
spotbugs now flags three suppression annotations as unnecessary: - Class-level AT_NONATOMIC_OPERATIONS_ON_SHARED_VARIABLE + AT_STALE_THREAD_WRITE_OF_PRIMITIVE — the int counter fields are no longer mutated cross-thread now that producer threads only enqueue SpanSnapshots and the aggregator thread is the sole writer. - clear() AT_NONATOMIC_64BIT_PRIMITIVE on the duration field — same reason; the long write is single-threaded. The class Javadoc already documents the single-writer invariant, so removing the annotations doesn't lose any documentation; the prose paragraph that referenced "the SuppressFBWarnings below" is updated in place. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "use the TestAggregateEntry subclass in src/test" reference pointed to a subclass that was replaced earlier in the stack by the AggregateEntryTestUtils helper class. Test-side value-equality is now a helper, not a subclass; AggregateEntry stayed final. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three small cleanups that the recent design review surfaced: - Move test-only AggregateEntry.forSnapshot(SpanSnapshot) to AggregateEntryTestUtils. Production callers (AggregateTable.findOrInsert) already use the two-arg forSnapshot(snap, keyHash); the no-keyHash overload existed for tests. AggregateEntryTest now goes through the test helper. MetricsIntegrationTest can't see src/test, so it inlines forSnapshot(snap, hashOf(snap)) using the production API directly. - Change AggregateEntry.recordOneDuration to return void. Returned `this` for fluent-style chaining but the only caller (Aggregator.accept) discards the return. - Remove PeerTagSchema.hashCode/equals + cachedHashCode field. Used only by AggregateEntry.hashOf, which now inlines Arrays.hashCode(schema.names) with an explicit null guard. Drops 42 lines from PeerTagSchema and three now-redundant equals tests from PeerTagSchemaTest -- the schema's identity contract is enforced by the hash function and hasSameTagsAs rather than the Object#equals contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ntions Five small cleanups surfaced by the design re-review: - Drop AggregateEntry.forSnapshot(SpanSnapshot, long). It wrapped the private constructor for no reason; make the constructor package- private and have AggregateTable.findOrInsert and AggregateEntryTestUtils.forSnapshot call it directly. - Class-level Javadoc now documents the required-vs-optional field absence convention: required fields canonicalize null -> EMPTY, optional fields stay null so the serializer's `!= null` check works. Previously a reader had to infer it from the constructor body. - Field Javadocs on `synthetic` (synthetic-monitoring origin tag) and `traceRoot` (parentId == 0). Both make it onto the wire; neither was obvious to a fresh reader. - Tighten the `peerTagNames` / `peerTagValues` field comment. The previous wording implied package-private was for "test-only" access; in fact production matches() reads them from within the class and the test helper is just one consumer. - Add a `canonicalizeOptional` helper that mirrors `canonicalize` but returns null (not EMPTY) for null input. Folds the four optional- field assignments in the constructor from three-line ternaries into one-liners. Keeps the `instanceof UTF8BytesString` short-circuit consistent across all label fields -- dead code for the String-typed optionals (httpMethod/Endpoint/grpcStatusCode), live for the CharSequence-typed serviceNameSource. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…dinality # Conflicts: # dd-trace-core/src/main/java/datadog/trace/common/metrics/AggregateEntry.java # dd-trace-core/src/main/java/datadog/trace/common/metrics/AggregateTable.java # dd-trace-core/src/main/java/datadog/trace/common/metrics/PeerTagSchema.java # dd-trace-core/src/test/java/datadog/trace/common/metrics/AggregateEntryTest.java # dd-trace-core/src/test/java/datadog/trace/common/metrics/AggregateEntryTestUtils.java # dd-trace-core/src/test/java/datadog/trace/common/metrics/PeerTagSchemaTest.java
Flagged by codenarcTraceAgentTest (UnusedImport rule). Left over from a prior rewrite of the entry-construction flow. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each AggregateEntry allocated two DDSketchHistograms in its constructor (ok + error latencies). DDSketchHistogram wraps a DDSketch + lazy store, roughly 60-80 bytes per histogram even when empty. Most spans aren't errors, so most entries' errorLatencies sit empty for life. Now the field starts null. recordOneDuration lazy-allocates on the first error; if no error ever lands on the entry, it stays null and ~80 bytes of empty-histogram overhead are reclaimed. Across a full 2048-entry table that's ~150 KB if 95% of entries never error -- the typical case. For the wire format, SerializingMetricWriter caches the serialized form of an empty histogram (~17 bytes) on first use and writes those cached bytes when an entry's errorLatencies is null. The cache is per-writer (not a global static) so each writer instance picks up the Histograms factory state at the time of its first report, avoiding races with test setup that registers the DDSketch factory at varying points. Trade-off: entries that DO see an error retain the histogram across clear() (just cleared, not nulled), so always-erroring entries allocate exactly once. Same total allocation as before for that case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 tasks
dougqh
changed the base branch from
dougqh/optimize-metric-key
to
dougqh/lazy-error-latencies
dougqh
force-pushed
the
dougqh/lazy-error-latencies
branch
from
f2ee559 to
0c658dd
Compare
Conflicts resolved by taking HEAD (cardinality branch) for all metrics-package files; master added ConflatingMetricsAggregatorDisableTest which duplicates the already-renamed ClientStatsAggregatorDisableTest — removed the old-named copy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Implements per-component / per-tag cadinality limits to improve user experience under high load.
Motivation
Previously, there was a single global cap and per-component / tag caches that helped curtail allocation and unbound live objects, but this approach had a couple problems.
While the aggregate table was sized capped, failure to insert into the aggregate table would lead to silent data loss. There weren't any obvious indications to the customer when metrics were lost.
And under extreme loads, the caches would degenerate to constantly missing and allocating which would result in long GC cycles.
The per-element limiting allows us to substitute a sentinel value to indicate what was dropped and why to the trace agent / backend. Additionally, this change includes logging and metrics to indicate to the user what is happening locally.
Additional Notes
The cardinality handlers introduced in this change serve dual roles. They both track cardinality and act as caches for UTF8 encodings.
By cardinality limiting first, constant allocation from string concatenation and UTF8 encoding is avoided. And given that a cache and cardinality limiter are basically both sets of recently used values, it seemed most efficient to combine them.
The one difference between the cardinality limiter and the cache is that the cardinality limiter is regularly fully reset -- which hurts the use of the limiter as a cache. To make up for that, the cardinality limiter also holds onto the values used in the previous cycle for reuse in the new cycle.
Claude's Summary
Stack: master → #11382 → #11478 → this PR. Bounds client-stats label cardinality, reworks peer-tag handling, renames the aggregator, and adds a design doc. The lazy-errorLatencies memory win that originally lived in this PR's downstream (#11389) was extracted ahead of #11387 into #11478 during a stack resequence, so the per-entry footprint reduction lands independently of the cardinality machinery.
DDCaches withPropertyCardinalityHandler/TagCardinalityHandler. Each has a per-field budget; once exhausted, the sentinel-substitution behavior is gated by the newtrace.stats.cardinality.limits.enabledflag (defaultfalse). With the flag on, overflow values canonicalize to ablocked_by_tracersentinelUTF8BytesStringand collapse to one bucket. With the flag off (the default), the cache size is still capped at the same budget but over-cap values get freshly-allocatedUTF8BytesStrings and flow to distinct buckets — so the wire format is identical to Update client-side stats to use light weight Hashtable #11382. Handlers reset every reporting cycle in either mode.AggregateTable.findOrInsertruns every label through its handler before computing the lookup hash, so cardinality-blocked values collapse into one bucket instead of fragmenting into N entries.String[]to aPeerTagSchema.namesarray).tag:valueinterning happens on the aggregator thread viaTagCardinalityHandler. The schema is synced once per trace viaPeerTagSchema.currentSyncedTo(Set)with an identity-check fast path.ConflatingMetricsAggregator→ClientStatsAggregator.span.kindbyte ordinal through a newCoreSpan.getSpanKindString()(skips a tag-map lookup per metrics-eligible span); hoistschema.namesout ofcapturePeerTagValues; avoidtoString()allocation inisSynthetic.TracerHealthMetrics.previousCountssize bug that would have silently dropped the newstatsInboxFullcounter; drop deadclearAggregates().docs/client_metrics_design.mdcovering the pipeline shape, the canonical-key trick, thread-safety contract, reporting cadence, failure modes, and benchmark numbers.Benchmark
ClientStatsAggregatorDDSpanBenchmark— producer publish() latency(64 client-kind DDSpans per op, real
CoreTracer)~3.2× over master end to end on the producer side.
Aggregator bench suite vs v1.62.0 + master + #11382
Re-measured 2026-05-27 with three benches in matrix form: full adversarial (all four label dimensions vary) and two cardinality-isolation companions (only
resourcevaries; onlypeer.hostnamevaries). Same machine state, same JMH config (8 producer threads, 2×15s warmup + 5×15s, 1 fork, throughput mode). TheHighCardinality*andAdversarialbenches were backported onto the v1.62.0 tag using itsConflatingMetricsAggregatorconstructor andHealthMetrics.NO_OP(v1.62.0 predates the inbox split so per-iteration drop counters are not directly comparable).AdversarialMetricsBenchmark(ops/s)HighCardinalityResource(ops/s)HighCardinalityPeer(ops/s)onStatsAggregateDroppedHealthMetrics.NO_OP)onStatsAggregateDroppedonStatsAggregateDroppedCustomer headline: vs the shipping v1.62.0 release, this branch at the default flag setting (limits OFF) delivers ~50× throughput on adversarial cardinality and ~5–6× on single-axis cardinality. With the flag ON (sentinel-substitution active), ~18× / 4–5× plus zero
onStatsAggregateDropped— i.e. the cardinality cap actually saves the bench from data loss. v1.62.0's Adversarial per-iteration progression shows the classic degradation curve (1.08M warmup → 277K → 199K) where the LRU cache thrashes catastrophically; this PR holds steady-state across iterations in either flag mode.Reading the trade-off:
onStatsAggregateDropped = 0only with limits ON. That's the safety guarantee the feature pays for. Every other config drops 10–225 M aggregate updates under adversarial cardinality because over-cap values fragment into distinct buckets and saturatetracerMetricsMaxAggregates.Adversarial-bench limits-on cost is real. All four label dimensions exhaust their cap simultaneously, so every snapshot pays the full sentinel-substitution +
blockedCounts+++warnedCardinalitybookkeeping on all four fields. Single-axis benches (HighCardinality*) show a much smaller limits-on penalty (~10%) because only one dimension is over-cap. Workloads with one runaway dimension and the rest bounded sit much closer to the limits-off throughput.Variance collapses dramatically with limits on. ±1.72 M / ±1.75 M / ±1.93 M on limits-on vs ±5–6.67 M without. Bounded cardinality means no eviction sweeps, stable table size, no per-cycle GC churn — predictable throughput. For workloads paged on p99 latency spikes during reporting cycles, this is often more valuable than peak throughput.
Benches are adversarial. Designed to saturate every capacity bound at once; realistic workloads with smaller working sets see proportionally smaller throughput gaps between configs. The 19–66% limits-on penalty vs Update client-side stats to use light weight Hashtable #11382 is an upper bound, not a steady-state cost.
Architecture note on the limits-off cost. Limits-off matches #11382's wire format exactly, but still costs ~19% on
HighCardinality*and ~28% onAdversarial. The gap comes fromAggregateTable.findOrInsertcanonicalizing every snapshot before lookup — required for the sentinel collapse in limits-on, but pure overhead in limits-off where the hash is content-stable across raw vs canonicalized forms. A two-pathfindOrInsert(hash-raw on limits-off, canonicalize-first on limits-on) would likely close most of the gap; deferred as a follow-up optimization if the default-off cost matters in practice.Test plan
:dd-trace-core:test— metrics tests pass (existing + newAggregateTableTestcases for cardinality collapse)🤖 Generated with Claude Code