-
Notifications
You must be signed in to change notification settings - Fork 338
Per-component / tag cardinality limits in client-side stats #11387
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
dougqh
wants to merge
205
commits into
dougqh/lazy-error-latencies
Choose a base branch
from
dougqh/control-tag-cardinality
base: dougqh/lazy-error-latencies
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
205 commits
Select commit
Hold shift + click to select a range
bc42093
Trim per-span work on metrics aggregator publish path
dougqh 808d63d
Add SpanKindFilter and CoreSpan.isKind for bitmask-based kind checks
dougqh 6aa620e
Use SpanKindFilter in ConflatingMetricsAggregator
dougqh a02d0a9
Add DDSpan-based variant of ConflatingMetricsAggregator JMH benchmark
dougqh ed38f18
Tighten SpanKindFilter encapsulation
dougqh 034afc0
Defer MetricKey construction and cache lookups to the aggregator thread
dougqh 3a056b3
Report aggregator inbox-full drops via health metrics
dougqh 3355865
Merge branch 'master' into dougqh/conflating-metrics-producer-wins
dougqh 950499c
Merge branch 'dougqh/conflating-metrics-producer-wins' into dougqh/co…
dougqh 0294058
Add Hashtable and LongHashingUtils to datadog.trace.util
dougqh f751ab4
Add AggregateTable + AggregateEntry backed by Hashtable
dougqh f1b030a
Swap LRUCache for AggregateTable in Aggregator + route disable() clear
dougqh 3738c85
Eliminate MetricKey: inline its fields onto AggregateEntry
dougqh 129ebd1
Cap per-field metric tag cardinality via Property/TagCardinalityHandler
dougqh 8aab88d
Canonicalize SpanSnapshot before hashing so blocked values collapse
dougqh 9b70705
Defer peer-tag pair construction; capture values + canonicalize via s…
dougqh ceec2af
Rename ConflatingMetricsAggregator to ClientStatsAggregator
dougqh dd372e7
Cleanups: fix previousCounts size, drop dead code
dougqh 3a1d67e
Hoist peer-tag schema sync to once per trace
dougqh fb32366
Use cached span.kind ordinal in metrics producer; drop tag-map lookup
dougqh 1221b2b
Add client metrics pipeline design doc
dougqh 46a9055
Add unit tests for Hashtable and LongHashingUtils
dougqh 031dc89
Apply spotless formatting to Hashtable and LongHashingUtils
dougqh f9e63b9
Add JMH benchmarks for Hashtable.D1 and D2
dougqh a534e4f
Add benchmark results to HashtableBenchmark header
dougqh ba66a36
Address review feedback on Hashtable
dougqh 3108941
Fix dropped argument in HashingUtils 5-arg Object hash
dougqh 1415f12
Address review feedback on Hashtable
dougqh b7cee2f
Drop reflection in iterator tests via package-private D1.buckets
dougqh 8414960
Resize previousCounts for inbox-full health metric
dougqh 553ffb8
Fold AggregateMetric into AggregateEntry
dougqh 07ed635
Avoid capturing lambda in Aggregator.report
dougqh df58ad7
Add context-passing forEach to Hashtable.D1 and D2
dougqh 9c6e95c
Move forEach loop body to Support helper
dougqh 7ab47f7
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
dougqh 590ab4a
Delegate AggregateTable.forEach to Support.forEach
dougqh 447ea33
Move bucket-head cast to Support.bucket helper
dougqh 46f147a
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
dougqh dd5e13f
Use Support.bucket and type chain walks as AggregateEntry
dougqh df7f98f
Drop d1_/d2_ prefix from per-table benchmark methods
dougqh e72fd01
Add DDAgentFeaturesDiscovery.peerTagsRevision()
dougqh dce4b2c
Move peer-tag schema cache from PeerTagSchema statics to ClientStatsA…
dougqh e6ecc16
Add Hashtable.Support helpers: MAX_RATIO, insertHeadEntry, MutatingTa…
dougqh a409d5d
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
dougqh 96b40b8
Simplify AggregateTable via new Hashtable.Support helpers
dougqh 55ca204
Swap MAX_RATIO numerator/denominator pair for a single float + scaled…
dougqh 7b9eccd
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
dougqh 192de0c
Address second-round review on AggregateTable / Aggregator
dougqh 4bac439
Tighten Hashtable docs + rename MAX_CAPACITY to MAX_BUCKETS
dougqh de289a0
Dedupe chain-head splice in D1/D2 via keyHash insertHeadEntry overload
dougqh 2dd65ed
Tighten Entry.next encapsulation; doc hasNext; add D1/D2 getOrCreate
dougqh cc8d98a
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
dougqh 6a9063b
Use keyHash insertHeadEntry overload in AggregateTable.findOrInsert
dougqh 8f4c6f4
Merge dougqh/optimize-metric-key into dougqh/control-tag-cardinality
dougqh 5a52622
Fold AggregateMetric into AggregateEntry
dougqh ece78c9
Remove accidentally-staged .claude/worktrees entries
dougqh dbb1702
Replace // nullable comments with @Nullable annotations on AggregateE…
dougqh 545e74c
Drop redundant load-factor comment from AggregateTable ctor
dougqh 9983a59
Import java.util.Objects in AggregateEntry instead of fully qualifying
dougqh d2e4477
Document evictOneStale cost and disable() best-effort offer
dougqh 24969db
Skip SpanSnapshot allocation when the inbox is already at capacity
dougqh b6c4f5f
Address review on AggregateEntry nullables + PeerTagSchema revision
dougqh 14f7f58
Consolidate cardinality-handler reset behind one entry point
dougqh b953b3a
Parameterize PropertyCardinalityHandler on T extends CharSequence
dougqh 40e8cbd
Add long-lived LRU cache to PropertyCardinalityHandler
dougqh d88a863
Centralize per-field cardinality limits in MetricCardinalityLimits
dougqh d01036f
Reimplement cardinality handlers as open-addressed flat arrays
dougqh 7b6c5f1
Drop parallel keys array in PropertyCardinalityHandler
dougqh 10ca111
Drop type parameter from PropertyCardinalityHandler
dougqh 4610078
Guard cardinality-handler ctor against pathological inputs
dougqh 713aa34
Make EMPTY the universal absent sentinel for AggregateEntry UTF8 fields
dougqh 617bc51
Use EMPTY consistently for absent values in peer-tag canonicalization
dougqh e1dec83
Tighten handler visibility + add tests for EMPTY-on-null contract
dougqh 2336bb5
Notify on peer-tag cardinality blocks
dougqh 5b6c5aa
Address PR #11387 review: dual-role docs, rename, @Nullable, consumer…
dougqh 3c3c8b1
Lock in cardinality-handler prior-cycle UTF8 reuse with explicit tests
dougqh 66ec7f6
Hashtable: add missing braces and detach removed/replaced entries
dougqh be823ab
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
dougqh 10956b2
Add Hashtable and LongHashingUtils to datadog.trace.util
dougqh 035dc09
Add unit tests for Hashtable and LongHashingUtils
dougqh 7728b60
Apply spotless formatting to Hashtable and LongHashingUtils
dougqh 8cd2d86
Add JMH benchmarks for Hashtable.D1 and D2
dougqh c689ef9
Add benchmark results to HashtableBenchmark header
dougqh 75790eb
Address review feedback on Hashtable
dougqh 6056ff7
Fix dropped argument in HashingUtils 5-arg Object hash
dougqh da55021
Address review feedback on Hashtable
dougqh 8b8b088
Drop reflection in iterator tests via package-private D1.buckets
dougqh 0fde7cd
Add context-passing forEach to Hashtable.D1 and D2
dougqh 6d6c2e0
Move forEach loop body to Support helper
dougqh 268de2b
Move bucket-head cast to Support.bucket helper
dougqh 93813b9
Drop d1_/d2_ prefix from per-table benchmark methods
dougqh 11a58bf
Add Hashtable.Support helpers: MAX_RATIO, insertHeadEntry, MutatingTa…
dougqh 8f1828d
Swap MAX_RATIO numerator/denominator pair for a single float + scaled…
dougqh c0d3e26
Tighten Hashtable docs + rename MAX_CAPACITY to MAX_BUCKETS
dougqh a0978ba
Dedupe chain-head splice in D1/D2 via keyHash insertHeadEntry overload
dougqh e604a8f
Tighten Entry.next encapsulation; doc hasNext; add D1/D2 getOrCreate
dougqh e2642cd
Hashtable: add missing braces and detach removed/replaced entries
dougqh 585ca56
Rename LongHashingUtils.hashCodeX(Object) to hash(Object) for API con…
dougqh aab5b3b
Merge remote-tracking branch 'origin/master' into dougqh/optimize-met…
dougqh 88f169f
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
dougqh d61a3c6
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh 795ba76
Merge remote-tracking branch 'origin/master' into dougqh/conflating-m…
dougqh e455801
Introduce slim PeerTagSchema; capture peer-tag values not pairs
dougqh c2ff012
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/…
dougqh 1df8bd2
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh e766fd3
Address PR #11381 review (round 2)
dougqh 8cfa4a5
Cover inbox-full fast-path in ConflatingMetricsAggregator.publish
dougqh 3644470
Reconcile PeerTagSchema once per reporting cycle on the aggregator th…
dougqh 1d1ba4d
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/…
dougqh 57cb018
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh e7d0b42
Add bootstrap + reconcile coverage for PeerTagSchema
dougqh 3d4f380
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/…
dougqh 9391c48
Use writer.finishBucket() count in bootstrap test for cascade compati…
dougqh ba3225c
Use writer.finishBucket() count in bootstrap test for cascade compati…
dougqh 517ef75
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh 9bbe2d0
Rename bootstrap test to ClientStatsAggregator + adapt PeerTagSchemaTest
dougqh ec857b2
Merge branch 'master' into dougqh/conflating-metrics-background-work
dougqh ba0d390
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/…
dougqh 0b86066
Preserve TRACER_METRICS_MAX_PENDING semantic + drop stale imports
dougqh f71384e
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/…
dougqh 5c78dbb
Add AdversarialMetricsBenchmark for capacity-bound stress testing
dougqh 471f8f7
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/…
dougqh 70c20ef
Trim AdversarialMetricsBenchmark counters and clarify printout
dougqh 7a074e0
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/…
dougqh 68848ad
Close PeerTagSchema reconcile race + cover the swap branch
dougqh 3fa7ac7
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/…
dougqh df3b31d
Adapt reconcileSwapsSchemaWhenTagSetChanges to AggregateEntry shape
dougqh 2ea61c5
Clarify materializePeerTags hit-counting loop
dougqh 8ec956d
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/…
dougqh a606692
Fix MetricsIntegrationTest entry recording call site
dougqh 913e7d7
Make ConflatingMetricAggregatorTest counter checks actually verify
dougqh 2dcea9a
Drop dead recordDurations(int, AtomicLongArray) batch API
dougqh 50b06e5
Warn about colon split in AggregateEntry.of test factory
dougqh b0f21bf
Add coverage for disable() -> ClearSignal threading path
dougqh 5a4685f
Drop unused Tags imports flagged by codenarc
dougqh 200028d
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/…
dougqh 0c50037
Drop unused Tags imports flagged by codenarc
dougqh 078382f
Drop unused SpanKindFilter imports flagged by codenarc
dougqh 7455fcc
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh 4171d15
Sync client_metrics_design doc with reconcile-on-aggregator-thread
dougqh a1863db
Update dd-trace-core/src/main/java/datadog/trace/common/metrics/PeerT…
dougqh 3fb86d3
Spread input hash before masking in cardinality-handler probes
dougqh e5cfb54
Apply Spotless Javadoc reflows on metrics files
dougqh a75bc4a
Address sarahchen6's review pass
dougqh 4871892
Clamp TRACER_METRICS_MAX_PENDING instead of throwing on overflow
dougqh 5f73c2d
Fix duplicate-entry bug for null-fielded SpanSnapshots
dougqh 9dddf0a
Clear dirty flag in ClearSignal handler
dougqh 2c8ad26
Merge branch 'master' into dougqh/conflating-metrics-background-work
dougqh 80778c4
Drop conditional null-skip from peer-tag hashing
dougqh 21e7545
Delete dead Aggregator.clearAggregates()
dougqh 877d95c
Cursor-resume eviction in AggregateTable via half-open MutatingTableI…
dougqh e2f2585
Move AggregateEntry.of() test factory out of production class
dougqh 2536aa2
Fix AggregateEntry equals/hashCode contract violation
dougqh c0449a3
Don't trample queued STOP in ClearSignal handler
dougqh be13431
Implement PeerTagSchema.equals symmetric with hashCode
dougqh d174938
Route service and spanKind through canonicalize for null-safety
dougqh 641b5ef
Suppress forbiddenApis for tearDown's System.err diagnostics
dougqh 7eedb25
Merge branch 'master' into dougqh/conflating-metrics-background-work
dougqh ce5b38b
Use DDAgentFeaturesDiscovery.state() hash for PeerTagSchema reconcile
dougqh 2eaa58f
Convert TRACER_METRICS_MAX_PENDING rationale to /* */ block comment
dougqh cd56981
Merge remote-tracking branch 'origin/dougqh/conflating-metrics-backgr…
dougqh d1df95e
Add cardinality-isolation companions to AdversarialMetricsBenchmark
dougqh 4a4b72e
Merge remote-tracking branch 'origin/dougqh/conflating-metrics-backgr…
dougqh b382df5
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh e4ed029
Reflow reconcilePeerTagSchema Javadoc after merge
dougqh 133d3c1
Consolidate contentEquals; remove redundant stringContentEquals
dougqh 2ae738b
Rename AggregateEntries -> AggregateEntryFixtures
dougqh f68ad4f
Tidy PR-iteration history out of test comments
dougqh 620241e
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh 6fdc2de
Merge remote-tracking branch 'origin/master' into dougqh/optimize-met…
dougqh a6b7854
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh 3f32dfd
Merge remote-tracking branch 'origin/master' into dougqh/optimize-met…
dougqh 3d8cccc
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh 2d1536c
Make AggregateEntry.ERROR_TAG / TOP_LEVEL_TAG package-private
dougqh ad01d7d
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh 5bf4d90
Move AggregateEntry equality contract to test-only helper
dougqh 60edaa4
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh 6b10345
Move AggregateEntry equality contract to test-only helper
dougqh a67ac23
Consolidate AggregateEntryFixtures into AggregateEntryTestUtils
dougqh 2c3730e
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh 1eb6826
Document deliberate cohesion + single-writer invariant on AggregateEntry
dougqh 49124e5
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh d5065f2
Avoid recomputing keyHash on AggregateTable miss
dougqh db08d58
Tighten client-stats cardinality plumbing
dougqh e6f2399
Document AggregateEntry.clear key-field persistence + SignalItem sing…
dougqh 553bfbd
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh 5e72808
Encapsulate EMPTY-as-absent sentinel + align hashOf field order with …
dougqh 703c9e1
Preserve warm TagCardinalityHandlers across peer-tag schema rebuilds
dougqh 92a9af3
Revert "Preserve warm TagCardinalityHandlers across peer-tag schema r…
dougqh 3f4c1c6
Add trace.stats.cardinality.limits.enabled flag (default off)
dougqh 6dc6701
Doc the regime shift after cardinality-limits flag landed
dougqh dcb1898
Drop useless @SuppressFBWarnings on AggregateEntry
dougqh a06c2a8
Update stale Javadoc on AggregateEntry's no-equals contract
dougqh acf2ffa
Tighten AggregateEntry / PeerTagSchema surface area
dougqh ccb4a4b
Tighten AggregateEntry surface — drop one-line factory, doc the conve…
dougqh 1469811
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh d0c0be8
Merge branch 'master' into dougqh/optimize-metric-key
dougqh 4af16c1
Drop unused UTF8BytesString import in MetricsIntegrationTest
dougqh c48399d
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh dcd3407
Merge branch 'master' into dougqh/optimize-metric-key
dougqh f2ee559
Lazy-allocate the error latency histogram on AggregateEntry
dougqh 690b796
Merge branch 'dougqh/lazy-error-latencies' into dougqh/control-tag-ca…
dougqh e61f9cd
Merge master into dougqh/control-tag-cardinality
dougqh 4bdfa62
Merge dougqh/lazy-error-latencies: lazy histogram + namesHash caching
dougqh 60a33f7
Clamp populatePeerTags iteration to Math.min(schema.size(), values.le…
dougqh File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Claude, this change is unnecessary