feat(ingester): Add cortex_ingester_active_metric_names gauge per user#7514
Open
yeya24 wants to merge 1 commit into
Open
feat(ingester): Add cortex_ingester_active_metric_names gauge per user#7514yeya24 wants to merge 1 commit into
yeya24 wants to merge 1 commit into
Conversation
Expose the number of unique metric names (distinct __name__ values) per tenant in the ingester head as a new Prometheus gauge metric. The data is sourced from the existing seriesInMetric counter which already tracks series counts per metric name via TSDB lifecycle callbacks. The metric is registered when -ingester.active-series-metrics-enabled is true (same gate as cortex_ingester_active_series) and updated in the same periodic loop alongside active series counts. This enables operators to monitor metric name cardinality per tenant without additional overhead, as the underlying data structure already exists. Signed-off-by: Ben Ye <benye@amazon.com>
dosubot
Bot
added
component/ingester
type/feature
type/observability
SungJin1212
reviewed
May 13, 2026
| // Not registered automatically, but only if activeSeriesEnabled is true. | ||
| activeMetricNamesPerUser: prometheus.NewGaugeVec(prometheus.GaugeOpts{ | ||
| Name: "cortex_ingester_active_metric_names", | ||
| Help: "Number of unique metric names in the ingester head per user.", |
Member
There was a problem hiding this comment.
Nit: What about a TSDB head?
Suggested change
| Help: "Number of unique metric names in the ingester head per user.", | |
| Help: "Number of unique metric names in the TSDB head per user.", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expose the number of unique metric names (distinct
__name__values) per tenant in the ingester head as a new Prometheus gauge metric:cortex_ingester_active_metric_names.Motivation
Operators need visibility into metric name cardinality per tenant to detect cardinality explosions at the metric name level (as opposed to series level which
cortex_ingester_active_seriesalready covers).Changes
pkg/ingester/user_state.go: AddedActiveMetricNames()method tometricCounterthat returns the total number of unique metric names across all shards.pkg/ingester/metrics.go: Addedcortex_ingester_active_metric_namesGaugeVec (labels:user), registered underactiveSeriesEnabledgate, with cleanup indeletePerUserMetrics.pkg/ingester/ingester.go: Set the gauge inupdateActiveSeriesloop and clean up on TSDB close.pkg/ingester/user_state_test.go: Unit test forActiveMetricNames().docs/configuration/v1-guarantees.md: Listed as experimental feature.How it works
The
seriesInMetric(metricCounter) already tracks a sharded map ofmetricName → seriesCountmaintained via TSDBPostCreation/PostDeletioncallbacks. The number of keys in this map equals the number of unique metric names in the head. No new data structures or tracking overhead is introduced.Testing
TestMetricCounter_ActiveMetricNames— verifies count increases/decreases correctly as series are added/removed.TestExpandedCachePostings_RaceandTestIngester_Pushare unrelated race conditions on master).