Skip to content

rename metric label values for fetched-data limit rejections #7517

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SungJin1212 wants to merge 1 commit into
base: master
Choose a base branch
from
+32 −29
Open

rename metric label values for fetched-data limit rejections #7517

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
# Changelog

## master / unreleased
* [CHANGE] Query Frontend: Rename `cortex_rejected_queries_total` metric label `reason` values for fetched-data limit rejections. #7517
* Old values: `series_fetched`, `chunks_fetched`, `chunk_bytes_fetched`, `data_bytes_fetched`.
* New values: `fetched_series_exceeded`, `fetched_chunks_exceeded`, `fetched_chunk_bytes_exceeded`, `fetched_data_bytes_exceeded`.
* [CHANGE] Querier: Make query time range configurations per-tenant: `query_ingesters_within`, `query_store_after`, and `shuffle_sharding_ingesters_lookback_period`. Uses `model.Duration` instead of `time.Duration` to support serialization but has minimum unit of 1ms (nanoseconds/microseconds not supported). #7160
* [CHANGE] Cache: Setting `-blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl` to 0 will disable the bucket-index cache. #7446
* [CHANGE] HA Tracker: Move `-distributor.ha-tracker.failover-timeout` from a global config to a per-tenant runtime config. The flag name and default value (30s) remain the same. #7481
Expand Down
42 changes: 21 additions & 21 deletions pkg/frontend/transport/handler.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,23 +49,23 @@ var (
)

const (
reasonTooManyTenants = "too_many_tenants"
reasonRequestBodySizeExceeded = "request_body_size_exceeded"
reasonResponseBodySizeExceeded = "response_body_size_exceeded"
reasonTooManyRequests = "too_many_requests"
reasonResourceExhausted = "resource_exhausted"
reasonTimeRangeExceeded = "time_range_exceeded"
reasonResponseSizeExceeded = "response_size_exceeded"
reasonTooManySamples = "too_many_samples"
reasonSeriesFetched = "series_fetched"
reasonChunksFetched = "chunks_fetched"
reasonChunkBytesFetched = "chunk_bytes_fetched"
reasonDataBytesFetched = "data_bytes_fetched"
reasonSeriesLimitStoreGateway = "store_gateway_series_limit"
reasonChunksLimitStoreGateway = "store_gateway_chunks_limit"
reasonBytesLimitStoreGateway = "store_gateway_bytes_limit"
reasonUnOptimizedRegexMatcher = `unoptimized_regex_matcher`
reasonQueryTooExpensive = "query_too_expensive"
reasonTooManyTenants = "too_many_tenants"
reasonRequestBodySizeExceeded = "request_body_size_exceeded"
reasonResponseBodySizeExceeded = "response_body_size_exceeded"
reasonTooManyRequests = "too_many_requests"
reasonResourceExhausted = "resource_exhausted"
reasonTimeRangeExceeded = "time_range_exceeded"
reasonResponseSizeExceeded = "response_size_exceeded"
reasonTooManySamples = "too_many_samples"
Copy link
Copy Markdown
Member

@friedrichg friedrichg May 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this also be fetch_too_many_samples_exceeded?

Copy link
Copy Markdown
Member Author

@SungJin1212 SungJin1212 May 15, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might break existing dashboards and alerts. However, I realized that the existing values cannot clearly explain why the query is being rejected (these are label values of the reason).

I think the too_many_samples is clear enough to explain why the query is rejected.
The fetch_too_many_samples_exceeded is clearer, but I'd prefer to keep the changes to a minimum.

reasonFetchedSeriesExceeded = "fetched_series_exceeded"
reasonFetchedChunksExceeded = "fetched_chunks_exceeded"
reasonFetchedChunkBytesExceeded = "fetched_chunk_bytes_exceeded"
reasonFetchedDataBytesExceeded = "fetched_data_bytes_exceeded"
reasonSeriesLimitStoreGateway = "store_gateway_series_limit"
reasonChunksLimitStoreGateway = "store_gateway_chunks_limit"
reasonBytesLimitStoreGateway = "store_gateway_bytes_limit"
reasonUnOptimizedRegexMatcher = `unoptimized_regex_matcher`
reasonQueryTooExpensive = "query_too_expensive"

limitTooManySamples = `query processing would load too many samples into memory`
limitTimeRangeExceeded = `the query time range exceeds the limit`
Expand Down Expand Up @@ -579,13 +579,13 @@ func (f *Handler) reportQueryStats(r *http.Request, source, userID string, query
} else if strings.Contains(errMsg, limitResponseSizeExceeded) {
reason = reasonResponseSizeExceeded
} else if strings.Contains(errMsg, limitSeriesFetched) {
reason = reasonSeriesFetched
reason = reasonFetchedSeriesExceeded
} else if strings.Contains(errMsg, limitChunksFetched) {
reason = reasonChunksFetched
reason = reasonFetchedChunksExceeded
} else if strings.Contains(errMsg, limitChunkBytesFetched) {
reason = reasonChunkBytesFetched
reason = reasonFetchedChunkBytesExceeded
} else if strings.Contains(errMsg, limitDataBytesFetched) {
reason = reasonDataBytesFetched
reason = reasonFetchedDataBytesExceeded
} else if strings.Contains(errMsg, limitSeriesStoreGateway) {
reason = reasonSeriesLimitStoreGateway
} else if strings.Contains(errMsg, limitChunksStoreGateway) {
Expand Down
16 changes: 8 additions & 8 deletions pkg/frontend/transport/handler_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ func TestHandler_ServeHTTP(t *testing.T) {
expectedStatusCode: http.StatusUnprocessableEntity,
},
{
name: "test handler with reasonSeriesFetched",
name: "test handler with reasonFetchedSeriesExceeded",
cfg: HandlerConfig{QueryStatsEnabled: true},
expectedMetrics: 6,
roundTripperFunc: roundTripperFunc(func(req *http.Request) (*http.Response, error) {
Expand All @@ -282,13 +282,13 @@ func TestHandler_ServeHTTP(t *testing.T) {
}, nil
}),
additionalMetricsCheckFunc: func(h *Handler) {
v := promtest.ToFloat64(h.rejectedQueries.WithLabelValues(reasonSeriesFetched, requestmeta.SourceAPI, userID))
v := promtest.ToFloat64(h.rejectedQueries.WithLabelValues(reasonFetchedSeriesExceeded, requestmeta.SourceAPI, userID))
assert.Equal(t, float64(1), v)
},
expectedStatusCode: http.StatusUnprocessableEntity,
},
{
name: "test handler with reasonChunksFetched",
name: "test handler with reasonFetchedChunksExceeded",
cfg: HandlerConfig{QueryStatsEnabled: true},
expectedMetrics: 6,
roundTripperFunc: roundTripperFunc(func(req *http.Request) (*http.Response, error) {
Expand All @@ -298,13 +298,13 @@ func TestHandler_ServeHTTP(t *testing.T) {
}, nil
}),
additionalMetricsCheckFunc: func(h *Handler) {
v := promtest.ToFloat64(h.rejectedQueries.WithLabelValues(reasonChunksFetched, requestmeta.SourceAPI, userID))
v := promtest.ToFloat64(h.rejectedQueries.WithLabelValues(reasonFetchedChunksExceeded, requestmeta.SourceAPI, userID))
assert.Equal(t, float64(1), v)
},
expectedStatusCode: http.StatusUnprocessableEntity,
},
{
name: "test handler with reasonChunkBytesFetched",
name: "test handler with reasonFetchedChunkBytesExceeded",
cfg: HandlerConfig{QueryStatsEnabled: true},
expectedMetrics: 6,
roundTripperFunc: roundTripperFunc(func(req *http.Request) (*http.Response, error) {
Expand All @@ -314,13 +314,13 @@ func TestHandler_ServeHTTP(t *testing.T) {
}, nil
}),
additionalMetricsCheckFunc: func(h *Handler) {
v := promtest.ToFloat64(h.rejectedQueries.WithLabelValues(reasonChunkBytesFetched, requestmeta.SourceAPI, userID))
v := promtest.ToFloat64(h.rejectedQueries.WithLabelValues(reasonFetchedChunkBytesExceeded, requestmeta.SourceAPI, userID))
assert.Equal(t, float64(1), v)
},
expectedStatusCode: http.StatusUnprocessableEntity,
},
{
name: "test handler with reasonDataBytesFetched",
name: "test handler with reasonFetchedDataBytesExceeded",
cfg: HandlerConfig{QueryStatsEnabled: true},
expectedMetrics: 6,
roundTripperFunc: roundTripperFunc(func(req *http.Request) (*http.Response, error) {
Expand All @@ -330,7 +330,7 @@ func TestHandler_ServeHTTP(t *testing.T) {
}, nil
}),
additionalMetricsCheckFunc: func(h *Handler) {
v := promtest.ToFloat64(h.rejectedQueries.WithLabelValues(reasonDataBytesFetched, requestmeta.SourceAPI, userID))
v := promtest.ToFloat64(h.rejectedQueries.WithLabelValues(reasonFetchedDataBytesExceeded, requestmeta.SourceAPI, userID))
assert.Equal(t, float64(1), v)
},
expectedStatusCode: http.StatusUnprocessableEntity,
Expand Down
Loading