cortexproject · CharlieTLe · Apr 12, 2026 · Apr 12, 2026 · Apr 12, 2026 · Apr 12, 2026
diff --git a/flaky-tests/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod.md b/flaky-tests/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod.md
@@ -0,0 +1,30 @@
+# Flaky Test: TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Multiple subtests failed on arm64 without `-race`. Errors at `blocks_cleaner_test.go:699` ("Not equal") and `blocks_cleaner_test.go:829`/`:866` ("Received unexpected error"). Likely timing-dependent block cleanup assertions that assume synchronous completion.
+
+## Occurrences
+
+### 2026-04-13T21:00:49Z
+- **Job**: [ci / test-no-race (arm64)](https://github.com/cortexproject/cortex/actions/runs/24366476213)
+- **Package**: `github.com/cortexproject/cortex/pkg/compactor`
+- **File**: `pkg/compactor/blocks_cleaner_test.go:699, :829, :866`
+- **Notes**: Failed on arm64 without `-race`. Passed on amd64 (both race and no-race) and arm64 with `-race`.
+
+<details><summary>Build logs</summary>
+
+```
+--- FAIL: TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod (0.47s)
+        Error Trace:	/__w/cortex/cortex/pkg/compactor/blocks_cleaner_test.go:699
+        Error:      	Not equal:
+        Error Trace:	/__w/cortex/cortex/pkg/compactor/blocks_cleaner_test.go:829
+        Error:      	Received unexpected error:
+        Error Trace:	/__w/cortex/cortex/pkg/compactor/blocks_cleaner_test.go:699
+        Error:      	Not equal:
+        Error Trace:	/__w/cortex/cortex/pkg/compactor/blocks_cleaner_test.go:866
+        Error:      	Received unexpected error:
+FAIL	github.com/cortexproject/cortex/pkg/compactor	90.399s
+```
+
+</details>
diff --git a/flaky-tests/TestDistributorQuerier_QueryIngestersWithinBoundary.md b/flaky-tests/TestDistributorQuerier_QueryIngestersWithinBoundary.md
@@ -0,0 +1,53 @@
+# Flaky Test: TestDistributorQuerier_QueryIngestersWithinBoundary
+
+**Status**: Fixed
+**Occurrences**: 2
+**Root Cause**: Timing-sensitive test. `now` is captured at test setup (line 561) via `time.Now()`, but the code under test in `newDistributorQueryable` also calls `time.Now()` internally to compute the lookback boundary. Under the race detector (slower execution), enough wall-clock time elapses between setup and execution that the internal boundary shifts. The "maxT well after lookback boundary" subtest uses only a 10-second margin (`now.Add(-lookback + 10*time.Second)`), which is not enough buffer when the race detector slows execution. The result is that `queryMaxT` falls before the internally-computed boundary, the query is skipped, and `distributor.Calls` is empty instead of having 1 call.
+
+## Occurrences
+
+### 2026-04-12T19:09:52Z
+- **Job**: [ci / test (amd64)](https://github.com/cortexproject/cortex/actions/runs/24314068155)
+- **Package**: `github.com/cortexproject/cortex/pkg/querier`
+- **File**: `pkg/querier/distributor_queryable_test.go:638`
+- **Subtest**: `maxT_well_after_lookback_boundary`
+- **Notes**: Passed on arm64, passed on test-no-race (amd64), failed only under `-race` on amd64.
+
+<details><summary>Build logs</summary>
+
+```
+--- FAIL: TestDistributorQuerier_QueryIngestersWithinBoundary (0.00s)
+    --- FAIL: TestDistributorQuerier_QueryIngestersWithinBoundary/maxT_well_after_lookback_boundary (0.01s)
+        distributor_queryable_test.go:638:
+            	Error Trace:	/__w/cortex/cortex/pkg/querier/distributor_queryable_test.go:638
+            	Error:      	"[]" should have 1 item(s), but has 0
+            	Test:       	TestDistributorQuerier_QueryIngestersWithinBoundary/maxT_well_after_lookback_boundary
+            	Messages:   	should manipulate when maxT is well after boundary
+FAIL
+FAIL	github.com/cortexproject/cortex/pkg/querier	34.451s
+```
+
+</details>
+
+### 2026-04-12T20:32:32Z
+- **Job**: [ci / test (amd64)](https://github.com/cortexproject/cortex/actions/runs/24315645679)
+- **Package**: `github.com/cortexproject/cortex/pkg/querier`
+- **File**: `pkg/querier/distributor_queryable_test.go:638`
+- **Subtest**: `maxT_well_after_lookback_boundary`
+- **Notes**: Same failure, again on amd64 with `-race`. Passed on arm64 and test-no-race.
+
+<details><summary>Build logs</summary>
+
+```
+--- FAIL: TestDistributorQuerier_QueryIngestersWithinBoundary (0.00s)
+    --- FAIL: TestDistributorQuerier_QueryIngestersWithinBoundary/maxT_well_after_lookback_boundary (0.00s)
+        distributor_queryable_test.go:638:
+            	Error Trace:	/__w/cortex/cortex/pkg/querier/distributor_queryable_test.go:638
+            	Error:      	"[]" should have 1 item(s), but has 0
+            	Test:       	TestDistributorQuerier_QueryIngestersWithinBoundary/maxT_well_after_lookback_boundary
+            	Messages:   	should manipulate when maxT is well after boundary
+FAIL
+FAIL	github.com/cortexproject/cortex/pkg/querier	33.505s
+```
+
+</details>
diff --git a/flaky-tests/TestIngesterMetadata.md b/flaky-tests/TestIngesterMetadata.md
@@ -0,0 +1,29 @@
+# Flaky Test: TestIngesterMetadata
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: `WaitSumMetricsWithOptions` timed out waiting for ingester metrics at `ingester_metadata_test.go:48`. The ingester service didn't report expected metrics in time on arm64. Likely a slow startup or metric registration race in the integration test environment.
+
+## Occurrences
+
+### 2026-04-17T18:02:58Z
+- **Job**: [ci / integration (ubuntu-24.04-arm, arm64, requires_docker)](https://github.com/cortexproject/cortex/actions/runs/24578865123)
+- **Package**: `github.com/cortexproject/cortex/integration`
+- **File**: `integration/ingester_metadata_test.go:48`
+- **Notes**: Timeout in WaitSumMetricsWithOptions on arm64.
+
+<details><summary>Build logs</summary>
+
+```
+        Error Trace:	/home/runner/work/cortex/cortex/integration/ingester_metadata_test.go:48
+        Error:      	Received unexpected error:
+            github.com/cortexproject/cortex/integration/e2e.(*HTTPService).WaitSumMetricsWithOptions
+                /home/runner/work/cortex/cortex/integration/e2e/service.go:611
+            github.com/cortexproject/cortex/integration.TestIngesterMetadata
+                /home/runner/work/cortex/cortex/integration/ingester_metadata_test.go:48
+        Test:       	TestIngesterMetadata
+--- FAIL: TestIngesterMetadata (4.88s)
+FAIL	github.com/cortexproject/cortex/integration	523.768s
+```
+
+</details>
diff --git a/flaky-tests/TestMinimizeSpreadTokenGenerator.md b/flaky-tests/TestMinimizeSpreadTokenGenerator.md
@@ -0,0 +1,29 @@
+# Flaky Test: TestMinimizeSpreadTokenGenerator
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Floating point precision boundary. The test asserts that token spread distance error is <= 0.01, but the result was 0.01097 — barely over the threshold. The token generation algorithm involves randomness and the assertion threshold is too tight for non-deterministic outcomes. Failed on amd64 without `-race`.
+
+## Occurrences
+
+### 2026-04-16T21:27:42Z
+- **Job**: [ci / test-no-race (amd64)](https://github.com/cortexproject/cortex/actions/runs/24534699120)
+- **Package**: `github.com/cortexproject/cortex/pkg/ring`
+- **File**: `pkg/ring/token_generator_test.go:202` (called from `:106`)
+- **Notes**: Error was 0.01097, threshold is 0.01. Passed on arm64 and amd64 with `-race`.
+
+<details><summary>Build logs</summary>
+
+```
+--- FAIL: TestMinimizeSpreadTokenGenerator (2.03s)
+    token_generator_test.go:202:
+        Error Trace:	/__w/cortex/cortex/pkg/ring/token_generator_test.go:202
+                    	/__w/cortex/cortex/pkg/ring/token_generator_test.go:106
+        Error:      	Condition failed!
+        Test:       	TestMinimizeSpreadTokenGenerator
+        Messages:   	[minimize-42-zone1] expected and real distance error is greater than 0.01 -> 0.010970902851567321[8.2595524e+07/8.3511723e+07]
+FAIL
+FAIL	github.com/cortexproject/cortex/pkg/ring	67.285s
+```
+
+</details>
diff --git a/flaky-tests/TestMultitenantAlertmanager_InitialSyncFailureWithSharding.md b/flaky-tests/TestMultitenantAlertmanager_InitialSyncFailureWithSharding.md
@@ -0,0 +1,29 @@
+# Flaky Test: TestMultitenantAlertmanager_InitialSyncFailureWithSharding
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Ring membership race condition. The test checks that initial sync failure is properly handled with sharding enabled, but the ring instance goes missing and gets re-added (`instance missing in the ring, adding it back`). The assertion at line 1778 (`Should be false`) fails because the ring state hasn't stabilized in time on arm64 with `-race`.
+
+## Occurrences
+
+### 2026-04-17T17:25:08Z
+- **Job**: [ci / test (arm64)](https://github.com/cortexproject/cortex/actions/runs/24577873547)
+- **Package**: `github.com/cortexproject/cortex/pkg/alertmanager`
+- **File**: `pkg/alertmanager/multitenant_test.go:1778`
+- **Notes**: Ring warnings about instance missing. Failed on arm64 with `-race`.
+
+<details><summary>Build logs</summary>
+
+```
+level=warn component=MultiTenantAlertmanager msg="instance missing in the ring, adding it back" ring=alertmanager
+level=warn component=MultiTenantAlertmanager msg="instance missing in the ring, adding it back" ring=alertmanager
+--- FAIL: TestMultitenantAlertmanager_InitialSyncFailureWithSharding (1.76s)
+    multitenant_test.go:1778:
+        Error Trace:	/__w/cortex/cortex/pkg/alertmanager/multitenant_test.go:1778
+        Error:      	Should be false
+        Test:       	TestMultitenantAlertmanager_InitialSyncFailureWithSharding
+FAIL
+FAIL	github.com/cortexproject/cortex/pkg/alertmanager	50.012s
+```
+
+</details>
diff --git a/flaky-tests/TestParquetFuzz.md b/flaky-tests/TestParquetFuzz.md
@@ -0,0 +1,26 @@
+# Flaky Test: TestParquetFuzz
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Fuzz test with non-deterministic query generation. The test generates random PromQL queries and compares Cortex results against Prometheus. 1 out of N randomly generated test cases failed, likely due to floating point precision differences or edge cases in query evaluation. This is inherently non-deterministic.
+
+## Occurrences
+
+### 2026-04-13T20:44:25Z
+- **Job**: [ci / integration (ubuntu-24.04, amd64, integration_query_fuzz)](https://github.com/cortexproject/cortex/actions/runs/24365433548)
+- **Package**: `github.com/cortexproject/cortex/integration`
+- **File**: `integration/parquet_querier_test.go:176`
+- **Notes**: "1 test cases failed" out of the fuzz suite. Non-deterministic by nature.
+
+<details><summary>Build logs</summary>
+
+```
+        Error Trace:	/home/runner/work/cortex/cortex/integration/parquet_querier_test.go:176
+        Error:      	finished query fuzzing tests
+        Test:       	TestParquetFuzz
+        Messages:   	1 test cases failed
+--- FAIL: TestParquetFuzz (24.12s)
+FAIL	github.com/cortexproject/cortex/integration	200.599s
+```
+
+</details>
diff --git a/flaky-tests/TestPushRace.md b/flaky-tests/TestPushRace.md
@@ -0,0 +1,29 @@
+# Flaky Test: TestPushRace
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Race condition in concurrent push operations. The test pushes samples concurrently and one goroutine receives an "out of bounds" error with `timestamp=1970-01-01T00:00:00Z` (zero value). This suggests a race where a sample's timestamp is read before it's fully initialized, or TSDB head boundaries shift between goroutines. Failed on arm64 without `-race`.
+
+## Occurrences
+
+### 2026-04-16T22:34:58Z
+- **Job**: [ci / test-no-race (arm64)](https://github.com/cortexproject/cortex/actions/runs/24537394343)
+- **Package**: `github.com/cortexproject/cortex/pkg/ingester`
+- **File**: `pkg/ingester/ingester_test.go:719`
+- **Notes**: Error: `rpc error: code = Code(400) desc = user=1: err: out of bounds. timestamp=1970-01-01T00:00:00Z`
+
+<details><summary>Build logs</summary>
+
+```
+--- FAIL: TestPushRace (0.12s)
+    ingester_test.go:719:
+        Error Trace:	/__w/cortex/cortex/pkg/ingester/ingester_test.go:719
+                    	/usr/local/go/src/runtime/asm_arm64.s:1268
+        Error:      	Received unexpected error:
+                    	rpc error: code = Code(400) desc = user=1: err: out of bounds. timestamp=1970-01-01T00:00:00Z, series={__name__="foo", k="10", userId="1"}
+        Test:       	TestPushRace
+FAIL
+FAIL	github.com/cortexproject/cortex/pkg/ingester	22.814s
+```
+
+</details>
diff --git a/flaky-tests/TestQuerierWithBlocksStorageLimits.md b/flaky-tests/TestQuerierWithBlocksStorageLimits.md
@@ -0,0 +1,27 @@
+# Flaky Test: TestQuerierWithBlocksStorageLimits
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: The test expects HTTP 422 (Unprocessable Entity) for a query exceeding limits but received HTTP 500 (Internal Server Error) instead. This is likely a race where the store-gateway or querier hasn't fully initialized its limit-checking path, causing the error to surface as a generic 500 rather than the expected 422. Failed on arm64.
+
+## Occurrences
+
+### 2026-04-16T18:18:08Z
+- **Job**: [ci / integration (ubuntu-24.04-arm, arm64, integration_querier)](https://github.com/cortexproject/cortex/actions/runs/24526116820)
+- **Package**: `github.com/cortexproject/cortex/integration`
+- **File**: `integration/querier_test.go:468`
+- **Notes**: Expected 422, got 500. Passed on amd64 in the same run.
+
+<details><summary>Build logs</summary>
+
+```
+        Error Trace:	/home/runner/work/cortex/cortex/integration/querier_test.go:468
+        Error:      	Not equal:
+                    	expected: 422
+                    	actual  : 500
+        Test:       	TestQuerierWithBlocksStorageLimits
+--- FAIL: TestQuerierWithBlocksStorageLimits (11.00s)
+FAIL	github.com/cortexproject/cortex/integration	193.371s
+```
+
+</details>
diff --git a/flaky-tests/TestQuerierWithBlocksStorageRunningInSingleBinaryMode.md b/flaky-tests/TestQuerierWithBlocksStorageRunningInSingleBinaryMode.md
@@ -0,0 +1,28 @@
+# Flaky Test: TestQuerierWithBlocksStorageRunningInSingleBinaryMode
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Transient Docker infrastructure failure. The Consul container (`e2e-cortex-test-consul`) disappeared mid-test, causing `Error response from daemon: No such container: e2e-cortex-test-consul`. This is a CI environment issue, not a code bug. The test passed on amd64 in the same run and passed on arm64 in run #6.
+
+## Occurrences
+
+### 2026-04-12T21:00:08Z
+- **Job**: [ci / integration (ubuntu-24.04-arm, arm64, integration_querier)](https://github.com/cortexproject/cortex/actions/runs/24316060467)
+- **Package**: `github.com/cortexproject/cortex/integration`
+- **File**: `integration/querier_test.go:217`
+- **Subtest**: `[TSDB]_blocks_sharding_enabled,_redis_index_cache,_bucket_index_enabled,thanosEngine=false`
+- **Notes**: Passed on amd64 in the same run. Docker container `e2e-cortex-test-consul` vanished during test execution.
+
+<details><summary>Build logs</summary>
+
+```
+20:57:39 Error response from daemon: No such container: e2e-cortex-test-consul
+        Error Trace:	/home/runner/work/cortex/cortex/integration/querier_test.go:217
+        Error:      	Received unexpected error:
+--- FAIL: TestQuerierWithBlocksStorageRunningInSingleBinaryMode (69.50s)
+    --- FAIL: TestQuerierWithBlocksStorageRunningInSingleBinaryMode/[TSDB]_blocks_sharding_enabled,_redis_index_cache,_bucket_index_enabled,thanosEngine=false (34.33s)
+FAIL
+FAIL	github.com/cortexproject/cortex/integration	279.595s
+```
+
+</details>
diff --git a/flaky-tests/TestQuerierWithStoreGatewayDataBytesLimits.md b/flaky-tests/TestQuerierWithStoreGatewayDataBytesLimits.md
@@ -0,0 +1,30 @@
+# Flaky Test: TestQuerierWithStoreGatewayDataBytesLimits
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Transient Docker infrastructure failure. Multiple containers vanished mid-test (`e2e-cortex-test-consul`, `e2e-cortex-test-distributor`, `e2e-cortex-test-minio-9000`, `e2e-cortex-test-store-gateway`, `e2e-cortex-test-ingester`, `e2e-cortex-test-querier`). Same infrastructure instability pattern as TestQuerierWithBlocksStorageRunningInSingleBinaryMode. The test passed on arm64 in the same run and on both arches in runs #1-#4.
+
+## Occurrences
+
+### 2026-04-12T21:37:26Z
+- **Job**: [ci / integration (ubuntu-24.04, amd64, integration_querier)](https://github.com/cortexproject/cortex/actions/runs/24316771541)
+- **Package**: `github.com/cortexproject/cortex/integration`
+- **File**: `integration/querier_test.go:565`
+- **Notes**: Widespread Docker container disappearance. All `e2e-cortex-test-*` containers returned "No such container" errors.
+
+<details><summary>Build logs</summary>
+
+```
+21:34:11 Error response from daemon: No such container: e2e-cortex-test-distributor
+21:34:11 Error response from daemon: No such container: e2e-cortex-test-minio-9000
+21:34:11 Error response from daemon: No such container: e2e-cortex-test-store-gateway
+21:34:11 Error response from daemon: No such container: e2e-cortex-test-ingester
+21:34:11 Error response from daemon: No such container: e2e-cortex-test-querier
+21:34:19 Error response from daemon: No such container: e2e-cortex-test-consul
+        Error Trace:	/home/runner/work/cortex/cortex/integration/querier_test.go:565
+        Error:      	Not equal:
+--- FAIL: TestQuerierWithStoreGatewayDataBytesLimits (10.69s)
+FAIL	github.com/cortexproject/cortex/integration	202.827s
+```
+
+</details>
diff --git a/flaky-tests/TestQueueConcurrency.md b/flaky-tests/TestQueueConcurrency.md
@@ -0,0 +1,31 @@
+# Flaky Test: TestQueueConcurrency
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Deadlock/hang in concurrent test. The test spawns 30 goroutines that concurrently enqueue, dequeue, and delete from a queue. Goroutines where `cnt%5 == 0` (and odd) call `dequeueRequest(0, false)` which blocks on a channel waiting for an item. If other goroutines have already drained the queue or called `deleteQueue`, the dequeue goroutine blocks forever — there will never be another enqueue to unblock it. This causes the entire test to hang until the 30-minute timeout. The issue manifests more on arm64 with `-race` due to slower execution and different goroutine scheduling.
+
+## Occurrences
+
+### 2026-04-12T20:23:09Z
+- **Job**: [ci / test (arm64)](https://github.com/cortexproject/cortex/actions/runs/24314927948)
+- **Package**: `github.com/cortexproject/cortex/pkg/scheduler/queue`
+- **File**: `pkg/scheduler/queue/user_queues_test.go:461`
+- **Notes**: Timed out after 30m on arm64 with `-race`. Passed on amd64 with `-race`, passed on both arches without `-race`. Goroutine dump shows `dequeueRequest` stuck waiting on channel at `user_request_queue.go:35`.
+
+<details><summary>Build logs</summary>
+
+```
+panic: test timed out after 30m0s
+	running tests:
+		TestQueueConcurrency (29m51s)
+
+goroutine 100 [chan receive, 29 minutes]:
+github.com/cortexproject/cortex/pkg/scheduler/queue.(*FIFORequestQueue).dequeueRequest(0xc000410080, 0xc8ea31?, 0x6?)
+	/__w/cortex/cortex/pkg/scheduler/queue/user_request_queue.go:35 +0x48
+github.com/cortexproject/cortex/pkg/scheduler/queue.TestQueueConcurrency.func1(0xf)
+	/__w/cortex/cortex/pkg/scheduler/queue/user_queues_test.go:477 +0x280
+
+FAIL	github.com/cortexproject/cortex/pkg/scheduler/queue	1800.101s
+```
+
+</details>
diff --git a/flaky-tests/TestRuler_rules.md b/flaky-tests/TestRuler_rules.md
@@ -0,0 +1,25 @@
+# Flaky Test: TestRuler_rules
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Same alert state race as TestRuler_rules_limit. The test expects `"state":"unknown"` for an alerting rule but gets `"state":"inactive"`. The ruler evaluates rules asynchronously after startup, and the alert state transitions before the HTTP response is captured. This manifested in the `integration-configs-db` job which runs the ruler tests with a database backend.
+
+## Occurrences
+
+### 2026-04-13T21:05:42Z
+- **Job**: [ci / integration-configs-db (ubuntu-24.04, amd64)](https://github.com/cortexproject/cortex/actions/runs/24366476213)
+- **Package**: `github.com/cortexproject/cortex/pkg/ruler`
+- **File**: `pkg/ruler/api_test.go:306`
+- **Notes**: Same `"state":"unknown"` vs `"state":"inactive"` pattern as TestRuler_rules_limit.
+
+<details><summary>Build logs</summary>
+
+```
+        Error Trace:	/go/src/github.com/cortexproject/cortex/pkg/ruler/api_test.go:306
+        Error:      	Not equal:
+                    	expected: ..."state":"unknown"...
+                    	actual  : ..."state":"inactive"...
+--- FAIL: TestRuler_rules (0.05s)
+```
+
+</details>