[MNT] Diagnose and address long test runtimes (#1633)

[Abhishek9639]

[MNT] diagnose and address long test runtimes. Closes #1633

Changes

Current CI test runs take 1–2+ hours. This PR diagnoses the bottleneck and implements several improvements:

Root Cause

The production_server tests (74 tests) make live API calls to openml.org, taking ~1h 23m in CI even with 4-worker parallelization.

Improvements

1. Global per-test timeout (pyproject.toml)

   • Added timeout = 600 (10 min) to [tool.pytest.ini_options]
   • Prevents any single test from hanging indefinitely

2. CI workflow improvements (.github/workflows/test.yml)

   • Changed --durations=20 → --durations=0 to report ALL test durations for diagnosis
   • Added explicit --timeout=600 to all 3 pytest invocations

3. Fixture optimization (tests/conftest.py)

   • Changed verify_cache_state fixture scope from function → module
   • Reduces redundant filesystem I/O (was running before/after EVERY test)

4. Benchmark script (scripts/profile_tests.sh)

   • New script for easy local test duration profiling
   • Configurable marker filters

Test Distribution Analysis

Category	Count	CI Time
All tests	368
production_server	74	~1h 23m (bottleneck)
test_server	196	excluded from CI
sklearn-only	6	~1 min
Non-server	99	fast

Verification

• All pre-commit checks pass (ruff, ruff-format, mypy)
• All 368 tests still collect correctly

[Abhishek9639]

Hii @geetu040 and @fkiraly,
Fixed the code quality checks. All pre-commit checks are now passing.
Please review it.

[geetu040]

Thanks, scripts/profile_tests.sh file will be used, but I not sure about other duration and timeout related changes. See comments below.

[geetu040]

you should mention the issue #1633 without the keyword Fixes #1633 since it doesn't close it, rather adds script to help debug this.

[Abhishek9639]

@geetu040,
I’ve made all the changes you suggested. Could you please review it once?
And if any further changes are needed, please let me know.
Thanks

[geetu040]

see the comment below

[Abhishek9639]

@geetu040,
Updated Added -n for workers with --dist=load and removed -q for full output the script now mimics the exact CI pytest command. Please review
If any further changes are needed, please let me know.
Thanks

[geetu040]

LGTM

@fkiraly please merge.

[PGijsbers]

Is this ready for review or not?:) I was pinged for this but I see there are still commits being pushed

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.67%. Comparing base (e653ef6) to head (144cee9).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1692   +/-   ##
=======================================
  Coverage   54.67%   54.67%           
=======================================
  Files          63       63           
  Lines        5108     5108           
=======================================
  Hits         2793     2793           
  Misses       2315     2315           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Abhishek9639]

Hi @PGijsbers,
Yes, it's ready for review now! Sorry about the confusion the force-push was just to clean up the commit history.
No more changes planned.
Thanks