Skip to content

gh-150114: Use get_process_memory_usage() in memory watchdog#150402

Merged
vstinner merged 4 commits into
python:mainfrom
vstinner:mem_watchdog
May 25, 2026
Merged

gh-150114: Use get_process_memory_usage() in memory watchdog#150402
vstinner merged 4 commits into
python:mainfrom
vstinner:mem_watchdog

Conversation

@vstinner
Copy link
Copy Markdown
Member

@vstinner vstinner commented May 25, 2026

In practice, this change adds Windows and FreeBSD support to the memory watchdog.

In practice, this change adds Windows and FreeBSD support to the
memory watchdog.
@vstinner
Copy link
Copy Markdown
Member Author

Example on Linux:

$ ./python -m test test_bigmem -u all -M 20G -v
== CPython 3.16.0a0 (tags/v3.15.0b1-20-g57ef2199503:57ef2199503, May 25 2026, 15:08:14) [GCC 16.1.1 20260515 (Red Hat 16.1.1-2)]
== Linux-7.0.9-204.fc44.x86_64-x86_64-with-glibc2.43 little-endian
== Python build: debug TraceRefs
== cwd: /home/vstinner/python/main/build/test_python_worker_308892æ
== CPU count: 12
== encodings: locale=UTF-8 FS=utf-8
== resources: all

Using random seed: 1234531347
0:00:00 load avg: 1.57 mem: 28.3 MiB Run 1 test sequentially in a single process
0:00:00 load avg: 1.57 mem: 28.3 MiB [1/1] test_bigmem
test_capitalize (test.test_bigmem.BytearrayTest.test_capitalize) ... 
 ... expected peak memory use: 4.0 GiB
 ... process data size: 0.5 GiB
 ... process data size: 2.4 GiB
 ... process data size: 4.0 GiB
 ... process data size: 4.0 GiB
 ... process data size: 4.0 GiB
(...)

cc @cmaloney

@vstinner
Copy link
Copy Markdown
Member Author

In practice, this change adds Windows and FreeBSD support to the memory watchdog.

macOS support is coming: see PR gh-150396.

@vstinner
Copy link
Copy Markdown
Member Author

"Tests / Windows (free-threading) / Build and test (x64, switch-case)" failed with:

FAIL: test_generator_not_under_consumer_arithmetic (test.test_profiling.test_sampling_profiler.test_blocking.TestBlockingModeStackAccuracy.test_generator_not_under_consumer_arithmetic)
Test that fibonacci_generator doesn't appear when consume_generator does arithmetic.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\test_profiling\test_sampling_profiler\test_blocking.py", line 137, in test_generator_not_under_consumer_arithmetic
    self.assertEqual(invalid_stacks, 0,
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
        f"Found {invalid_stacks}/{arithmetic_samples} invalid stacks where "
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        f"fibonacci_generator appears in the stack when consume_generator "
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        f"is on an arithmetic line. This indicates torn/inconsistent stack "
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        f"traces are being captured.")
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 639 != 0 : Found 639/4508 invalid stacks where fibonacci_generator appears in the stack when consume_generator is on an arithmetic line. This indicates torn/inconsistent stack traces are being captured.


======================================================================
ERROR: test_getwche (test.test_msvcrt.TestConsoleIO.test_getwche)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\test_msvcrt.py", line 102, in test_getwche
    self.check_getwch('getwche')
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "D:\a\cpython\cpython\Lib\test\test_msvcrt.py", line 92, in check_getwch
    self.run_in_separated_process(code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "D:\a\cpython\cpython\Lib\test\test_msvcrt.py", line 70, in run_in_separated_process
    subprocess.run(cmd, check=True, capture_output=True,
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                   creationflags=subprocess.CREATE_NEW_CONSOLE)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\cpython\cpython\Lib\subprocess.py", line 692, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['D:\\a\\cpython\\cpython\\PCbuild\\amd64t\\python_d.exe', '-c', '\nimport msvcrt\nfrom _testconsole import write_input\nwith open("CONIN$", "rb", buffering=0) as stdin:\n    write_input(stdin, b\'W[\')\n    assert msvcrt.getwche() == "å­—"\n']' returned non-zero exit status 3221225794.

======================================================================
FAIL: test_python_legacy_windows_stdio (test.test_cmd_line.CmdLineTest.test_python_legacy_windows_stdio)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\test_cmd_line.py", line 1047, in test_python_legacy_windows_stdio
    self.assertEqual(p.returncode, 0)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
AssertionError: 3221225794 != 0

3221225794 is 0xc0000142 (STATUS_DLL_INIT_FAILED) error code.

I restarted the job.

Comment thread Lib/test/memory_watchdog.py
Comment thread Lib/test/memory_watchdog.py Outdated
Comment thread Lib/test/memory_watchdog.py
Comment thread Lib/test/memory_watchdog.py
@vstinner
Copy link
Copy Markdown
Member Author

@cmaloney: I updated my PR. Thanks for the review. Please review the updated PR.

Copy link
Copy Markdown
Contributor

@cmaloney cmaloney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvement overall; I don't have a BSD system to test. Definitely an improvement on macOS.

@vstinner vstinner enabled auto-merge (squash) May 25, 2026 20:50
@vstinner vstinner merged commit cb60f48 into python:main May 25, 2026
88 of 91 checks passed
@vstinner vstinner deleted the mem_watchdog branch May 25, 2026 21:15
@vstinner
Copy link
Copy Markdown
Member Author

FAIL: test_python_legacy_windows_stdio (test.test_cmd_line.CmdLineTest.test_python_legacy_windows_stdio)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\test_cmd_line.py", line 1047, in test_python_legacy_windows_stdio
    self.assertEqual(p.returncode, 0)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
AssertionError: 3221225794 != 0

I created issue gh-150436 to track this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants