gh-144503: Pass sys.argv via a route less limited by single command line argument length#144508
gh-144503: Pass sys.argv via a route less limited by single command line argument length#144508sjoerdjob wants to merge 6 commits intopython:mainfrom
sys.argv via a route less limited by single command line argument length#144508Conversation
The maximum length of a single command line argument is more restricted than the total size of all command line arguments together.
|
All commit authors signed the Contributor License Agreement. |
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
The `resource` module is not available in a WASI environment.
sjoerdjob
force-pushed
the
issue-144503-multiprocessing-forkserver-arglen
branch
from
c375314 to
0507aa0
Compare
Lib/multiprocessing/forkserver.py
Outdated
| else: | ||
| authkey = b'' | ||
|
|
||
| if preload: |
There was a problem hiding this comment.
I'm kind-of inclined to think that this logic should be in _handle_preload, but I'm not sure how to combine that with the tests that were introduced in gh-141859.
…ad of argv The allowed size for argv is limited, while the amount of data that can be sent over a pipe is virtually unlimited.
| main_kws = {} | ||
| if self._preload_modules: | ||
| data = spawn.get_preparation_data('ignore') | ||
| if 'sys_path' in data: |
There was a problem hiding this comment.
The if ... in data have been removed, because a cursory reading of get_preparation_data shows that the sys_path, sys_argv are always present in the returned dictionary.
The init_main_from_path is not always present, but the default argument in _handle_preload is None, so I think using that same value here is fine.
Similarly to on_error: this was checking if it was equal to the default value of _handle_preload, and if so, would not pass it. If the default value of _handle_preload would be changed, and this statement here would not be, that would cause an inconsistency. I think it's better to just always pass the value here to prevent future drift.
sjoerdjob
changed the title
sys.argv as separate command line arguments.sys.argv via a route less limited by single command line argument length
- Use buffered os.fdopen() for both writing and reading the init pipe so that preload payloads larger than PIPE_BUF are written and read fully. A bare os.write() may short-write large buffers. - Use data.get() for preparation-data keys rather than direct indexing to avoid coupling to spawn.get_preparation_data() internals.
| main_kws['on_error'] = self._preload_on_error | ||
| preload_kwargs = { | ||
| "preload": self._preload_modules, | ||
| "sys_path": data.get("sys_path"), |
There was a problem hiding this comment.
This data.get("sys_path") can just be data["sys_path"]. I feel quite certain that there is no flow through get_preparation_data where the key "sys_path" is not in the returned dictionary.
…rocessing-forkserver-arglen
2 participants
The maximum length of a single command line argument is more restricted than the total size of all command line arguments together.
multiprocessing.Poolfails if command has a long list of arguments (regression in 3.14.3, 3.13.12) #144503