requests: HTTP/1.1 response bodies and max_body limit#1119
requests: HTTP/1.1 response bodies and max_body limit#1119pablogventura wants to merge 7 commits into
Conversation
Introduce read_headers(), read_body(), and chunked decoding without changing the public request() API yet. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Parse Content-Length and chunked Transfer-Encoding; store body on Response.content. Default request line to HTTP/1.1. Add max_body limit (32 KiB default). Fix relative redirect Location URLs. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Cover Content-Length, chunked encoding, max_body, and relative redirects using the socket mock. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Document HTTP/1.1 response support, max_body, and remaining limits. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Order-independent mock asserts, live TCP tests, unix/ESP32 runners, prefer /lib over frozen requests on ESP32, and HEAD/auth/encoding fixes. Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
|
Thanks for the contribution.
Unfortunately this change is a non-starter. Microcontrollers need to have simple code -- yes that's true -- but more importantly they need to be able to run in limited RAM. The current Response object has a This PR removes I suggest you start instead with a new PR and the simplest set of changes that implement HTTP/1.1. That will be very useful. |
|
Closing in favor of #1124. Reworked to preserve |
Summary
I use
requestson small boards fairly often, and the current package left me hanging onchunked responses (
raw.read()never finished) and still spoke HTTP/1.0 on the wire. This PRis the first slice of the HTTP work I outlined in #1118: enough client behaviour for real
servers without pulling in TLS or streaming yet.
What I changed:
requests/_http.pyto parse status/headers and read bodies (Content-Length andchunked). The full body lands in
Response.contentso callers do not have to fight thesocket after the headers.
Connection: close(no keep-alive in this PR).resp.headersis a smallHeadersmapping with case-insensitive keys.Locationvalues on redirects are resolved against the original URL (fixes "requests" does not handle redirection with relative path correctly #931).max_bodyargument, default 32 KiB; passmax_body=Noneif you really want no cap.Closes #1118.
Testing
I started with the existing socket-mock tests and extended them for bodies, chunked encoding,
redirects, and
max_body. On my machine:All 14 mock tests pass on CPython 3.12.
I also ran the same file on the unix port (
build-standard) and addedtest_requests_live.py(local TCP server in a thread) plus
run-unix-tests.sh— 14 + 4 tests, all green.On hardware I flashed ESP32_GENERIC (ESP32-D0WD, 4 MiB flash), copied this branch’s package
to
/lib/requests(the board still ships an older frozenrequests, so tests prefer/lib),and ran
run-esp32-tests.sh(mock + live). Same counts, all pass.Finally I connected the board to my home WiFi and hit real HTTP endpoints (
example.com,httpbin.orgGET + redirect) — status 200, bodies as expected. I have not added HTTPS here;that stays for a follow-up (#838).
CI in this repo should pick up
python-ecosys/requests/test_requests.pywith the project’sMicroPython build; the live/ESP32 scripts are mainly for local regression.
Trade-offs and alternatives
documented it and added
max_bodyso a runaway response does not silently exhaust memory.(SECURITY: Requests module HTTPS - no server certificate verification. #838, requests: stream parameter is noop #777).
headers for diff stability).
Generative AI
I used generative AI tools while working on this PR (including help polishing this
description). I reviewed all of the code and take responsibility for the changes and the
text above.