Skip to content

requests: HTTP/1.1 response bodies and max_body limit#1119

Closed
pablogventura wants to merge 7 commits into
micropython:masterfrom
pablogventura:requests-http11-essentials
Closed

requests: HTTP/1.1 response bodies and max_body limit#1119
pablogventura wants to merge 7 commits into
micropython:masterfrom
pablogventura:requests-http11-essentials

Conversation

@pablogventura

@pablogventura pablogventura commented Jun 1, 2026

Copy link
Copy Markdown

Summary

I use requests on small boards fairly often, and the current package left me hanging on
chunked responses (raw.read() never finished) and still spoke HTTP/1.0 on the wire. This PR
is the first slice of the HTTP work I outlined in #1118: enough client behaviour for real
servers without pulling in TLS or streaming yet.

What I changed:

  • Added requests/_http.py to parse status/headers and read bodies (Content-Length and
    chunked). The full body lands in Response.content so callers do not have to fight the
    socket after the headers.
  • Requests now use HTTP/1.1 with Connection: close (no keep-alive in this PR).
  • resp.headers is a small Headers mapping with case-insensitive keys.
  • Relative Location values on redirects are resolved against the original URL (fixes "requests" does not handle redirection with relative path correctly #931).
  • New max_body argument, default 32 KiB; pass max_body=None if you really want no cap.

Closes #1118.

Testing

I started with the existing socket-mock tests and extended them for bodies, chunked encoding,
redirects, and max_body. On my machine:

python3 -S python-ecosys/requests/test_requests.py

All 14 mock tests pass on CPython 3.12.

I also ran the same file on the unix port (build-standard) and added test_requests_live.py
(local TCP server in a thread) plus run-unix-tests.sh14 + 4 tests, all green.

On hardware I flashed ESP32_GENERIC (ESP32-D0WD, 4 MiB flash), copied this branch’s package
to /lib/requests (the board still ships an older frozen requests, so tests prefer /lib),
and ran run-esp32-tests.sh (mock + live). Same counts, all pass.

Finally I connected the board to my home WiFi and hit real HTTP endpoints (example.com,
httpbin.org GET + redirect) — status 200, bodies as expected. I have not added HTTPS here;
that stays for a follow-up (#838).

CI in this repo should pick up python-ecosys/requests/test_requests.py with the project’s
MicroPython build; the live/ESP32 scripts are mainly for local regression.

Trade-offs and alternatives

Generative AI

I used generative AI tools while working on this PR (including help polishing this
description). I reviewed all of the code and take responsibility for the changes and the
text above.

Introduce read_headers(), read_body(), and chunked decoding
without changing the public request() API yet.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Parse Content-Length and chunked Transfer-Encoding; store body on
Response.content. Default request line to HTTP/1.1. Add max_body
limit (32 KiB default). Fix relative redirect Location URLs.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Cover Content-Length, chunked encoding, max_body, and relative
redirects using the socket mock.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Document HTTP/1.1 response support, max_body, and remaining limits.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Order-independent mock asserts, live TCP tests, unix/ESP32 runners,
prefer /lib over frozen requests on ESP32, and HEAD/auth/encoding fixes.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
@dpgeorge

Copy link
Copy Markdown
Member

Thanks for the contribution.

The entire body is buffered in RAM. That is deliberate for simplicity on microcontrollers;

Unfortunately this change is a non-starter. Microcontrollers need to have simple code -- yes that's true -- but more importantly they need to be able to run in limited RAM.

The current Response object has a .raw member which points to the underlying stream and allows the user to stream out the body as necessary. Eg reading a small amount of data at a time into a pre-allocated buffer. And then there's also .content, .text and .json properties/methods which allow convenient access to the body if you don't care to access .raw. This is all compatible with the upstream CPython requests module and is microcontroller friendly (those two things are hard to achieve!).

This PR removes .raw (makes it a closed socket) which is a non-starter.

I suggest you start instead with a new PR and the simplest set of changes that implement HTTP/1.1. That will be very useful.

@pablogventura

Copy link
Copy Markdown
Author

Closing in favor of #1124. Reworked to preserve .raw streaming per review feedback. This PR is intentionally minimal: HTTP/1.1 + Content-Length only; chunked and other features will follow in separate PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

requests: HTTP/1.1 response bodies and max_body limit "requests" does not handle redirection with relative path correctly

2 participants