Add orcarouter_chat example#323
Conversation
Greptile SummaryThis PR adds a new
Confidence Score: 3/5The main app file has a concurrency issue in the submit path that should be resolved before this example ships. In orcarouter_chat/orcarouter_chat/orcarouter_chat.py — specifically the Important Files Changed
Reviews (3): Last reviewed commit: "Guard against Clear during streaming" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
- load_models: run as background task (release state lock during HTTP fetch) - Enter key submits via rx.form (was: button-only) - pinned custom models persist across catalog refreshes - assets/favicon.ico: include reflex default
masenf
left a comment
masenf
left a comment
There was a problem hiding this comment.
it might be worthwhile to separate the processing loop and message streaming to its own state. that way other elements on the page won't constantly be re-rendering as the model streams partial messages.
| try: | ||
| from dotenv import load_dotenv | ||
|
|
||
| _here = Path(__file__).resolve().parent | ||
| for candidate in (_here / ".env", _here.parent / ".env"): | ||
| if candidate.exists(): | ||
| load_dotenv(candidate, override=False) | ||
| break | ||
| except ImportError: | ||
| pass |
There was a problem hiding this comment.
reflex does have dotenv handling built in if you have python-dotenv installed.
you have env_file in rxconfig.py, so is this boilerplate actually needed here?
- drop dotenv boilerplate: reflex already loads .env via Config(env_file=".env") + python-dotenv from requirements - split State into ConfigState (model picker, fallback toggle, catalog) and ChatState (messages, streaming, prompt, tokens) so chunk-by-chunk message updates do not re-render config components
| self.prompt = "" | ||
| history = list(self.messages[:-1]) | ||
|
|
||
| config = await self.get_state(ConfigState) |
There was a problem hiding this comment.
i tried to submit a prompt and it crashed here
[Reflex Backend Exception]
Traceback (most recent call last):
File
"/Users/masenf/code/reflex-dev/reflex-examples/orcarouter_chat/.venv/lib/python3
.14/site-packages/reflex_base/event/processor/event_processor.py", line 737, in
_finish_task
future.result()
~~~~~~~~~~~~~^^
File
"/Users/masenf/code/reflex-dev/reflex-examples/orcarouter_chat/.venv/lib/python3
.14/site-packages/reflex_base/event/processor/event_processor.py", line 728, in
_finish_task
result = task.result()
File
"/Users/masenf/code/reflex-dev/reflex-examples/orcarouter_chat/.venv/lib/python3
.14/site-packages/reflex_base/event/processor/event_processor.py", line 565, in
_process_event_queue_entry
await self._execute_event(entry=entry,
registered_handler=registered_handler)
File
"/Users/masenf/code/reflex-dev/reflex-examples/orcarouter_chat/.venv/lib/python3
.14/site-packages/reflex_base/event/processor/base_state_processor.py", line
371, in _execute_event
await process_event(
...<4 lines>...
)
File
"/Users/masenf/code/reflex-dev/reflex-examples/orcarouter_chat/.venv/lib/python3
.14/site-packages/reflex_base/event/processor/base_state_processor.py", line
238, in process_event
events = await fn(**payload)
^^^^^^^^^^^^^^^^^^^
File
"/Users/masenf/code/reflex-dev/reflex-examples/orcarouter_chat/orcarouter_chat/o
rcarouter_chat.py", line 251, in submit
config = await self.get_state(ConfigState)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/masenf/code/reflex-dev/reflex-examples/orcarouter_chat/.venv/lib/python3
.14/site-packages/reflex/istate/proxy.py", line 308, in get_state
raise ImmutableStateError(msg)
reflex_base.utils.exceptions.ImmutableStateError: Background task StateProxy is
immutable outside of a context manager. Use `async with self` to modify state.
get_state calls have to be done inside async with self
There was a problem hiding this comment.
Thanks, fixed in b6b5006. Moved the get_state call inside async with self. Confirmed locally with a few prompts.
Background-task StateProxy is immutable outside of the locking context; calling get_state outside `async with self` raises ImmutableStateError (reported by @masenf running the example locally).
| @rx.event(background=True) | ||
| async def submit(self): | ||
| async with self: | ||
| prompt = self.prompt.strip() | ||
| if not prompt or self.streaming: | ||
| return | ||
| api_key = os.environ.get("ORCAROUTER_API_KEY", "").strip() | ||
| if not api_key: | ||
| self.error = ( | ||
| "ORCAROUTER_API_KEY is not set. Create a .env next to " | ||
| "rxconfig.py (or export the env var before running) and " | ||
| "restart `reflex run`." | ||
| ) | ||
| return | ||
| config = await self.get_state(ConfigState) | ||
| model = config.model | ||
| use_fallback = config.use_fallback_route | ||
| fallback_csv = config.fallback_models_csv | ||
| self.error = "" |
There was a problem hiding this comment.
get_state(ConfigState) is awaited on line 241 while the ChatState state lock is still held from async with self: on line 229. Any other event that needs ChatState's lock (e.g., set_prompt as the user types) is blocked until ConfigState's lock is also free. Moving get_state before the async with self: block eliminates this cross-lock hold.
| @rx.event(background=True) | |
| async def submit(self): | |
| async with self: | |
| prompt = self.prompt.strip() | |
| if not prompt or self.streaming: | |
| return | |
| api_key = os.environ.get("ORCAROUTER_API_KEY", "").strip() | |
| if not api_key: | |
| self.error = ( | |
| "ORCAROUTER_API_KEY is not set. Create a .env next to " | |
| "rxconfig.py (or export the env var before running) and " | |
| "restart `reflex run`." | |
| ) | |
| return | |
| config = await self.get_state(ConfigState) | |
| model = config.model | |
| use_fallback = config.use_fallback_route | |
| fallback_csv = config.fallback_models_csv | |
| self.error = "" | |
| @rx.event(background=True) | |
| async def submit(self): | |
| config = await self.get_state(ConfigState) | |
| model = config.model | |
| use_fallback = config.use_fallback_route | |
| fallback_csv = config.fallback_models_csv | |
| async with self: | |
| prompt = self.prompt.strip() | |
| if not prompt or self.streaming: | |
| return | |
| api_key = os.environ.get("ORCAROUTER_API_KEY", "").strip() | |
| if not api_key: | |
| self.error = ( | |
| "ORCAROUTER_API_KEY is not set. Create a .env next to " | |
| "rxconfig.py (or export the env var before running) and " | |
| "restart `reflex run`." | |
| ) | |
| return | |
| self.error = "" |
Rule Used: API calls should be made as background tasks to av... (source)
Learned From
reflex-dev/flexgen#2091
There was a problem hiding this comment.
Keeping get_state inside async with self, since moving it outside the lock raises ImmutableStateError on the StateProxy (already caught earlier on this PR). Clear-during-streaming race is fixed in 1fadc07: early return in clear_chat when streaming is active, plus a length check inside the consume loop.
Radix Themes' Button only greys the element visually when disabled is set; it still fires on_click. Belt and suspenders: - visual disabled=ChatState.streaming on the Clear button - early return in clear_chat when streaming is active - length and role check in the consume loop so a stray clear cannot IndexError on msgs[-1]
2 participants
Adds an orcarouter_chat example: a minimal Reflex chat app that calls
OrcaRouter, an OpenAI-compatible LLM gateway, and lets you switch the model
from a dropdown populated at startup from OrcaRouter's public pricing
catalog, with a curated fallback list when that endpoint is unreachable.
Disclosure: I'm an engineer on the OrcaRouter team.