[LMI v20] ModuleNotFoundError: vllm.entrypoints.openai.protocol breaks chat completions #5719
Description
Cross-posting from deepjavalibrary/djl-serving#3000 as this affects the AWS LMI DLC images and i'm not getting any traction there.
In LMI v19/v20 (DJL 0.36.0), any request using messages format fails with ModuleNotFoundError: No module named 'vllm.entrypoints.openai.protocol' because vllm_chat_utils.py references a module path that was refactored in vLLM 0.13.0+.
Affected images: 763104351884.dkr.ecr.ca-central-1.amazonaws.com/djl-inference:0.36.0-lmi20.0.0-cu128-v1.0
I am able to get some predictions by formatting my requests in sagemaker with "inputs" instead of "messages" but it seems to have the side effects of breaking some other parameters such as guided_choices, log probs and so on.