Whisper models are incorrectly mapped to Seq2SeqLM

[irudachirath]

Problem

The auto-generated inference notebook/code on the model page is incorrect.

It uses:

from transformers import AutoModelForSeq2SeqLM

base_model = AutoModelForSeq2SeqLM.from_pretrained("openai/whisper-small")

This fails with:

ValueError: Unrecognized configuration class WhisperConfig for AutoModelForSeq2SeqLM

because Whisper is a speech-to-text model and must be loaded with WhisperForConditionalGeneration or AutoModelForSpeechSeq2Seq.

Correct code

from transformers import WhisperForConditionalGeneration
from peft import PeftModel

base_model = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-small"
)

model = PeftModel.from_pretrained(
    base_model,
    "<your-peft-adapter-in-huggingface>"
)

[ariG23498]

Hey @irudachirath

Thanks for reporting this issue with us. The model page for whisper-small autogenerates this notebook.

The code in it:

# Load model directly
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq

processor = AutoProcessor.from_pretrained("openai/whisper-small")
model = AutoModelForSpeechSeq2Seq.from_pretrained("openai/whisper-small")

I don't see an issue here. Could you help me with links of model page and the generated notebook that you are referring to?