Implement Whisper Speech-to-Text

You've built the prompt pipeline and the retrieval layer, but the moment a user tries to feed a voice memo into your chatbot, the whole system stalls. Audio is the most natural human interface, and yet, parsing it reliably into text is a problem that trips up even senior engineers. In this lesson, you'll learn how to implement speech-to-text with Whisper — OpenAI's open-source ASR model — turning raw audio into clean, timestamped text in minutes, not weeks. By the end, you'll have a working transcription script, a clear mental model of how Whisper processes sound, and the knowledge to integrate it into your AI products.

The problem this lesson solves

Speech recognition is deceptively hard. Accents, background noise, overlapping speakers, and technical jargon all conspire to produce garbled text. Traditional approaches required specialized hardware or expensive cloud APIs with opaque pricing. Whisper changes that: it's a free, open-source model that runs on your own hardware and handles a wide range of languages and audio conditions with surprising accuracy.

But "running Whisper" isn't the same as "implementing Whisper properly." Many developers hit the same wall: they install the package, try the first audio file, get a transcription that's 50% wrong, and give up. The real problem isn't the model — it's the implementation: choosing the right model size, preprocessing the audio correctly, handling long files, and interpreting the output structure.

This lesson gives you a practical, step-by-step blueprint to implement speech-to-text with Whisper, avoiding the pitfalls and building a foundation you can extend to real-time, multi-language, or summary-driven applications.

Core concept / mental model

Think of Whisper as a three-stage pipeline, not a single black box.

1. Audio to log-mel spectrogram — The raw waveform is transformed into a visual-like representation (a spectrogram) that highlights frequency patterns over time.
2. Encoder — A transformer encoder processes the spectrogram, extracting high-level features about the sound.
3. Decoder — A transformer decoder autoregressively generates the text transcript, token by token, using the encoder's output and previously generated words.

A helpful analogy: Whisper is like a simultaneous interpreter who doesn't just translate words, but also captures tone, pauses, and emphasis. The spectrogram is the interpreter's shorthand notes; the encoder is their memory; the decoder is their voice. But unlike a human interpreter, Whisper operates on token probabilities — it doesn't understand meaning, it predicts the most likely next token based on patterns learned from millions of hours of audio.

Understanding this mental model is crucial because it explains Whisper's behavior: - Why background noise hurts accuracy (it adds confusing features to the spectrogram) - Why it sometimes 'hallucinates' words (the decoder can generate plausible but nonexistent text) - Why chunking long audio helps (the model has a fixed context window)

Pro tip: Whisper is not a speech-to-text API — it's a model. You control the preprocessing, the decoding, and the integration. That's both the power and the responsibility.

How it works step by step

Here's the logical sequence to implement speech-to-text with Whisper, from raw audio to final transcript:

1. Install the dependencies — openai-whisper (the Python package) and ffmpeg (for audio decoding).
2. Load the model — Choose a size (tiny, base, small, medium, large). Bigger models are more accurate but slower and need more memory.
3. Preprocess the audio (optional but recommended) — Resample to 16 kHz, convert to mono, and trim silences. Whisper works with multiple formats, but clean input yields better output.
4. Run transcription — Call model.transcribe() with the audio file path. The function returns a dictionary with the full text and segment-level details.
5. Handle long files — Whisper can process up to ~30 seconds at a time natively. The package splits longer audio automatically, but you might want to manage chunks yourself for control.
6. Post-process the output — Extract segments, filter based on confidence, or join with timestamps for alignment.

Each step builds on the last. Skipping preprocessing or choosing the wrong model size are the most common causes of poor results.

Installing Whisper and ffmpeg

# Install the Whisper Python package
pip install openai-whisper
# Install ffmpeg (on Ubuntu/Debian)
sudo apt update && sudo apt install ffmpeg
# On macOS with Homebrew
brew install ffmpeg

Hands-on walkthrough

Now let's implement speech-to-text with Whisper in practice. We'll start with a minimal script, then expand with preprocessing and error handling.

Minimal transcription script

import whisper

model = whisper.load_model("base")
result = model.transcribe("meeting.mp3")

print("Full transcript:")
print(result["text"])

print("\nSegments:")
for seg in result["segments"]:
    print(f"[{seg['start']:.2f} - {seg['end']:.2f}] {seg['text']}")

Expected output:

Full transcript:
 Welcome everyone, let's start the meeting... 

Segments:
[0.00 - 4.12]  Welcome everyone, let's start the meeting...
[4.12 - 8.55]  First on the agenda is the Q3 report.

Pro tip: The base model is a good balance of speed and accuracy for most practical use cases. Start there, and only move to small or medium if accuracy is critical.

Advanced: preprocessing and custom decoding

Here's a more realistic implementation that includes resampling and language detection:

import whisper
from pydub import AudioSegment  # optional, useful for preprocessing
import io

def preprocess_audio(input_path, output_path, target_sr=16000):
    """Convert audio to 16kHz mono WAV for best Whisper performance."""
    audio = AudioSegment.from_file(input_path)
    audio = audio.set_frame_rate(target_sr).set_channels(1)
    audio.export(output_path, format="wav")
    return output_path

# Load the model once
tiny_model = whisper.load_model("tiny")
base_model = whisper.load_model("base")

# Preprocess the audio
preprocessed = preprocess_audio("noisy_interview.m4a", "clean_input.wav")

# Transcribe with timestamps and language detection
result = base_model.transcribe(
    preprocessed,
    language="en",          # optional: force language
    fp16=False,              # use FP32 on CPU for better stability
    condition_on_previous_text=False  # avoids hallucination loops
)

# Write transcript to a file
with open("transcript.txt", "w") as f:
    f.write(result["text"])

print(result["language"])

Expected output:

en

Handling long audio files

Whisper's built-in chunking works, but for files over an hour, you might want more control:

import whisper
from pydub import AudioSegment

CHUNK_LENGTH_MS = 30000  # 30 seconds

def transcribe_long_audio(model, audio_path):
    audio = AudioSegment.from_file(audio_path)
    full_text = []

    for i, chunk_start in enumerate(range(0, len(audio), CHUNK_LENGTH_MS)):
        chunk = audio[chunk_start:chunk_start + CHUNK_LENGTH_MS]
        chunk_path = f"chunk_{i}.wav"
        chunk.export(chunk_path, format="wav")

        result = model.transcribe(chunk_path, fp16=False)
        full_text.append(result["text"])
        print(f"Processed chunk {i+1}", flush=True)

    return " ".join(full_text)

model = whisper.load_model("medium")
transcript = transcribe_long_audio(model, "podcast_2h.mp3")
print(transcript)

Expected output: a long string of the full podcast transcript, printed after progress messages. (The exact text depends on the audio.)

Compare options / when to choose what

Not all transcription needs are the same. Here's how Whisper stacks up against alternatives:

When to choose what:

• Choose local Whisper when you need privacy, offline capability, or want to fine-tune the model.
• Choose Faster-Whisper if you need lower latency on CPU or want to process long audio quickly.
• Choose cloud APIs when you need massive scale and don't want to manage infrastructure.
• Choose Whisper API for a quick prototype without GPU.

Variations and alternatives

• Faster-Whisper — A reimplementation using CTranslate2, up to 4x faster, same accuracy.
• WhisperX — Adds word-level timestamps and voice activity detection (VAD) for better alignment.
• Distil-Whisper — A distilled version of Whisper, 5x smaller and faster, slightly lower accuracy.

Troubleshooting & edge cases

You'll inevitably run into issues. Here are the most common ones and how to fix them.

"FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg'"

This means ffmpeg is not installed or not in your PATH. Install it via your package manager (as shown earlier). On Windows, download the binary and add it to PATH.

Model outputs empty or gibberish

This usually happens when the audio is too noisy or the language is not detected correctly. Try: - Forcing the language: model.transcribe(audio, language="en") - Preprocessing the audio (noise reduction, normalization) - Using a larger model

Transcription slows down or runs out of memory

Larger models consume significant RAM/VRAM. Use model.transcribe(..., fp16=False) on CPU. For GPU, ensure you have CUDA installed and PyTorch with CUDA support.

Long files cause repeated text (hallucinations)

Whisper sometimes loops on repeated words for long audio. Set condition_on_previous_text=False to reduce this, and chunk the audio yourself.

Timestamps are misaligned

If you need precise word-level alignment, use WhisperX. Whisper's segment-level timestamps can be off by a few hundred milliseconds.

What you learned & what's next

In this lesson, you learned how to implement speech-to-text with Whisper — from understanding its core architecture to running real transcribing scripts, handling long audio, and choosing the right configuration for your use case. You can now explain how the spectrogram-encoder-decoder pipeline works, and you've completed a hands-on exercise that produces transcripts with timestamps.

This is the foundation for many AI applications: voice assistants, meeting summary systems, content search, and more. Your next step is to connect Whisper to larger workflows — for example, passing the transcript to an LLM for summarization or building a RAG pipeline over audio content.