faster-whisper CUDA Setup: Local Whisper Transcription, VAD, and Batching

faster-whisper is a Whisper inference implementation maintained by SYSTRAN. It uses CTranslate2 as the backend, keeping the workflow close to Whisper while making inference speed, memory use, and deployment flexibility more suitable for engineering work.

If you have used openai/whisper, you can think of faster-whisper as a more production-oriented alternative. The interface still centers on loading a model, transcribing audio, and reading segmented results, but the execution layer is faster and easier to tune around CPU, GPU, quantization, and batching.

Quick Answer

Use faster-whisper when you want local or server-side speech-to-text that is faster and easier to tune than the original Whisper implementation. Start with large-v3 on CUDA with float16 when quality matters, use int8_float16 if GPU memory is tight, and use smaller models with int8 for CPU-only background jobs.

For subtitle workflows, enable VAD for long silence and word-level timestamps when you need more precise alignment.

What problem does it solve?

Whisper works well, but the original implementation often runs into a few issues when deployed directly:

1. Long audio can take a noticeable amount of time to transcribe.
2. GPU memory usage can be high.
3. CPU execution works, but speed may not be ideal.
4. Throughput is not always easy to scale when processing large batches of audio or video.

faster-whisper mainly optimizes around these problems. Its README states that, with the same accuracy, it can be up to 4 times faster than openai/whisper while using less memory. With 8-bit quantization, speed can improve further.

Installation

In a regular Python environment, install it directly:

1

pip install faster-whisper

If you want to use a GPU, make sure your local CUDA, cuDNN, and CTranslate2 versions are compatible. This is the easiest place to stumble: the code itself may be fine, but inference can fail when loading the model or running the first request if the GPU driver and CUDA runtime do not match.

Basic usage

The minimal example is straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

from faster_whisper import WhisperModel

model_size = "large-v3"

model = WhisperModel(model_size, device="cuda", compute_type="float16")

segments, info = model.transcribe("audio.mp3", beam_size=5)

print("Detected language '%s' with probability %f" % (info.language, info.language_probability))

for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

The key parameters are:

If your goal is quick local transcription, start by testing medium or large-v3. If GPU memory is tight, then consider quantization.

Choosing CPU or GPU

With an NVIDIA GPU, prefer:

1

model = WhisperModel("large-v3", device="cuda", compute_type="float16")

If GPU memory is not enough, switch to:

1

model = WhisperModel("large-v3", device="cuda", compute_type="int8_float16")

Without a GPU, run it on CPU:

1

model = WhisperModel("small", device="cpu", compute_type="int8")

CPU mode is better for lightweight jobs, low-frequency background tasks, or servers without a graphics card. For a large amount of long audio, GPU is still the better fit.

Batched transcription

faster-whisper also provides batched transcription. Batching is useful for many short audio files or when you need higher GPU throughput:

1
2
3
4
5
6
7
8

from faster_whisper import WhisperModel, BatchedInferencePipeline

model = WhisperModel("turbo", device="cuda", compute_type="float16")
batched_model = BatchedInferencePipeline(model=model)
segments, info = batched_model.transcribe("audio.mp3", batch_size=16)

for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

batch_size is not always better when larger. It improves throughput, but also increases GPU memory pressure. In practice, test values like 4, 8, and 16 step by step until you find a stable point for your machine.

VAD and word-level timestamps

Speech-to-text often has to deal with long silence, background noise, and subtitle alignment. faster-whisper includes practical parameters that can be enabled directly during transcription.

Enable VAD:

1

segments, info = model.transcribe("audio.mp3", vad_filter=True)

Get word-level timestamps:

1
2
3
4
5

segments, info = model.transcribe("audio.mp3", word_timestamps=True)

for segment in segments:
    for word in segment.words:
        print("[%.2fs -> %.2fs] %s" % (word.start, word.end, word.word))

VAD is useful for meeting recordings, podcasts, and livestream replays that contain long silent sections. Word-level timestamps are useful for subtitles, transcript proofreading, and player-side word highlighting.

Choosing a model

Model choice mainly depends on accuracy, speed, and machine resources.

For Chinese speech, start with large-v3 if quality matters. If the machine is under too much pressure, then lower the model size or use quantization. Do not look only at speed at the beginning; if transcription quality drops, the extra manual proofreading time may cancel out the inference time you saved.

Suitable use cases

faster-whisper is well suited for:

1. Generating video subtitles.
2. Transcribing podcasts, meetings, and course recordings.
3. Building local transcription workflows for Bilibili, YouTube, and similar videos.
4. Batch archiving and searching audio content.
5. Feeding speech content into RAG, knowledge bases, or search systems.

It does not directly solve higher-level tasks such as speaker diarization, summarization, or chapter segmentation, but it can serve as a stable transcription layer. You can add pyannote for speaker diarization and an LLM for summarization and structured cleanup.

Deployment suggestions

For real use, debug in this order:

1. Use a 1-to-3-minute audio clip to confirm the environment runs correctly.
2. Test accuracy with samples that match your target language and audio quality.
3. Check GPU memory usage before deciding whether to enable quantization.
4. Split long audio first, so a failed task does not require rerunning everything.
5. Save both TXT and SRT outputs to make later proofreading easier.

For server-side tasks, load the model during service startup instead of reloading it for every request. Model loading takes time, and frequent reloading can also make GPU memory management less stable.

Related Speech And AI Workflow Guides

• OpenMontage Tutorial: turn an AI coding assistant into a video production pipeline
• Microsoft MarkItDown Tutorial: Convert Documents to Markdown for AI Knowledge Bases
• Headroom Guide: Compress AI Agent Context for Claude Code, Codex, and MCP

OpenAI Whisper: Model Scope and Practical Limits

openai/whisper is an OpenAI open source speech recognition project. The thesis direction is Robust Speech Recognition via Large-Scale Weak Supervision. It allows many people to obtain, for the first time at a low threshold, multi-lingual speech transliteration capabilities that can be run locally.

Although today there are faster-whisper, whisper.cpp, various cloud ASR and new generation speech models, the original Whisper is still the starting point for understanding the open source ASR ecosystem.

What is it suitable for?

Common uses for Whisper include:

• Audio to text;
• Video subtitle generation;
• Podcast transcription;
• Minutes of meetings;
• Multilingual speech recognition;
• Voice translation to English;
• Subtitle draft and content retrieval.

Its advantages are robustness, multi-language, open source, and ecological maturity. Many subsequent tools are optimized around the Whisper model or interface.

Use boundaries

Whisper is not a universal dictator:

• Noise, accents, and overlap of multiple people will affect the results;
• Professional terms and names require post-processing;
• Long audio should be segmented;
• Timestamps may not always be perfect;
• The inference speed and resource usage of the original version may not be suitable for production;
• Pay attention to local processing and storage of private audio.

If you need high-throughput production services, you might want to look at faster-whisper, whisper.cpp, batch, quantization, and GPU deployments.

Who is it suitable for?

Suitable:

• Subtitle and transliteration tools;
• Process podcasts, courses, and conference recordings;
• Study ASR models;
• Build local voice-to-text service;
• Organize multi-language content.

If you only occasionally transcribe a piece of audio, a hosted service may be more trouble-free; if you care about privacy and cost, a local deployment is more attractive.

Reference sources

• openai/whisper - GitHub

Deploy faster-whisper on a NAS

The most suitable scenario for deploying Whisper to NAS is not to pursue “real-time conference subtitles”, but to put home videos, interviews, course recordings, podcasts, and conference files into a private directory to generate transcripts and SRT subtitles on demand or in batches.

For most NAS, it is recommended to start with faster-whisper. It uses the CTranslate2 inference engine, can use CPU or CUDA GPU, and supports quantization, VAD silence filtering, and word-level timestamping. First use a small model to run through the process, and then decide whether to upgrade the model based on accuracy and waiting time.

Let’s talk about the conclusion first

The safest route to NAS local speech recognition is:

1
2
3
4
5
6

准备输入 / 输出目录
-> 安装 ffmpeg 与 faster-whisper
-> 用短音频跑 small 或 base
-> 检查 TXT 和 SRT
-> 开启 VAD 过滤
-> 再做批量任务、GPU 加速或 Docker 封装

NAS without discrete graphics can also transcribe, but it should be treated as an offline task. If you want to frequently transcribe long videos, record multiple people, or pursue faster response, the role of the GPU is usually more obvious than simply increasing the hard drive capacity.

How to choose a model

Whisper comes in different size models. Larger models are generally better at handling accents, noise, and complex context, but have slower inference and higher memory requirements.

Don’t judge the effect just by the model name. Recording clarity, speaker overlap, background music, microphone distance, and proper nouns often affect the final transcript more than moving up from small to a larger model.

Directory planning: separate input, output and model cache

Take Linux NAS as an example:

1
2
3

mkdir -p ~/asr/inbox
mkdir -p ~/asr/out
mkdir -p ~/asr/models

Suggested agreement:

1
2
3

~/asr/inbox/   放 MP3、M4A、WAV、MP4 等输入文件
~/asr/out/     放 TXT、SRT、JSON 等结果
~/asr/models/  放下载的模型缓存

Batch tasks are most afraid of mixing input, output and model cache in the same directory. After separation, backup, cleanup and permission control are easier.

Installation environment

In a Debian/Ubuntu-like NAS or Linux container, first install the basic tools:

1
2

sudo apt update
sudo apt install -y python3 python3-venv ffmpeg

Create an independent virtual environment:

1
2
3
4

python3 -m venv ~/asr/.venv
source ~/asr/.venv/bin/activate
pip install --upgrade pip
pip install faster-whisper

ffmpeg is used to read common audio and video formats. Without it, MP4, M4A, or some encoding formats may not be transcoded or read properly.

Minimal transliteration script

Write the following code in ~/asr/transcribe.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

from pathlib import Path
from faster_whisper import WhisperModel

input_file = Path("/home/USER/asr/inbox/sample.mp3")
output_dir = Path("/home/USER/asr/out")
output_dir.mkdir(parents=True, exist_ok=True)

model = WhisperModel(
    "small",
    device="cpu",
    compute_type="int8",
    download_root="/home/USER/asr/models",
)

segments, info = model.transcribe(
    str(input_file),
    language="zh",
    vad_filter=True,
    word_timestamps=True,
)

segments = list(segments)

lines = []
for segment in segments:
    text = segment.text.strip()
    lines.append(text)
    print(f"[{segment.start:.2f} -> {segment.end:.2f}] {text}")

(output_dir / f"{input_file.stem}.txt").write_text(
    "\n".join(lines), encoding="utf-8"
)

Change /home/USER in the code to your actual Linux user directory, then run:

1
2

source ~/asr/.venv/bin/activate
python ~/asr/transcribe.py

When running for the first time, faster-whisper will download the corresponding model. After the model is successfully downloaded, it will be saved in the download_root you specified, and there is no need to download it again in the future.

CPU NAS recommended parameters

When there is no GPU, priority is given to controlling the model size and calculation accuracy:

1
2
3
4
5
6

model = WhisperModel(
    "small",
    device="cpu",
    compute_type="int8",
    download_root="/home/USER/asr/models",
)

int8 is often suitable as a starting point for CPU inference. If the NAS also provides file synchronization, photo indexing, downloading or media services, it is recommended that:

• Start with base or small;
• Avoid running multiple transcription tasks at the same time;
• Long videos are scheduled during off-peak hours;
• Observe system memory, CPU temperature and swap;
• Do not allow the NAS to perform large amounts of transcoding, verification, and transcoding at the same time.

The goal of a CPU NAS should be “stable completion” rather than catching up with audio playback progress in real time.

How to configure when there is NVIDIA GPU

When your NAS or Linux host has an NVIDIA GPU available, try:

1
2
3
4
5
6

model = WhisperModel(
    "small",
    device="cuda",
    compute_type="float16",
    download_root="/home/USER/asr/models",
)

When memory is tight, you can try:

1
2
3
4
5
6

model = WhisperModel(
    "small",
    device="cuda",
    compute_type="int8_float16",
    download_root="/home/USER/asr/models",
)

The actual compute_type available depends on CTranslate2, CUDA, driver and GPU. Verify with short audio first, then process long files. If you put it into Docker, you need to first confirm that the container can see the GPU; when nvidia-smi is not available in the container, the transcriber will not receive CUDA acceleration.

Why is it recommended to turn on VAD filtering

VAD (Voice Activity Detection) skips segments that are obviously speechless. For conference recordings, long pauses, background music, or files with a lot of blank space at the beginning and end of the recording, there are usually two advantages:

• Reduce meaningless reasoning time;
• Reduce the probability of silent segments being recognized as irrelevant text.

Enabled in faster-whisper:

1
2
3
4

segments, info = model.transcribe(
    "input.mp3",
    vad_filter=True,
)

But VAD is not a “never fail” switch. When the voice is very soft, music and human voices are mixed, or the phone recording is intermittent, you should randomly check whether the VAD has accidentally cut off the valid content.

Generate SRT subtitles

Each segment of Whisper has a start and end time. A simple SRT can be output based on the above script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

def srt_time(seconds: float) -> str:
    milliseconds = round(seconds * 1000)
    hours, milliseconds = divmod(milliseconds, 3_600_000)
    minutes, milliseconds = divmod(milliseconds, 60_000)
    seconds, milliseconds = divmod(milliseconds, 1000)
    return f"{hours:02}:{minutes:02}:{seconds:02},{milliseconds:03}"

blocks = []
for index, segment in enumerate(segments, start=1):
    blocks.append(
        f"{index}\n"
        f"{srt_time(segment.start)} --> {srt_time(segment.end)}\n"
        f"{segment.text.strip()}\n"
    )

(output_dir / f"{input_file.stem}.srt").write_text(
    "\n".join(blocks), encoding="utf-8"
)

If you want to do word-by-word highlighting or finer editing, you can read the word-level timestamp after word_timestamps=True. However, word-level results also require manual spot checks, especially mixed Chinese and English, names of people, places and abbreviations.

Do not run full batch transfers in one go

It is recommended to process batch files serially or with low concurrency by queue:

1
2
3

for file in ~/asr/inbox/*; do
  python ~/asr/transcribe.py "$file"
done

Logic for extension filtering, success/failure logging, and skipping completed files should be added to the actual script. The most important thing is: process 5 samples first, confirm that the output directory, language, model and subtitle timeline are correct before starting the entire batch of tasks.

Blindly concurrently running multiple large-v3 transfer tasks on the NAS can easily cause memory, GPU memory, or heat dissipation to become bottlenecks. If the transcription service also shares resources with photo albums, backups, and downloaders, the queue should be limited.

How to use Docker stably

Many NAS are better suited to managing services with Container Manager or Docker. It is recommended to run the Python version on the host or an ordinary Linux container first, and then package the container. When containerizing, at least:

• Mount the input directory read-only;
• Mount the output directory separately and writably;
• Mount the model cache directory persistently;
• Fixed faster-whisper, CTranslate2 and Python versions;
• If using a GPU, first verify container runtime and device passthrough.

Do not mount the entire NAS shared directory into the transcoding container in read-write mode. The speech recognition task only requires reading audio and video and writing text results. The smaller the permissions, the smaller the impact of misoperation.

Accuracy and privacy boundaries

The advantage of local deployment is that the audio does not have to be uploaded to a third-party service, but this does not mean that the output is naturally accurate. Whisper may mishear names, jargon, numbers, and silent segments, as well as unreliable text when exposed to noise or long whitespace.

The following content should be reviewed manually:

• medical, legal, financial and security records;
• Subtitles released to the public;
• Interview quotes and meeting resolutions;
• Name, amount, date, phone number and product model.

Keeping the original audio and treating the transcript as a first draft is a safer process than “delete the original file immediately after transcribing.”

A set of default configurations suitable for NAS

If you don’t have a discrete graphics card:

1
2
3
4
5

model: base 或 small
device: cpu
compute_type: int8
vad_filter: true
并发: 1

If you have an NVIDIA GPU available:

1
2
3
4
5

model: small 起步，按效果再试 medium
device: cuda
compute_type: float16 或 int8_float16
vad_filter: true
并发: 从 1 开始压测

First use the same 10 to 30-minute real recording to compare the time-consuming, omitted words, typos and subtitles timeline before deciding whether to upgrade the model. Don’t just rely on a few seconds of clear samples to judge whether a NAS is suitable for long-term transcription.

Summarize

Whisper NAS deployment can start with faster-whisper: small model, low concurrency, VAD filtering, input and output directory separation. CPU NAS can handle low-frequency offline transcription; GPU NAS with a correct CUDA environment is more suitable for a large number of videos and larger models.

What really determines the experience is not “Whisper can be installed”, but your audio quality, model size, actual transcoding queue and how many other services the NAS is responsible for. Run through a piece of real audio first, and then scale it up.

refer to:

• faster-whisper official warehouse
• OpenAI Whisper official repository
• NAS deployment Ollama performance is not enough

Summary

The value of faster-whisper is that it turns Whisper into a transcription component better suited for long-term use. It is not a different model; it is a more efficient inference backend and engineering interface.

For personal workflows, it can quickly turn videos, meetings, and course audio into text. For server-side tasks, you can tune performance with GPU execution, quantization, batching, and VAD. As long as the machine environment is configured correctly, it is better suited than the original Whisper implementation for stable, batch speech-to-text work.

Project: https://github.com/SYSTRAN/faster-whisper