High-Performance Speech Recognition in .NET with FasterWhisper.NET

Integrating automatic speech recognition (ASR) into enterprise .NET applications has historically been a challenge. Developers were forced to choose between calling heavy external Python microservices or dealing with slow, unoptimized native wrapper libraries.

To bridge this gap, we engineered FasterWhisper.NET—a production-ready, cross-platform .NET SDK for OpenAI's Whisper model built on top of the high-performance CTranslate2 C++ inference engine.

In this article, we’ll explore the architecture of FasterWhisper.NET, analyze its performance scaling, and walk through how to implement streaming transcription and replica pooling in modern .NET 8.0/9.0/10.0 applications.

---

CTranslate2: Under the Hood

Under the hood, FasterWhisper.NET leverages CTranslate2, an inference engine optimized for Transformer models. Unlike raw PyTorch or standard C++ ports, CTranslate2 implements:

1. Weight Quantization: Custom C++ kernel implementations supporting 8-bit integer (INT8) and 16-bit float (FP16) execution. This reduces memory footprint by up to 4x and dramatically speeds up matrix multiplication.
2. Memory-Mapped Loading: Models are loaded into memory using OS-level mmap systems, allowing instances to share model weights directly in memory without duplicating RAM.
3. Parallel Execution & Shared Weight Replicas: Multiple worker threads (replicas) can concurrently run inference on different audio files while sharing the exact same loaded model parameters in VRAM/RAM.

---

Getting Started with FasterWhisper.NET

Getting started requires only adding the NuGet package. The package ships with pre-compiled native runtimes for Windows, Linux, macOS, Android, and iOS.

dotnet add package FasterWhisper.NET

1. Basic Transcription

Loading a model and transcribing an audio file is clean and idiomatic in C#:

using Qourex.FasterWhisper.NET;

// 1. Download and load the model (cached locally)
using var model = await WhisperModel.LoadAsync(
    modelNameOrPath: "base",      // tiny, base, small, medium, large-v3
    device:          "cpu",       // cpu or cuda
    computeType:     "default"    // float32, float16, int8
);

// 2. Configure options
var options = new WhisperOptions
{
    BeamSize = 5,
    Language = "en",
    WordTimestamps = true
};

// 3. Transcribe audio file (WAV)
var result = await model.TranscribeAsync("sample.wav", options);

foreach (var segment in result.Segments)
{
    Console.WriteLine($"[{segment.Start:hh\\:mm\\:ss} -> {segment.End:hh\\:mm\\:ss}]: {segment.Text}");
}

---

Real-Time Streaming & Voice Activity Detection (VAD)

For live applications, such as real-time captioning or voice assistants, waiting for a full audio file to complete is unacceptable. FasterWhisper.NET implements an asynchronous streaming pipeline powered by Silero VAD v5 running locally via ONNX Runtime.

The VAD pipeline monitors incoming audio segments, determines exactly when speech starts and stops, and feeds active audio frames directly into the Whisper model.

using Qourex.FasterWhisper.NET;

// Load model and VAD engine
using var model = await WhisperModel.LoadAsync("base", "cpu");
using var stream = model.CreateStream(new WhisperOptions { Language = "en" });

// Push audio buffers as they arrive from your mic/network
_ = Task.Run(async () => {
    byte[] wavBuffer = new byte[6400]; // 16kHz mono PCM
    while (await micStream.ReadAsync(wavBuffer) > 0)
    {
        await stream.WriteAsync(wavBuffer);
    }
    stream.Complete(); // Stop streaming
});

// Consume transcribed segments in real-time
await foreach (var segment in stream.GetSegmentsAsync())
{
    Console.WriteLine($"[Live Transcript]: {segment.Text}");
}

---

Performance & Scaling Benchmarks

Below are the official repository benchmark metrics executed on a long-form audio file (972.29 seconds / 16.2 minutes) using the faster-distil-whisper-large-v3 model (756 million parameters).

Test Environment

• CPU: Intel Core i7-4790 (4 Cores / 8 Threads)
• RAM: 32 GB DDR3
• GPU: NVIDIA GeForce GTX 1070 Ti (8 GB VRAM)
• CUDA/cuDNN: 12.4 / 9.1
• OS: Windows 11 Pro

1. Quantization Performance (GPU Execution)

Quantization speeds up matrix multiplication and reduces compute constraints:

Quantization / Compute Type	Inference Duration	Real-Time Factor (RTF)	Speedup vs float32
float32 (default)	44,403.2 ms	0.0457	Baseline
int8	31,708.0 ms	0.0326	1.40x Faster

2. Model Loading Strategies (Memory-Mapped vs Standard)

Memory-mapped loading optimizes process initialization by mapping files directly into address spaces:

Loading Strategy	Initialization Time	CPU RAM Delta	GPU VRAM Delta
Standard Load (No-Mmap)	3,172.8 ms	110.1 MB	3,814.0 MB
Memory-Mapped Load	2,506.2 ms	963.1 MB	3,713.0 MB

3. Multi-Replica Scaling (Shared Memory Footprint)

CTranslate2 allows workers within the same process to share loaded model weights. As the number of replicas (NumReplicas) increases, memory footprint remains virtually constant:

Configuration	Load Time	CPU RAM	GPU VRAM
NumReplicas = 1	2,516.7 ms	963.1 MB	3,712.0 MB
NumReplicas = 2	2,406.2 ms	1,454.2 MB	3,700.0 MB
NumReplicas = 4	2,496.4 ms	1,447.3 MB	3,712.0 MB

---

Conclusion

FasterWhisper.NET brings production-grade speech-to-text directly to C#. By combining CTranslate2's fast native performance with local ONNX-based Voice Activity Detection and clean asynchronous C# interfaces, you can build speech-recognition platforms that deploy anywhere—from standard shared hosting to high-scale GPU clusters.

Check out the source code and give it a star on GitHub.