TechResearch/Chatterbox_Experiment/chatterbox_implementation_log

🛠️ Implementation Log: Chatterbox (Resemble AI) Voice Cloning — From Research to Working Pipeline

Companion doc to chatterbox_resemble_ai.md (the research doc). That doc answered "what is Chatterbox and should I use it." This doc is the record of actually building it — every command, every failure, every judgment call, in the order they happened. Written for a future blog post on what self-hosting a real voice-cloning model on Apple Silicon actually involves, past the "pip install and go" pitch.

0. Starting point

Prior session (same day) built a Kokoro-82M TTS integration into this project's video-production dashboard: a Web Worker running the model fully client-side via WebAssembly/ONNX, no backend involved. Two real bugs were found and fixed there (a truncation: true token-limit silently cutting off long scripts, and a cached 304 Not Modified response being treated as "model not found"), which is a separate story — see git history on frontend/src/workers/kokoro.worker.ts.

The ask: do the same thing for Chatterbox, Resemble AI's zero-shot voice-cloning model, per the research doc's recommendation of the Turbo variant (350M params, fastest, native paralinguistic tags like [chuckle]).

First finding: "the same thing" isn't structurally possible. Kokoro is 82M params with a browser-compatible ONNX export — it runs entirely in a Web Worker, zero backend. Chatterbox is 350–500M params, PyTorch-native, with no browser export. It needs a real running Python process. That single fact drove every architectural decision below.

1. The Docker/MPS problem

This project's whole stack (mongo, redis, backend, frontend) runs in docker-compose on macOS (Apple M3 Max, 36GB RAM). The obvious move would've been to add Chatterbox as a fifth service in that same compose file.

That doesn't work well here: Docker Desktop on macOS cannot pass the host's Metal/MPS GPU through to containers. A container-run PyTorch process on this machine is forced to CPU-only, regardless of the M3 Max sitting right there. For an 82M-param Kokoro that's tolerable (if slow); for a 350–500M-param generative model with a diffusion decoder, CPU-only was judged likely to be painfully slow.

Decision (asked the user, not assumed): run Chatterbox as a native macOS process, outside docker-compose, to get real MPS acceleration. Trade-off accepted: it's a separate long-running server you start/stop independently, installed via a third-party repo's install steps rather than one docker-compose up.

Model variant: Turbo, confirmed — fastest, and the only variant with paralinguistic tag support ([laugh], [chuckle], [sigh], [gasp], [cough], [clear throat], [sniff], [groan], [shush]).

Server choice: devnen/Chatterbox-TTS-Server — a community FastAPI wrapper around the official resemble-ai/chatterbox, chosen because it already has a documented Apple Silicon (MPS) install path and a fork (chatterbox-v2) with a specific MPS float64 crash patch pre-applied.

Cloned to a sibling directory outside this git repo — /Users/mihirzz/Codes/mihirzzsolutions/chatterbox-tts-server — rather than nesting a third-party repo inside this one. It's infrastructure, not video content.

2. The Python-version saga

devnen/Chatterbox-TTS-Server's own docs are explicit: Python 3.10 required, 3.11+ unsupported — several pinned dependencies (onnx==1.16.0, descript-audio-codec==1.0.0) predate cp311+ wheel availability.

First attempt: installed a dedicated Python 3.10.20 via pyenv, specifically to match this requirement, and started building a venv for it.

Then, mid-task, the ground shifted: in a separate session, the user did a machine-wide Python consolidation — deleted env311, env39, the pyenv 3.10.20 install, and a stray /Library/Frameworks/Python.framework/Versions/3.11, leaving only Python 3.13 anywhere on the system (conda base + a lean env313 used for this project's own backend/). That pyenv 3.10.20 environment this task had just built was gone.

Told to use env313 (Python 3.13.14) instead. Pushed back initially — 3.13 is further from the documented-supported 3.10 than the already-unsupported 3.11, real risk of hitting packages with no cp313 wheels. The user's counter: the official resemble-ai/chatterbox repo's own README suggests python=3.11 for its base install, and its GitHub had commits within the last 3 weeks — reasonable confidence 3.13 would work for the model even if devnen's wrapper repo hadn't explicitly tested that far.

Called it correctly. Every single dependency in the manual install sequence — torch, torchvision, torchaudio, conformer, diffusers, transformers, einops, omegaconf, resampy, resemble-perth, librosa, safetensors, soundfile, pydub, audiotsm, praat-parselmouth, descript-audio-codec, and about 70 transitive packages — installed cleanly on cp313 with prebuilt wheels, first try. The only real casualty was one specific pinned version (onnx==1.16.0), not the Python version itself (see §4).

3. The `fastapi` conflict

Before installing anything, a real version collision surfaced: devnen's requirements.txt pins fastapi>=0.100.0,<0.116.0 (needed because its bundled web UI code calls Starlette's Jinja2Templates.TemplateResponse with an older calling convention). env313 already had fastapi==0.139.0 installed, matching this project's own backend/requirements.txt (fastapi>=0.139.0).

Installing the pin as-is would have downgraded fastapi/starlette in the shared env313 environment, which this project's own backend dev tooling depends on — exactly the kind of cross-contamination a prior, separate decision (keeping env313 deliberately lean, one project's deps only) was meant to prevent.

Presented three options: (a) build a dedicated Python 3.10 env just for Chatterbox, (b) install into env313 but skip the fastapi pin, accepting the risk that devnen's own bundled web UI might break, (c) install into env313 and just let the downgrade happen.

Decision: skip the pin. Reasoning given: install chatterbox-tts and pull in everything except forcing an old fastapi. Confirmed later this was exactly right — see §5, the bundled web UI page does throw an error, and it doesn't matter, because this project was always going to build its own UI in TTSPanel.tsx and only call the JSON /tts endpoint, which doesn't touch Jinja2 templates at all.

4. The `onnx==1.16.0` build failure

Running the documented manual install sequence for Apple Silicon:

pip install --no-deps git+https://github.com/devnen/chatterbox-v2.git@master \
  s3tokenizer==0.3.0 onnx==1.16.0

failed:

Command '['...cmake', '-DPYTHON_INCLUDE_DIR=.../python3.13', ...]' returned non-zero exit status 1.
ERROR: Failed building wheel for onnx

onnx==1.16.0 (the version pinned in requirements.txt, chosen for protobuf compatibility with an older Python) has no prebuilt wheel for cp313 on PyPI — pip fell back to a from-source cmake build, which failed. Because pip builds all wheels in a single invocation before installing any of them, this also silently prevented chatterbox-tts and s3tokenizer from installing at all, even though their wheel builds had succeeded.

Checked PyPI's release history directly for the first onnx version shipping a cp313 wheel:

curl -s https://pypi.org/pypi/onnx/json | python3 -c "... filter releases for cp313 macos wheels ..."
# → onnx 1.18.0 is the first version with a real (non-rc) cp313 wheel

Fix: bump to onnx==1.18.0. Retried the same install command — all three packages (chatterbox-tts, s3tokenizer, onnx) installed cleanly.

5. The `protobuf` fallout from that fix

Bumping onnx didn't fully resolve things — importing the package chain now failed differently:

File ".../onnx/onnx_ml_pb2.py", line 9, in <module>
    from google.protobuf.internal import builder as _builder
ImportError: cannot import name 'builder' from 'google.protobuf.internal'

Root cause: onnx==1.18.0 requires protobuf>=4.25.1 (the builder module doesn't exist before protobuf 3.20). But descript-audiotools — installed earlier as a transitive dependency, pinned via its own metadata to protobuf<3.20,>=3.9.2 — had already pulled in protobuf==3.19.6. This is the exact conflict devnen's original pinned versions (onnx==1.16.0 + old protobuf) were designed to avoid; bumping onnx for Python-version reasons reintroduced it from the other side.

pip install "protobuf>=4.25.1"
# → installs protobuf-7.35.1, prints a conflict warning for descript-audiotools, but proceeds

pip warned about the conflict but didn't block it. Re-tested the import chain — worked cleanly. descript-audiotools's own protobuf usage (likely a TensorBoard logging path, unused here) never got exercised in a way that broke anything.

chatterbox import OK

Lesson for the blog post: version pins in someone else's requirements.txt are frequently load-bearing for reasons that don't apply to your situation (here: an old Python target), and un-pinning one thing can just move the same conflict to a different pair of packages. The fix isn't "respect all pins" or "ignore all pins" — it's understanding why each pin exists before deciding whether it still applies.

6. MPS verification

python -c "import torch; print(torch.backends.mps.is_available(), torch.backends.mps.is_built())"
# → True True

Set config.yaml's tts_engine.device explicitly to mps (belt-and-suspenders over the default auto, which the changelog says correctly detects MPS anyway as of recent versions).

7. First server start

python server.py

Log output confirmed the important things in order:

[INFO] engine: MPS requested and functional. Using MPS.
[INFO] engine: Final device selection: mps
[INFO] engine: Model selector 'chatterbox-turbo' resolved to Turbo model (ChatterboxTurboTTS)
[INFO] engine: Turbo model supports paralinguistic tags: ['laugh', 'chuckle', 'sigh', 'gasp', 'cough', 'clear throat', 'sniff', 'groan', 'shush']
... (downloads 10 model files from Hugging Face, first run only) ...
[INFO] engine: TTS Model loaded successfully on mps. Engine sample rate: 24000 Hz.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8004

One error, exactly as predicted in §3: hitting the server's own root page (/) threw TypeError: unhashable type: 'dict' inside Starlette's TemplateResponse caching — the fastapi-version-skew consequence of skipping the pin. Confirmed via GET /api/model-info and GET /docs that the actual JSON API and FastAPI's auto-generated Swagger docs (which don't depend on the app's custom Jinja2 setup) both work fine. Irrelevant to this project either way — the plan was always to build a custom UI, not use devnen's bundled one.

Smoke test — predefined voice, no cloning yet:

curl -X POST http://localhost:8004/tts -H "Content-Type: application/json" \
  -d '{"text":"Hey, quick test [chuckle] just making sure this actually works.","voice_mode":"predefined","predefined_voice_id":"Michael.wav","output_format":"wav"}' \
  --output smoketest.wav
# → HTTP 200, valid 24kHz mono 16-bit PCM WAV, generated in ~1 second on MPS

8. Reference clip → voice cloning

Reference clip: mihir.m4a, sitting at the project root. Checked its actual format first (macOS's built-in afinfo, no ffmpeg needed for reading metadata):

Data format: 1 ch, 48000 Hz, aac
estimated duration: 24.661333 sec

Mono, 48kHz AAC, ~24.7 seconds — within the server's max_reference_duration_sec: 30 config ceiling and the research doc's own "keep it under ~1 minute" guidance, comfortably past the "~10-15s is the reliable floor" note too.

Problem: devnen's server documents .wav/.mp3 reference clips — not .m4a — and its underlying audio libraries (soundfile/librosa) can't decode AAC without ffmpeg, which wasn't installed on this machine at all (a warning about this had already surfaced in the server's own startup log: Couldn't find ffmpeg or avconv).

brew install ffmpeg
ffmpeg -i mihir.m4a -ar 24000 -ac 1 -y reference_audio/mihir.wav
# → mono, 24000 Hz, 16-bit PCM WAV, ~24.68s, placed directly in the server's reference_audio/ folder

(Copied the converted file straight into the server's reference_audio/ directory rather than going through its /upload_reference HTTP endpoint — since this is a local native process we already have filesystem access to, that's simpler than round-tripping through multipart upload.)

Cloning test:

curl -X POST http://localhost:8004/tts -H "Content-Type: application/json" \
  -d '{"text":"Hey, this is a test of voice cloning [chuckle] using my own reference clip.","voice_mode":"clone","reference_audio_filename":"mihir.wav","output_format":"wav"}' \
  --output clone_test.wav
# → HTTP 200, 24kHz mono WAV, 7.64 seconds of generated audio

Worked end-to-end on the first real attempt once the format/ffmpeg issue was resolved.

9. Frontend integration — `TTSPanel.tsx`

Added a third engine option ("Chatterbox ✦") alongside the existing "Browser TTS" and "Kokoro AI ✦" toggle in frontend/src/components/TTS/TTSPanel.tsx. Structurally different from Kokoro's integration because of §0/§1: no Web Worker, just a direct fetch() from the browser to http://localhost:8004 (the native process, reachable directly since the browser — not the containerized frontend — makes the request).

What got built:

Voice-mode toggle: Clone Reference vs Predefined, mirroring the server's own voice_mode field
Reference-clip and predefined-voice dropdowns, populated live from the server's GET /get_reference_files and GET /get_predefined_voices — mihir.wav auto-selected by default when present
Two sliders specific to Chatterbox: Exaggeration and CFG Weight — the two knobs the original research doc called out as Chatterbox's actual differentiator over plain cloning
Reused the existing shared Rate slider as the API's speed_factor
A footer note listing the 9 supported paralinguistic tags, so they're discoverable without re-reading the research doc

Infrastructure pieces:

VITE_CHATTERBOX_URL added to docker-compose.yml's frontend service (defaults to http://localhost:8004), same pattern as the existing VITE_API_URL
Confirmed CORS was already wide open server-side (allow_origins=["*", "null"]) — no extra config needed for the cross-origin browser fetch (localhost:3000 → localhost:8004)

10. Where things actually live (for reproducing this later)

What	Where
Chatterbox server code	`/Users/mihirzz/Codes/mihirzzsolutions/chatterbox-tts-server` (sibling to this repo, not inside it)
Python environment	conda `env313` (Python 3.13.14) — shared with this project's `backend/`, not a dedicated env
Reference voice clip	`chatterbox-tts-server/reference_audio/mihir.wav` (converted from root-level `mihir.m4a`)
Server config	`chatterbox-tts-server/config.yaml` — `model.repo_id: chatterbox-turbo`, `tts_engine.device: mps`
Run command	`cd chatterbox-tts-server && conda run -n env313 python server.py` (port 8004)
Frontend integration	`frontend/src/components/TTS/TTSPanel.tsx`, "Chatterbox ✦" engine

11. Trade-offs and failure modes, for the actual write-up

MPS-on-native vs CPU-in-Docker: chose real GPU acceleration over stack consistency. Cost: one more process to remember to start, outside docker-compose up.
Skipping the fastapi pin: chose to keep the shared dev environment's newer fastapi over the third-party server's own bundled UI working. Cost: that server's own web page is broken (confirmed, doesn't matter for this use case).
Bumping onnx past its pin: chose Python-3.13 wheel availability over the exact pinned version. Cost: reintroduced the protobuf conflict from the other direction, needed a second fix.
env313 reuse instead of a dedicated env: chose the user's machine-wide "one Python version" policy over environment isolation for this specific tool. Risk accepted: a heavier, more ML-laden env313 than originally intended ("kept deliberately lean" no longer strictly true).
Net result: every one of these was a real, non-obvious judgment call with an actual cost — not a clean "just pip install it" story, despite the research doc's TL;DR making it sound that simple. That gap between "docs say this is easy" and "here's what it actually took" is the blog post.