gateway: fix TTS latency + restore audio warm path

- patch_audio_default_mp3.py: default /v1/audio/speech to response_format=mp3
  when the caller omits it. LibreChat sends no response_format, and Tinfoil's
  voxtral-tts then serves WAV: measured ~8-17s warm (145-300KB) vs ~2.6-3.1s
  for mp3 (28-39KB, provider streams it). LibreChat labels the reply
  audio/mpeg regardless, so WAV bytes rode under an mp3 label. Explicit
  response_format from any caller still wins.
- start.sh: scheduler tasks arrive as ARGUMENTS to this ENTRYPOINT script
  (docker run <image> <command>); they were booting a second tinfoil-proxy +
  LiteLLM and dying on the bind FATAL every run, so audio_keepalive never
  executed. Detect the arg and exec the task directly.
- start.sh: own the warm path in the app container regardless of the
  scheduler container: ping tinfoil/voxtral-tts every 480s in the background
  (silent on success; failures append to /app/data/keepalive.log).
This commit is contained in:
inference-bot committed 2026-11-07 20:09:54 +00:00
1 parent 943701581c
commit 5d38403892
3 files changed
+139

No files matched your search

+8
View File
@@ -101,6 +101,14 @@ RUN chmod +x /app/code/audio_keepalive.py
COPY patch_transcription_response_format.py /app/code/patch_transcription_response_format.py
RUN python3 /app/code/patch_transcription_response_format.py
# TTS default response format: LibreChat sends no response_format on its TTS
# request and Tinfoil's voxtral-tts then serves WAV (~8-17s warm measured) —
# while LibreChat labels the reply audio/mpeg regardless. Default to mp3
# (OpenAI's documented default, ~3s warm measured). Non-fatal; check the
# "[coop-patch] applied" line in the build log on every LiteLLM upgrade.
COPY patch_audio_default_mp3.py /app/code/patch_audio_default_mp3.py
RUN python3 /app/code/patch_audio_default_mp3.py
# Override the base image's ENTRYPOINT (which is `litellm`) so our
# startup script runs instead of being passed as an argument to litellm.
ENTRYPOINT ["/app/code/start.sh"]