gateway: fix TTS latency + restore audio warm path

- patch_audio_default_mp3.py: default /v1/audio/speech to response_format=mp3
  when the caller omits it. LibreChat sends no response_format, and Tinfoil's
  voxtral-tts then serves WAV: measured ~8-17s warm (145-300KB) vs ~2.6-3.1s
  for mp3 (28-39KB, provider streams it). LibreChat labels the reply
  audio/mpeg regardless, so WAV bytes rode under an mp3 label. Explicit
  response_format from any caller still wins.
- start.sh: scheduler tasks arrive as ARGUMENTS to this ENTRYPOINT script
  (docker run <image> <command>); they were booting a second tinfoil-proxy +
  LiteLLM and dying on the bind FATAL every run, so audio_keepalive never
  executed. Detect the arg and exec the task directly.
- start.sh: own the warm path in the app container regardless of the
  scheduler container: ping tinfoil/voxtral-tts every 480s in the background
  (silent on success; failures append to /app/data/keepalive.log).
This commit is contained in:
inference-bot committed 2026-11-07 20:09:54 +00:00
1 parent 943701581c
commit 5d38403892
3 files changed
+139

No files matched your search

+28
View File
@@ -6,6 +6,18 @@ set -eu
# Configures LiteLLM with Cloudron's PostgreSQL and Redis addons
# ============================================================================
# --- Cloudron scheduler tasks arrive as ARGUMENTS to this script ---
# The manifest's scheduler commands run as `docker run <image> <command>`, and
# this script is the image ENTRYPOINT — so the task command lands in "$@".
# Without this branch every scheduled task booted a second tinfoil-proxy and
# LiteLLM (whose ports are already bound in the app's network namespace) and
# died on the bind FATAL before doing its job — which looked exactly like a
# scheduler that never fired. Exec the task directly instead of booting.
if [ "$#" -gt 0 ]; then
echo "start.sh: scheduler-task mode — exec: $*"
exec "$@"
fi
echo "=== LiteLLM Cloudron Startup ==="
# --- Cloudron PostgreSQL addon provides:
@@ -227,4 +239,20 @@ if [ "$proxy_up" != "1" ]; then
fi
echo "Tinfoil proxy is up on 127.0.0.1:3301"
# --- In-container audio keep-alive loop ---
# The scheduler task (handled above in scheduler-task mode) runs in a SEPARATE
# container whose ability to reach 127.0.0.1:4000 and read /app/data is not
# guaranteed, so the warm path is owned HERE — the app container, where the
# network namespace, the master key and the sidecar are all proven. Every 8
# minutes ping tinfoil/voxtral-tts ($0.002/request) so voice mode skips the
# ~24s Tinfoil cold-start after idle. audio_keepalive.py is silent on success
# and prints on failure, so /app/data/keepalive.log only ever grows on failure.
(
while true; do
python3 /app/code/audio_keepalive.py >> /app/data/keepalive.log 2>&1 || true
sleep 480
done
) &
echo "Audio keep-alive loop started (every 480s)"
exec litellm --config "$CONFIG_FILE" --port 4000 --host 0.0.0.0