gateway: fix TTS latency + restore audio warm path
- patch_audio_default_mp3.py: default /v1/audio/speech to response_format=mp3 when the caller omits it. LibreChat sends no response_format, and Tinfoil's voxtral-tts then serves WAV: measured ~8-17s warm (145-300KB) vs ~2.6-3.1s for mp3 (28-39KB, provider streams it). LibreChat labels the reply audio/mpeg regardless, so WAV bytes rode under an mp3 label. Explicit response_format from any caller still wins. - start.sh: scheduler tasks arrive as ARGUMENTS to this ENTRYPOINT script (docker run <image> <command>); they were booting a second tinfoil-proxy + LiteLLM and dying on the bind FATAL every run, so audio_keepalive never executed. Detect the arg and exec the task directly. - start.sh: own the warm path in the app container regardless of the scheduler container: ping tinfoil/voxtral-tts every 480s in the background (silent on success; failures append to /app/data/keepalive.log).
This commit is contained in:
1 parent
943701581c
commit
5d38403892
3 files changed
+139
No files matched your search
@@ -6,6 +6,18 @@ set -eu
|
||||
# Configures LiteLLM with Cloudron's PostgreSQL and Redis addons
|
||||
# ============================================================================
|
||||
|
||||
# --- Cloudron scheduler tasks arrive as ARGUMENTS to this script ---
|
||||
# The manifest's scheduler commands run as `docker run <image> <command>`, and
|
||||
# this script is the image ENTRYPOINT — so the task command lands in "$@".
|
||||
# Without this branch every scheduled task booted a second tinfoil-proxy and
|
||||
# LiteLLM (whose ports are already bound in the app's network namespace) and
|
||||
# died on the bind FATAL before doing its job — which looked exactly like a
|
||||
# scheduler that never fired. Exec the task directly instead of booting.
|
||||
if [ "$#" -gt 0 ]; then
|
||||
echo "start.sh: scheduler-task mode — exec: $*"
|
||||
exec "$@"
|
||||
fi
|
||||
|
||||
echo "=== LiteLLM Cloudron Startup ==="
|
||||
|
||||
# --- Cloudron PostgreSQL addon provides:
|
||||
@@ -227,4 +239,20 @@ if [ "$proxy_up" != "1" ]; then
|
||||
fi
|
||||
echo "Tinfoil proxy is up on 127.0.0.1:3301"
|
||||
|
||||
# --- In-container audio keep-alive loop ---
|
||||
# The scheduler task (handled above in scheduler-task mode) runs in a SEPARATE
|
||||
# container whose ability to reach 127.0.0.1:4000 and read /app/data is not
|
||||
# guaranteed, so the warm path is owned HERE — the app container, where the
|
||||
# network namespace, the master key and the sidecar are all proven. Every 8
|
||||
# minutes ping tinfoil/voxtral-tts ($0.002/request) so voice mode skips the
|
||||
# ~24s Tinfoil cold-start after idle. audio_keepalive.py is silent on success
|
||||
# and prints on failure, so /app/data/keepalive.log only ever grows on failure.
|
||||
(
|
||||
while true; do
|
||||
python3 /app/code/audio_keepalive.py >> /app/data/keepalive.log 2>&1 || true
|
||||
sleep 480
|
||||
done
|
||||
) &
|
||||
echo "Audio keep-alive loop started (every 480s)"
|
||||
|
||||
exec litellm --config "$CONFIG_FILE" --port 4000 --host 0.0.0.0
|
||||
Reference in new issue
Block a user