Skip to content
Smelter logo Smelter logo Smelter

Overview

Side channel is a Unix-socket stream of decoded media that Smelter exposes alongside its normal output. An external process (typically Python) can read decoded RGBA video frames and PCM audio batches from a Smelter input. A common use is running ML inference on that data and feeding the results back to Smelter via the HTTP API.

How it works

On startup Smelter prepares a side-channel socket directory pointed to by SMELTER_SIDE_CHANNEL_SOCKET_DIR. The directory must be empty (Smelter creates it if missing).

When you register an input with side_channel.video: true and/or side_channel.audio: true, Smelter creates one Unix socket per enabled track inside that directory:

  • video_<input_id>.sock: decoded RGBA video frames.
  • audio_<input_id>.sock: decoded PCM audio batches.

A consumer process connects to the socket and receives a stream of decoded frames / batches. Acting on that data happens outside the side channel protocol, by calling Smelter’s regular HTTP API.

Enabling side channel

Enable side channel per track when registering an input:

Example request 

POST: /api/input/:input_id/register
Content-Type: application/json


{
  "type": "mp4",
  "url": "https://example.com/video.mp4",
  "side_channel": {
    "video": true,
    "audio": true
  }
}

The side_channel field is supported by every server-side input type. See each input’s reference page for the exact field definition (e.g. MP4).

Buffering frames ahead with delay_ms

By default the side channel publishes each frame at the moment the queue consumes it, which leaves no time for the subscriber to process it before it appears in the composed output. Set side_channel.delay_ms to buffer frames for that many milliseconds ahead of the queue, so the subscriber receives them early and has roughly that long to react (e.g. run inference) before the frame is due.

{
  "type": "mp4",
  "url": "https://example.com/video.mp4",
  "side_channel": {
    "video": true,
    "delay_ms": 500
  }
}

Higher delays give the consumer more processing time at the cost of additional end-to-end latency on the composed output.

Configuring the socket directory

Smelter writes side-channel sockets into the directory pointed to by the SMELTER_SIDE_CHANNEL_SOCKET_DIR environment variable. The same path must be visible to the consuming process. See SMELTER_SIDE_CHANNEL_SOCKET_DIR in the server configuration reference.

Consuming side channel data

Install smelter-sdk and subscribe to a channel by input id:

pip install smelter-sdk
from smelter import subscribe_video_channel


for frame in subscribe_video_channel("input_1"):
    # frame.rgba is the decoded RGBA buffer — run inference / analysis
    # and post results back via Smelter's HTTP API as needed.
    ...

Run the consuming process with SMELTER_SIDE_CHANNEL_SOCKET_DIR pointing at the same directory Smelter writes its sockets to:

SMELTER_SIDE_CHANNEL_SOCKET_DIR=/path/to/sockets python my_consumer.py

See the Python API reference for the full surface (sync + async, audio batches, channel discovery).

Guides