Quick start - Node.js

In this guide, you’ll learn how to create a basic Smelter setup in Node.js using TypeScript SDK. We will show you how to take two input videos and produce an output video with both of the inputs displayed side-by-side.

Create Smelter object and initialize it.

import Smelter from "@swmansion/smelter-node"

async function start() {
  const smelter = new Smelter();
  await smelter.init();
}

It will download appropriate binaries and start the Smelter server locally.

Alternatively, you can connect to an existing instance.

import Smelter, { ExistingInstanceManager } from "@swmansion/smelter-node"

async function start() {
  const manager = new ExistingInstanceManager({
    ip: "127.0.0.1",
    port: 8000,
    protocol: 'http',
  });
  const smelter = new Smelter(manager);
  await smelter.init();
}

import { View } from "@swmansion/smelter"
import Smelter from "@swmansion/smelter-node"

async function start() {
  const smelter = new Smelter();
  await smelter.init();

  await smelter.registerOutput("output", <View />, {
    type: "rtp_stream",
    transportProtocol: "udp",
    port: 9004,
    ip: "127.0.0.1",
    video: {
      resolution: { width: 1280, height: 720 },
      encoder: { type: "ffmpeg_h264" },
    },
    audio: {
      encoder: { type: "opus", channels: "stereo" },
    }
  });

  await smelter.start();
}

This step creates an output that renders just an empty <View /> component. Resulting blank image is encoded and sent over UDP to local port 9004 via RTP protocol.

Calling smelter.start() will cause all registered outputs to start producing frames. All user-facing timestamp or offset are relative to this start call.

How to receive and display this stream?

Above example sends that over UDP, so you need to start listening on that port before registering the output.

GStreamer

To view the output stream with GStreamer run the following command:

gst-launch-1.0 rtpptdemux name=demux \\
  udpsrc port=9004 ! \"application/x-rtp\" ! queue ! demux. \\
  demux.src_96 ! \"application/x-rtp,media=video,clock-rate=90000,encoding-name=H264\" ! queue \\
  ! rtph264depay ! decodebin ! videoconvert ! autovideosink \\
  demux.src_97 ! \"application/x-rtp,media=audio,clock-rate=48000,encoding-name=OPUS\" ! queue \\
  ! rtpopusdepay ! decodebin ! audioconvert ! autoaudiosink sync=false

FFmpeg (`ffplay`)

To view the output stream with FFmpeg:

Create the SDP file.

v=0
o=- 0 0 IN IP4 127.0.0.1
s=No Name
c=IN IP4 127.0.0.1
m=video 9004 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1
a=rtcp-mux

Run the following command

ffplay -protocol_whitelist "file,rtp,udp" output.sdp

Note: FFmpeg does not support multiplexing audio and video on the same port, so this example will only work with video.

Or checkout other alternative protocols:

Option 1: WHIP output

With WHIP output you can send it e.g. to Twitch.

import { View } from "@swmansion/smelter"
import Smelter from "@swmansion/smelter-node"

async function start() {
  const smelter = new Smelter();
  await smelter.init();

  await smelter.registerOutput("output", <View />, {
    type: "whip",
    endpointUrl: "https://g.webrtc.live-video.net:4443/v2/offer",
    bearerToken: "<YOUR STREAM TOKEN>",
    video: {
      resolution: { width: 1280, height: 720 },
      encoder: { type: "ffmpeg_h264" },
    },
    audio: {
      encoder: { type: "opus", channels: "stereo" },
    }
  });

  await smelter.start();
}

Option 2: MP4 output

With mp4 output you can record the output to file.

import { View } from "@swmansion/smelter"
import Smelter from "@swmansion/smelter-node"

async function start() {
  const smelter = new Smelter();
  await smelter.init();

  await smelter.registerOutput("output", <View />, {
    type: "mp4",
    serverPath: "output.mp4",
    video: {
      resolution: { width: 1280, height: 720 },
      encoder: { type: "ffmpeg_h264" },
    },
    audio: {
      encoder: { type: "aac", channels: "stereo" },
    }
  });

  await smelter.start();
}

Note that mp4 must be gracefully terminated to write metadata, so if you just kill the application the file will be corrupted. When you want to finish recording, run:

await smelter.unregisterOutput('output');

For simplicity, in this example that will be two mp4 files, but the same logic would apply for any other input type.

import { View } from "@swmansion/smelter"
import Smelter from "@swmansion/smelter-node"

async function start() {
  const smelter = new Smelter();
  await smelter.init();

  await smelter.registerInput("input_1", {
    type: "mp4",
    serverPath: "input1.mp4"
  });

  await smelter.registerInput("input_2", {
    type: "mp4",
    serverPath: "input2.mp4"
  });

  await smelter.registerOutput("output", <View />, {
11 collapsed lines
    type: "rtp_stream",
    transportProtocol: "udp",
    port: 9004,
    ip: "127.0.0.1",
    video: {
      resolution: { width: 1280, height: 720 },
      encoder: { type: "ffmpeg_h264" },
    },
    audio: {
      encoder: { type: "opus", channels: "stereo" },
    }
  });

  await smelter.start();
}

If you run this example now, you will see the same output as in the previous step. Input streams are registered, but they are not used anywhere in the scene definition yet.

Render one of the registered input streams on the output.

In the earlier steps we passed <View /> to the registerOutput call. Now let’s switch that to something more complex. We are defining here new React component called App, and using it as an output.

import { Tiles, InputStream } from "@swmansion/smelter"
import Smelter from "@swmansion/smelter-node"

function App() {
  return <InputStream inputId="input_1" volume={0.9} />
}

async function start() {
  const smelter = new Smelter();
11 collapsed lines
  await smelter.init();

  await smelter.registerInput("input_1", {
    type: "mp4",
    serverPath: "input1.mp4"
  });

  await smelter.registerInput("input_2", {
    type: "mp4",
    serverPath: "input2.mp4"
  })

  await smelter.registerOutput("output", <App />, {
    type: "rtp_stream",
10 collapsed lines
    transportProtocol: "udp",
    port: 9004,
    ip: "127.0.0.1",
    video: {
      resolution: { width: 1280, height: 720 },
      encoder: { type: "ffmpeg_h264" },
    },
    audio: {
      encoder: { type: "opus", channels: "stereo" },
    }
  });

  await smelter.start();
}

Resulting output stream will show input_1 input stream and play the audio from that source with slightly lowered volume. (If input has a different resolution it won’t be automatically rescaled).

Modify the components to display both inputs.

Tiles component is a simple way to display element in a layout that uses available space efficiently (similar to stream layout in most video call apps).

import { Tiles, InputStream } from "@swmansion/smelter"
import Smelter from "@swmansion/smelter-node"

function App() {
  return (
    <Tiles style={{ backgroundColor: "#4d4d4d" }}>
      <InputStream inputId="input_1" volume={0.9} />
      <InputStream inputId="input_2" />
    </Tiles>
  )
}

30 collapsed lines
async function start() {
    const smelter = new Smelter();
    await smelter.init();

    await smelter.registerInput("input_1", {
      type: "mp4",
      serverPath: "input1.mp4"
    });

    await smelter.registerInput("input_2", {
      type: "mp4",
      serverPath: "input2.mp4"
    })

    await smelter.registerOutput("output", <App />, {
      type: "rtp_stream",
      transportProtocol: "udp",
      port: 9004,
      ip: "127.0.0.1",
      video: {
        resolution: { width: 1280, height: 720 },
        encoder: { type: "ffmpeg_h264" },
      },
      audio: {
        encoder: { type: "opus", channels: "stereo" },
      }
    });

    await smelter.start();
}

Resulting output stream will show both inputs side-by-side as shown below. Audio from both streams is mixed together, but input_1 audio has slightly lowered volume.

Quick start - Node.js

GStreamer

FFmpeg (ffplay)

FFmpeg (`ffplay`)