Skip to main content

Audio Formats - Real-time API

Audio data specifications for real-time streaming.

Audio Data Format​

The real-time WebSocket API accepts raw audio data sent as binary (ArrayBuffer).

TypeScript Type​

sendAudioData: (audioData: ArrayBuffer) => void;

Sending Audio Data​

Audio data is sent as binary through the WebSocket connection when the connection state is 'connected'.

The sendAudioData function sends raw audio ArrayBuffer data:

const sendAudioData = useCallback(
  (audioData: ArrayBuffer) => {
    if (
      websocketRef.current?.readyState === WebSocket.OPEN &&
      connectionState === 'connected'
    ) {
      websocketRef.current.send(audioData);
    }
  },
  [connectionState]
);

Audio Source Modes​

Microphone Mode​

Audio is captured from the client's microphone and sent as binary ArrayBuffer data to the WebSocket after the connection is established and the start command is sent.

Stream Mode​

Audio is streamed from a URL provided by the client. The server fetches and processes the audio stream directly.

Supported Stream Types:

  • Direct MP3/M3U8 file URLs
  • Azure Blob Storage URLs
  • Video files (.mp4)

Notes​

  • Audio data must be sent as ArrayBuffer (binary)
  • Only send audio when connection state is 'connected'
  • WebSocket must be in OPEN state (readyState === WebSocket.OPEN)
  • For microphone mode, send start command before sending audio data
  • For stream mode, server handles audio fetching from the provided URL