peddlers-of-ketran/MEDIACONTROL_API.md

MediaControl WebRTC Signaling Protocol

This document describes the clean, pluggable API for the MediaControl component used for peer-to-peer audio/video communication.

Overview

The MediaControl component provides a reusable WebRTC-based media communication system that can be integrated into any application. It handles:

• Peer-to-peer audio and video streaming
• WebRTC connection management (offers, answers, ICE candidates)
• Signaling via WebSocket
• User controls for muting/unmuting and video on/off

Server-Side Protocol

Required WebSocket Message Types

1. Join Request (Client → Server)

{
  type: "join",
  data: {
    has_media?: boolean  // Whether this peer provides audio/video (default: true)
  }
}

2. Join Status Response (Server → Client)

{
  type: "join_status",
  status: "Joined" | "Joining" | "Error",
  message?: string
}

3. Add Peer (Server → Client)

Sent to all existing peers when a new peer joins, and sent to the new peer for each existing peer.

{
  type: "addPeer",
  data: {
    peer_id: string,        // Unique identifier (session name)
    peer_name: string,      // Display name for the peer
    has_media: boolean,     // Whether this peer provides media
    should_create_offer: boolean,  // If true, create an RTC offer to this peer
    // Legacy fields (optional, for backward compatibility):
    hasAudio?: boolean,
    hasVideo?: boolean
  }
}

4. Remove Peer (Server → Clients)

Sent when a peer disconnects.

{
  type: "removePeer",
  data: {
    peer_id: string,
    peer_name: string
  }
}

5. Relay ICE Candidate (Client → Server → Peer)

// Client sends:
{
  type: "relayICECandidate",
  data: {
    peer_id: string,              // Target peer
    candidate: RTCIceCandidateInit
  }
}

// Server relays to target peer:
{
  type: "iceCandidate",
  data: {
    peer_id: string,              // Source peer
    peer_name: string,
    candidate: RTCIceCandidateInit
  }
}

6. Relay Session Description (Client → Server → Peer)

// Client sends:
{
  type: "relaySessionDescription",
  data: {
    peer_id: string,                    // Target peer
    session_description: RTCSessionDescriptionInit
  }
}

// Server relays to target peer:
{
  type: "sessionDescription",
  data: {
    peer_id: string,                    // Source peer
    peer_name: string,
    session_description: RTCSessionDescriptionInit
  }
}

7. Peer State Update (Client → Server → All Peers)

// Client sends:
{
  type: "peer_state_update",
  data: {
    peer_id: string,
    muted: boolean,
    video_on: boolean
  }
}

// Server broadcasts to all other peers:
{
  type: "peer_state_update",
  data: {
    peer_id: string,
    peer_name: string,
    muted: boolean,
    video_on: boolean
  }
}

Server Implementation Requirements

Peer Registry

The server must maintain a registry of connected peers for each room/game:

interface PeerInfo {
  ws: WebSocket;           // WebSocket connection
  has_media: boolean;      // Whether peer provides media
  hasAudio?: boolean;      // Legacy: has audio
  hasVideo?: boolean;      // Legacy: has video
}

const peers: Record<string, PeerInfo> = {};

Join Handler

function join(peers: any, session: any, config: {
  has_media?: boolean,
  hasVideo?: boolean,
  hasAudio?: boolean
}) {
  const { has_media, hasVideo, hasAudio } = config;
  const peerHasMedia = has_media ?? (hasVideo || hasAudio);

  // Notify all existing peers about new peer
  for (const peer in peers) {
    peers[peer].ws.send(JSON.stringify({
      type: "addPeer",
      data: {
        peer_id: session.name,
        peer_name: session.name,
        has_media: peerHasMedia,
        should_create_offer: false
      }
    }));
  }

  // Notify new peer about all existing peers
  for (const peer in peers) {
    session.ws.send(JSON.stringify({
      type: "addPeer",
      data: {
        peer_id: peer,
        peer_name: peer,
        has_media: peers[peer].has_media,
        should_create_offer: true
      }
    }));
  }

  // Add new peer to registry
  peers[session.name] = {
    ws: session.ws,
    has_media: peerHasMedia,
    hasAudio,
    hasVideo
  };

  // Send success status
  session.ws.send(JSON.stringify({
    type: "join_status",
    status: "Joined",
    message: "Successfully joined"
  }));
}

Client-Side Integration

Session Type

interface Session {
  id: string;
  name: string | null;
  has_media?: boolean;  // Whether this session provides audio/video
  // ... other fields
}

MediaControl Components

MediaAgent

Handles WebRTC signaling and connection management. Does not render UI.

import { MediaAgent } from './MediaControl';

<MediaAgent
  socketUrl={socketUrl}
  session={session}
  peers={peers}
  setPeers={setPeers}
/>

MediaControl

Renders video feed and controls for a single peer.

import { MediaControl } from './MediaControl';

<MediaControl
  isSelf={peer.local}
  peer={peer}
  sendJsonMessage={sendJsonMessage}
  remoteAudioMuted={peer.muted}
  remoteVideoOff={!peer.video_on}
/>

Peer State Management

const [peers, setPeers] = useState<Record<string, Peer>>({});

API Endpoints

GET /api/v1/games/

Returns session information including media capability:

{
  id: string,
  name: string | null,
  has_media: boolean,  // Default: true for regular users
  lobbies: Room[]
}

Migration Guide

From Legacy API

If your server currently uses hasVideo/hasAudio:

1. Add has_media field to all messages (computed as hasVideo || hasAudio)
2. Add peer_name field to all peer-related messages (same as peer_id)
3. Add join_status response to join requests
4. Update session endpoint to return has_media

Backward Compatibility

The protocol supports both old and new field names:

• Server accepts has_media, hasVideo, and hasAudio
• Server sends both old and new fields during transition period

Reconnection Handling

The server handles WebSocket reconnections gracefully:

1. On Reconnection: When a peer reconnects (new WebSocket for existing session):

   • Old peer is removed from registry via part()
   • New WebSocket reference is updated in peer registry
   • join_status response sent with "Reconnected" message
   • All existing peers are sent to reconnecting client via addPeer
   • All other peers are notified about the reconnected peer

2. Client Behavior: Clients should:

   • Wait for join_status before considering themselves joined
   • Handle addPeer messages even after initial join (for reconnections)
   • Re-establish peer connections when receiving addPeer for known peers

Security Considerations

1. STUN/TURN Configuration: Configure ICE servers in MediaControl.tsx (lines 462-474)
2. Peer Authentication: Validate peer identities before relaying signaling messages
3. Rate Limiting: Limit signaling message frequency to prevent abuse
4. Room Isolation: Ensure peers can only connect to others in the same room/game

Troubleshooting

Peers Not Connecting

1. Check join_status response is "Joined" or "Reconnected"
2. Verify addPeer messages include both peer_id and peer_name
3. Check ICE candidates are being relayed correctly
4. Verify STUN/TURN servers are accessible
5. Check server logs for "Already joined" messages (indicates reconnection scenario)

No Video/Audio

1. Check has_media is set correctly in session
2. Verify browser permissions for camera/microphone
3. Check peer state updates are being broadcast
4. Verify tracks are enabled in MediaStream

"Already Joined" Issues

If peers show as "Already joined" but can't connect:

1. Check that old WebSocket connections are being cleaned up on reconnect
2. Verify part() is called when WebSocket is replaced
3. Ensure peer registry is updated with new WebSocket reference

Testing

Manual Test Checklist

• Join shows "Joined" status
• Existing peers appear in peer list
• New peer appears to existing peers
• Video/audio streams connect
• Mute/unmute works locally and remotely
• Video on/off works locally and remotely
• Peer removal cleans up connections
• Reconnection after disconnect works