peddlers-of-ketran/MEDIACONTROL_API.md

# MediaControl WebRTC Signaling Protocol

This document describes the clean, pluggable API for the MediaControl component used for peer-to-peer audio/video communication.

## Overview

The MediaControl component provides a reusable WebRTC-based media communication system that can be integrated into any application. It handles:
- Peer-to-peer audio and video streaming
- WebRTC connection management (offers, answers, ICE candidates)
- Signaling via WebSocket
- User controls for muting/unmuting and video on/off

## Server-Side Protocol

### Required WebSocket Message Types

#### 1. Join Request (Client → Server)
```typescript
{
  type: "join",
  data: {
    has_media?: boolean  // Whether this peer provides audio/video (default: true)
  }
}
```

#### 2. Join Status Response (Server → Client)
```typescript
{
  type: "join_status",
  status: "Joined" | "Joining" | "Error",
  message?: string
}
```

#### 3. Add Peer (Server → Client)
Sent to all existing peers when a new peer joins, and sent to the new peer for each existing peer.
```typescript
{
  type: "addPeer",
  data: {
    peer_id: string,        // Unique identifier (session name)
    peer_name: string,      // Display name for the peer
    has_media: boolean,     // Whether this peer provides media
    should_create_offer: boolean,  // If true, create an RTC offer to this peer
    // Legacy fields (optional, for backward compatibility):
    hasAudio?: boolean,
    hasVideo?: boolean
  }
}
```

#### 4. Remove Peer (Server → Clients)
Sent when a peer disconnects.
```typescript
{
  type: "removePeer",
  data: {
    peer_id: string,
    peer_name: string
  }
}
```

#### 5. Relay ICE Candidate (Client → Server → Peer)
```typescript
// Client sends:
{
  type: "relayICECandidate",
  data: {
    peer_id: string,              // Target peer
    candidate: RTCIceCandidateInit
  }
}

// Server relays to target peer:
{
  type: "iceCandidate",
  data: {
    peer_id: string,              // Source peer
    peer_name: string,
    candidate: RTCIceCandidateInit
  }
}
```

#### 6. Relay Session Description (Client → Server → Peer)
```typescript
// Client sends:
{
  type: "relaySessionDescription",
  data: {
    peer_id: string,                    // Target peer
    session_description: RTCSessionDescriptionInit
  }
}

// Server relays to target peer:
{
  type: "sessionDescription",
  data: {
    peer_id: string,                    // Source peer
    peer_name: string,
    session_description: RTCSessionDescriptionInit
  }
}
```

#### 7. Peer State Update (Client → Server → All Peers)
```typescript
// Client sends:
{
  type: "peer_state_update",
  data: {
    peer_id: string,
    muted: boolean,
    video_on: boolean
  }
}

// Server broadcasts to all other peers:
{
  type: "peer_state_update",
  data: {
    peer_id: string,
    peer_name: string,
    muted: boolean,
    video_on: boolean
  }
}
```

## Server Implementation Requirements

### Peer Registry
The server must maintain a registry of connected peers for each room/game:

```typescript
interface PeerInfo {
  ws: WebSocket;           // WebSocket connection
  has_media: boolean;      // Whether peer provides media
  hasAudio?: boolean;      // Legacy: has audio
  hasVideo?: boolean;      // Legacy: has video
}

const peers: Record<string, PeerInfo> = {};
```

### Join Handler
```typescript
function join(peers: any, session: any, config: {
  has_media?: boolean,
  hasVideo?: boolean,
  hasAudio?: boolean
}) {
  const { has_media, hasVideo, hasAudio } = config;
  const peerHasMedia = has_media ?? (hasVideo || hasAudio);

  // Notify all existing peers about new peer
  for (const peer in peers) {
    peers[peer].ws.send(JSON.stringify({
      type: "addPeer",
      data: {
        peer_id: session.name,
        peer_name: session.name,
        has_media: peerHasMedia,
        should_create_offer: false
      }
    }));
  }

  // Notify new peer about all existing peers
  for (const peer in peers) {
    session.ws.send(JSON.stringify({
      type: "addPeer",
      data: {
        peer_id: peer,
        peer_name: peer,
        has_media: peers[peer].has_media,
        should_create_offer: true
      }
    }));
  }

  // Add new peer to registry
  peers[session.name] = {
    ws: session.ws,
    has_media: peerHasMedia,
    hasAudio,
    hasVideo
  };

  // Send success status
  session.ws.send(JSON.stringify({
    type: "join_status",
    status: "Joined",
    message: "Successfully joined"
  }));
}
```

## Client-Side Integration

### Session Type
```typescript
interface Session {
  id: string;
  name: string | null;
  has_media?: boolean;  // Whether this session provides audio/video
  // ... other fields
}
```

### MediaControl Components

#### MediaAgent
Handles WebRTC signaling and connection management. Does not render UI.

```typescript
import { MediaAgent } from './MediaControl';

<MediaAgent
  socketUrl={socketUrl}
  session={session}
  peers={peers}
  setPeers={setPeers}
/>
```

#### MediaControl
Renders video feed and controls for a single peer.

```typescript
import { MediaControl } from './MediaControl';

<MediaControl
  isSelf={peer.local}
  peer={peer}
  sendJsonMessage={sendJsonMessage}
  remoteAudioMuted={peer.muted}
  remoteVideoOff={!peer.video_on}
/>
```

### Peer State Management
```typescript
const [peers, setPeers] = useState<Record<string, Peer>>({});
```

## API Endpoints

### GET /api/v1/games/
Returns session information including media capability:
```typescript
{
  id: string,
  name: string | null,
  has_media: boolean,  // Default: true for regular users
  lobbies: Room[]
}
```

## Migration Guide

### From Legacy API
If your server currently uses `hasVideo`/`hasAudio`:

1. Add `has_media` field to all messages (computed as `hasVideo || hasAudio`)
2. Add `peer_name` field to all peer-related messages (same as `peer_id`)
3. Add `join_status` response to join requests
4. Update session endpoint to return `has_media`

### Backward Compatibility
The protocol supports both old and new field names:
- Server accepts `has_media`, `hasVideo`, and `hasAudio`
- Server sends both old and new fields during transition period

## Reconnection Handling

The server handles WebSocket reconnections gracefully:

1. **On Reconnection**: When a peer reconnects (new WebSocket for existing session):
   - Old peer is removed from registry via `part()`
   - New WebSocket reference is updated in peer registry
   - `join_status` response sent with "Reconnected" message
   - All existing peers are sent to reconnecting client via `addPeer`
   - All other peers are notified about the reconnected peer

2. **Client Behavior**: Clients should:
   - Wait for `join_status` before considering themselves joined
   - Handle `addPeer` messages even after initial join (for reconnections)
   - Re-establish peer connections when receiving `addPeer` for known peers

## Security Considerations

1. **STUN/TURN Configuration**: Configure ICE servers in MediaControl.tsx (lines 462-474)
2. **Peer Authentication**: Validate peer identities before relaying signaling messages
3. **Rate Limiting**: Limit signaling message frequency to prevent abuse
4. **Room Isolation**: Ensure peers can only connect to others in the same room/game

## Troubleshooting

### Peers Not Connecting
1. Check `join_status` response is "Joined" or "Reconnected"
2. Verify `addPeer` messages include both `peer_id` and `peer_name`
3. Check ICE candidates are being relayed correctly
4. Verify STUN/TURN servers are accessible
5. Check server logs for "Already joined" messages (indicates reconnection scenario)

### No Video/Audio
1. Check `has_media` is set correctly in session
2. Verify browser permissions for camera/microphone
3. Check peer state updates are being broadcast
4. Verify tracks are enabled in MediaStream

### "Already Joined" Issues
If peers show as "Already joined" but can't connect:
1. Check that old WebSocket connections are being cleaned up on reconnect
2. Verify `part()` is called when WebSocket is replaced
3. Ensure peer registry is updated with new WebSocket reference

## Testing

### Manual Test Checklist
- [ ] Join shows "Joined" status
- [ ] Existing peers appear in peer list
- [ ] New peer appears to existing peers
- [ ] Video/audio streams connect
- [ ] Mute/unmute works locally and remotely
- [ ] Video on/off works locally and remotely
- [ ] Peer removal cleans up connections
- [ ] Reconnection after disconnect works