James Ketrenos
e0548a128c
``` POST https://ketrenos.com/ai-voicebot/api/bots/ai_chatbot/join 404 (Not Found) ``` The issue was caused by three main problems: 1. **Incorrect Provider Registration Check**: The voicebot service was checking provider registration using the wrong API endpoint (`/api/bots` instead of `/api/bots/providers`) 2. **No Persistence for Bot Providers**: Bot providers were stored only in memory and lost on server restart, requiring re-registration 3. **AsyncIO Task Initialization Issue**: The cleanup task was being created during `__init__` when no event loop was running, causing FastAPI route registration failures **File**: `voicebot/bot_orchestrator.py` **Problem**: The `check_provider_registration` function was calling `/api/bots` (which returns available bots) instead of `/api/bots/providers` (which returns registered providers). **Fix**: Updated the function to use the correct endpoint and parse the response properly: ```python async def check_provider_registration(server_url: str, provider_id: str, insecure: bool = False) -> bool: """Check if the bot provider is still registered with the server.""" try: import httpx verify = not insecure async with httpx.AsyncClient(verify=verify) as client: # Check if our provider is still in the provider list response = await client.get(f"{server_url}/api/bots/providers", timeout=5.0) if response.status_code == 200: data = response.json() providers = data.get("providers", []) # providers is a list of BotProviderModel objects, check if our provider_id is in the list is_registered = any(provider.get("provider_id") == provider_id for provider in providers) logger.debug(f"Registration check: provider_id={provider_id}, found_providers={len(providers)}, is_registered={is_registered}") return is_registered else: logger.warning(f"Registration check failed: HTTP {response.status_code}") return False except Exception as e: logger.debug(f"Provider registration check failed: {e}") return False ``` **File**: `server/core/bot_manager.py` **Problem**: Bot providers were stored only in memory and lost on server restart. **Fix**: Added persistence functionality to save/load bot providers to/from `bot_providers.json`: ```python def _save_bot_providers(self): """Save bot providers to disk""" try: with self.lock: providers_data = {} for provider_id, provider in self.bot_providers.items(): providers_data[provider_id] = provider.model_dump() with open(self.bot_providers_file, 'w') as f: json.dump(providers_data, f, indent=2) logger.debug(f"Saved {len(providers_data)} bot providers to {self.bot_providers_file}") except Exception as e: logger.error(f"Failed to save bot providers: {e}") def _load_bot_providers(self): """Load bot providers from disk""" try: if not os.path.exists(self.bot_providers_file): logger.debug(f"No bot providers file found at {self.bot_providers_file}") return with open(self.bot_providers_file, 'r') as f: providers_data = json.load(f) with self.lock: for provider_id, provider_dict in providers_data.items(): try: provider = BotProviderModel.model_validate(provider_dict) self.bot_providers[provider_id] = provider except Exception as e: logger.warning(f"Failed to load bot provider {provider_id}: {e}") logger.info(f"Loaded {len(self.bot_providers)} bot providers from {self.bot_providers_file}") except Exception as e: logger.error(f"Failed to load bot providers: {e}") ``` **Integration**: The persistence functions are automatically called: - `_load_bot_providers()` during `BotManager.__init__()` - `_save_bot_providers()` when registering new providers or removing stale ones **File**: `server/core/bot_manager.py` **Problem**: The cleanup task was being created during `BotManager.__init__()` when no event loop was running, causing the FastAPI application to fail to register routes properly. **Fix**: Deferred the cleanup task creation until it's actually needed: ```python def __init__(self): # ... other initialization ... # Load persisted bot providers self._load_bot_providers() # Note: Don't start cleanup task here - will be started when needed def start_cleanup(self): """Start the cleanup task""" try: if self.cleanup_task is None: self.cleanup_task = asyncio.create_task(self._periodic_cleanup()) logger.debug("Bot provider cleanup task started") except RuntimeError: # No event loop running yet, cleanup will be started later logger.debug("No event loop available for bot provider cleanup task") async def register_provider(self, request: BotProviderRegisterRequest) -> BotProviderRegisterResponse: # ... registration logic ... # Start cleanup task if not already running self.start_cleanup() return BotProviderRegisterResponse(provider_id=provider_id) ``` **File**: `server/core/bot_manager.py` **Enhancement**: Added a background task that periodically removes providers that haven't been seen in 15 minutes: ```python async def _periodic_cleanup(self): """Periodically clean up stale bot providers""" cleanup_interval = 300 # 5 minutes stale_threshold = 900 # 15 minutes while not self._shutdown_event.is_set(): try: await asyncio.sleep(cleanup_interval) now = time.time() providers_to_remove = [] with self.lock: for provider_id, provider in self.bot_providers.items(): if now - provider.last_seen > stale_threshold: providers_to_remove.append(provider_id) logger.info(f"Marking stale bot provider for removal: {provider.name} (ID: {provider_id}, last_seen: {now - provider.last_seen:.1f}s ago)") if providers_to_remove: with self.lock: for provider_id in providers_to_remove: if provider_id in self.bot_providers: del self.bot_providers[provider_id] self._save_bot_providers() logger.info(f"Cleaned up {len(providers_to_remove)} stale bot providers") except asyncio.CancelledError: break except Exception as e: logger.error(f"Error in bot provider cleanup: {e}") ``` **File**: `client/src/BotManager.tsx` **Enhancement**: Added retry logic to handle temporary 404s during service restarts: ```typescript // Retry logic for handling service restart scenarios let retries = 3; let response; while (retries > 0) { try { response = await botsApi.requestJoinLobby(selectedBot, request); break; // Success, exit retry loop } catch (err: any) { retries--; // If it's a 404 error and we have retries left, wait and retry if (err?.status === 404 && retries > 0) { console.log(`Bot join failed with 404, retrying... (${retries} attempts left)`); await new Promise(resolve => setTimeout(resolve, 1000)); // Wait 1 second continue; } // If it's not a 404 or we're out of retries, throw the error throw err; } } ``` 1. **Persistence**: Bot providers now survive server restarts and don't need to re-register immediately 2. **Correct Registration Checks**: Provider registration checks use the correct API endpoint 3. **Proper AsyncIO Task Management**: Cleanup tasks are started only when an event loop is available 4. **Automatic Cleanup**: Stale providers are automatically removed to prevent accumulation of dead entries 5. **Client Resilience**: Frontend can handle temporary 404s during service restarts with automatic retries 6. **Reduced Downtime**: Users experience fewer failed bot additions during service restarts After implementing these fixes: 1. Bot providers are correctly persisted in `bot_providers.json` 2. Server restarts load existing providers from disk 3. Provider registration checks use the correct `/api/bots/providers` endpoint 4. AsyncIO cleanup tasks start properly without interfering with route registration 5. Client retries failed requests with 404 errors 6. Periodic cleanup prevents accumulation of stale providers 7. Bot join requests work correctly: `POST /api/bots/{bot_name}/join` returns 200 OK Test the fix with these commands: ```bash curl -k https://ketrenos.com/ai-voicebot/api/lobby curl -k -X POST https://ketrenos.com/ai-voicebot/api/bots/ai_chatbot/join \ -H "Content-Type: application/json" \ -d '{"lobby_id":"<lobby_id>","nick":"test-bot","provider_id":"<provider_id>"}' curl -k https://ketrenos.com/ai-voicebot/api/bots/providers curl -k https://ketrenos.com/ai-voicebot/api/bots ``` 1. `voicebot/bot_orchestrator.py` - Fixed registration check endpoint 2. `server/core/bot_manager.py` - Added persistence and cleanup 3. `client/src/BotManager.tsx` - Added retry logic No additional configuration is required. The fixes work with existing environment variables and settings.
AI Voicebot
A WebRTC-enabled AI voicebot system with speech recognition and synthetic media capabilities. The voicebot can run in two modes: as a client connecting to lobbies or as a provider serving bots to other applications.
Features
- Speech Recognition: Uses Whisper models for real-time audio transcription
- Synthetic Media: Generates animated video and audio tracks
- WebRTC Integration: Real-time peer-to-peer communication
- Bot Provider System: Can register with a main server to provide bot services
- Flexible Deployment: Docker-based with development and production modes
Quick Start
Prerequisites
- Docker and Docker Compose
- Python 3.12+ (if running locally)
- Access to a compatible signaling server
Running with Docker
1. Bot Provider Mode (Recommended)
Run the voicebot as a bot provider that registers with the main server:
# Development mode with auto-reload
VOICEBOT_MODE=provider PRODUCTION=false docker-compose up voicebot
# Production mode
VOICEBOT_MODE=provider PRODUCTION=true docker-compose up voicebot
2. Direct Client Mode
Run the voicebot as a direct client connecting to a lobby:
# Development mode
VOICEBOT_MODE=client PRODUCTION=false docker-compose up voicebot
# Production mode
VOICEBOT_MODE=client PRODUCTION=true docker-compose up voicebot
Running Locally
1. Setup Environment
cd voicebot/
# Create virtual environment
uv init --python /usr/bin/python3.12 --name "ai-voicebot-agent"
uv add -r requirements.txt
# Activate environment
source .venv/bin/activate
2. Bot Provider Mode
# Development with auto-reload
python main.py --mode provider --server-url https://your-server.com/ai-voicebot --reload --insecure
# Production
python main.py --mode provider --server-url https://your-server.com/ai-voicebot
3. Direct Client Mode
python main.py --mode client \
--server-url https://your-server.com/ai-voicebot \
--lobby "my-lobby" \
--session-name "My Bot" \
--insecure
Configuration
Environment Variables
| Variable | Description | Default | Example |
|---|---|---|---|
VOICEBOT_MODE |
Operating mode: client or provider |
client |
provider |
PRODUCTION |
Production mode flag | false |
true |
Command Line Arguments
Common Arguments
--mode: Run asclientorprovider--server-url: Main server URL--insecure: Allow insecure SSL connections--help: Show all available options
Provider Mode Arguments
--host: Host to bind the provider server (default:0.0.0.0)--port: Port for the provider server (default:8788)--reload: Enable auto-reload for development
Client Mode Arguments
--lobby: Lobby name to join (default:default)--session-name: Display name for the bot (default:Python Bot)--session-id: Existing session ID to reuse--password: Password for protected names--private: Create/join private lobby
Available Bots
The voicebot system includes the following bot types:
1. Whisper Bot
- Name:
whisper - Description: Speech recognition agent using OpenAI Whisper models
- Capabilities: Real-time audio transcription, multiple language support
- Models: Supports various Whisper and Distil-Whisper models
2. Synthetic Media Bot
- Name:
synthetic_media - Description: Generates animated video and audio tracks
- Capabilities: Animated video generation, synthetic audio, edge detection on incoming video
Architecture
Bot Provider System
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Main Server │ │ Bot Provider │ │ Client App │
│ │◄───┤ (Voicebot) │ │ │
│ - Bot Registry │ │ - Whisper Bot │ │ - Bot Manager │
│ - Lobby Management │ - Synthetic Bot │ │ - UI Controls │
│ - API Endpoints │ │ - API Server │ │ - Lobby View │
└─────────────────┘ └──────────────────┘ └─────────────────┘
Flow
- Voicebot registers as bot provider with main server
- Main server discovers available bots from providers
- Client requests bot to join lobby via main server
- Main server forwards request to appropriate provider
- Provider creates bot instance that connects to the lobby
Development
Auto-Reload
In development mode, the bot provider supports auto-reload using uvicorn:
# Watches /voicebot and /shared directories for changes
python main.py --mode provider --reload
Adding New Bots
- Create a new module in
voicebot/bots/ - Implement required functions:
def agent_info() -> dict: return {"name": "my_bot", "description": "My custom bot"} def create_agent_tracks(session_name: str) -> dict: # Return MediaStreamTrack instances return {"audio": my_audio_track, "video": my_video_track} - The bot will be automatically discovered and available
Testing
# Test bot discovery
python test_bot_api.py
# Test client connection
python main.py --mode client --lobby test --session-name "Test Bot"
Production Deployment
Docker Compose
version: '3.8'
services:
voicebot-provider:
build: .
environment:
- VOICEBOT_MODE=provider
- PRODUCTION=true
ports:
- "8788:8788"
volumes:
- ./cache:/voicebot/cache
Kubernetes
apiVersion: apps/v1
kind: Deployment
metadata:
name: voicebot-provider
spec:
replicas: 1
selector:
matchLabels:
app: voicebot-provider
template:
metadata:
labels:
app: voicebot-provider
spec:
containers:
- name: voicebot
image: ai-voicebot:latest
env:
- name: VOICEBOT_MODE
value: "provider"
- name: PRODUCTION
value: "true"
ports:
- containerPort: 8788
API Reference
Bot Provider Endpoints
The voicebot provider exposes the following HTTP API:
GET /bots- List available botsPOST /bots/{bot_name}/join- Request bot to join lobbyGET /bots/runs- List active bot instancesPOST /bots/runs/{run_id}/stop- Stop a bot instance
Example API Usage
# List available bots
curl http://localhost:8788/bots
# Request whisper bot to join lobby
curl -X POST http://localhost:8788/bots/whisper/join \
-H "Content-Type: application/json" \
-d '{
"lobby_id": "lobby-123",
"session_id": "session-456",
"nick": "Speech Bot",
"server_url": "https://server.com/ai-voicebot"
}'
Troubleshooting
Common Issues
Bot provider not registering:
- Check server URL is correct and accessible
- Verify network connectivity between provider and server
- Check logs for registration errors
Auto-reload not working:
- Ensure
--reloadflag is used in development - Check file permissions on watched directories
- Verify uvicorn version supports reload functionality
WebRTC connection issues:
- Check STUN/TURN server configuration
- Verify network ports are not blocked
- Check browser console for ICE connection errors
Logs
Logs are written to stdout and include:
- Bot registration status
- WebRTC connection events
- Media track creation/destruction
- API request/response details
Debug Mode
Enable verbose logging:
python main.py --mode provider --server-url https://server.com --debug
Contributing
- Fork the repository
- Create a feature branch
- Make your changes
- Add tests for new functionality
- Submit a pull request
License
This project is licensed under the MIT License - see the LICENSE file for details.