Compare commits
10 Commits
2c81fe1b0d
...
8b747fd667
| Author | SHA1 | Date | |
|---|---|---|---|
|
8b747fd667 | ||
|
|
86fa7a9564 | ||
|
|
b8f839cfe4 | ||
|
|
03cbb5eb1f | ||
|
|
71cd6a07b7 | ||
|
|
6a30a99c62 | ||
|
|
37b2972231 | ||
|
|
801644f396 | ||
|
|
fff1718478 | ||
|
|
715934f157 |
2
.env.web
2
.env.web
@@ -2,4 +2,4 @@ NUXT_PORT=3001
|
||||
NUXT_HOST=0.0.0.0
|
||||
|
||||
# Point Nuxt to the API container (not localhost)
|
||||
NUXT_PUBLIC_API_BASE_URL=http://jupiter.routebox.co:3000
|
||||
NUXT_PUBLIC_API_BASE_URL=https://tenant1.routebox.co
|
||||
|
||||
83
DEBUG_INCOMING_CALL.md
Normal file
83
DEBUG_INCOMING_CALL.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# Debugging Incoming Call Issue
|
||||
|
||||
## Current Problem
|
||||
- Hear "Connecting to your call" message (TwiML is executing)
|
||||
- No ring on mobile after "Connecting" message
|
||||
- Click Accept button does nothing
|
||||
- Call never connects
|
||||
|
||||
## Root Cause Hypothesis
|
||||
The Twilio Device SDK is likely **NOT receiving the incoming call event** from Twilio's Signaling Server. This could be because:
|
||||
|
||||
1. **Identity Mismatch**: The Device's identity (from JWT token) doesn't match the `<Client>ID</Client>` in TwiML
|
||||
2. **Device Not Registered**: Device registration isn't completing before the call arrives
|
||||
3. **Twilio Signaling Issue**: Device isn't connected to Twilio Signaling Server
|
||||
|
||||
## How to Debug
|
||||
|
||||
### Step 1: Check Device Identity in Console
|
||||
When you open the softphone dialog, **open Browser DevTools Console (F12)**
|
||||
|
||||
You should see logs like:
|
||||
```
|
||||
Token received, creating Device...
|
||||
Token identity: e6d45fa3-a108-4085-81e5-a8e05e85e6fb
|
||||
Token grants: {voice: {...}}
|
||||
Registering Twilio Device...
|
||||
✓ Twilio Device registered - ready to receive calls
|
||||
Device identity: e6d45fa3-a108-4085-81e5-a8e05e85e6fb
|
||||
Device state: ready
|
||||
```
|
||||
|
||||
**Note the Device identity value** - e.g., "e6d45fa3-a108-4085-81e5-a8e05e85e6fb"
|
||||
|
||||
### Step 2: Check Backend Logs
|
||||
When you make an inbound call, look for backend logs showing:
|
||||
|
||||
```
|
||||
╔════════════════════════════════════════╗
|
||||
║ === INBOUND CALL RECEIVED ===
|
||||
╚════════════════════════════════════════╝
|
||||
...
|
||||
Client IDs to dial: e6d45fa3-a108-4085-81e5-a8e05e85e6fb
|
||||
First Client ID format check: "e6d45fa3-a108-4085-81e5-a8e05e85e6fb" (length: 36)
|
||||
```
|
||||
|
||||
### Step 3: Compare Identities
|
||||
The Device identity from frontend console MUST MATCH the Client ID from backend logs.
|
||||
|
||||
**If they match**: The issue is with Twilio Signaling or Device SDK configuration
|
||||
**If they don't match**: We found the bug - identity mismatch
|
||||
|
||||
### Step 4: Monitor Incoming Event
|
||||
When you make the inbound call, keep watching the browser console for:
|
||||
|
||||
```
|
||||
🔔 Twilio Device INCOMING event received: {...}
|
||||
```
|
||||
|
||||
**If this appears**: The Device SDK IS receiving the call, so the Accept button issue is frontend
|
||||
**If this doesn't appear**: The Device SDK is NOT receiving the call, so it's an identity/registration issue
|
||||
|
||||
## What Changed
|
||||
- Frontend now relies on **Twilio Device SDK `incoming` event** (not Socket.IO) for showing incoming call
|
||||
- Added comprehensive logging to Device initialization
|
||||
- Added logging to Accept button handler
|
||||
- Backend logs Device ID format for comparison
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Make an inbound call
|
||||
2. Check browser console for the 5 logs above
|
||||
3. Check backend logs for Client ID
|
||||
4. Look for "🔔 Twilio Device INCOMING event" in browser console
|
||||
5. Try clicking Accept and watch console for "📞 Accepting call" logs
|
||||
6. Report back with:
|
||||
- Device identity from console
|
||||
- Client ID from backend logs
|
||||
- Whether "🔔 Twilio Device INCOMING event" appears
|
||||
- Whether any accept logs appear
|
||||
|
||||
## Important Files
|
||||
- Backend: `/backend/src/voice/voice.controller.ts` (lines 205-210 show Client ID logging)
|
||||
- Frontend: `/frontend/composables/useSoftphone.ts` (Device initialization and incoming handler)
|
||||
173
SOFTPHONE_AI_ASSISTANT.md
Normal file
173
SOFTPHONE_AI_ASSISTANT.md
Normal file
@@ -0,0 +1,173 @@
|
||||
# Softphone AI Assistant - Complete Implementation
|
||||
|
||||
## 🎉 Features Implemented
|
||||
|
||||
### ✅ Real-time AI Call Assistant
|
||||
- **OpenAI Realtime API Integration** - Listens to live calls and provides suggestions
|
||||
- **Audio Streaming** - Twilio Media Streams fork audio to backend for AI processing
|
||||
- **Real-time Transcription** - Speech-to-text during calls
|
||||
- **Smart Suggestions** - AI analyzes conversation and advises the agent
|
||||
|
||||
## 🔧 Architecture
|
||||
|
||||
### Backend Flow
|
||||
```
|
||||
Inbound Call → TwiML (<Start><Stream> + <Dial>)
|
||||
→ Media Stream WebSocket → OpenAI Realtime API
|
||||
→ AI Processing → Socket.IO → Frontend
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
1. **TwiML Structure** (`voice.controller.ts:226-234`)
|
||||
- `<Start><Stream>` - Forks audio for AI processing
|
||||
- `<Dial><Client>` - Connects call to agent's softphone
|
||||
|
||||
2. **OpenAI Integration** (`voice.service.ts:431-519`)
|
||||
- WebSocket connection to `wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01`
|
||||
- Session config with custom instructions for agent assistance
|
||||
- Handles transcripts and generates suggestions
|
||||
|
||||
3. **AI Message Handler** (`voice.service.ts:609-707`)
|
||||
- Processes OpenAI events (transcripts, suggestions, audio)
|
||||
- Routes suggestions to frontend via Socket.IO
|
||||
- Saves transcripts to database
|
||||
|
||||
4. **Voice Gateway** (`voice.gateway.ts:272-289`)
|
||||
- `notifyAiTranscript()` - Real-time transcript chunks
|
||||
- `notifyAiSuggestion()` - AI suggestions to agent
|
||||
|
||||
### Frontend Components
|
||||
|
||||
1. **Softphone Dialog** (`SoftphoneDialog.vue:104-135`)
|
||||
- AI Assistant section with badge showing suggestion count
|
||||
- Color-coded suggestions (blue=response, green=action, purple=insight)
|
||||
- Animated highlight for newest suggestion
|
||||
|
||||
2. **Softphone Composable** (`useSoftphone.ts:515-535`)
|
||||
- Socket.IO event handlers for `ai:suggestion` and `ai:transcript`
|
||||
- Maintains history of last 10 suggestions
|
||||
- Maintains history of last 50 transcript items
|
||||
|
||||
## 📋 AI Prompt Configuration
|
||||
|
||||
The AI is instructed to:
|
||||
- **Listen, not talk** - It advises the agent, not the caller
|
||||
- **Provide concise suggestions** - 1-2 sentences max
|
||||
- **Use formatted output**:
|
||||
- `💡 Suggestion: [advice]`
|
||||
- `⚠️ Alert: [important notice]`
|
||||
- `📋 Action: [CRM action]`
|
||||
|
||||
## 🎨 UI Features
|
||||
|
||||
### Suggestion Types
|
||||
- **Response** (Blue) - Suggested replies or approaches
|
||||
- **Action** (Green) - Recommended CRM actions
|
||||
- **Insight** (Purple) - Important alerts or observations
|
||||
|
||||
### Visual Feedback
|
||||
- Badge showing number of suggestions
|
||||
- Newest suggestion pulses for attention
|
||||
- Auto-scrolling suggestion list
|
||||
- Timestamp on each suggestion
|
||||
|
||||
## 🔍 How to Monitor
|
||||
|
||||
### 1. Backend Logs
|
||||
```bash
|
||||
# Watch for AI events
|
||||
docker logs -f neo-backend-1 | grep -E "AI|OpenAI|transcript|suggestion"
|
||||
```
|
||||
|
||||
Key log markers:
|
||||
- `📝 Transcript chunk:` - Real-time speech detection
|
||||
- `✅ Final transcript:` - Complete transcript saved
|
||||
- `💡 AI Suggestion:` - AI-generated advice
|
||||
|
||||
### 2. Database
|
||||
```sql
|
||||
-- View call transcripts
|
||||
SELECT call_sid, ai_transcript, created_at
|
||||
FROM calls
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 5;
|
||||
```
|
||||
|
||||
### 3. Frontend Console
|
||||
- Open browser DevTools Console
|
||||
- Watch for: "AI suggestion:", "AI transcript:"
|
||||
|
||||
## 🚀 Testing
|
||||
|
||||
1. **Make a test call** to your Twilio number
|
||||
2. **Accept the call** in the softphone dialog
|
||||
3. **Talk during the call** - Say something like "I need to schedule a follow-up"
|
||||
4. **Watch the UI** - AI suggestions appear in real-time
|
||||
5. **Check logs** - See transcription and suggestion generation
|
||||
|
||||
## 📊 Current Status
|
||||
|
||||
✅ **Working**:
|
||||
- Inbound calls ring softphone
|
||||
- Media stream forks audio to backend
|
||||
- OpenAI processes audio (1300+ packets/call)
|
||||
- AI generates suggestions
|
||||
- Suggestions appear in frontend
|
||||
- Transcripts saved to database
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Required Environment Variables
|
||||
```env
|
||||
# OpenAI API Key (set in tenant integrations config)
|
||||
OPENAI_API_KEY=sk-...
|
||||
|
||||
# Optional overrides
|
||||
OPENAI_MODEL=gpt-4o-realtime-preview-2024-10-01
|
||||
OPENAI_VOICE=alloy
|
||||
```
|
||||
|
||||
### Tenant Configuration
|
||||
Set in Settings > Integrations:
|
||||
- OpenAI API Key
|
||||
- Model (optional)
|
||||
- Voice (optional)
|
||||
|
||||
## 🎯 Next Steps (Optional Enhancements)
|
||||
|
||||
1. **CRM Tool Execution** - Implement actual tool calls (search contacts, create tasks)
|
||||
2. **Audio Response** - Send OpenAI audio back to caller (two-way AI interaction)
|
||||
3. **Sentiment Analysis** - Track call sentiment in real-time
|
||||
4. **Call Summary** - Generate post-call summary automatically
|
||||
5. **Custom Prompts** - Allow agents to customize AI instructions per call type
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### No suggestions appearing?
|
||||
1. Check OpenAI API key is configured
|
||||
2. Verify WebSocket connection logs show "OpenAI Realtime connected"
|
||||
3. Check frontend Socket.IO connection is established
|
||||
4. Verify user ID matches between backend and frontend
|
||||
|
||||
### Transcripts not saving?
|
||||
1. Check tenant database connection
|
||||
2. Verify `calls` table has `ai_transcript` column
|
||||
3. Check logs for "Failed to update transcript" errors
|
||||
|
||||
### OpenAI connection fails?
|
||||
1. Verify API key is valid
|
||||
2. Check model name is correct
|
||||
3. Review WebSocket close codes in logs
|
||||
|
||||
## 📝 Files Modified
|
||||
|
||||
**Backend:**
|
||||
- `/backend/src/voice/voice.service.ts` - OpenAI integration & AI message handling
|
||||
- `/backend/src/voice/voice.controller.ts` - TwiML generation with stream fork
|
||||
- `/backend/src/voice/voice.gateway.ts` - Socket.IO event emission
|
||||
- `/backend/src/main.ts` - Media stream WebSocket handler
|
||||
|
||||
**Frontend:**
|
||||
- `/frontend/components/SoftphoneDialog.vue` - AI suggestions UI
|
||||
- `/frontend/composables/useSoftphone.ts` - Socket.IO event handlers
|
||||
249
backend/package-lock.json
generated
249
backend/package-lock.json
generated
@@ -10,7 +10,7 @@
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@casl/ability": "^6.7.5",
|
||||
"@fastify/websocket": "^11.2.0",
|
||||
"@fastify/websocket": "^10.0.1",
|
||||
"@nestjs/bullmq": "^10.1.0",
|
||||
"@nestjs/common": "^10.3.0",
|
||||
"@nestjs/config": "^3.1.1",
|
||||
@@ -19,6 +19,7 @@
|
||||
"@nestjs/passport": "^10.0.3",
|
||||
"@nestjs/platform-fastify": "^10.3.0",
|
||||
"@nestjs/platform-socket.io": "^10.4.20",
|
||||
"@nestjs/serve-static": "^4.0.2",
|
||||
"@nestjs/websockets": "^10.4.20",
|
||||
"@prisma/client": "^5.8.0",
|
||||
"bcrypt": "^5.1.1",
|
||||
@@ -981,42 +982,16 @@
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@fastify/websocket": {
|
||||
"version": "11.2.0",
|
||||
"resolved": "https://registry.npmjs.org/@fastify/websocket/-/websocket-11.2.0.tgz",
|
||||
"integrity": "sha512-3HrDPbAG1CzUCqnslgJxppvzaAZffieOVbLp1DAy1huCSynUWPifSvfdEDUR8HlJLp3sp1A36uOM2tJogADS8w==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/fastify"
|
||||
},
|
||||
{
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/fastify"
|
||||
}
|
||||
],
|
||||
"version": "10.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@fastify/websocket/-/websocket-10.0.1.tgz",
|
||||
"integrity": "sha512-8/pQIxTPRD8U94aILTeJ+2O3el/r19+Ej5z1O1mXlqplsUH7KzCjAI0sgd5DM/NoPjAi5qLFNIjgM5+9/rGSNw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"duplexify": "^4.1.3",
|
||||
"fastify-plugin": "^5.0.0",
|
||||
"ws": "^8.16.0"
|
||||
"duplexify": "^4.1.2",
|
||||
"fastify-plugin": "^4.0.0",
|
||||
"ws": "^8.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@fastify/websocket/node_modules/fastify-plugin": {
|
||||
"version": "5.1.0",
|
||||
"resolved": "https://registry.npmjs.org/fastify-plugin/-/fastify-plugin-5.1.0.tgz",
|
||||
"integrity": "sha512-FAIDA8eovSt5qcDgcBvDuX/v0Cjz0ohGhENZ/wpc3y+oZCY2afZ9Baqql3g/lC+OHRnciQol4ww7tuthOb9idw==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/fastify"
|
||||
},
|
||||
{
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/fastify"
|
||||
}
|
||||
],
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@humanwhocodes/config-array": {
|
||||
"version": "0.13.0",
|
||||
"resolved": "https://registry.npmjs.org/@humanwhocodes/config-array/-/config-array-0.13.0.tgz",
|
||||
@@ -2200,6 +2175,39 @@
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@nestjs/serve-static": {
|
||||
"version": "4.0.2",
|
||||
"resolved": "https://registry.npmjs.org/@nestjs/serve-static/-/serve-static-4.0.2.tgz",
|
||||
"integrity": "sha512-cT0vdWN5ar7jDI2NKbhf4LcwJzU4vS5sVpMkVrHuyLcltbrz6JdGi1TfIMMatP2pNiq5Ie/uUdPSFDVaZX/URQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"path-to-regexp": "0.2.5"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@fastify/static": "^6.5.0 || ^7.0.0",
|
||||
"@nestjs/common": "^9.0.0 || ^10.0.0",
|
||||
"@nestjs/core": "^9.0.0 || ^10.0.0",
|
||||
"express": "^4.18.1",
|
||||
"fastify": "^4.7.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@fastify/static": {
|
||||
"optional": true
|
||||
},
|
||||
"express": {
|
||||
"optional": true
|
||||
},
|
||||
"fastify": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/@nestjs/serve-static/node_modules/path-to-regexp": {
|
||||
"version": "0.2.5",
|
||||
"resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.2.5.tgz",
|
||||
"integrity": "sha512-l6qtdDPIkmAmzEO6egquYDfqQGPMRNGjYtrU13HAXb3YSRrt7HSb1sJY0pKp6o2bAa86tSB6iwaW2JbthPKr7Q==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@nestjs/testing": {
|
||||
"version": "10.4.20",
|
||||
"resolved": "https://registry.npmjs.org/@nestjs/testing/-/testing-10.4.20.tgz",
|
||||
@@ -2359,14 +2367,14 @@
|
||||
"version": "5.22.0",
|
||||
"resolved": "https://registry.npmjs.org/@prisma/debug/-/debug-5.22.0.tgz",
|
||||
"integrity": "sha512-AUt44v3YJeggO2ZU5BkXI7M4hu9BF2zzH2iF2V5pyXT/lRTyWiElZ7It+bRH1EshoMRxHgpYg4VB6rCM+mG5jQ==",
|
||||
"dev": true,
|
||||
"devOptional": true,
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/@prisma/engines": {
|
||||
"version": "5.22.0",
|
||||
"resolved": "https://registry.npmjs.org/@prisma/engines/-/engines-5.22.0.tgz",
|
||||
"integrity": "sha512-UNjfslWhAt06kVL3CjkuYpHAWSO6L4kDCVPegV6itt7nD1kSJavd3vhgAEhjglLJJKEdJ7oIqDJ+yHk6qO8gPA==",
|
||||
"dev": true,
|
||||
"devOptional": true,
|
||||
"hasInstallScript": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
@@ -2380,14 +2388,14 @@
|
||||
"version": "5.22.0-44.605197351a3c8bdd595af2d2a9bc3025bca48ea2",
|
||||
"resolved": "https://registry.npmjs.org/@prisma/engines-version/-/engines-version-5.22.0-44.605197351a3c8bdd595af2d2a9bc3025bca48ea2.tgz",
|
||||
"integrity": "sha512-2PTmxFR2yHW/eB3uqWtcgRcgAbG1rwG9ZriSvQw+nnb7c4uCr3RAcGMb6/zfE88SKlC1Nj2ziUvc96Z379mHgQ==",
|
||||
"dev": true,
|
||||
"devOptional": true,
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/@prisma/fetch-engine": {
|
||||
"version": "5.22.0",
|
||||
"resolved": "https://registry.npmjs.org/@prisma/fetch-engine/-/fetch-engine-5.22.0.tgz",
|
||||
"integrity": "sha512-bkrD/Mc2fSvkQBV5EpoFcZ87AvOgDxbG99488a5cexp5Ccny+UM6MAe/UFkUC0wLYD9+9befNOqGiIJhhq+HbA==",
|
||||
"dev": true,
|
||||
"devOptional": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@prisma/debug": "5.22.0",
|
||||
@@ -2399,7 +2407,7 @@
|
||||
"version": "5.22.0",
|
||||
"resolved": "https://registry.npmjs.org/@prisma/get-platform/-/get-platform-5.22.0.tgz",
|
||||
"integrity": "sha512-pHhpQdr1UPFpt+zFfnPazhulaZYCUqeIcPpJViYoq9R+D/yw4fjE+CtnsnKzPYm0ddUbeXUzjGVGIRVgPDCk4Q==",
|
||||
"dev": true,
|
||||
"devOptional": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@prisma/debug": "5.22.0"
|
||||
@@ -3286,6 +3294,20 @@
|
||||
"node": ">=0.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/acorn-import-phases": {
|
||||
"version": "1.0.4",
|
||||
"resolved": "https://registry.npmjs.org/acorn-import-phases/-/acorn-import-phases-1.0.4.tgz",
|
||||
"integrity": "sha512-wKmbr/DDiIXzEOiWrTTUcDm24kQ2vGfZQvM2fwg2vXqR5uW6aapr7ObPtj1th32b9u90/Pf4AItvdTh42fBmVQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=10.13.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"acorn": "^8.14.0"
|
||||
}
|
||||
},
|
||||
"node_modules/acorn-jsx": {
|
||||
"version": "5.3.2",
|
||||
"resolved": "https://registry.npmjs.org/acorn-jsx/-/acorn-jsx-5.3.2.tgz",
|
||||
@@ -3716,9 +3738,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/baseline-browser-mapping": {
|
||||
"version": "2.8.31",
|
||||
"resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.8.31.tgz",
|
||||
"integrity": "sha512-a28v2eWrrRWPpJSzxc+mKwm0ZtVx/G8SepdQZDArnXYU/XS+IF6mp8aB/4E+hH1tyGCoDo3KlUCdlSxGDsRkAw==",
|
||||
"version": "2.9.11",
|
||||
"resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.9.11.tgz",
|
||||
"integrity": "sha512-Sg0xJUNDU1sJNGdfGWhVHX0kkZ+HWcvmVymJbj6NSgZZmW/8S9Y2HQ5euytnIgakgxN6papOAWiwDo1ctFDcoQ==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"bin": {
|
||||
@@ -3788,9 +3810,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/browserslist": {
|
||||
"version": "4.28.0",
|
||||
"resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.0.tgz",
|
||||
"integrity": "sha512-tbydkR/CxfMwelN0vwdP/pLkDwyAASZ+VfWm4EOwlB6SWhx1sYnWLqo8N5j0rAzPfzfRaxt0mM/4wPU/Su84RQ==",
|
||||
"version": "4.28.1",
|
||||
"resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.1.tgz",
|
||||
"integrity": "sha512-ZC5Bd0LgJXgwGqUknZY/vkUQ04r8NXnJZ3yYi4vDmSiZmC/pdSN0NbNRPxZpbtO4uAfDUAFffO8IZoM3Gj8IkA==",
|
||||
"dev": true,
|
||||
"funding": [
|
||||
{
|
||||
@@ -3808,11 +3830,11 @@
|
||||
],
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"baseline-browser-mapping": "^2.8.25",
|
||||
"caniuse-lite": "^1.0.30001754",
|
||||
"electron-to-chromium": "^1.5.249",
|
||||
"baseline-browser-mapping": "^2.9.0",
|
||||
"caniuse-lite": "^1.0.30001759",
|
||||
"electron-to-chromium": "^1.5.263",
|
||||
"node-releases": "^2.0.27",
|
||||
"update-browserslist-db": "^1.1.4"
|
||||
"update-browserslist-db": "^1.2.0"
|
||||
},
|
||||
"bin": {
|
||||
"browserslist": "cli.js"
|
||||
@@ -3966,9 +3988,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/caniuse-lite": {
|
||||
"version": "1.0.30001757",
|
||||
"resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001757.tgz",
|
||||
"integrity": "sha512-r0nnL/I28Zi/yjk1el6ilj27tKcdjLsNqAOZr0yVjWPrSQyHgKI2INaEWw21bAQSv2LXRt1XuCS/GomNpWOxsQ==",
|
||||
"version": "1.0.30001762",
|
||||
"resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001762.tgz",
|
||||
"integrity": "sha512-PxZwGNvH7Ak8WX5iXzoK1KPZttBXNPuaOvI2ZYU7NrlM+d9Ov+TUvlLOBNGzVXAntMSMMlJPd+jY6ovrVjSmUw==",
|
||||
"dev": true,
|
||||
"funding": [
|
||||
{
|
||||
@@ -4679,9 +4701,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/electron-to-chromium": {
|
||||
"version": "1.5.260",
|
||||
"resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.260.tgz",
|
||||
"integrity": "sha512-ov8rBoOBhVawpzdre+Cmz4FB+y66Eqrk6Gwqd8NGxuhv99GQ8XqMAr351KEkOt7gukXWDg6gJWEMKgL2RLMPtA==",
|
||||
"version": "1.5.267",
|
||||
"resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.267.tgz",
|
||||
"integrity": "sha512-0Drusm6MVRXSOJpGbaSVgcQsuB4hEkMpHXaVstcPmhu5LIedxs1xNK/nIxmQIU/RPC0+1/o0AVZfBTkTNJOdUw==",
|
||||
"dev": true,
|
||||
"license": "ISC"
|
||||
},
|
||||
@@ -8852,7 +8874,7 @@
|
||||
"version": "5.22.0",
|
||||
"resolved": "https://registry.npmjs.org/prisma/-/prisma-5.22.0.tgz",
|
||||
"integrity": "sha512-vtpjW3XuYCSnMsNVBjLMNkTj6OZbudcPPTPYHqX0CJfpcdWciI1dM8uHETwmDxxiqEwCIE6WvXucWUetJgfu/A==",
|
||||
"dev": true,
|
||||
"devOptional": true,
|
||||
"hasInstallScript": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
@@ -10006,9 +10028,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/terser-webpack-plugin": {
|
||||
"version": "5.3.14",
|
||||
"resolved": "https://registry.npmjs.org/terser-webpack-plugin/-/terser-webpack-plugin-5.3.14.tgz",
|
||||
"integrity": "sha512-vkZjpUjb6OMS7dhV+tILUW6BhpDR7P2L/aQSAv+Uwk+m8KATX9EccViHTJR2qDtACKPIYndLGCyl3FMo+r2LMw==",
|
||||
"version": "5.3.16",
|
||||
"resolved": "https://registry.npmjs.org/terser-webpack-plugin/-/terser-webpack-plugin-5.3.16.tgz",
|
||||
"integrity": "sha512-h9oBFCWrq78NyWWVcSwZarJkZ01c2AyGrzs1crmHZO3QUg9D61Wu4NPjBy69n7JqylFF5y+CsUZYmYEIZ3mR+Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -10581,9 +10603,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/update-browserslist-db": {
|
||||
"version": "1.1.4",
|
||||
"resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.1.4.tgz",
|
||||
"integrity": "sha512-q0SPT4xyU84saUX+tomz1WLkxUbuaJnR1xWt17M7fJtEJigJeWUNGUqrauFXsHnqev9y9JTRGwk13tFBuKby4A==",
|
||||
"version": "1.2.3",
|
||||
"resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.2.3.tgz",
|
||||
"integrity": "sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w==",
|
||||
"dev": true,
|
||||
"funding": [
|
||||
{
|
||||
@@ -10728,6 +10750,56 @@
|
||||
"integrity": "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==",
|
||||
"license": "BSD-2-Clause"
|
||||
},
|
||||
"node_modules/webpack": {
|
||||
"version": "5.104.1",
|
||||
"resolved": "https://registry.npmjs.org/webpack/-/webpack-5.104.1.tgz",
|
||||
"integrity": "sha512-Qphch25abbMNtekmEGJmeRUhLDbe+QfiWTiqpKYkpCOWY64v9eyl+KRRLmqOFA2AvKPpc9DC6+u2n76tQLBoaA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@types/eslint-scope": "^3.7.7",
|
||||
"@types/estree": "^1.0.8",
|
||||
"@types/json-schema": "^7.0.15",
|
||||
"@webassemblyjs/ast": "^1.14.1",
|
||||
"@webassemblyjs/wasm-edit": "^1.14.1",
|
||||
"@webassemblyjs/wasm-parser": "^1.14.1",
|
||||
"acorn": "^8.15.0",
|
||||
"acorn-import-phases": "^1.0.3",
|
||||
"browserslist": "^4.28.1",
|
||||
"chrome-trace-event": "^1.0.2",
|
||||
"enhanced-resolve": "^5.17.4",
|
||||
"es-module-lexer": "^2.0.0",
|
||||
"eslint-scope": "5.1.1",
|
||||
"events": "^3.2.0",
|
||||
"glob-to-regexp": "^0.4.1",
|
||||
"graceful-fs": "^4.2.11",
|
||||
"json-parse-even-better-errors": "^2.3.1",
|
||||
"loader-runner": "^4.3.1",
|
||||
"mime-types": "^2.1.27",
|
||||
"neo-async": "^2.6.2",
|
||||
"schema-utils": "^4.3.3",
|
||||
"tapable": "^2.3.0",
|
||||
"terser-webpack-plugin": "^5.3.16",
|
||||
"watchpack": "^2.4.4",
|
||||
"webpack-sources": "^3.3.3"
|
||||
},
|
||||
"bin": {
|
||||
"webpack": "bin/webpack.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=10.13.0"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/webpack"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"webpack-cli": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/webpack-node-externals": {
|
||||
"version": "3.0.0",
|
||||
"resolved": "https://registry.npmjs.org/webpack-node-externals/-/webpack-node-externals-3.0.0.tgz",
|
||||
@@ -10748,6 +10820,61 @@
|
||||
"node": ">=10.13.0"
|
||||
}
|
||||
},
|
||||
"node_modules/webpack/node_modules/es-module-lexer": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-2.0.0.tgz",
|
||||
"integrity": "sha512-5POEcUuZybH7IdmGsD8wlf0AI55wMecM9rVBTI/qEAy2c1kTOm3DjFYjrBdI2K3BaJjJYfYFeRtM0t9ssnRuxw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/webpack/node_modules/eslint-scope": {
|
||||
"version": "5.1.1",
|
||||
"resolved": "https://registry.npmjs.org/eslint-scope/-/eslint-scope-5.1.1.tgz",
|
||||
"integrity": "sha512-2NxwbF/hZ0KpepYN0cNbo+FN6XoK7GaHlQhgx/hIZl6Va0bF45RQOOwhLIy8lQDbuCiadSLCBnH2CFYquit5bw==",
|
||||
"dev": true,
|
||||
"license": "BSD-2-Clause",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"esrecurse": "^4.3.0",
|
||||
"estraverse": "^4.1.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=8.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/webpack/node_modules/estraverse": {
|
||||
"version": "4.3.0",
|
||||
"resolved": "https://registry.npmjs.org/estraverse/-/estraverse-4.3.0.tgz",
|
||||
"integrity": "sha512-39nnKffWz8xN1BU/2c79n9nB9HDzo0niYUqx6xyqUnyoAnQyyWpOTdZEeiCch8BBu515t4wp9ZmgVfVhn9EBpw==",
|
||||
"dev": true,
|
||||
"license": "BSD-2-Clause",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/webpack/node_modules/schema-utils": {
|
||||
"version": "4.3.3",
|
||||
"resolved": "https://registry.npmjs.org/schema-utils/-/schema-utils-4.3.3.tgz",
|
||||
"integrity": "sha512-eflK8wEtyOE6+hsaRVPxvUKYCpRgzLqDTb8krvAsRIwOGlHoSgYLgBXoubGgLd2fT41/OUYdb48v4k4WWHQurA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@types/json-schema": "^7.0.9",
|
||||
"ajv": "^8.9.0",
|
||||
"ajv-formats": "^2.1.1",
|
||||
"ajv-keywords": "^5.1.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 10.13.0"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/webpack"
|
||||
}
|
||||
},
|
||||
"node_modules/whatwg-url": {
|
||||
"version": "5.0.0",
|
||||
"resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-5.0.0.tgz",
|
||||
|
||||
@@ -27,7 +27,7 @@
|
||||
},
|
||||
"dependencies": {
|
||||
"@casl/ability": "^6.7.5",
|
||||
"@fastify/websocket": "^11.2.0",
|
||||
"@fastify/websocket": "^10.0.1",
|
||||
"@nestjs/bullmq": "^10.1.0",
|
||||
"@nestjs/common": "^10.3.0",
|
||||
"@nestjs/config": "^3.1.1",
|
||||
@@ -36,6 +36,7 @@
|
||||
"@nestjs/passport": "^10.0.3",
|
||||
"@nestjs/platform-fastify": "^10.3.0",
|
||||
"@nestjs/platform-socket.io": "^10.4.20",
|
||||
"@nestjs/serve-static": "^4.0.2",
|
||||
"@nestjs/websockets": "^10.4.20",
|
||||
"@prisma/client": "^5.8.0",
|
||||
"bcrypt": "^5.1.1",
|
||||
|
||||
@@ -3,13 +3,15 @@ import {
|
||||
FastifyAdapter,
|
||||
NestFastifyApplication,
|
||||
} from '@nestjs/platform-fastify';
|
||||
import { ValidationPipe } from '@nestjs/common';
|
||||
import { ValidationPipe, Logger } from '@nestjs/common';
|
||||
import { AppModule } from './app.module';
|
||||
import { VoiceService } from './voice/voice.service';
|
||||
import { AudioConverterService } from './voice/audio-converter.service';
|
||||
|
||||
async function bootstrap() {
|
||||
const app = await NestFactory.create<NestFastifyApplication>(
|
||||
AppModule,
|
||||
new FastifyAdapter({ logger: false }),
|
||||
new FastifyAdapter({ logger: true }),
|
||||
);
|
||||
|
||||
// Global validation pipe
|
||||
@@ -33,6 +35,145 @@ async function bootstrap() {
|
||||
const port = process.env.PORT || 3000;
|
||||
await app.listen(port, '0.0.0.0');
|
||||
|
||||
// After app is listening, register WebSocket handler
|
||||
const fastifyInstance = app.getHttpAdapter().getInstance();
|
||||
const logger = new Logger('MediaStreamWS');
|
||||
const voiceService = app.get(VoiceService);
|
||||
const audioConverter = app.get(AudioConverterService);
|
||||
|
||||
const WebSocketServer = require('ws').Server;
|
||||
const wss = new WebSocketServer({ noServer: true });
|
||||
|
||||
// Handle WebSocket upgrades at the server level
|
||||
const server = (fastifyInstance.server as any);
|
||||
|
||||
// Track active Media Streams connections: streamSid -> WebSocket
|
||||
const mediaStreams: Map<string, any> = new Map();
|
||||
|
||||
server.on('upgrade', (request: any, socket: any, head: any) => {
|
||||
if (request.url === '/api/voice/media-stream') {
|
||||
logger.log('=== MEDIA STREAM WEBSOCKET UPGRADE REQUEST ===');
|
||||
logger.log(`Path: ${request.url}`);
|
||||
|
||||
wss.handleUpgrade(request, socket, head, (ws: any) => {
|
||||
logger.log('=== MEDIA STREAM WEBSOCKET UPGRADED SUCCESSFULLY ===');
|
||||
handleMediaStreamSocket(ws);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
async function handleMediaStreamSocket(ws: any) {
|
||||
let streamSid: string | null = null;
|
||||
let callSid: string | null = null;
|
||||
let tenantDomain: string | null = null;
|
||||
let mediaPacketCount = 0;
|
||||
|
||||
ws.on('message', async (message: Buffer) => {
|
||||
try {
|
||||
const msg = JSON.parse(message.toString());
|
||||
|
||||
switch (msg.event) {
|
||||
case 'connected':
|
||||
logger.log('=== MEDIA STREAM EVENT: CONNECTED ===');
|
||||
logger.log(`Protocol: ${msg.protocol}`);
|
||||
logger.log(`Version: ${msg.version}`);
|
||||
break;
|
||||
|
||||
case 'start':
|
||||
streamSid = msg.streamSid;
|
||||
callSid = msg.start.callSid;
|
||||
tenantDomain = msg.start.customParameters?.tenantId || 'tenant1';
|
||||
|
||||
logger.log(`=== MEDIA STREAM EVENT: START ===`);
|
||||
logger.log(`StreamSid: ${streamSid}`);
|
||||
logger.log(`CallSid: ${callSid}`);
|
||||
logger.log(`Tenant: ${tenantDomain}`);
|
||||
logger.log(`MediaFormat: ${JSON.stringify(msg.start.mediaFormat)}`);
|
||||
|
||||
mediaStreams.set(streamSid, ws);
|
||||
logger.log(`Stored WebSocket for streamSid: ${streamSid}. Total active streams: ${mediaStreams.size}`);
|
||||
|
||||
// Initialize OpenAI Realtime connection
|
||||
logger.log(`Initializing OpenAI Realtime for call ${callSid}...`);
|
||||
try {
|
||||
await voiceService.initializeOpenAIRealtime({
|
||||
callSid,
|
||||
tenantId: tenantDomain,
|
||||
userId: msg.start.customParameters?.userId || 'system',
|
||||
});
|
||||
logger.log(`✓ OpenAI Realtime initialized for call ${callSid}`);
|
||||
} catch (error: any) {
|
||||
logger.error(`Failed to initialize OpenAI: ${error.message}`);
|
||||
}
|
||||
break;
|
||||
|
||||
case 'media':
|
||||
mediaPacketCount++;
|
||||
// Only log every 500 packets to reduce noise
|
||||
if (mediaPacketCount % 500 === 0) {
|
||||
logger.log(`Received media packet #${mediaPacketCount} for StreamSid: ${streamSid}`);
|
||||
}
|
||||
|
||||
if (!callSid || !tenantDomain) {
|
||||
logger.warn('Received media before start event');
|
||||
break;
|
||||
}
|
||||
|
||||
try {
|
||||
// Convert Twilio audio (μ-law 8kHz) to OpenAI format (PCM16 24kHz)
|
||||
const twilioAudio = msg.media.payload;
|
||||
const openaiAudio = audioConverter.twilioToOpenAI(twilioAudio);
|
||||
|
||||
// Send audio to OpenAI Realtime API
|
||||
await voiceService.sendAudioToOpenAI(callSid, openaiAudio);
|
||||
} catch (error: any) {
|
||||
logger.error(`Error processing media: ${error.message}`);
|
||||
}
|
||||
break;
|
||||
|
||||
case 'stop':
|
||||
logger.log(`=== MEDIA STREAM EVENT: STOP ===`);
|
||||
logger.log(`StreamSid: ${streamSid}`);
|
||||
logger.log(`Total media packets received: ${mediaPacketCount}`);
|
||||
|
||||
if (streamSid) {
|
||||
mediaStreams.delete(streamSid);
|
||||
logger.log(`Removed WebSocket for streamSid: ${streamSid}`);
|
||||
}
|
||||
|
||||
// Clean up OpenAI connection
|
||||
if (callSid) {
|
||||
try {
|
||||
logger.log(`Cleaning up OpenAI connection for call ${callSid}...`);
|
||||
await voiceService.cleanupOpenAIConnection(callSid);
|
||||
logger.log(`✓ OpenAI connection cleaned up`);
|
||||
} catch (error: any) {
|
||||
logger.error(`Failed to cleanup OpenAI: ${error.message}`);
|
||||
}
|
||||
}
|
||||
break;
|
||||
|
||||
default:
|
||||
logger.debug(`Unknown media stream event: ${msg.event}`);
|
||||
}
|
||||
} catch (error: any) {
|
||||
logger.error(`Error processing media stream message: ${error.message}`);
|
||||
}
|
||||
});
|
||||
|
||||
ws.on('close', () => {
|
||||
logger.log(`=== MEDIA STREAM WEBSOCKET CLOSED ===`);
|
||||
if (streamSid) {
|
||||
mediaStreams.delete(streamSid);
|
||||
}
|
||||
});
|
||||
|
||||
ws.on('error', (error: Error) => {
|
||||
logger.error(`=== MEDIA STREAM WEBSOCKET ERROR ===`);
|
||||
logger.error(`Error message: ${error.message}`);
|
||||
});
|
||||
}
|
||||
|
||||
console.log(`🚀 Application is running on: http://localhost:${port}/api`);
|
||||
}
|
||||
|
||||
|
||||
@@ -20,21 +20,22 @@ export class TenantController {
|
||||
* Get integrations configuration for the current tenant
|
||||
*/
|
||||
@Get('integrations')
|
||||
async getIntegrationsConfig(@TenantId() tenantId: string) {
|
||||
async getIntegrationsConfig(@TenantId() domain: string) {
|
||||
const centralPrisma = getCentralPrisma();
|
||||
|
||||
const tenant = await centralPrisma.tenant.findUnique({
|
||||
where: { id: tenantId },
|
||||
select: { integrationsConfig: true },
|
||||
// Look up tenant by domain
|
||||
const domainRecord = await centralPrisma.domain.findUnique({
|
||||
where: { domain },
|
||||
include: { tenant: { select: { id: true, integrationsConfig: true } } },
|
||||
});
|
||||
|
||||
if (!tenant || !tenant.integrationsConfig) {
|
||||
if (!domainRecord?.tenant || !domainRecord.tenant.integrationsConfig) {
|
||||
return { data: null };
|
||||
}
|
||||
|
||||
// Decrypt the config
|
||||
const config = this.tenantDbService.decryptIntegrationsConfig(
|
||||
tenant.integrationsConfig as any,
|
||||
domainRecord.tenant.integrationsConfig as any,
|
||||
);
|
||||
|
||||
// Return config with sensitive fields masked
|
||||
@@ -48,20 +49,45 @@ export class TenantController {
|
||||
*/
|
||||
@Put('integrations')
|
||||
async updateIntegrationsConfig(
|
||||
@TenantId() tenantId: string,
|
||||
@TenantId() domain: string,
|
||||
@Body() body: { integrationsConfig: any },
|
||||
) {
|
||||
const { integrationsConfig } = body;
|
||||
|
||||
if (!domain) {
|
||||
throw new Error('Domain is missing from request');
|
||||
}
|
||||
|
||||
// Look up tenant by domain
|
||||
const centralPrisma = getCentralPrisma();
|
||||
const domainRecord = await centralPrisma.domain.findUnique({
|
||||
where: { domain },
|
||||
include: { tenant: { select: { id: true, integrationsConfig: true } } },
|
||||
});
|
||||
|
||||
if (!domainRecord?.tenant) {
|
||||
throw new Error(`Tenant with domain ${domain} not found`);
|
||||
}
|
||||
|
||||
// Merge with existing config to preserve masked values
|
||||
let finalConfig = integrationsConfig;
|
||||
if (domainRecord.tenant.integrationsConfig) {
|
||||
const existingConfig = this.tenantDbService.decryptIntegrationsConfig(
|
||||
domainRecord.tenant.integrationsConfig as any,
|
||||
);
|
||||
|
||||
// Replace masked values with actual values from existing config
|
||||
finalConfig = this.unmaskConfig(integrationsConfig, existingConfig);
|
||||
}
|
||||
|
||||
// Encrypt the config
|
||||
const encryptedConfig = this.tenantDbService.encryptIntegrationsConfig(
|
||||
integrationsConfig,
|
||||
finalConfig,
|
||||
);
|
||||
|
||||
// Update in database
|
||||
const centralPrisma = getCentralPrisma();
|
||||
await centralPrisma.tenant.update({
|
||||
where: { id: tenantId },
|
||||
where: { id: domainRecord.tenant.id },
|
||||
data: {
|
||||
integrationsConfig: encryptedConfig as any,
|
||||
},
|
||||
@@ -73,6 +99,32 @@ export class TenantController {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Unmask config by replacing masked values with actual values from existing config
|
||||
*/
|
||||
private unmaskConfig(newConfig: any, existingConfig: any): any {
|
||||
const result = { ...newConfig };
|
||||
|
||||
// Unmask Twilio credentials
|
||||
if (result.twilio && existingConfig.twilio) {
|
||||
if (result.twilio.authToken === '••••••••' && existingConfig.twilio.authToken) {
|
||||
result.twilio.authToken = existingConfig.twilio.authToken;
|
||||
}
|
||||
if (result.twilio.apiSecret === '••••••••' && existingConfig.twilio.apiSecret) {
|
||||
result.twilio.apiSecret = existingConfig.twilio.apiSecret;
|
||||
}
|
||||
}
|
||||
|
||||
// Unmask OpenAI credentials
|
||||
if (result.openai && existingConfig.openai) {
|
||||
if (result.openai.apiKey === '••••••••' && existingConfig.openai.apiKey) {
|
||||
result.openai.apiKey = existingConfig.openai.apiKey;
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Mask sensitive fields for API responses
|
||||
*/
|
||||
@@ -86,6 +138,7 @@ export class TenantController {
|
||||
masked.twilio = {
|
||||
...masked.twilio,
|
||||
authToken: masked.twilio.authToken ? '••••••••' : '',
|
||||
apiSecret: masked.twilio.apiSecret ? '••••••••' : '',
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
214
backend/src/voice/audio-converter.service.ts
Normal file
214
backend/src/voice/audio-converter.service.ts
Normal file
@@ -0,0 +1,214 @@
|
||||
import { Injectable, Logger } from '@nestjs/common';
|
||||
|
||||
/**
|
||||
* Audio format converter for Twilio <-> OpenAI audio streaming
|
||||
*
|
||||
* Twilio Media Streams format:
|
||||
* - Codec: μ-law (G.711)
|
||||
* - Sample rate: 8kHz
|
||||
* - Encoding: base64
|
||||
* - Chunk size: 20ms (160 bytes)
|
||||
*
|
||||
* OpenAI Realtime API format:
|
||||
* - Codec: PCM16
|
||||
* - Sample rate: 24kHz
|
||||
* - Encoding: base64
|
||||
* - Mono channel
|
||||
*/
|
||||
@Injectable()
|
||||
export class AudioConverterService {
|
||||
private readonly logger = new Logger(AudioConverterService.name);
|
||||
|
||||
// μ-law decode lookup table
|
||||
private readonly MULAW_DECODE_TABLE = this.buildMuLawDecodeTable();
|
||||
|
||||
// μ-law encode lookup table
|
||||
private readonly MULAW_ENCODE_TABLE = this.buildMuLawEncodeTable();
|
||||
|
||||
/**
|
||||
* Build μ-law to linear PCM16 decode table
|
||||
*/
|
||||
private buildMuLawDecodeTable(): Int16Array {
|
||||
const table = new Int16Array(256);
|
||||
for (let i = 0; i < 256; i++) {
|
||||
const mulaw = ~i;
|
||||
const exponent = (mulaw >> 4) & 0x07;
|
||||
const mantissa = mulaw & 0x0f;
|
||||
let sample = (mantissa << 3) + 0x84;
|
||||
sample <<= exponent;
|
||||
sample -= 0x84;
|
||||
if ((mulaw & 0x80) === 0) {
|
||||
sample = -sample;
|
||||
}
|
||||
table[i] = sample;
|
||||
}
|
||||
return table;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build linear PCM16 to μ-law encode table
|
||||
*/
|
||||
private buildMuLawEncodeTable(): Uint8Array {
|
||||
const table = new Uint8Array(65536);
|
||||
for (let i = 0; i < 65536; i++) {
|
||||
const sample = (i - 32768);
|
||||
const sign = sample < 0 ? 0x80 : 0x00;
|
||||
const magnitude = Math.abs(sample);
|
||||
|
||||
// Add bias
|
||||
let biased = magnitude + 0x84;
|
||||
|
||||
// Find exponent
|
||||
let exponent = 7;
|
||||
for (let exp = 0; exp < 8; exp++) {
|
||||
if (biased <= (0xff << exp)) {
|
||||
exponent = exp;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Extract mantissa
|
||||
const mantissa = (biased >> (exponent + 3)) & 0x0f;
|
||||
|
||||
// Combine sign, exponent, mantissa
|
||||
const mulaw = ~(sign | (exponent << 4) | mantissa);
|
||||
table[i] = mulaw & 0xff;
|
||||
}
|
||||
return table;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode μ-law audio to linear PCM16
|
||||
* @param mulawData - Buffer containing μ-law encoded audio
|
||||
* @returns Buffer containing PCM16 audio (16-bit little-endian)
|
||||
*/
|
||||
decodeMuLaw(mulawData: Buffer): Buffer {
|
||||
const pcm16 = Buffer.allocUnsafe(mulawData.length * 2);
|
||||
|
||||
for (let i = 0; i < mulawData.length; i++) {
|
||||
const sample = this.MULAW_DECODE_TABLE[mulawData[i]];
|
||||
pcm16.writeInt16LE(sample, i * 2);
|
||||
}
|
||||
|
||||
return pcm16;
|
||||
}
|
||||
|
||||
/**
|
||||
* Encode linear PCM16 to μ-law
|
||||
* @param pcm16Data - Buffer containing PCM16 audio (16-bit little-endian)
|
||||
* @returns Buffer containing μ-law encoded audio
|
||||
*/
|
||||
encodeMuLaw(pcm16Data: Buffer): Buffer {
|
||||
const mulaw = Buffer.allocUnsafe(pcm16Data.length / 2);
|
||||
|
||||
for (let i = 0; i < pcm16Data.length; i += 2) {
|
||||
const sample = pcm16Data.readInt16LE(i);
|
||||
const index = (sample + 32768) & 0xffff;
|
||||
mulaw[i / 2] = this.MULAW_ENCODE_TABLE[index];
|
||||
}
|
||||
|
||||
return mulaw;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resample audio from 8kHz to 24kHz (linear interpolation)
|
||||
* @param pcm16Data - Buffer containing 8kHz PCM16 audio
|
||||
* @returns Buffer containing 24kHz PCM16 audio
|
||||
*/
|
||||
resample8kTo24k(pcm16Data: Buffer): Buffer {
|
||||
const inputSamples = pcm16Data.length / 2;
|
||||
const outputSamples = Math.floor(inputSamples * 3); // 8k * 3 = 24k
|
||||
const output = Buffer.allocUnsafe(outputSamples * 2);
|
||||
|
||||
for (let i = 0; i < outputSamples; i++) {
|
||||
const srcIndex = i / 3;
|
||||
const srcIndexFloor = Math.floor(srcIndex);
|
||||
const srcIndexCeil = Math.min(srcIndexFloor + 1, inputSamples - 1);
|
||||
const fraction = srcIndex - srcIndexFloor;
|
||||
|
||||
const sample1 = pcm16Data.readInt16LE(srcIndexFloor * 2);
|
||||
const sample2 = pcm16Data.readInt16LE(srcIndexCeil * 2);
|
||||
|
||||
// Linear interpolation
|
||||
const interpolated = Math.round(sample1 + (sample2 - sample1) * fraction);
|
||||
output.writeInt16LE(interpolated, i * 2);
|
||||
}
|
||||
|
||||
return output;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resample audio from 24kHz to 8kHz (decimation with averaging)
|
||||
* @param pcm16Data - Buffer containing 24kHz PCM16 audio
|
||||
* @returns Buffer containing 8kHz PCM16 audio
|
||||
*/
|
||||
resample24kTo8k(pcm16Data: Buffer): Buffer {
|
||||
const inputSamples = pcm16Data.length / 2;
|
||||
const outputSamples = Math.floor(inputSamples / 3); // 24k / 3 = 8k
|
||||
const output = Buffer.allocUnsafe(outputSamples * 2);
|
||||
|
||||
for (let i = 0; i < outputSamples; i++) {
|
||||
// Average 3 samples for anti-aliasing
|
||||
const idx1 = Math.min(i * 3, inputSamples - 1);
|
||||
const idx2 = Math.min(i * 3 + 1, inputSamples - 1);
|
||||
const idx3 = Math.min(i * 3 + 2, inputSamples - 1);
|
||||
|
||||
const sample1 = pcm16Data.readInt16LE(idx1 * 2);
|
||||
const sample2 = pcm16Data.readInt16LE(idx2 * 2);
|
||||
const sample3 = pcm16Data.readInt16LE(idx3 * 2);
|
||||
|
||||
const averaged = Math.round((sample1 + sample2 + sample3) / 3);
|
||||
output.writeInt16LE(averaged, i * 2);
|
||||
}
|
||||
|
||||
return output;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert Twilio μ-law 8kHz to OpenAI PCM16 24kHz
|
||||
* @param twilioBase64 - Base64-encoded μ-law audio from Twilio
|
||||
* @returns Base64-encoded PCM16 24kHz audio for OpenAI
|
||||
*/
|
||||
twilioToOpenAI(twilioBase64: string): string {
|
||||
try {
|
||||
// Decode base64
|
||||
const mulawBuffer = Buffer.from(twilioBase64, 'base64');
|
||||
|
||||
// μ-law -> PCM16
|
||||
const pcm16_8k = this.decodeMuLaw(mulawBuffer);
|
||||
|
||||
// 8kHz -> 24kHz
|
||||
const pcm16_24k = this.resample8kTo24k(pcm16_8k);
|
||||
|
||||
// Encode to base64
|
||||
return pcm16_24k.toString('base64');
|
||||
} catch (error) {
|
||||
this.logger.error('Error converting Twilio to OpenAI audio', error);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert OpenAI PCM16 24kHz to Twilio μ-law 8kHz
|
||||
* @param openaiBase64 - Base64-encoded PCM16 24kHz audio from OpenAI
|
||||
* @returns Base64-encoded μ-law 8kHz audio for Twilio
|
||||
*/
|
||||
openAIToTwilio(openaiBase64: string): string {
|
||||
try {
|
||||
// Decode base64
|
||||
const pcm16_24k = Buffer.from(openaiBase64, 'base64');
|
||||
|
||||
// 24kHz -> 8kHz
|
||||
const pcm16_8k = this.resample24kTo8k(pcm16_24k);
|
||||
|
||||
// PCM16 -> μ-law
|
||||
const mulawBuffer = this.encodeMuLaw(pcm16_8k);
|
||||
|
||||
// Encode to base64
|
||||
return mulawBuffer.toString('base64');
|
||||
} catch (error) {
|
||||
this.logger.error('Error converting OpenAI to Twilio audio', error);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2,8 +2,9 @@ export interface TwilioConfig {
|
||||
accountSid: string;
|
||||
authToken: string;
|
||||
phoneNumber: string;
|
||||
apiKeySid?: string;
|
||||
apiKeySecret?: string;
|
||||
apiKey?: string; // API Key SID for generating access tokens
|
||||
apiSecret?: string; // API Key Secret
|
||||
twimlAppSid?: string; // TwiML App SID for Voice SDK
|
||||
}
|
||||
|
||||
export interface OpenAIConfig {
|
||||
|
||||
@@ -13,6 +13,7 @@ import { FastifyRequest, FastifyReply } from 'fastify';
|
||||
import { JwtAuthGuard } from '../auth/jwt-auth.guard';
|
||||
import { VoiceService } from './voice.service';
|
||||
import { VoiceGateway } from './voice.gateway';
|
||||
import { AudioConverterService } from './audio-converter.service';
|
||||
import { InitiateCallDto } from './dto/initiate-call.dto';
|
||||
import { TenantId } from '../tenant/tenant.decorator';
|
||||
|
||||
@@ -20,9 +21,13 @@ import { TenantId } from '../tenant/tenant.decorator';
|
||||
export class VoiceController {
|
||||
private readonly logger = new Logger(VoiceController.name);
|
||||
|
||||
// Track active Media Streams connections: streamSid -> WebSocket
|
||||
private mediaStreams: Map<string, any> = new Map();
|
||||
|
||||
constructor(
|
||||
private readonly voiceService: VoiceService,
|
||||
private readonly voiceGateway: VoiceGateway,
|
||||
private readonly audioConverter: AudioConverterService,
|
||||
) {}
|
||||
|
||||
/**
|
||||
@@ -49,6 +54,25 @@ export class VoiceController {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate Twilio access token for browser client
|
||||
*/
|
||||
@Get('token')
|
||||
@UseGuards(JwtAuthGuard)
|
||||
async getAccessToken(
|
||||
@Req() req: any,
|
||||
@TenantId() tenantId: string,
|
||||
) {
|
||||
const userId = req.user?.userId || req.user?.sub;
|
||||
|
||||
const token = await this.voiceService.generateAccessToken(tenantId, userId);
|
||||
|
||||
return {
|
||||
success: true,
|
||||
data: { token },
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Get call history
|
||||
*/
|
||||
@@ -73,22 +97,61 @@ export class VoiceController {
|
||||
}
|
||||
|
||||
/**
|
||||
* TwiML for outbound calls
|
||||
* TwiML for outbound calls from browser (Twilio Device)
|
||||
*/
|
||||
@Post('twiml/outbound')
|
||||
async outboundTwiml(@Req() req: FastifyRequest, @Res() res: FastifyReply) {
|
||||
const twiml = `<?xml version="1.0" encoding="UTF-8"?>
|
||||
const body = req.body as any;
|
||||
const to = body.To;
|
||||
const from = body.From;
|
||||
const callSid = body.CallSid;
|
||||
|
||||
this.logger.log(`=== TwiML OUTBOUND REQUEST RECEIVED ===`);
|
||||
this.logger.log(`CallSid: ${callSid}, Body From: ${from}, Body To: ${to}`);
|
||||
this.logger.log(`Full body: ${JSON.stringify(body)}`);
|
||||
|
||||
try {
|
||||
// Extract tenant domain from Host header
|
||||
const host = req.headers.host || '';
|
||||
const tenantDomain = host.split('.')[0]; // e.g., "tenant1" from "tenant1.routebox.co"
|
||||
|
||||
this.logger.log(`Extracted tenant domain: ${tenantDomain}`);
|
||||
|
||||
// Look up tenant's Twilio phone number from config
|
||||
let callerId = to; // Fallback (will cause error if not found)
|
||||
try {
|
||||
// Get Twilio config to find the phone number
|
||||
const { config } = await this.voiceService['getTwilioClient'](tenantDomain);
|
||||
callerId = config.phoneNumber;
|
||||
this.logger.log(`Retrieved Twilio phone number for tenant: ${callerId}`);
|
||||
} catch (error: any) {
|
||||
this.logger.error(`Failed to get Twilio config: ${error.message}`);
|
||||
}
|
||||
|
||||
const dialNumber = to;
|
||||
|
||||
this.logger.log(`Using callerId: ${callerId}, dialNumber: ${dialNumber}`);
|
||||
|
||||
// Return TwiML to DIAL the phone number with proper callerId
|
||||
const twiml = `<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Response>
|
||||
<Start>
|
||||
<Stream url="wss://${req.headers.host}/api/voice/stream" />
|
||||
</Start>
|
||||
<Say>Connecting your call</Say>
|
||||
<Dial>
|
||||
<Number>${(req.body as any).To}</Number>
|
||||
<Dial callerId="${callerId}">
|
||||
<Number>${dialNumber}</Number>
|
||||
</Dial>
|
||||
</Response>`;
|
||||
|
||||
res.type('text/xml').send(twiml);
|
||||
this.logger.log(`Returning TwiML with Dial verb - callerId: ${callerId}, to: ${dialNumber}`);
|
||||
res.type('text/xml').send(twiml);
|
||||
} catch (error: any) {
|
||||
this.logger.error(`=== ERROR GENERATING TWIML ===`);
|
||||
this.logger.error(`Error: ${error.message}`);
|
||||
this.logger.error(`Stack: ${error.stack}`);
|
||||
const errorTwiml = `<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Response>
|
||||
<Say>An error occurred while processing your call.</Say>
|
||||
</Response>`;
|
||||
res.type('text/xml').send(errorTwiml);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -101,24 +164,90 @@ export class VoiceController {
|
||||
const fromNumber = body.From;
|
||||
const toNumber = body.To;
|
||||
|
||||
this.logger.log(`Incoming call: ${callSid} from ${fromNumber} to ${toNumber}`);
|
||||
this.logger.log(`\n\n╔════════════════════════════════════════╗`);
|
||||
this.logger.log(`║ === INBOUND CALL RECEIVED ===`);
|
||||
this.logger.log(`╚════════════════════════════════════════╝`);
|
||||
this.logger.log(`CallSid: ${callSid}`);
|
||||
this.logger.log(`From: ${fromNumber}`);
|
||||
this.logger.log(`To: ${toNumber}`);
|
||||
this.logger.log(`Full body: ${JSON.stringify(body)}`);
|
||||
|
||||
// TODO: Determine tenant from phone number mapping
|
||||
// TODO: Find available user to route call to
|
||||
// For now, return a simple TwiML response
|
||||
try {
|
||||
// Extract tenant domain from Host header
|
||||
const host = req.headers.host || '';
|
||||
const tenantDomain = host.split('.')[0]; // e.g., "tenant1" from "tenant1.routebox.co"
|
||||
|
||||
this.logger.log(`Extracted tenant domain: ${tenantDomain}`);
|
||||
|
||||
const twiml = `<?xml version="1.0" encoding="UTF-8"?>
|
||||
// Get all connected users for this tenant
|
||||
const connectedUsers = this.voiceGateway.getConnectedUsers(tenantDomain);
|
||||
|
||||
this.logger.log(`Connected users for tenant ${tenantDomain}: ${connectedUsers.length}`);
|
||||
if (connectedUsers.length > 0) {
|
||||
this.logger.log(`Connected user IDs: ${connectedUsers.join(', ')}`);
|
||||
}
|
||||
|
||||
if (connectedUsers.length === 0) {
|
||||
// No users online - send to voicemail or play message
|
||||
const twiml = `<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Response>
|
||||
<Say>Sorry, no agents are currently available. Please try again later.</Say>
|
||||
<Hangup/>
|
||||
</Response>`;
|
||||
this.logger.log(`❌ No users online - returning unavailable message`);
|
||||
return res.type('text/xml').send(twiml);
|
||||
}
|
||||
|
||||
// Build TwiML to dial all connected clients with Media Streams for AI
|
||||
const clientElements = connectedUsers.map(userId => ` <Client>${userId}</Client>`).join('\n');
|
||||
|
||||
// Use wss:// for secure WebSocket (Traefik handles HTTPS)
|
||||
const streamUrl = `wss://${host}/api/voice/media-stream`;
|
||||
|
||||
this.logger.log(`Stream URL: ${streamUrl}`);
|
||||
this.logger.log(`Dialing ${connectedUsers.length} client(s)...`);
|
||||
this.logger.log(`Client IDs to dial: ${connectedUsers.join(', ')}`);
|
||||
|
||||
// Verify we have client IDs in proper format
|
||||
if (connectedUsers.length > 0) {
|
||||
this.logger.log(`First Client ID format check: "${connectedUsers[0]}" (length: ${connectedUsers[0].length})`);
|
||||
}
|
||||
|
||||
// Notify connected users about incoming call via Socket.IO
|
||||
connectedUsers.forEach(userId => {
|
||||
this.voiceGateway.notifyIncomingCall(userId, {
|
||||
callSid,
|
||||
fromNumber,
|
||||
toNumber,
|
||||
tenantDomain,
|
||||
});
|
||||
});
|
||||
|
||||
const twiml = `<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Response>
|
||||
<Start>
|
||||
<Stream url="wss://${req.headers.host}/api/voice/stream" />
|
||||
<Stream url="${streamUrl}">
|
||||
<Parameter name="tenantId" value="${tenantDomain}"/>
|
||||
<Parameter name="userId" value="${connectedUsers[0]}"/>
|
||||
</Stream>
|
||||
</Start>
|
||||
<Say>Please wait while we connect you to an agent</Say>
|
||||
<Dial>
|
||||
<Queue>support</Queue>
|
||||
<Dial timeout="30">
|
||||
${clientElements}
|
||||
</Dial>
|
||||
</Response>`;
|
||||
|
||||
res.type('text/xml').send(twiml);
|
||||
this.logger.log(`✓ Returning inbound TwiML with Media Streams - dialing ${connectedUsers.length} client(s)`);
|
||||
this.logger.log(`Generated TwiML:\n${twiml}\n`);
|
||||
res.type('text/xml').send(twiml);
|
||||
} catch (error: any) {
|
||||
this.logger.error(`Error generating inbound TwiML: ${error.message}`);
|
||||
const errorTwiml = `<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Response>
|
||||
<Say>Sorry, we are unable to connect your call at this time.</Say>
|
||||
<Hangup/>
|
||||
</Response>`;
|
||||
res.type('text/xml').send(errorTwiml);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -131,30 +260,8 @@ export class VoiceController {
|
||||
const status = body.CallStatus;
|
||||
const duration = body.CallDuration ? parseInt(body.CallDuration) : undefined;
|
||||
|
||||
this.logger.log(`Call status update: ${callSid} -> ${status}`);
|
||||
|
||||
// TODO: Extract tenant ID from call record
|
||||
// For now, we'll need to lookup the call to get tenant ID
|
||||
// This is a limitation - we should store tenantId in call metadata
|
||||
|
||||
try {
|
||||
// Update call status
|
||||
// await this.voiceService.updateCallStatus({
|
||||
// callSid,
|
||||
// tenantId: 'LOOKUP_NEEDED',
|
||||
// status,
|
||||
// duration,
|
||||
// });
|
||||
|
||||
// Notify user via WebSocket
|
||||
// await this.voiceGateway.notifyCallUpdate(userId, {
|
||||
// callSid,
|
||||
// status,
|
||||
// duration,
|
||||
// });
|
||||
} catch (error) {
|
||||
this.logger.error('Failed to process status webhook', error);
|
||||
}
|
||||
this.logger.log(`Call status webhook - CallSid: ${callSid}, Status: ${status}, Duration: ${duration}`);
|
||||
this.logger.log(`Full status webhook body:`, JSON.stringify(body));
|
||||
|
||||
return { success: true };
|
||||
}
|
||||
@@ -166,30 +273,223 @@ export class VoiceController {
|
||||
async recordingWebhook(@Req() req: FastifyRequest) {
|
||||
const body = req.body as any;
|
||||
const callSid = body.CallSid;
|
||||
const recordingUrl = body.RecordingUrl;
|
||||
const recordingSid = body.RecordingSid;
|
||||
const recordingStatus = body.RecordingStatus;
|
||||
|
||||
this.logger.log(`Recording available for call ${callSid}: ${recordingUrl}`);
|
||||
|
||||
// TODO: Update call record with recording URL
|
||||
// TODO: Trigger transcription if needed
|
||||
this.logger.log(`Recording webhook - CallSid: ${callSid}, RecordingSid: ${recordingSid}, Status: ${recordingStatus}`);
|
||||
|
||||
return { success: true };
|
||||
}
|
||||
|
||||
/**
|
||||
* WebSocket endpoint for Twilio Media Streams
|
||||
* Twilio Media Streams WebSocket endpoint
|
||||
* Receives real-time audio from Twilio and forwards to OpenAI Realtime API
|
||||
*
|
||||
* This handles the HTTP GET request and upgrades it to WebSocket manually.
|
||||
*/
|
||||
@Post('stream')
|
||||
async mediaStream(@Req() req: FastifyRequest, @Res() res: FastifyReply) {
|
||||
// Twilio Media Streams use WebSocket protocol
|
||||
// This would need to be handled by the WebSocket server
|
||||
// In Fastify, we need to upgrade the connection
|
||||
@Get('media-stream')
|
||||
mediaStream(@Req() req: FastifyRequest) {
|
||||
// For WebSocket upgrade, we need to access the raw socket
|
||||
let socket: any;
|
||||
|
||||
try {
|
||||
this.logger.log(`=== MEDIA STREAM REQUEST ===`);
|
||||
this.logger.log(`URL: ${req.url}`);
|
||||
this.logger.log(`Headers keys: ${Object.keys(req.headers).join(', ')}`);
|
||||
this.logger.log(`Headers: ${JSON.stringify(req.headers)}`);
|
||||
|
||||
// Check if this is a WebSocket upgrade request
|
||||
const hasWebSocketKey = 'sec-websocket-key' in req.headers;
|
||||
const hasWebSocketVersion = 'sec-websocket-version' in req.headers;
|
||||
|
||||
this.logger.log(`hasWebSocketKey: ${hasWebSocketKey}`);
|
||||
this.logger.log(`hasWebSocketVersion: ${hasWebSocketVersion}`);
|
||||
|
||||
if (!hasWebSocketKey || !hasWebSocketVersion) {
|
||||
this.logger.log('Not a WebSocket upgrade request - returning');
|
||||
return;
|
||||
}
|
||||
|
||||
this.logger.log('Media stream connection requested');
|
||||
this.logger.log('✓ WebSocket upgrade detected');
|
||||
|
||||
// Get the socket - try different ways
|
||||
socket = (req.raw as any).socket;
|
||||
this.logger.log(`Socket obtained: ${!!socket}`);
|
||||
|
||||
if (!socket) {
|
||||
this.logger.error('Failed to get socket from req.raw');
|
||||
return;
|
||||
}
|
||||
|
||||
const rawRequest = req.raw;
|
||||
const head = Buffer.alloc(0);
|
||||
|
||||
this.logger.log('Creating WebSocketServer...');
|
||||
const WebSocketServer = require('ws').Server;
|
||||
const wss = new WebSocketServer({ noServer: true });
|
||||
|
||||
this.logger.log('Calling handleUpgrade...');
|
||||
|
||||
// handleUpgrade will send the 101 response and take over the socket
|
||||
wss.handleUpgrade(rawRequest, socket, head, (ws: any) => {
|
||||
this.logger.log('=== TWILIO MEDIA STREAM WEBSOCKET UPGRADED SUCCESSFULLY ===');
|
||||
this.handleMediaStreamSocket(ws);
|
||||
});
|
||||
|
||||
this.logger.log('handleUpgrade completed');
|
||||
} catch (error: any) {
|
||||
this.logger.error(`=== FAILED TO UPGRADE TO WEBSOCKET ===`);
|
||||
this.logger.error(`Error message: ${error.message}`);
|
||||
this.logger.error(`Error stack: ${error.stack}`);
|
||||
}
|
||||
}
|
||||
|
||||
// TODO: Implement WebSocket upgrade for media streams
|
||||
// This will handle bidirectional audio streaming between Twilio and OpenAI
|
||||
/**
|
||||
* Handle incoming Media Stream WebSocket messages
|
||||
*/
|
||||
private handleMediaStreamSocket(ws: any) {
|
||||
let streamSid: string | null = null;
|
||||
let callSid: string | null = null;
|
||||
let tenantDomain: string | null = null;
|
||||
let mediaPacketCount = 0;
|
||||
|
||||
res.send({ message: 'WebSocket upgrade required' });
|
||||
// WebSocket message handler
|
||||
ws.on('message', async (message: Buffer) => {
|
||||
try {
|
||||
const msg = JSON.parse(message.toString());
|
||||
|
||||
switch (msg.event) {
|
||||
case 'connected':
|
||||
this.logger.log('=== MEDIA STREAM EVENT: CONNECTED ===');
|
||||
this.logger.log(`Protocol: ${msg.protocol}`);
|
||||
this.logger.log(`Version: ${msg.version}`);
|
||||
break;
|
||||
|
||||
case 'start':
|
||||
streamSid = msg.streamSid;
|
||||
callSid = msg.start.callSid;
|
||||
|
||||
// Extract tenant from customParameters if available
|
||||
tenantDomain = msg.start.customParameters?.tenantId || 'tenant1';
|
||||
|
||||
this.logger.log(`=== MEDIA STREAM EVENT: START ===`);
|
||||
this.logger.log(`StreamSid: ${streamSid}`);
|
||||
this.logger.log(`CallSid: ${callSid}`);
|
||||
this.logger.log(`Tenant: ${tenantDomain}`);
|
||||
this.logger.log(`AccountSid: ${msg.start.accountSid}`);
|
||||
this.logger.log(`MediaFormat: ${JSON.stringify(msg.start.mediaFormat)}`);
|
||||
this.logger.log(`Custom Parameters: ${JSON.stringify(msg.start.customParameters)}`);
|
||||
|
||||
// Store WebSocket connection
|
||||
this.mediaStreams.set(streamSid, ws);
|
||||
this.logger.log(`Stored WebSocket for streamSid: ${streamSid}. Total active streams: ${this.mediaStreams.size}`);
|
||||
|
||||
// Initialize OpenAI Realtime connection for this call
|
||||
this.logger.log(`Initializing OpenAI Realtime for call ${callSid}...`);
|
||||
await this.voiceService.initializeOpenAIRealtime({
|
||||
callSid,
|
||||
tenantId: tenantDomain,
|
||||
userId: msg.start.customParameters?.userId || 'system',
|
||||
});
|
||||
|
||||
this.logger.log(`✓ OpenAI Realtime initialized for call ${callSid}`);
|
||||
break;
|
||||
|
||||
case 'media':
|
||||
mediaPacketCount++;
|
||||
if (mediaPacketCount % 50 === 0) {
|
||||
// Log every 50th packet to avoid spam
|
||||
this.logger.log(`Received media packet #${mediaPacketCount} for StreamSid: ${streamSid}, CallSid: ${callSid}, PayloadSize: ${msg.media.payload?.length || 0} bytes`);
|
||||
}
|
||||
|
||||
if (!callSid || !tenantDomain) {
|
||||
this.logger.warn('Received media before start event');
|
||||
break;
|
||||
}
|
||||
|
||||
// msg.media.payload is base64-encoded μ-law audio from Twilio
|
||||
const twilioAudio = msg.media.payload;
|
||||
|
||||
// Convert Twilio audio (μ-law 8kHz) to OpenAI format (PCM16 24kHz)
|
||||
const openaiAudio = this.audioConverter.twilioToOpenAI(twilioAudio);
|
||||
|
||||
// Send audio to OpenAI Realtime API
|
||||
await this.voiceService.sendAudioToOpenAI(callSid, openaiAudio);
|
||||
break;
|
||||
|
||||
case 'stop':
|
||||
this.logger.log(`=== MEDIA STREAM EVENT: STOP ===`);
|
||||
this.logger.log(`StreamSid: ${streamSid}`);
|
||||
this.logger.log(`Total media packets received: ${mediaPacketCount}`);
|
||||
|
||||
if (streamSid) {
|
||||
this.mediaStreams.delete(streamSid);
|
||||
this.logger.log(`Removed WebSocket for streamSid: ${streamSid}. Remaining active streams: ${this.mediaStreams.size}`);
|
||||
}
|
||||
|
||||
// Clean up OpenAI connection
|
||||
if (callSid) {
|
||||
this.logger.log(`Cleaning up OpenAI connection for call ${callSid}...`);
|
||||
await this.voiceService.cleanupOpenAIConnection(callSid);
|
||||
this.logger.log(`✓ OpenAI connection cleaned up for call ${callSid}`);
|
||||
}
|
||||
break;
|
||||
|
||||
default:
|
||||
this.logger.debug(`Unknown media stream event: ${msg.event}`);
|
||||
}
|
||||
} catch (error: any) {
|
||||
this.logger.error(`Error processing media stream message: ${error.message}`);
|
||||
this.logger.error(`Stack: ${error.stack}`);
|
||||
}
|
||||
});
|
||||
|
||||
ws.on('close', () => {
|
||||
this.logger.log(`=== MEDIA STREAM WEBSOCKET CLOSED ===`);
|
||||
this.logger.log(`StreamSid: ${streamSid}`);
|
||||
this.logger.log(`Total media packets in this stream: ${mediaPacketCount}`);
|
||||
if (streamSid) {
|
||||
this.mediaStreams.delete(streamSid);
|
||||
this.logger.log(`Cleaned up streamSid on close. Remaining active streams: ${this.mediaStreams.size}`);
|
||||
}
|
||||
});
|
||||
|
||||
ws.on('error', (error: Error) => {
|
||||
this.logger.error(`=== MEDIA STREAM WEBSOCKET ERROR ===`);
|
||||
this.logger.error(`StreamSid: ${streamSid}`);
|
||||
this.logger.error(`Error message: ${error.message}`);
|
||||
this.logger.error(`Error stack: ${error.stack}`);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Send audio from OpenAI back to Twilio Media Stream
|
||||
*/
|
||||
async sendAudioToTwilio(streamSid: string, openaiAudioBase64: string) {
|
||||
const ws = this.mediaStreams.get(streamSid);
|
||||
|
||||
if (!ws) {
|
||||
this.logger.warn(`No Media Stream found for streamSid: ${streamSid}`);
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
// Convert OpenAI audio (PCM16 24kHz) to Twilio format (μ-law 8kHz)
|
||||
const twilioAudio = this.audioConverter.openAIToTwilio(openaiAudioBase64);
|
||||
|
||||
// Send to Twilio Media Stream
|
||||
const message = {
|
||||
event: 'media',
|
||||
streamSid,
|
||||
media: {
|
||||
payload: twilioAudio,
|
||||
},
|
||||
};
|
||||
|
||||
ws.send(JSON.stringify(message));
|
||||
} catch (error: any) {
|
||||
this.logger.error(`Error sending audio to Twilio: ${error.message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -40,7 +40,10 @@ export class VoiceGateway
|
||||
private readonly jwtService: JwtService,
|
||||
private readonly voiceService: VoiceService,
|
||||
private readonly tenantDbService: TenantDatabaseService,
|
||||
) {}
|
||||
) {
|
||||
// Set gateway reference in service to avoid circular dependency
|
||||
this.voiceService.setGateway(this);
|
||||
}
|
||||
|
||||
async handleConnection(client: AuthenticatedSocket) {
|
||||
try {
|
||||
@@ -49,21 +52,42 @@ export class VoiceGateway
|
||||
client.handshake.auth.token || client.handshake.headers.authorization?.split(' ')[1];
|
||||
|
||||
if (!token) {
|
||||
this.logger.warn('Client connection rejected: No token provided');
|
||||
this.logger.warn('❌ Client connection rejected: No token provided');
|
||||
client.disconnect();
|
||||
return;
|
||||
}
|
||||
|
||||
// Verify JWT token
|
||||
const payload = await this.jwtService.verifyAsync(token);
|
||||
client.tenantId = payload.tenantId;
|
||||
|
||||
// Extract domain from origin header (e.g., http://tenant1.routebox.co:3001)
|
||||
// The domains table stores just the subdomain part (e.g., "tenant1")
|
||||
const origin = client.handshake.headers.origin || client.handshake.headers.referer;
|
||||
let domain = 'localhost';
|
||||
|
||||
if (origin) {
|
||||
try {
|
||||
const url = new URL(origin);
|
||||
const hostname = url.hostname; // e.g., tenant1.routebox.co or localhost
|
||||
|
||||
// Extract first part of subdomain as domain
|
||||
// tenant1.routebox.co -> tenant1
|
||||
// localhost -> localhost
|
||||
domain = hostname.split('.')[0];
|
||||
} catch (error) {
|
||||
this.logger.warn(`Failed to parse origin: ${origin}`);
|
||||
}
|
||||
}
|
||||
|
||||
client.tenantId = domain; // Store the subdomain as tenantId
|
||||
client.userId = payload.sub;
|
||||
client.tenantSlug = payload.tenantSlug;
|
||||
client.tenantSlug = domain; // Same as subdomain
|
||||
|
||||
this.connectedUsers.set(client.userId, client);
|
||||
this.logger.log(
|
||||
`Client connected: ${client.id} (User: ${client.userId}, Tenant: ${client.tenantSlug})`,
|
||||
`✓ Client connected: ${client.id} (User: ${client.userId}, Domain: ${domain})`,
|
||||
);
|
||||
this.logger.log(`Total connected users in ${domain}: ${this.getConnectedUsers(domain).length}`);
|
||||
|
||||
// Send current call state if any active call
|
||||
const activeCallSid = this.activeCallsByUser.get(client.userId);
|
||||
@@ -75,7 +99,7 @@ export class VoiceGateway
|
||||
client.emit('call:state', callState);
|
||||
}
|
||||
} catch (error) {
|
||||
this.logger.error('Authentication failed', error);
|
||||
this.logger.error('❌ Authentication failed', error);
|
||||
client.disconnect();
|
||||
}
|
||||
}
|
||||
@@ -83,7 +107,8 @@ export class VoiceGateway
|
||||
handleDisconnect(client: AuthenticatedSocket) {
|
||||
if (client.userId) {
|
||||
this.connectedUsers.delete(client.userId);
|
||||
this.logger.log(`Client disconnected: ${client.id} (User: ${client.userId})`);
|
||||
this.logger.log(`✓ Client disconnected: ${client.id} (User: ${client.userId})`);
|
||||
this.logger.log(`Remaining connected users: ${this.connectedUsers.size}`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -256,8 +281,13 @@ export class VoiceGateway
|
||||
*/
|
||||
async notifyAiSuggestion(userId: string, data: any) {
|
||||
const socket = this.connectedUsers.get(userId);
|
||||
this.logger.log(`notifyAiSuggestion - userId: ${userId}, socket connected: ${!!socket}, total connected users: ${this.connectedUsers.size}`);
|
||||
if (socket) {
|
||||
this.logger.log(`Emitting ai:suggestion event with data:`, JSON.stringify(data));
|
||||
socket.emit('ai:suggestion', data);
|
||||
} else {
|
||||
this.logger.warn(`No socket connection found for userId: ${userId}`);
|
||||
this.logger.log(`Connected users: ${Array.from(this.connectedUsers.keys()).join(', ')}`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -270,4 +300,20 @@ export class VoiceGateway
|
||||
socket.emit('ai:action', data);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get connected users for a tenant
|
||||
*/
|
||||
getConnectedUsers(tenantDomain?: string): string[] {
|
||||
const userIds: string[] = [];
|
||||
|
||||
for (const [userId, socket] of this.connectedUsers.entries()) {
|
||||
// If tenantDomain specified, filter by tenant
|
||||
if (!tenantDomain || socket.tenantSlug === tenantDomain) {
|
||||
userIds.push(userId);
|
||||
}
|
||||
}
|
||||
|
||||
return userIds;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ import { JwtModule } from '@nestjs/jwt';
|
||||
import { VoiceGateway } from './voice.gateway';
|
||||
import { VoiceService } from './voice.service';
|
||||
import { VoiceController } from './voice.controller';
|
||||
import { AudioConverterService } from './audio-converter.service';
|
||||
import { TenantModule } from '../tenant/tenant.module';
|
||||
import { AuthModule } from '../auth/auth.module';
|
||||
|
||||
@@ -15,7 +16,7 @@ import { AuthModule } from '../auth/auth.module';
|
||||
signOptions: { expiresIn: process.env.JWT_EXPIRES_IN || '24h' },
|
||||
}),
|
||||
],
|
||||
providers: [VoiceGateway, VoiceService],
|
||||
providers: [VoiceGateway, VoiceService, AudioConverterService],
|
||||
controllers: [VoiceController],
|
||||
exports: [VoiceService],
|
||||
})
|
||||
|
||||
@@ -6,54 +6,82 @@ import * as Twilio from 'twilio';
|
||||
import { WebSocket } from 'ws';
|
||||
import { v4 as uuidv4 } from 'uuid';
|
||||
|
||||
const AccessToken = Twilio.jwt.AccessToken;
|
||||
const VoiceGrant = AccessToken.VoiceGrant;
|
||||
|
||||
@Injectable()
|
||||
export class VoiceService {
|
||||
private readonly logger = new Logger(VoiceService.name);
|
||||
private twilioClients: Map<string, Twilio.Twilio> = new Map();
|
||||
private openaiConnections: Map<string, WebSocket> = new Map(); // callSid -> WebSocket
|
||||
private callStates: Map<string, any> = new Map(); // callSid -> call state
|
||||
private voiceGateway: any; // Reference to gateway (to avoid circular dependency)
|
||||
|
||||
constructor(
|
||||
private readonly tenantDbService: TenantDatabaseService,
|
||||
) {}
|
||||
|
||||
/**
|
||||
* Set gateway reference (called by gateway on init)
|
||||
*/
|
||||
setGateway(gateway: any) {
|
||||
this.voiceGateway = gateway;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get Twilio client for a tenant
|
||||
*/
|
||||
private async getTwilioClient(tenantId: string): Promise<{ client: Twilio.Twilio; config: TwilioConfig }> {
|
||||
private async getTwilioClient(tenantIdOrDomain: string): Promise<{ client: Twilio.Twilio; config: TwilioConfig; tenantId: string }> {
|
||||
// Check cache first
|
||||
if (this.twilioClients.has(tenantId)) {
|
||||
if (this.twilioClients.has(tenantIdOrDomain)) {
|
||||
const centralPrisma = getCentralPrisma();
|
||||
const tenant = await centralPrisma.tenant.findUnique({
|
||||
where: { id: tenantId },
|
||||
select: { integrationsConfig: true },
|
||||
|
||||
// Look up tenant by domain
|
||||
const domainRecord = await centralPrisma.domain.findUnique({
|
||||
where: { domain: tenantIdOrDomain },
|
||||
include: { tenant: { select: { id: true, integrationsConfig: true } } },
|
||||
});
|
||||
|
||||
const config = this.getIntegrationConfig(tenant?.integrationsConfig as any);
|
||||
return { client: this.twilioClients.get(tenantId), config: config.twilio };
|
||||
const config = this.getIntegrationConfig(domainRecord?.tenant?.integrationsConfig as any);
|
||||
return {
|
||||
client: this.twilioClients.get(tenantIdOrDomain),
|
||||
config: config.twilio,
|
||||
tenantId: domainRecord.tenant.id
|
||||
};
|
||||
}
|
||||
|
||||
// Fetch tenant integrations config
|
||||
const centralPrisma = getCentralPrisma();
|
||||
const tenant = await centralPrisma.tenant.findUnique({
|
||||
where: { id: tenantId },
|
||||
select: { integrationsConfig: true },
|
||||
|
||||
this.logger.log(`Looking up domain: ${tenantIdOrDomain}`);
|
||||
|
||||
const domainRecord = await centralPrisma.domain.findUnique({
|
||||
where: { domain: tenantIdOrDomain },
|
||||
include: { tenant: { select: { id: true, integrationsConfig: true } } },
|
||||
});
|
||||
|
||||
if (!tenant?.integrationsConfig) {
|
||||
throw new Error('Tenant integrations config not found');
|
||||
this.logger.log(`Domain record found: ${!!domainRecord}, Tenant: ${!!domainRecord?.tenant}, Config: ${!!domainRecord?.tenant?.integrationsConfig}`);
|
||||
|
||||
if (!domainRecord?.tenant) {
|
||||
throw new Error(`Domain ${tenantIdOrDomain} not found`);
|
||||
}
|
||||
|
||||
const config = this.getIntegrationConfig(tenant.integrationsConfig as any);
|
||||
if (!domainRecord.tenant.integrationsConfig) {
|
||||
throw new Error('Tenant integrations config not found. Please configure Twilio credentials in Settings > Integrations');
|
||||
}
|
||||
|
||||
const config = this.getIntegrationConfig(domainRecord.tenant.integrationsConfig as any);
|
||||
|
||||
this.logger.log(`Config decrypted: ${!!config.twilio}, AccountSid: ${config.twilio?.accountSid?.substring(0, 10)}..., AuthToken: ${config.twilio?.authToken?.substring(0, 10)}..., Phone: ${config.twilio?.phoneNumber}`);
|
||||
|
||||
if (!config.twilio?.accountSid || !config.twilio?.authToken) {
|
||||
throw new Error('Twilio credentials not configured for tenant');
|
||||
}
|
||||
|
||||
const client = Twilio.default(config.twilio.accountSid, config.twilio.authToken);
|
||||
this.twilioClients.set(tenantId, client);
|
||||
this.twilioClients.set(tenantIdOrDomain, client);
|
||||
|
||||
return { client, config: config.twilio };
|
||||
return { client, config: config.twilio, tenantId: domainRecord.tenant.id };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -77,6 +105,35 @@ export class VoiceService {
|
||||
return {};
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate Twilio access token for browser Voice SDK
|
||||
*/
|
||||
async generateAccessToken(tenantDomain: string, userId: string): Promise<string> {
|
||||
const { config, tenantId } = await this.getTwilioClient(tenantDomain);
|
||||
|
||||
if (!config.accountSid || !config.apiKey || !config.apiSecret) {
|
||||
throw new Error('Twilio API credentials not configured. Please add API Key and Secret in Settings > Integrations');
|
||||
}
|
||||
|
||||
// Create an access token
|
||||
const token = new AccessToken(
|
||||
config.accountSid,
|
||||
config.apiKey,
|
||||
config.apiSecret,
|
||||
{ identity: userId, ttl: 3600 } // 1 hour expiry
|
||||
);
|
||||
|
||||
// Create a Voice grant
|
||||
const voiceGrant = new VoiceGrant({
|
||||
outgoingApplicationSid: config.twimlAppSid, // TwiML App SID for outbound calls
|
||||
incomingAllow: true, // Allow incoming calls
|
||||
});
|
||||
|
||||
token.addGrant(voiceGrant);
|
||||
|
||||
return token.toJwt();
|
||||
}
|
||||
|
||||
/**
|
||||
* Initiate outbound call
|
||||
*/
|
||||
@@ -85,36 +142,63 @@ export class VoiceService {
|
||||
userId: string;
|
||||
toNumber: string;
|
||||
}) {
|
||||
const { tenantId, userId, toNumber } = params;
|
||||
const { tenantId: tenantDomain, userId, toNumber } = params;
|
||||
|
||||
try {
|
||||
const { client, config } = await this.getTwilioClient(tenantId);
|
||||
this.logger.log(`=== INITIATING CALL ===`);
|
||||
this.logger.log(`Domain: ${tenantDomain}, To: ${toNumber}, User: ${userId}`);
|
||||
|
||||
// Validate phone number
|
||||
if (!toNumber.match(/^\+?[1-9]\d{1,14}$/)) {
|
||||
throw new Error(`Invalid phone number format: ${toNumber}. Use E.164 format (e.g., +1234567890)`);
|
||||
}
|
||||
|
||||
const { client, config, tenantId } = await this.getTwilioClient(tenantDomain);
|
||||
this.logger.log(`Twilio client obtained for tenant: ${tenantId}`);
|
||||
|
||||
// Get from number
|
||||
const fromNumber = config.phoneNumber;
|
||||
if (!fromNumber) {
|
||||
throw new Error('Twilio phone number not configured');
|
||||
}
|
||||
this.logger.log(`From number: ${fromNumber}`);
|
||||
|
||||
// Construct tenant-specific webhook URLs using HTTPS (for Traefik)
|
||||
const backendUrl = `https://${tenantDomain}`;
|
||||
const twimlUrl = `${backendUrl}/api/voice/twiml/outbound?phoneNumber=${encodeURIComponent(fromNumber)}&toNumber=${encodeURIComponent(toNumber)}`;
|
||||
const statusUrl = `${backendUrl}/api/voice/webhook/status`;
|
||||
|
||||
this.logger.log(`TwiML URL: ${twimlUrl}`);
|
||||
this.logger.log(`Status URL: ${statusUrl}`);
|
||||
|
||||
// Create call record in database
|
||||
const tenantKnex = await this.tenantDbService.getTenantKnexById(tenantId);
|
||||
const callId = uuidv4();
|
||||
|
||||
// Generate TwiML URL for call flow
|
||||
const twimlUrl = `${process.env.BACKEND_URL || 'http://localhost:3000'}/api/voice/twiml/outbound`;
|
||||
|
||||
// Initiate call via Twilio
|
||||
this.logger.log(`Calling Twilio API...`);
|
||||
|
||||
// For Device-to-Number calls, we need to use a TwiML App SID
|
||||
// The Twilio SDK will handle the Device connection, and we return TwiML with Dial
|
||||
const call = await client.calls.create({
|
||||
to: toNumber,
|
||||
from: config.phoneNumber,
|
||||
from: fromNumber, // Your Twilio phone number
|
||||
url: twimlUrl,
|
||||
statusCallback: `${process.env.BACKEND_URL || 'http://localhost:3000'}/api/voice/webhook/status`,
|
||||
statusCallback: statusUrl,
|
||||
statusCallbackEvent: ['initiated', 'ringing', 'answered', 'completed'],
|
||||
statusCallbackMethod: 'POST',
|
||||
record: true,
|
||||
recordingStatusCallback: `${process.env.BACKEND_URL || 'http://localhost:3000'}/api/voice/webhook/recording`,
|
||||
record: false,
|
||||
machineDetection: 'Enable', // Optional: detect answering machines
|
||||
});
|
||||
|
||||
this.logger.log(`Call created successfully: ${call.sid}, Status: ${call.status}`);
|
||||
|
||||
// Store call in database
|
||||
await tenantKnex('calls').insert({
|
||||
id: callId,
|
||||
call_sid: call.sid,
|
||||
direction: 'outbound',
|
||||
from_number: config.phoneNumber,
|
||||
from_number: fromNumber,
|
||||
to_number: toNumber,
|
||||
status: 'queued',
|
||||
user_id: userId,
|
||||
@@ -352,12 +436,30 @@ export class VoiceService {
|
||||
const { callSid, tenantId, userId } = params;
|
||||
|
||||
try {
|
||||
// Get OpenAI config
|
||||
// Get OpenAI config - tenantId might be a domain, so look it up
|
||||
const centralPrisma = getCentralPrisma();
|
||||
const tenant = await centralPrisma.tenant.findUnique({
|
||||
where: { id: tenantId },
|
||||
select: { integrationsConfig: true },
|
||||
});
|
||||
|
||||
// Try to find tenant by domain first (if tenantId is like "tenant1")
|
||||
let tenant;
|
||||
if (!tenantId.match(/^[0-9a-f]{8}-[0-9a-f]{4}-/i)) {
|
||||
// Looks like a domain, not a UUID
|
||||
const domainRecord = await centralPrisma.domain.findUnique({
|
||||
where: { domain: tenantId },
|
||||
include: { tenant: { select: { id: true, integrationsConfig: true } } },
|
||||
});
|
||||
tenant = domainRecord?.tenant;
|
||||
} else {
|
||||
// It's a UUID
|
||||
tenant = await centralPrisma.tenant.findUnique({
|
||||
where: { id: tenantId },
|
||||
select: { id: true, integrationsConfig: true },
|
||||
});
|
||||
}
|
||||
|
||||
if (!tenant) {
|
||||
this.logger.warn(`Tenant not found for identifier: ${tenantId}`);
|
||||
return;
|
||||
}
|
||||
|
||||
const config = this.getIntegrationConfig(tenant?.integrationsConfig as any);
|
||||
|
||||
@@ -367,7 +469,8 @@ export class VoiceService {
|
||||
}
|
||||
|
||||
// Connect to OpenAI Realtime API
|
||||
const ws = new WebSocket('wss://api.openai.com/v1/realtime', {
|
||||
const model = config.openai.model || 'gpt-4o-realtime-preview-2024-10-01';
|
||||
const ws = new WebSocket(`wss://api.openai.com/v1/realtime?model=${model}`, {
|
||||
headers: {
|
||||
'Authorization': `Bearer ${config.openai.apiKey}`,
|
||||
'OpenAI-Beta': 'realtime=v1',
|
||||
@@ -377,13 +480,39 @@ export class VoiceService {
|
||||
ws.on('open', () => {
|
||||
this.logger.log(`OpenAI Realtime connected for call ${callSid}`);
|
||||
|
||||
// Add to connections map only after it's open
|
||||
this.openaiConnections.set(callSid, ws);
|
||||
|
||||
// Store call state with userId for later use
|
||||
this.callStates.set(callSid, {
|
||||
callSid,
|
||||
tenantId: tenant.id,
|
||||
userId,
|
||||
status: 'in-progress',
|
||||
});
|
||||
this.logger.log(`📝 Stored call state for ${callSid} with userId: ${userId}`);
|
||||
|
||||
// Initialize session
|
||||
ws.send(JSON.stringify({
|
||||
type: 'session.update',
|
||||
session: {
|
||||
model: config.openai.model || 'gpt-4o-realtime-preview',
|
||||
voice: config.openai.voice || 'alloy',
|
||||
instructions: 'You are a helpful AI assistant providing real-time support during phone calls. Provide concise, actionable suggestions to help the user.',
|
||||
instructions: `You are an AI assistant in LISTENING MODE, helping a sales/support agent during their phone call.
|
||||
|
||||
IMPORTANT: You are NOT talking to the caller. You are advising the agent who is handling the call.
|
||||
|
||||
Your role:
|
||||
- Listen to the conversation between the agent and the caller
|
||||
- Provide concise, actionable suggestions to help the agent
|
||||
- Recommend CRM actions (search contacts, create tasks, update records)
|
||||
- Alert the agent to important information or next steps
|
||||
- Keep suggestions brief (1-2 sentences max)
|
||||
|
||||
Format your suggestions like:
|
||||
"💡 Suggestion: [your advice]"
|
||||
"⚠️ Alert: [important notice]"
|
||||
"📋 Action: [recommended CRM action]"`,
|
||||
turn_detection: {
|
||||
type: 'server_vad',
|
||||
},
|
||||
@@ -393,24 +522,84 @@ export class VoiceService {
|
||||
});
|
||||
|
||||
ws.on('message', (data: Buffer) => {
|
||||
this.handleOpenAIMessage(callSid, tenantId, userId, JSON.parse(data.toString()));
|
||||
// Pass the tenant UUID (tenant.id) instead of the domain string
|
||||
this.handleOpenAIMessage(callSid, tenant.id, userId, JSON.parse(data.toString()));
|
||||
});
|
||||
|
||||
ws.on('error', (error) => {
|
||||
this.logger.error(`OpenAI WebSocket error for call ${callSid}`, error);
|
||||
});
|
||||
|
||||
ws.on('close', () => {
|
||||
this.logger.log(`OpenAI Realtime disconnected for call ${callSid}`);
|
||||
this.logger.error(`OpenAI WebSocket error for call ${callSid}:`, error);
|
||||
this.openaiConnections.delete(callSid);
|
||||
});
|
||||
|
||||
this.openaiConnections.set(callSid, ws);
|
||||
ws.on('close', (code, reason) => {
|
||||
this.logger.log(`OpenAI Realtime disconnected for call ${callSid} - Code: ${code}, Reason: ${reason.toString()}`);
|
||||
this.openaiConnections.delete(callSid);
|
||||
});
|
||||
|
||||
// Don't add to connections here - wait for 'open' event
|
||||
} catch (error) {
|
||||
this.logger.error('Failed to initialize OpenAI Realtime', error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Send audio data to OpenAI Realtime API
|
||||
*/
|
||||
async sendAudioToOpenAI(callSid: string, audioBase64: string) {
|
||||
const ws = this.openaiConnections.get(callSid);
|
||||
|
||||
if (!ws) {
|
||||
this.logger.warn(`No OpenAI connection for call ${callSid}`);
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
// Send audio chunk to OpenAI
|
||||
ws.send(JSON.stringify({
|
||||
type: 'input_audio_buffer.append',
|
||||
audio: audioBase64,
|
||||
}));
|
||||
} catch (error) {
|
||||
this.logger.error(`Failed to send audio to OpenAI for call ${callSid}`, error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Commit audio buffer to OpenAI (trigger processing)
|
||||
*/
|
||||
async commitAudioBuffer(callSid: string) {
|
||||
const ws = this.openaiConnections.get(callSid);
|
||||
|
||||
if (!ws) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
ws.send(JSON.stringify({
|
||||
type: 'input_audio_buffer.commit',
|
||||
}));
|
||||
} catch (error) {
|
||||
this.logger.error(`Failed to commit audio buffer for call ${callSid}`, error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Clean up OpenAI connection for a call
|
||||
*/
|
||||
async cleanupOpenAIConnection(callSid: string) {
|
||||
const ws = this.openaiConnections.get(callSid);
|
||||
|
||||
if (ws) {
|
||||
try {
|
||||
ws.close();
|
||||
this.openaiConnections.delete(callSid);
|
||||
this.logger.log(`Cleaned up OpenAI connection for call ${callSid}`);
|
||||
} catch (error) {
|
||||
this.logger.error(`Error cleaning up OpenAI connection for call ${callSid}`, error);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle OpenAI Realtime messages
|
||||
*/
|
||||
@@ -423,21 +612,65 @@ export class VoiceService {
|
||||
try {
|
||||
switch (message.type) {
|
||||
case 'conversation.item.created':
|
||||
if (message.item.type === 'message' && message.item.role === 'assistant') {
|
||||
// AI response generated
|
||||
this.logger.log(`AI response for call ${callSid}`);
|
||||
// Skip logging for now
|
||||
break;
|
||||
|
||||
case 'response.audio.delta':
|
||||
// OpenAI is sending audio response (skip logging)
|
||||
const state = this.callStates.get(callSid);
|
||||
if (state?.streamSid && message.delta) {
|
||||
if (!state.pendingAudio) {
|
||||
state.pendingAudio = [];
|
||||
}
|
||||
state.pendingAudio.push(message.delta);
|
||||
}
|
||||
break;
|
||||
|
||||
case 'response.audio.done':
|
||||
// Skip logging
|
||||
break;
|
||||
|
||||
case 'response.audio_transcript.delta':
|
||||
// Real-time transcript
|
||||
// TODO: Emit to gateway
|
||||
// Skip - not transmitting individual words to frontend
|
||||
break;
|
||||
|
||||
case 'response.audio_transcript.done':
|
||||
// Final transcript
|
||||
// Final transcript - this contains the AI's actual text suggestions!
|
||||
const transcript = message.transcript;
|
||||
this.logger.log(`💡 AI Suggestion: "${transcript}"`);
|
||||
|
||||
// Save to database
|
||||
await this.updateCallTranscript(callSid, tenantId, transcript);
|
||||
|
||||
// Also send as suggestion to frontend if it looks like a suggestion
|
||||
if (transcript && transcript.length > 0) {
|
||||
// Determine suggestion type
|
||||
let suggestionType: 'response' | 'action' | 'insight' = 'insight';
|
||||
if (transcript.includes('💡') || transcript.toLowerCase().includes('suggest')) {
|
||||
suggestionType = 'response';
|
||||
} else if (transcript.includes('📋') || transcript.toLowerCase().includes('action')) {
|
||||
suggestionType = 'action';
|
||||
} else if (transcript.includes('⚠️') || transcript.toLowerCase().includes('alert')) {
|
||||
suggestionType = 'insight';
|
||||
}
|
||||
|
||||
// Emit to frontend
|
||||
const state = this.callStates.get(callSid);
|
||||
this.logger.log(`📊 Call state - userId: ${state?.userId}, gateway: ${!!this.voiceGateway}`);
|
||||
|
||||
if (state?.userId && this.voiceGateway) {
|
||||
this.logger.log(`📤 Sending to user ${state.userId}`);
|
||||
await this.voiceGateway.notifyAiSuggestion(state.userId, {
|
||||
type: suggestionType,
|
||||
text: transcript,
|
||||
callSid,
|
||||
timestamp: new Date().toISOString(),
|
||||
});
|
||||
this.logger.log(`✅ Suggestion sent to agent`);
|
||||
} else {
|
||||
this.logger.warn(`❌ Cannot send - userId: ${state?.userId}, gateway: ${!!this.voiceGateway}, callStates has ${this.callStates.size} entries`);
|
||||
}
|
||||
}
|
||||
break;
|
||||
|
||||
case 'response.function_call_arguments.done':
|
||||
@@ -445,8 +678,26 @@ export class VoiceService {
|
||||
await this.handleToolCall(callSid, tenantId, userId, message);
|
||||
break;
|
||||
|
||||
case 'session.created':
|
||||
case 'session.updated':
|
||||
case 'response.created':
|
||||
case 'response.output_item.added':
|
||||
case 'response.content_part.added':
|
||||
case 'response.content_part.done':
|
||||
case 'response.output_item.done':
|
||||
case 'response.done':
|
||||
case 'input_audio_buffer.speech_started':
|
||||
case 'input_audio_buffer.speech_stopped':
|
||||
case 'input_audio_buffer.committed':
|
||||
// Skip logging for these (too noisy)
|
||||
break;
|
||||
|
||||
case 'error':
|
||||
this.logger.error(`OpenAI error for call ${callSid}: ${JSON.stringify(message.error)}`);
|
||||
break;
|
||||
|
||||
default:
|
||||
// Handle other message types
|
||||
// Only log unhandled types occasionally
|
||||
break;
|
||||
}
|
||||
} catch (error) {
|
||||
|
||||
65
docs/TWILIO_SETUP.md
Normal file
65
docs/TWILIO_SETUP.md
Normal file
@@ -0,0 +1,65 @@
|
||||
# Twilio Setup Guide for Softphone
|
||||
|
||||
## Prerequisites
|
||||
- Twilio account with a phone number
|
||||
- Account SID and Auth Token
|
||||
|
||||
## Basic Setup (Current - Makes calls but no browser audio)
|
||||
|
||||
Currently, the softphone initiates calls through Twilio's REST API, but the audio doesn't flow through the browser. The calls go directly to your mobile device with a simple TwiML message.
|
||||
|
||||
## Full Browser Audio Setup (Requires additional configuration)
|
||||
|
||||
To enable actual softphone functionality where audio flows through your browser's microphone and speakers, you need:
|
||||
|
||||
### Option 1: Twilio Client SDK (Recommended)
|
||||
|
||||
1. **Create a TwiML App in Twilio Console**
|
||||
- Go to https://console.twilio.com/us1/develop/voice/manage/twiml-apps
|
||||
- Click "Create new TwiML App"
|
||||
- Name it (e.g., "RouteBox Softphone")
|
||||
- Set Voice URL to: `https://yourdomain.com/api/voice/twiml/outbound`
|
||||
- Set Voice Method to: `POST`
|
||||
- Save and copy the TwiML App SID
|
||||
|
||||
2. **Create an API Key**
|
||||
- Go to https://console.twilio.com/us1/account/keys-credentials/api-keys
|
||||
- Click "Create API key"
|
||||
- Give it a friendly name
|
||||
- Copy both the SID and Secret (you won't be able to see the secret again)
|
||||
|
||||
3. **Add credentials to Settings > Integrations**
|
||||
- Account SID (from main dashboard)
|
||||
- Auth Token (from main dashboard)
|
||||
- Phone Number (your Twilio number)
|
||||
- API Key SID (from step 2)
|
||||
- API Secret (from step 2)
|
||||
- TwiML App SID (from step 1)
|
||||
|
||||
### Option 2: Twilio Media Streams (Alternative - More complex)
|
||||
|
||||
Uses WebSocket to stream audio bidirectionally:
|
||||
- Requires WebSocket server setup
|
||||
- More control over audio processing
|
||||
- Can integrate with OpenAI Realtime API more easily
|
||||
|
||||
## Current Status
|
||||
|
||||
The system works but audio doesn't flow through browser because:
|
||||
1. Calls are made via REST API only
|
||||
2. No Twilio Client SDK integration yet
|
||||
3. TwiML returns simple voice message
|
||||
|
||||
To enable browser audio, you need to:
|
||||
1. Complete the Twilio setup above
|
||||
2. Implement the frontend Twilio Device connection
|
||||
3. Modify TwiML to dial the browser client instead of just the phone number
|
||||
|
||||
## Quick Test (Current Setup)
|
||||
|
||||
1. Save your Account SID, Auth Token, and Phone Number in Settings > Integrations
|
||||
2. Click the phone icon in sidebar
|
||||
3. Enter a phone number and click "Call"
|
||||
4. You should receive a call that says "This is a test call from your softphone"
|
||||
|
||||
The call works, but audio doesn't route through your browser - it's just a regular phone call initiated by the API.
|
||||
@@ -117,6 +117,11 @@ const staticMenuItems = [
|
||||
url: '/setup/roles',
|
||||
icon: Layers,
|
||||
},
|
||||
{
|
||||
title: 'Integrations',
|
||||
url: '/settings/integrations',
|
||||
icon: Settings,
|
||||
},
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
@@ -85,39 +85,39 @@
|
||||
{{ digit }}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- AI Transcript -->
|
||||
<div v-if="softphone.transcript.value.length > 0" class="space-y-2">
|
||||
<h3 class="text-sm font-semibold">Transcript</h3>
|
||||
<div class="max-h-40 overflow-y-auto p-3 rounded-lg border bg-gray-50 space-y-1">
|
||||
<p
|
||||
v-for="(item, index) in softphone.transcript.value.slice(-10)"
|
||||
:key="index"
|
||||
class="text-sm"
|
||||
:class="{ 'text-gray-400': !item.isFinal }"
|
||||
>
|
||||
{{ item.text }}
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- AI Suggestions -->
|
||||
<div v-if="softphone.aiSuggestions.value.length > 0" class="space-y-2">
|
||||
<h3 class="text-sm font-semibold">AI Suggestions</h3>
|
||||
<div class="space-y-2 max-h-32 overflow-y-auto">
|
||||
<div
|
||||
v-for="(suggestion, index) in softphone.aiSuggestions.value.slice(0, 5)"
|
||||
:key="index"
|
||||
class="p-2 rounded-lg border text-sm"
|
||||
:class="{
|
||||
'bg-blue-50 border-blue-200': suggestion.type === 'response',
|
||||
'bg-green-50 border-green-200': suggestion.type === 'action',
|
||||
'bg-purple-50 border-purple-200': suggestion.type === 'insight'
|
||||
}"
|
||||
>
|
||||
<span class="text-xs font-medium uppercase text-gray-600">{{ suggestion.type }}</span>
|
||||
<p class="mt-1">{{ suggestion.text }}</p>
|
||||
<!-- AI Suggestions - Show whenever there are suggestions, not just during active call -->
|
||||
<div v-if="softphone.aiSuggestions.value.length > 0" class="space-y-2">
|
||||
<h3 class="text-sm font-semibold flex items-center gap-2">
|
||||
<span>AI Assistant</span>
|
||||
<span class="px-2 py-0.5 text-xs bg-blue-100 text-blue-700 rounded-full">
|
||||
{{ softphone.aiSuggestions.value.length }}
|
||||
</span>
|
||||
</h3>
|
||||
<div class="space-y-2 max-h-40 overflow-y-auto">
|
||||
<div
|
||||
v-for="(suggestion, index) in softphone.aiSuggestions.value.slice(0, 5)"
|
||||
:key="index"
|
||||
class="p-3 rounded-lg border text-sm transition-all"
|
||||
:class="{
|
||||
'bg-blue-50 border-blue-200 animate-pulse': suggestion.type === 'response' && index === 0,
|
||||
'bg-blue-50 border-blue-200': suggestion.type === 'response' && index !== 0,
|
||||
'bg-green-50 border-green-200 animate-pulse': suggestion.type === 'action' && index === 0,
|
||||
'bg-green-50 border-green-200': suggestion.type === 'action' && index !== 0,
|
||||
'bg-purple-50 border-purple-200 animate-pulse': suggestion.type === 'insight' && index === 0,
|
||||
'bg-purple-50 border-purple-200': suggestion.type === 'insight' && index !== 0
|
||||
}"
|
||||
>
|
||||
<div class="flex items-center gap-2 mb-1">
|
||||
<span class="text-xs font-semibold uppercase" :class="{
|
||||
'text-blue-700': suggestion.type === 'response',
|
||||
'text-green-700': suggestion.type === 'action',
|
||||
'text-purple-700': suggestion.type === 'insight'
|
||||
}">{{ suggestion.type }}</span>
|
||||
<span class="text-xs text-gray-400">just now</span>
|
||||
</div>
|
||||
<p class="leading-relaxed">{{ suggestion.text }}</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -156,6 +156,11 @@
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
<!-- Debug: Test AI Suggestions -->
|
||||
<Button @click="testAiSuggestion" variant="outline" size="sm" class="w-full">
|
||||
🧪 Test AI Suggestion
|
||||
</Button>
|
||||
|
||||
<!-- Recent Calls -->
|
||||
<div v-if="softphone.callHistory.value.length > 0" class="space-y-2">
|
||||
<h3 class="text-sm font-semibold">Recent Calls</h3>
|
||||
@@ -243,6 +248,21 @@ const handleEndCall = async () => {
|
||||
}
|
||||
};
|
||||
|
||||
// Debug: Test AI suggestions display
|
||||
const testAiSuggestion = () => {
|
||||
console.log('🧪 Testing AI suggestion display');
|
||||
console.log('Current suggestions:', softphone.aiSuggestions.value);
|
||||
|
||||
// Add a test suggestion
|
||||
softphone.aiSuggestions.value.unshift({
|
||||
type: 'response',
|
||||
text: '💡 Test suggestion: This is a test AI suggestion to verify UI display'
|
||||
});
|
||||
|
||||
console.log('After test:', softphone.aiSuggestions.value);
|
||||
toast.success('Test suggestion added');
|
||||
};
|
||||
|
||||
const handleDtmf = async (digit: string) => {
|
||||
if (!softphone.currentCall.value) return;
|
||||
|
||||
|
||||
@@ -10,7 +10,8 @@ export const useApi = () => {
|
||||
// In browser, use current hostname but with port 3000 for API
|
||||
const currentHost = window.location.hostname
|
||||
const protocol = window.location.protocol
|
||||
return `${protocol}//${currentHost}:3000`
|
||||
//return `${protocol}//${currentHost}:3000`
|
||||
return `${protocol}//${currentHost}`
|
||||
}
|
||||
// Fallback for SSR
|
||||
return config.public.apiBaseUrl
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { ref, computed, onMounted, onUnmounted } from 'vue';
|
||||
import { ref, computed, onMounted, onUnmounted, shallowRef } from 'vue';
|
||||
import { io, Socket } from 'socket.io-client';
|
||||
import { Device, Call as TwilioCall } from '@twilio/voice-sdk';
|
||||
import { useAuth } from './useAuth';
|
||||
import { toast } from 'vue-sonner';
|
||||
|
||||
@@ -27,6 +28,8 @@ interface AiSuggestion {
|
||||
|
||||
// Module-level shared state for global access
|
||||
const socket = ref<Socket | null>(null);
|
||||
const twilioDevice = shallowRef<Device | null>(null);
|
||||
const twilioCall = shallowRef<TwilioCall | null>(null);
|
||||
const isConnected = ref(false);
|
||||
const isOpen = ref(false);
|
||||
const currentCall = ref<Call | null>(null);
|
||||
@@ -35,6 +38,8 @@ const transcript = ref<CallTranscript[]>([]);
|
||||
const aiSuggestions = ref<AiSuggestion[]>([]);
|
||||
const callHistory = ref<Call[]>([]);
|
||||
const isInitialized = ref(false);
|
||||
const isMuted = ref(false);
|
||||
const volume = ref(100);
|
||||
|
||||
export function useSoftphone() {
|
||||
const auth = useAuth();
|
||||
@@ -55,6 +60,171 @@ export function useSoftphone() {
|
||||
const hasIncomingCall = computed(() => incomingCall.value !== null);
|
||||
const callStatus = computed(() => currentCall.value?.status || 'idle');
|
||||
|
||||
/**
|
||||
* Request microphone permission explicitly
|
||||
*/
|
||||
const requestMicrophonePermission = async () => {
|
||||
try {
|
||||
// Check if mediaDevices is supported
|
||||
if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
|
||||
toast.error('Microphone access requires HTTPS. Please access the app via https:// or use localhost for testing.');
|
||||
console.error('navigator.mediaDevices not available. This typically means the page is not served over HTTPS.');
|
||||
return false;
|
||||
}
|
||||
|
||||
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
|
||||
// Stop the stream immediately, we just wanted the permission
|
||||
stream.getTracks().forEach(track => track.stop());
|
||||
return true;
|
||||
} catch (error: any) {
|
||||
console.error('Microphone permission denied:', error);
|
||||
if (error.name === 'NotAllowedError') {
|
||||
toast.error('Microphone access denied. Please allow microphone access in your browser settings.');
|
||||
} else if (error.name === 'NotFoundError') {
|
||||
toast.error('No microphone found. Please connect a microphone and try again.');
|
||||
} else {
|
||||
toast.error('Microphone access is required for calls. Please ensure you are using HTTPS or localhost.');
|
||||
}
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Initialize Twilio Device
|
||||
*/
|
||||
const initializeTwilioDevice = async () => {
|
||||
try {
|
||||
// First, explicitly request microphone permission
|
||||
const hasPermission = await requestMicrophonePermission();
|
||||
if (!hasPermission) {
|
||||
return;
|
||||
}
|
||||
|
||||
const { api } = useApi();
|
||||
console.log('Requesting Twilio token from /api/voice/token...');
|
||||
const response = await api.get('/voice/token');
|
||||
const token = response.data.token;
|
||||
|
||||
console.log('Token received, creating Device...');
|
||||
|
||||
// Log the token payload to see what identity is being used
|
||||
try {
|
||||
const tokenPayload = JSON.parse(atob(token.split('.')[1]));
|
||||
console.log('Token identity:', tokenPayload.sub);
|
||||
console.log('Token grants:', tokenPayload.grants);
|
||||
} catch (e) {
|
||||
console.log('Could not parse token payload');
|
||||
}
|
||||
|
||||
twilioDevice.value = new Device(token, {
|
||||
logLevel: 1,
|
||||
codecPreferences: ['opus', 'pcmu'],
|
||||
enableImprovedSignalingErrorPrecision: true,
|
||||
edge: 'ashburn',
|
||||
});
|
||||
|
||||
// Device events
|
||||
twilioDevice.value.on('registered', () => {
|
||||
console.log('✓ Twilio Device registered - ready to receive calls');
|
||||
toast.success('Softphone ready');
|
||||
});
|
||||
|
||||
twilioDevice.value.on('unregistered', () => {
|
||||
console.log('⚠ Twilio Device unregistered');
|
||||
});
|
||||
|
||||
twilioDevice.value.on('error', (error) => {
|
||||
console.error('❌ Twilio Device error:', error);
|
||||
toast.error('Device error: ' + error.message);
|
||||
});
|
||||
|
||||
twilioDevice.value.on('incoming', (call: TwilioCall) => {
|
||||
console.log('🔔 Twilio Device INCOMING event received:', call.parameters);
|
||||
console.log('Call parameters:', {
|
||||
CallSid: call.parameters.CallSid,
|
||||
From: call.parameters.From,
|
||||
To: call.parameters.To,
|
||||
});
|
||||
twilioCall.value = call;
|
||||
|
||||
// Update state
|
||||
incomingCall.value = {
|
||||
callSid: call.parameters.CallSid || '',
|
||||
direction: 'inbound',
|
||||
fromNumber: call.parameters.From || '',
|
||||
toNumber: call.parameters.To || '',
|
||||
status: 'ringing',
|
||||
};
|
||||
|
||||
// Open softphone dialog
|
||||
isOpen.value = true;
|
||||
|
||||
// Show notification
|
||||
toast.info(`Incoming call from ${incomingCall.value.fromNumber}`, {
|
||||
duration: 30000,
|
||||
});
|
||||
|
||||
// Setup call handlers
|
||||
setupCallHandlers(call);
|
||||
|
||||
// Play ringtone
|
||||
playRingtone();
|
||||
});
|
||||
|
||||
// Register the device
|
||||
console.log('Registering Twilio Device...');
|
||||
await twilioDevice.value.register();
|
||||
console.log('✓ Twilio Device register() completed');
|
||||
console.log('Device identity:', twilioDevice.value.identity);
|
||||
console.log('Device state:', twilioDevice.value.state);
|
||||
|
||||
} catch (error: any) {
|
||||
console.error('Failed to initialize Twilio Device:', error);
|
||||
toast.error('Failed to initialize voice device: ' + error.message);
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Setup handlers for a Twilio call
|
||||
*/
|
||||
const setupCallHandlers = (call: TwilioCall) => {
|
||||
call.on('accept', () => {
|
||||
console.log('Call accepted');
|
||||
currentCall.value = {
|
||||
callSid: call.parameters.CallSid || '',
|
||||
direction: twilioCall.value === call ? 'inbound' : 'outbound',
|
||||
fromNumber: call.parameters.From || '',
|
||||
toNumber: call.parameters.To || '',
|
||||
status: 'in-progress',
|
||||
startedAt: new Date().toISOString(),
|
||||
};
|
||||
incomingCall.value = null;
|
||||
});
|
||||
|
||||
call.on('disconnect', () => {
|
||||
console.log('Call disconnected');
|
||||
currentCall.value = null;
|
||||
twilioCall.value = null;
|
||||
});
|
||||
|
||||
call.on('cancel', () => {
|
||||
console.log('Call cancelled');
|
||||
incomingCall.value = null;
|
||||
twilioCall.value = null;
|
||||
});
|
||||
|
||||
call.on('reject', () => {
|
||||
console.log('Call rejected');
|
||||
incomingCall.value = null;
|
||||
twilioCall.value = null;
|
||||
});
|
||||
|
||||
call.on('error', (error) => {
|
||||
console.error('Call error:', error);
|
||||
toast.error('Call error: ' + error.message);
|
||||
});
|
||||
};
|
||||
|
||||
/**
|
||||
* Initialize WebSocket connection
|
||||
*/
|
||||
@@ -70,7 +240,7 @@ export function useSoftphone() {
|
||||
if (typeof window !== 'undefined') {
|
||||
const currentHost = window.location.hostname;
|
||||
const protocol = window.location.protocol;
|
||||
return `${protocol}//${currentHost}:3000`;
|
||||
return `${protocol}//${currentHost}`;
|
||||
}
|
||||
return 'http://localhost:3000';
|
||||
};
|
||||
@@ -89,8 +259,12 @@ export function useSoftphone() {
|
||||
|
||||
// Connection events
|
||||
socket.value.on('connect', () => {
|
||||
console.log('Softphone WebSocket connected');
|
||||
console.log('🔌 Softphone WebSocket connected');
|
||||
console.log('📋 Token payload (check userId):', parseJwt(token));
|
||||
isConnected.value = true;
|
||||
|
||||
// Initialize Twilio Device after WebSocket connects
|
||||
initializeTwilioDevice();
|
||||
});
|
||||
|
||||
socket.value.on('disconnect', () => {
|
||||
@@ -115,7 +289,10 @@ export function useSoftphone() {
|
||||
|
||||
// AI events
|
||||
socket.value.on('ai:transcript', handleAiTranscript);
|
||||
socket.value.on('ai:suggestion', handleAiSuggestion);
|
||||
socket.value.on('ai:suggestion', (data: any) => {
|
||||
console.log('🎯 AI Suggestion received:', data.text);
|
||||
handleAiSuggestion(data);
|
||||
});
|
||||
socket.value.on('ai:action', handleAiAction);
|
||||
|
||||
isInitialized.value = true;
|
||||
@@ -151,122 +328,124 @@ export function useSoftphone() {
|
||||
};
|
||||
|
||||
/**
|
||||
* Initiate outbound call
|
||||
* Initiate outbound call using Twilio Device
|
||||
*/
|
||||
const initiateCall = async (toNumber: string) => {
|
||||
if (!socket.value?.connected) {
|
||||
toast.error('Not connected to voice service');
|
||||
if (!twilioDevice.value) {
|
||||
toast.error('Voice device not initialized');
|
||||
return;
|
||||
}
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
socket.value!.emit('call:initiate', { toNumber }, (response: any) => {
|
||||
if (response.success) {
|
||||
resolve(response);
|
||||
} else {
|
||||
reject(new Error(response.error));
|
||||
try {
|
||||
// Make call using Twilio Device
|
||||
const call = await twilioDevice.value.connect({
|
||||
params: {
|
||||
To: toNumber,
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
twilioCall.value = call;
|
||||
setupCallHandlers(call);
|
||||
|
||||
toast.success('Calling ' + toNumber);
|
||||
} catch (error: any) {
|
||||
console.error('Failed to initiate call:', error);
|
||||
toast.error('Failed to initiate call: ' + error.message);
|
||||
throw error;
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Accept incoming call
|
||||
*/
|
||||
const acceptCall = async (callSid: string) => {
|
||||
if (!socket.value?.connected) {
|
||||
toast.error('Not connected to voice service');
|
||||
console.log('📞 Accepting call - callSid:', callSid);
|
||||
console.log('twilioCall.value:', twilioCall.value);
|
||||
|
||||
if (!twilioCall.value) {
|
||||
console.error('❌ No incoming call to accept - twilioCall.value is null');
|
||||
toast.error('No incoming call');
|
||||
return;
|
||||
}
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
socket.value!.emit('call:accept', { callSid }, (response: any) => {
|
||||
if (response.success) {
|
||||
resolve(response);
|
||||
} else {
|
||||
reject(new Error(response.error));
|
||||
}
|
||||
});
|
||||
});
|
||||
try {
|
||||
console.log('Calling twilioCall.value.accept()...');
|
||||
await twilioCall.value.accept();
|
||||
console.log('✓ Call accepted successfully');
|
||||
toast.success('Call accepted');
|
||||
} catch (error: any) {
|
||||
console.error('❌ Failed to accept call:', error);
|
||||
toast.error('Failed to accept call: ' + error.message);
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Reject incoming call
|
||||
*/
|
||||
const rejectCall = async (callSid: string) => {
|
||||
if (!socket.value?.connected) {
|
||||
toast.error('Not connected to voice service');
|
||||
if (!twilioCall.value) {
|
||||
toast.error('No incoming call');
|
||||
return;
|
||||
}
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
socket.value!.emit('call:reject', { callSid }, (response: any) => {
|
||||
if (response.success) {
|
||||
resolve(response);
|
||||
} else {
|
||||
reject(new Error(response.error));
|
||||
}
|
||||
});
|
||||
});
|
||||
try {
|
||||
twilioCall.value.reject();
|
||||
incomingCall.value = null;
|
||||
twilioCall.value = null;
|
||||
toast.info('Call rejected');
|
||||
} catch (error: any) {
|
||||
console.error('Failed to reject call:', error);
|
||||
toast.error('Failed to reject call: ' + error.message);
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* End active call
|
||||
*/
|
||||
const endCall = async (callSid: string) => {
|
||||
if (!socket.value?.connected) {
|
||||
toast.error('Not connected to voice service');
|
||||
if (!twilioCall.value) {
|
||||
toast.error('No active call');
|
||||
return;
|
||||
}
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
socket.value!.emit('call:end', { callSid }, (response: any) => {
|
||||
if (response.success) {
|
||||
resolve(response);
|
||||
} else {
|
||||
reject(new Error(response.error));
|
||||
}
|
||||
});
|
||||
});
|
||||
try {
|
||||
twilioCall.value.disconnect();
|
||||
currentCall.value = null;
|
||||
twilioCall.value = null;
|
||||
toast.info('Call ended');
|
||||
} catch (error: any) {
|
||||
console.error('Failed to end call:', error);
|
||||
toast.error('Failed to end call: ' + error.message);
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Toggle mute
|
||||
*/
|
||||
const toggleMute = () => {
|
||||
if (!twilioCall.value) return;
|
||||
|
||||
isMuted.value = !isMuted.value;
|
||||
twilioCall.value.mute(isMuted.value);
|
||||
};
|
||||
|
||||
/**
|
||||
* Send DTMF tone
|
||||
*/
|
||||
const sendDtmf = async (callSid: string, digit: string) => {
|
||||
if (!socket.value?.connected) {
|
||||
if (!twilioCall.value) {
|
||||
return;
|
||||
}
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
socket.value!.emit('call:dtmf', { callSid, digit }, (response: any) => {
|
||||
if (response.success) {
|
||||
resolve(response);
|
||||
} else {
|
||||
reject(new Error(response.error));
|
||||
}
|
||||
});
|
||||
});
|
||||
twilioCall.value.sendDigits(digit);
|
||||
};
|
||||
|
||||
// Event handlers
|
||||
const handleIncomingCall = (data: Call) => {
|
||||
console.log('Incoming call:', data);
|
||||
incomingCall.value = data;
|
||||
isOpen.value = true;
|
||||
|
||||
toast.info(`Incoming call from ${data.fromNumber}`, {
|
||||
duration: 30000,
|
||||
action: {
|
||||
label: 'Answer',
|
||||
onClick: () => {
|
||||
acceptCall(data.callSid);
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
// Play ringtone
|
||||
playRingtone();
|
||||
// Socket.IO notification that a call is coming
|
||||
// The actual call object will come from Twilio Device SDK's 'incoming' event
|
||||
console.log('Socket.IO call notification:', data);
|
||||
// Don't set incomingCall here - wait for the Device SDK incoming event
|
||||
};
|
||||
|
||||
const handleCallInitiated = (data: any) => {
|
||||
@@ -334,7 +513,6 @@ export function useSoftphone() {
|
||||
};
|
||||
|
||||
const handleAiTranscript = (data: { transcript: string; isFinal: boolean }) => {
|
||||
console.log('AI transcript:', data);
|
||||
transcript.value.push({
|
||||
text: data.transcript,
|
||||
isFinal: data.isFinal,
|
||||
@@ -348,7 +526,6 @@ export function useSoftphone() {
|
||||
};
|
||||
|
||||
const handleAiSuggestion = (data: AiSuggestion) => {
|
||||
console.log('AI suggestion:', data);
|
||||
aiSuggestions.value.unshift(data);
|
||||
|
||||
// Keep only last 10 suggestions
|
||||
@@ -357,6 +534,15 @@ export function useSoftphone() {
|
||||
}
|
||||
};
|
||||
|
||||
// Helper to parse JWT (for debugging)
|
||||
const parseJwt = (token: string) => {
|
||||
try {
|
||||
return JSON.parse(atob(token.split('.')[1]));
|
||||
} catch (e) {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
const handleAiAction = (data: any) => {
|
||||
console.log('AI action:', data);
|
||||
toast.info(`AI: ${data.action}`);
|
||||
@@ -366,12 +552,31 @@ export function useSoftphone() {
|
||||
let ringtoneAudio: HTMLAudioElement | null = null;
|
||||
|
||||
const playRingtone = () => {
|
||||
// Optional: Play a simple beep tone using Web Audio API
|
||||
// This is a nice-to-have enhancement but not required for incoming calls to work
|
||||
try {
|
||||
ringtoneAudio = new Audio('/ringtone.mp3');
|
||||
ringtoneAudio.loop = true;
|
||||
ringtoneAudio.play();
|
||||
const audioContext = new (window.AudioContext || (window as any).webkitAudioContext)();
|
||||
const oscillator = audioContext.createOscillator();
|
||||
const gainNode = audioContext.createGain();
|
||||
|
||||
oscillator.connect(gainNode);
|
||||
gainNode.connect(audioContext.destination);
|
||||
|
||||
// Phone ringtone frequency (440 Hz)
|
||||
oscillator.frequency.value = 440;
|
||||
oscillator.type = 'sine';
|
||||
|
||||
const now = audioContext.currentTime;
|
||||
gainNode.gain.setValueAtTime(0.15, now);
|
||||
gainNode.gain.setValueAtTime(0, now + 0.5);
|
||||
gainNode.gain.setValueAtTime(0.15, now + 1.0);
|
||||
gainNode.gain.setValueAtTime(0, now + 1.5);
|
||||
|
||||
oscillator.start(now);
|
||||
oscillator.stop(now + 2);
|
||||
} catch (error) {
|
||||
console.error('Failed to play ringtone:', error);
|
||||
// Silent fail - incoming call still works without audio
|
||||
console.debug('Audio notification skipped:', error);
|
||||
}
|
||||
};
|
||||
|
||||
@@ -402,20 +607,23 @@ export function useSoftphone() {
|
||||
hasIncomingCall,
|
||||
currentCall,
|
||||
incomingCall,
|
||||
callStatus,
|
||||
transcript,
|
||||
aiSuggestions,
|
||||
callStatus,
|
||||
callHistory,
|
||||
isMuted,
|
||||
volume,
|
||||
|
||||
// Methods
|
||||
// Actions
|
||||
open,
|
||||
close,
|
||||
connect,
|
||||
disconnect,
|
||||
initiateCall,
|
||||
acceptCall,
|
||||
rejectCall,
|
||||
endCall,
|
||||
sendDtmf,
|
||||
toggleMute,
|
||||
connect,
|
||||
disconnect,
|
||||
};
|
||||
}
|
||||
|
||||
928
frontend/package-lock.json
generated
928
frontend/package-lock.json
generated
File diff suppressed because it is too large
Load Diff
@@ -17,6 +17,7 @@
|
||||
"dependencies": {
|
||||
"@internationalized/date": "^3.10.1",
|
||||
"@nuxtjs/tailwindcss": "^6.11.4",
|
||||
"@twilio/voice-sdk": "^2.11.2",
|
||||
"@vueuse/core": "^10.11.1",
|
||||
"class-variance-authority": "^0.7.0",
|
||||
"clsx": "^2.1.0",
|
||||
|
||||
@@ -1,106 +1,135 @@
|
||||
<template>
|
||||
<div class="max-w-4xl mx-auto space-y-6">
|
||||
<div>
|
||||
<h1 class="text-3xl font-bold">Integrations</h1>
|
||||
<p class="text-muted-foreground mt-2">
|
||||
Configure third-party service integrations for your tenant
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<!-- Twilio Configuration -->
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle class="flex items-center gap-2">
|
||||
<Phone class="w-5 h-5" />
|
||||
Twilio Voice
|
||||
</CardTitle>
|
||||
<CardDescription>
|
||||
Configure Twilio for voice calling capabilities
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent class="space-y-4">
|
||||
<div class="space-y-2">
|
||||
<Label for="twilio-account-sid">Account SID</Label>
|
||||
<Input
|
||||
id="twilio-account-sid"
|
||||
v-model="twilioConfig.accountSid"
|
||||
placeholder="ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
||||
/>
|
||||
</div>
|
||||
<div class="space-y-2">
|
||||
<Label for="twilio-auth-token">Auth Token</Label>
|
||||
<Input
|
||||
id="twilio-auth-token"
|
||||
v-model="twilioConfig.authToken"
|
||||
type="password"
|
||||
placeholder="Enter your Twilio auth token"
|
||||
/>
|
||||
</div>
|
||||
<div class="space-y-2">
|
||||
<Label for="twilio-phone-number">Phone Number</Label>
|
||||
<Input
|
||||
id="twilio-phone-number"
|
||||
v-model="twilioConfig.phoneNumber"
|
||||
placeholder="+1234567890"
|
||||
/>
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<!-- OpenAI Configuration -->
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle class="flex items-center gap-2">
|
||||
<Bot class="w-5 h-5" />
|
||||
OpenAI Realtime
|
||||
</CardTitle>
|
||||
<CardDescription>
|
||||
Configure OpenAI for AI-assisted calling features
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent class="space-y-4">
|
||||
<div class="space-y-2">
|
||||
<Label for="openai-api-key">API Key</Label>
|
||||
<Input
|
||||
id="openai-api-key"
|
||||
v-model="openaiConfig.apiKey"
|
||||
type="password"
|
||||
placeholder="sk-..."
|
||||
/>
|
||||
</div>
|
||||
<div class="space-y-2">
|
||||
<Label for="openai-model">Model</Label>
|
||||
<Input
|
||||
id="openai-model"
|
||||
v-model="openaiConfig.model"
|
||||
placeholder="gpt-4o-realtime-preview"
|
||||
/>
|
||||
<p class="text-xs text-muted-foreground">
|
||||
Default: gpt-4o-realtime-preview
|
||||
<NuxtLayout name="default">
|
||||
<main class="container mx-auto px-4 py-8">
|
||||
<div class="flex items-center justify-between mb-8">
|
||||
<div>
|
||||
<h1 class="text-3xl font-bold">Integrations</h1>
|
||||
<p class="text-muted-foreground mt-2">
|
||||
Configure third-party service integrations for your tenant
|
||||
</p>
|
||||
</div>
|
||||
<div class="space-y-2">
|
||||
<Label for="openai-voice">Voice</Label>
|
||||
<Input
|
||||
id="openai-voice"
|
||||
v-model="openaiConfig.voice"
|
||||
placeholder="alloy"
|
||||
/>
|
||||
<p class="text-xs text-muted-foreground">
|
||||
Options: alloy, echo, fable, onyx, nova, shimmer
|
||||
</p>
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
<Button @click="saveConfig" :disabled="saving">
|
||||
<Save class="mr-2 h-4 w-4" />
|
||||
{{ saving ? 'Saving...' : 'Save Configuration' }}
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
<!-- Save Button -->
|
||||
<div class="flex justify-end">
|
||||
<Button @click="saveConfig" :disabled="saving">
|
||||
<Save class="w-4 h-4 mr-2" />
|
||||
{{ saving ? 'Saving...' : 'Save Configuration' }}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
<!-- Services Grid -->
|
||||
<div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
|
||||
<!-- Twilio Configuration -->
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle class="flex items-center gap-2">
|
||||
<Phone class="w-5 h-5" />
|
||||
Twilio Voice
|
||||
</CardTitle>
|
||||
<CardDescription>
|
||||
Configure Twilio for voice calling
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent class="space-y-4">
|
||||
<div class="space-y-2">
|
||||
<Label for="twilio-account-sid">Account SID</Label>
|
||||
<Input
|
||||
id="twilio-account-sid"
|
||||
v-model="twilioConfig.accountSid"
|
||||
placeholder="ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
||||
/>
|
||||
</div>
|
||||
<div class="space-y-2">
|
||||
<Label for="twilio-auth-token">Auth Token</Label>
|
||||
<Input
|
||||
id="twilio-auth-token"
|
||||
v-model="twilioConfig.authToken"
|
||||
type="password"
|
||||
placeholder="Enter your Twilio auth token"
|
||||
/>
|
||||
</div>
|
||||
<div class="space-y-2">
|
||||
<Label for="twilio-phone-number">Phone Number</Label>
|
||||
<Input
|
||||
id="twilio-phone-number"
|
||||
v-model="twilioConfig.phoneNumber"
|
||||
placeholder="+1234567890"
|
||||
/>
|
||||
</div>
|
||||
<div class="space-y-2">
|
||||
<Label for="twilio-api-key">API Key SID (for browser calls)</Label>
|
||||
<Input
|
||||
id="twilio-api-key"
|
||||
v-model="twilioConfig.apiKey"
|
||||
placeholder="SKxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
||||
/>
|
||||
</div>
|
||||
<div class="space-y-2">
|
||||
<Label for="twilio-api-secret">API Secret</Label>
|
||||
<Input
|
||||
id="twilio-api-secret"
|
||||
v-model="twilioConfig.apiSecret"
|
||||
type="password"
|
||||
placeholder="Enter your API Key Secret"
|
||||
/>
|
||||
</div>
|
||||
<div class="space-y-2">
|
||||
<Label for="twilio-twiml-app">TwiML App SID</Label>
|
||||
<Input
|
||||
id="twilio-twiml-app"
|
||||
v-model="twilioConfig.twimlAppSid"
|
||||
placeholder="APxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
||||
/>
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<!-- OpenAI Configuration -->
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle class="flex items-center gap-2">
|
||||
<Bot class="w-5 h-5" />
|
||||
OpenAI Realtime
|
||||
</CardTitle>
|
||||
<CardDescription>
|
||||
Configure OpenAI for AI features
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent class="space-y-4">
|
||||
<div class="space-y-2">
|
||||
<Label for="openai-api-key">API Key</Label>
|
||||
<Input
|
||||
id="openai-api-key"
|
||||
v-model="openaiConfig.apiKey"
|
||||
type="password"
|
||||
placeholder="sk-..."
|
||||
/>
|
||||
</div>
|
||||
<div class="space-y-2">
|
||||
<Label for="openai-model">Model</Label>
|
||||
<Input
|
||||
id="openai-model"
|
||||
v-model="openaiConfig.model"
|
||||
placeholder="gpt-4o-realtime-preview"
|
||||
/>
|
||||
</div>
|
||||
<div class="space-y-2">
|
||||
<Label for="openai-voice">Voice</Label>
|
||||
<select
|
||||
id="openai-voice"
|
||||
v-model="openaiConfig.voice"
|
||||
class="w-full px-3 py-2 border rounded-md bg-background"
|
||||
>
|
||||
<option value="alloy">Alloy</option>
|
||||
<option value="echo">Echo</option>
|
||||
<option value="fable">Fable</option>
|
||||
<option value="onyx">Onyx</option>
|
||||
<option value="nova">Nova</option>
|
||||
<option value="shimmer">Shimmer</option>
|
||||
</select>
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
</main>
|
||||
</NuxtLayout>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
@@ -119,6 +148,9 @@ const twilioConfig = ref({
|
||||
accountSid: '',
|
||||
authToken: '',
|
||||
phoneNumber: '',
|
||||
apiKey: '',
|
||||
apiSecret: '',
|
||||
twimlAppSid: '',
|
||||
});
|
||||
|
||||
const openaiConfig = ref({
|
||||
|
||||
0
infra/.env.api
Normal file
0
infra/.env.api
Normal file
116
validate-softphone.sh
Executable file
116
validate-softphone.sh
Executable file
@@ -0,0 +1,116 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Softphone Incoming Call System Validation Script
|
||||
# This script verifies that all components are properly configured and running
|
||||
|
||||
echo "╔════════════════════════════════════════════════════════════════╗"
|
||||
echo "║ SOFTPHONE INCOMING CALL SYSTEM VALIDATION ║"
|
||||
echo "╚════════════════════════════════════════════════════════════════╝"
|
||||
echo ""
|
||||
|
||||
# Colors for output
|
||||
GREEN='\033[0;32m'
|
||||
RED='\033[0;31m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
check() {
|
||||
local name=$1
|
||||
local command=$2
|
||||
local expected=$3
|
||||
|
||||
if eval "$command" > /dev/null 2>&1; then
|
||||
if [ -z "$expected" ] || eval "$command" | grep -q "$expected"; then
|
||||
echo -e "${GREEN}✓${NC} $name"
|
||||
((PASS++))
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
echo -e "${RED}✗${NC} $name"
|
||||
((FAIL++))
|
||||
return 1
|
||||
}
|
||||
|
||||
echo "🔍 Checking Services..."
|
||||
echo ""
|
||||
|
||||
# Check backend is running
|
||||
check "Backend running on port 3000" "netstat -tuln | grep ':3000'" "3000"
|
||||
|
||||
# Check frontend is running
|
||||
check "Frontend running on port 3001" "netstat -tuln | grep ':3001'" "3001"
|
||||
|
||||
echo ""
|
||||
echo "🔍 Checking Backend Configuration..."
|
||||
echo ""
|
||||
|
||||
# Check backend files exist
|
||||
check "Voice controller exists" "test -f /root/neo/backend/src/voice/voice.controller.ts"
|
||||
check "Voice gateway exists" "test -f /root/neo/backend/src/voice/voice.gateway.ts"
|
||||
|
||||
# Check for inbound TwiML handler
|
||||
check "inboundTwiml handler defined" "grep -q '@Post.*twiml/inbound' /root/neo/backend/src/voice/voice.controller.ts"
|
||||
|
||||
# Check for notifyIncomingCall method
|
||||
check "notifyIncomingCall method exists" "grep -q 'notifyIncomingCall' /root/neo/backend/src/voice/voice.gateway.ts"
|
||||
|
||||
# Check for Socket.IO emit in notifyIncomingCall
|
||||
check "notifyIncomingCall emits call:incoming" "grep -A3 'notifyIncomingCall' /root/neo/backend/src/voice/voice.gateway.ts | grep -q \"call:incoming\""
|
||||
|
||||
echo ""
|
||||
echo "🔍 Checking Frontend Configuration..."
|
||||
echo ""
|
||||
|
||||
# Check frontend files exist
|
||||
check "Softphone composable exists" "test -f /root/neo/frontend/composables/useSoftphone.ts"
|
||||
check "Softphone dialog component exists" "test -f /root/neo/frontend/components/SoftphoneDialog.vue"
|
||||
|
||||
# Check for Socket.IO listener
|
||||
check "call:incoming event listener registered" "grep -q \"'call:incoming'\" /root/neo/frontend/composables/useSoftphone.ts"
|
||||
|
||||
# Check for handler function
|
||||
check "handleIncomingCall function defined" "grep -q 'const handleIncomingCall' /root/neo/frontend/composables/useSoftphone.ts"
|
||||
|
||||
# Check that handler updates incomingCall ref
|
||||
check "Handler updates incomingCall.value" "grep -A5 'const handleIncomingCall' /root/neo/frontend/composables/useSoftphone.ts | grep -q 'incomingCall.value = data'"
|
||||
|
||||
echo ""
|
||||
echo "🔍 Checking End-to-End Flow..."
|
||||
echo ""
|
||||
|
||||
# Check that backend calls notifyIncomingCall in handler
|
||||
check "inboundTwiml calls notifyIncomingCall" "grep -A50 '@Post.*twiml/inbound' /root/neo/backend/src/voice/voice.controller.ts | grep -q 'notifyIncomingCall'"
|
||||
|
||||
# Check TwiML generation includes Dial
|
||||
check "TwiML includes Dial element" "grep -A50 '@Post.*twiml/inbound' /root/neo/backend/src/voice/voice.controller.ts | grep -q '<Dial'"
|
||||
|
||||
# Check TwiML includes Client elements
|
||||
check "TwiML includes Client dial targets" "grep -A50 '@Post.*twiml/inbound' /root/neo/backend/src/voice/voice.controller.ts | grep -q '<Client>'"
|
||||
|
||||
echo ""
|
||||
echo "╔════════════════════════════════════════════════════════════════╗"
|
||||
echo "║ VALIDATION SUMMARY ║"
|
||||
echo "╠════════════════════════════════════════════════════════════════╣"
|
||||
printf "║ %-50s %s ║\n" "Tests Passed" "${GREEN}${PASS}${NC}"
|
||||
printf "║ %-50s %s ║\n" "Tests Failed" "${RED}${FAIL}${NC}"
|
||||
echo "╚════════════════════════════════════════════════════════════════╝"
|
||||
|
||||
if [ $FAIL -eq 0 ]; then
|
||||
echo ""
|
||||
echo -e "${GREEN}✓ All checks passed! System is properly configured.${NC}"
|
||||
echo ""
|
||||
echo "Next Steps:"
|
||||
echo "1. Connect to softphone at http://localhost:3001"
|
||||
echo "2. Open softphone dialog and verify it shows 'Connected' status"
|
||||
echo "3. Make an inbound call to your Twilio number"
|
||||
echo "4. Verify incoming call dialog appears in softphone UI"
|
||||
echo "5. Test accepting/rejecting the call"
|
||||
exit 0
|
||||
else
|
||||
echo ""
|
||||
echo -e "${RED}✗ Some checks failed. Review the configuration.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
Reference in New Issue
Block a user