Voice-Powered AI Storytelling, Built & Deployed
A real-time interactive narration app where users speak to an AI storyteller that crafts personalized audio adventures — complete with dual engine modes, live transcription, and full session analytics.
The Vision
Speak your story into existence
The Infinite Story Engine is an interactive platform that lets users create personalized audio adventures using just their voice. They needed an MVP to demonstrate real-time voice-to-AI narration in the browser.
The client envisioned a platform where anyone could step into a story just by speaking. No typing, no menus — just a conversation with an AI narrator who listens, adapts, and brings the adventure to life in real time.
Building a real-time voice AI app is complex. It requires low-latency WebRTC connections, speech recognition, language model orchestration, text-to-speech synthesis, session management, and mobile compatibility. They needed a team that could architect, build, and deploy the full stack.
Your voice brings the story to life
How It Works
The real-time story loop
User Interaction Flow:
- 01SpeakUser speaks directly to the AI narrator via browser microphone.
- 02NarrateThe AI processes the voice intent and generates the next chapter of the story.
- 03ChooseThe user reacts, makes a choice, or asks a question about their surroundings.
- 04ContinueThe story evolves infinitely based entirely on conversational input.
Live session — real-time transcript, voice controls, and AI narration
Under The Hood
How we built it
Two Engine Modes
We built two distinct engine modes so the client could test different approaches with real users and compare quality versus latency.
Single voice-to-voice AI model via xAI's Grok Realtime API. Ultra-low latency with a natural conversational feel. One model handles listening, thinking, and speaking.
Three specialized models working in sequence: Deepgram Nova-3 for speech recognition, Claude Sonnet 3.5 for story generation, and ElevenLabs for cinematic narration.
Full-Stack Implementation
Real-Time Voice via LiveKit
WebRTC-powered voice connection between browser and AI agent. Handles audio streaming, connection state, and mobile audio compatibility.
Next.js Frontend
TypeScript and Tailwind CSS. Responsive UI with live transcript, session controls, and engine mode selection.
Deployed on DigitalOcean
Dockerized app on the client's own VPS. User authentication via Supabase. Nginx reverse proxy, SSL, connected to custom domain.
The Results
What this delivered
2
Engine Modes
Grok Voice and HD Pipeline for real-world A/B testing
$0.06
Per Minute
Both modes optimized to ~$0.06/min with prompt caching
Live
MVP Deployed
Hosted on client's own server, domain, and infrastructure
"Sage was great. He understood the project and worked with me on my lack of knowledge on certain issues. He delivered my project on time and worked with me to fine tune little details. I'm excited to work with him and his team again in the future."
Have a voice AI or
real-time app idea?
We build and deploy AI-powered applications end-to-end. From LiveKit voice agents to full-stack MVPs, let's talk about what we can build for you.