AI Multi-Modal Visual Novel Engine

MirAI School of Technology — AI Builder Track
Capstone Mini-Project: The Multi-Modal Visual Novel

---

Overview

The AI Multi-Modal Visual Novel Engine is a stateful, AI-driven interactive "Choose Your Own Adventure" web application built with Streamlit. It orchestrates LLM narrative generation (via Google Gemini API or free Pollinations AI), real-time scene artwork synthesis (via Pollinations Image API), and text-to-speech audio narration (via gTTS).

Instead of traditional text chat interfaces, the engine parses structured JSON from the AI model to dynamically build its own interactive button choices per scene turn. It includes disk-backed multi-session persistence, allowing players to create multiple story sessions, switch between them seamlessly, and continue their adventures across browser refreshes.

---

Demo video

demo.mp4

---

Key Capabilities & Architecture

• Dual Provider Support: Supports Google Gemini API (gemini-2.5-flash, gemini-1.5-flash) or keyless Pollinations Free AI out of the box.
• Multi-Session Storage & Disk Persistence: Automatically snapshots active sessions into a local .saved_sessions.json file with base64-encoded visual artwork. Resume any adventure at any time.
• Single-Screen Visual Novel Layout: Optimized 1:1 square artwork (768x768) and side-by-side narrative pane designed for zero-scroll visual immersion.
• Structured JSON Engine: System prompts enforce strict JSON responses containing narrative text, visual prompts, and dynamic action options.
• Multi-Modal Narration: Automatically synthesizes MP3 voice narration for every scene using gTTS and renders playback via st.audio().
• Fault-Tolerant Fallbacks: Built-in 3-stage prompt retry mechanism and st.toast() notifications ensure uninterrupted gameplay even during API latency spikes.

---

Feature Verification & Audit Matrix

All core requirements from Phases 1 through 5 specified in the Capstone rubric are fully implemented, verified, and integrated into app.py.

Phase	Requirement	Implementation Details	Code Reference (app.py)	Status
Phase 1: Director's Cut	Cache AI Client	Uses @st.cache_resource on get_ai_client() to prevent client re-initialization on Streamlit reruns.	Lines 75–86 (get_ai_client)	Verified
Phase 1: Director's Cut	Sidebar Settings	"Story Settings" with Gemini key validation, model selection, Genre, and Art Style selectors.	Lines 323–360 (render_sidebar)	Verified
Phase 1: Director's Cut	Session State Initialization	Initializes messages, history, game_started, turn_count, saved_sessions, and current_session_id.	Lines 142–157 (init_session_state)	Verified
Phase 2: Structured JSON Engine	System Prompt Engineering	Prompt enforces strict output schema (no markdown fences, strictly single JSON object).	Lines 49–68 (SYSTEM_PROMPT_TEMPLATE)	Verified
Phase 2: Structured JSON Engine	JSON Schema (3 Keys)	Requires story_text (narrative), image_prompt (visual prompt with art style), and options (2–3 choice strings).	Lines 57–62, 175–196 (parse_story_json)	Verified
Phase 2: Structured JSON Engine	Python json Parsing	Uses import json with defensive parsing (json.loads) and regex fallback cleaning to prevent invalid JSON crashes.	Lines 160–196 (parse_story_json)	Verified
Phase 3: Dynamic UI Generation	No st.chat_input	Replaces standard text inputs with dynamic visual novel decision buttons.	Lines 445–460 (render_history)	Verified
Phase 3: Dynamic UI Generation	Dynamic st.button() Loop	Iterates over the options array in the latest parsed response to generate interactive choice buttons via st.columns().	Lines 445–460 (render_history)	Verified
Phase 3: Dynamic UI Generation	Stateful Action Handlers	Button clicks update st.session_state.pending_action, feeding chosen actions back to the LLM turn context.	Lines 495–515 (main)	Verified
Phase 4: Multi-Media & TTS	Pollinations Visual Engine	Constructs prompt URL, fetches illustration via requests.get(), and displays image via st.image().	Lines 250–276 (generate_scene_image)	Verified
Phase 4: Multi-Media & TTS	Audio TTS Narration	Uses gTTS library to synthesize story_text into MP3 audio in a temporary file.	Lines 279–289 (generate_narration_audio)	Verified
Phase 4: Multi-Media & TTS	st.audio() Playback	Plays synthesized voice narration directly inside the browser for every scene.	Lines 435–442 (render_history)	Verified
Phase 4: Multi-Media & TTS	State & Disk Persistence	Saves images, audio, and chat logs into st.session_state.history and .saved_sessions.json so previous scenes stay visible and persist across refreshes.	Lines 90–140, 475–515	Verified
Phase 5: Graceful Failures	Network & API Error Handling	All external API calls (LLM API, Pollinations, gTTS) wrapped in Python try...except blocks.	Lines 200–240, 250–276, 279–289	Verified
Phase 5: Graceful Failures	Toast Notifications	Displays non-intrusive Streamlit toasts (e.g. st.toast("Image server is busy, skipping visual...")) instead of crashing with Traceback errors.	Lines 242, 275, 288	Verified

---

Installation & Setup Guide

1. Install Dependencies

Ensure Python 3.10+ is installed on your system. Install all required dependencies:

pip install -r requirements.txt

Required Libraries:

• streamlit>=1.38: Web application framework & state management
• openai>=1.0.0: LLM API client for Gemini and Pollinations OpenAI endpoints
• requests>=2.31: HTTP client for fetching scene illustrations from Pollinations API
• gTTS>=2.5: Google Text-to-Speech audio synthesis engine

2. Configure API Key (Optional for Gemini)

You can run the engine in two modes:

1. Pollinations Free AI (Keyless): Default mode. Instant play with zero API keys required.
2. Google Gemini API: Enter a Gemini API key from Google AI Studio.

Set your key via Environment Variable:

export GEMINI_API_KEY="AIzaSy..."

Or enter it directly into the Gemini API Key field in the Streamlit sidebar at runtime.

3. Launch the Application

Run the Streamlit app:

streamlit run app.py

Open your browser at http://localhost:8501.

---

Video Demonstration Workflow

Follow this sequence when recording your screen capture submission:

1. Sidebar Configuration (0:00–0:10)

   • Show the Story Settings sidebar.
   • Choose a Story Genre (e.g., Cyberpunk Noir) and an Art Style (e.g., Studio Ghibli Watercolor).
   • Enter a Gemini API key or use Pollinations Free AI, then click Begin Your Adventure.

2. First Chapter Render & Voice Narration (0:10–0:25)

   • Show the rendered 1:1 scene illustration, narrative text block, and audio player.
   • Press play on st.audio() to demonstrate gTTS voice narration.

3. Dynamic Choice Progression (0:25–0:45)

   • Highlight the dynamic option buttons generated from the AI's JSON output.
   • Click one of the choices.
   • Show Chapter 2 loading with a new image, updated narrative text, and new decision choices.

4. Multi-Session & Refresh Persistence (0:45–1:00)

   • Click + New Story Session in the sidebar to start another adventure in a different genre.
   • Open Saved Story Sessions and click Load & Continue to instantly resume your first adventure.
   • Refresh the browser to prove all saved sessions remain intact from disk storage.