Skip to content

fix(onboarding): notify user when voice intro falls back due to #233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
maciej-konczal wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix(onboarding): notify user when voice intro falls back due to #233

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f3f05ba
fix(onboarding): notify user when voice intro falls back due to
maciej-konczal May 28, 2026
ef0ee85
Merge remote-tracking branch 'upstream/main' into fix/onboarding-gemi…
maciej-konczal May 28, 2026
90e5415
fix(onboarding): gemini setup updates
maciej-konczal May 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,13 @@
# Required: Anthropic API key for kernel AI features
ANTHROPIC_API_KEY=sk-ant-...

# Optional: Gemini API key for image generation (Nano Banana)
# Optional: Gemini API key. Powers two features:
# 1. The voice-first first-boot onboarding (Gemini Live, "Aoede" voice).
# Without this key the intro silently falls back to a text-only mode
# and shows a notice in the shell.
# 2. Image generation (Nano Banana). The IPC tool returns a "not configured"
# message if invoked without a key.
# Get one at https://aistudio.google.com/apikey
GEMINI_API_KEY=

# Gateway
Expand Down
5 changes: 5 additions & 0 deletions packages/gateway/src/onboarding/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,11 @@ export const GatewayToShellSchema = z.discriminatedUnion("type", [
z.object({ type: z.literal("safe_action_summary"), action: z.record(z.string(), z.unknown()) }),
z.object({ type: z.literal("api_key_result"), valid: z.boolean(), error: z.string().optional() }),
z.object({ type: z.literal("onboarding_already_complete") }),
z.object({
type: z.literal("notice"),
code: z.enum(["gemini_unavailable"]),
message: z.string(),
}),
z.object({
type: z.literal("error"),
code: z.string(),
Expand Down
16 changes: 16 additions & 0 deletions packages/gateway/src/onboarding/ws-handler.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,15 @@ type SendFn = (msg: GatewayToShell) => void;
const ONBOARDING_COMPLETE_FILE = "system/onboarding-complete.json";
const ONBOARDING_STATE_FILE = "system/onboarding-state.json";

const GEMINI_VOICE_UNAVAILABLE_NOTICE =
"Voice intro is unavailable — set GEMINI_API_KEY for the spoken onboarding, or continue in text below.";
const GEMINI_VOICE_CONNECT_FAILED_NOTICE =
"Voice intro could not connect — continue in text below, or try again in a moment.";

function sendGeminiVoiceNotice(send: SendFn, message: string): void {
send({ type: "notice", code: "gemini_unavailable", message });
}

const TAIL_TO_DONE: Record<OnboardingStage, OnboardingStage[]> = {
greeting: ["interview", "extract_profile", "suggest_apps", "done"],
interview: ["extract_profile", "suggest_apps", "done"],
Expand Down Expand Up @@ -226,6 +235,9 @@ export function createOnboardingHandler(deps: OnboardingDeps) {
client.on("disconnected", () => {
if (gemini !== client) return;
if (sm.current !== "done") {
if (audioMode) {
sendGeminiVoiceNotice(send, GEMINI_VOICE_CONNECT_FAILED_NOTICE);
}
send({ type: "mode_change", mode: "text" });
audioMode = false;
}
Expand Down Expand Up @@ -282,12 +294,16 @@ export function createOnboardingHandler(deps: OnboardingDeps) {
console.error("[onboarding] Gemini Live connection FAILED:", err instanceof Error ? err.message : String(err));
if (gemini !== client) return;
gemini = null;
sendGeminiVoiceNotice(send, GEMINI_VOICE_CONNECT_FAILED_NOTICE);
send({ type: "mode_change", mode: "text" });
audioMode = false;
}
} else {
console.log("[onboarding] No voice mode — Gemini Live connection:", hasVoiceConnection ? "set" : "MISSING");
audioMode = false;
if (audioFormat === "pcm16" && !hasVoiceConnection) {
sendGeminiVoiceNotice(send, GEMINI_VOICE_UNAVAILABLE_NOTICE);
}
send({ type: "mode_change", mode: "text" });
}
}
Expand Down
7 changes: 7 additions & 0 deletions shell/src/components/OnboardingScreen.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -451,6 +451,13 @@ export function OnboardingScreen({ onComplete, onOpenManualSetup }: OnboardingSc
</div>
)}

{/* Notice (informational, e.g. voice unavailable) */}
{ob.notice && !ob.error && (
<div className="absolute bottom-4 left-1/2 -translate-x-1/2 px-4 py-2 rounded-lg bg-muted/40 border border-border text-muted-foreground text-xs max-w-md text-center">
{ob.notice}
</div>
)}

{/* Error */}
{ob.error && (
<div className="absolute bottom-4 left-1/2 -translate-x-1/2 px-4 py-2 rounded-lg bg-destructive/10 border border-destructive/20 text-destructive text-xs">
Expand Down
8 changes: 8 additions & 0 deletions shell/src/hooks/useOnboarding.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ export interface OnboardingHook {
transcripts: Transcript[];
suggestedApps: SuggestedApp[];
error: string | null;
notice: string | null;
isVoiceMode: boolean;
alreadyComplete: boolean;
apiKeyResult: { valid: boolean; error?: string } | null;
Expand All @@ -192,6 +193,7 @@ export function useOnboarding(): OnboardingHook {
const [transcripts, setTranscripts] = useState<Transcript[]>([]);
const [suggestedApps, setSuggestedApps] = useState<SuggestedApp[]>([]);
const [error, setError] = useState<string | null>(null);
const [notice, setNotice] = useState<string | null>(null);
const [isVoiceMode, setIsVoiceMode] = useState(false);
const [alreadyComplete, setAlreadyComplete] = useState(false);
const [apiKeyResult, setApiKeyResult] = useState<{ valid: boolean; error?: string } | null>(null);
Expand Down Expand Up @@ -534,6 +536,9 @@ export function useOnboarding(): OnboardingHook {
case "onboarding_already_complete":
setAlreadyComplete(true);
break;
case "notice":
setNotice(msg.message as string);
break;
Comment on lines +539 to +541
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot May 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Stale notice persists across reconnect/restart

notice is set on receipt but never cleared. If handleStart is called a second time (e.g., the user retries with a valid key), the old gemini_unavailable banner will remain visible until a new notice arrives or the component unmounts. error has the same pattern, but a stale "voice unavailable" message while voice is actually working would be misleading. Clearing notice (and error) at the start of a new session would match the intent.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: shell/src/hooks/useOnboarding.ts
Line: 336-338

Comment:
**Stale notice persists across reconnect/restart**

`notice` is set on receipt but never cleared. If `handleStart` is called a second time (e.g., the user retries with a valid key), the old `gemini_unavailable` banner will remain visible until a new `notice` arrives or the component unmounts. `error` has the same pattern, but a stale "voice unavailable" message while voice is actually working would be misleading. Clearing `notice` (and `error`) at the start of a new session would match the intent.

How can I resolve this? If you propose a fix, please make it concise.

case "error":
setError(msg.message as string);
break;
Expand Down Expand Up @@ -573,6 +578,8 @@ export function useOnboarding(): OnboardingHook {

// Public API
const start = useCallback((useVoice: boolean) => {
setNotice(null);
setError(null);
setIsVoiceMode(useVoice);

void connect()
Expand Down Expand Up @@ -672,6 +679,7 @@ export function useOnboarding(): OnboardingHook {
transcripts,
suggestedApps,
error,
notice,
isVoiceMode,
alreadyComplete,
apiKeyResult,
Expand Down
Loading
Loading