-
Notifications
You must be signed in to change notification settings - Fork 153
Add two skills to the genui package #800
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
e1b150f
fe840c6
10265bc
c695204
2bced77
1b140d7
5cae9ea
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
| --- | ||
| name: create-catalog-item | ||
| description: Use this skill when the user asks to create a new CatalogItem, data class, and/or widget class based on a JSON Schema definition in an application that uses Flutter's `genui` package. | ||
| --- | ||
|
|
||
| # Create CatalogItem | ||
|
|
||
| ## Goal | ||
| To correctly implement a GenUI CatalogItem based on a provided json_schema_builder Schema, including its corresponding data class, CatalogItem instance, and Widget class. This ensures the AI model can properly generate and interact with the UI component. | ||
|
|
||
| ## Instructions | ||
| When tasked with creating a CatalogItem from a `Schema`, follow these steps: | ||
|
|
||
| 1. **Create the Data Class**: | ||
| - Name it `_<SchemaName>Data` (e.g., if schema is `myCardSchema`, data class is `_MyCardData`). | ||
| - Add final fields for each property defined in the schema. | ||
| - Create a `factory _<SchemaName>Data.fromJson(Map<String, Object?> json)` method. | ||
| - Use a `try-catch` block to parse the properties and return a new instance. | ||
| - Cast each property from the `json` map to its expected type, e.g., `title: json['title'] as String,` or `action: json['action'] as JsonMap?,`. | ||
| - Throw an `Exception('Invalid JSON for _<SchemaName>Data')` in the `catch` block if an error occurs. | ||
|
|
||
| 2. **Create the CatalogItem Instance**: | ||
| - Name it identical to the schema name but without the "Schema" suffix (e.g., `myCard` for `myCardSchema`). | ||
| - Declare as a `final CatalogItem`. | ||
| - Set `name` to the capitalized version of the name (e.g., `'MyCard'`). | ||
| - Set `dataSchema` to the provided schema. | ||
| - Implement the `widgetBuilder: (itemContext)`: | ||
| - Cast `itemContext.data` to `Map<String, Object?>`. | ||
| - Parse the data using the data class `fromJson` method: `_<SchemaName>Data.fromJson(json)`. | ||
| - Return the corresponding Widget class and pass the required data. | ||
| - If the schema includes an action callback (like `onCompleted`), implement it here. You must parse the action context using `resolveContext` and dispatch an event using `itemContext.dispatchEvent(...)`. | ||
|
|
||
| 3. **Create the Widget Class**: | ||
| - Name it `_<CapitalizedSchemaName>` (e.g., `_MyCard`). | ||
| - Inherit from `StatelessWidget` or `StatefulWidget` depending on state requirements. | ||
| - Add the Data Class as a required property (e.g., `final _<SchemaName>Data data;`). | ||
| - Add any required callback properties (e.g., `final void Function(int) onCompleted;`). | ||
| - Implement the `build` method using Flutter Material components (e.g., Card, Column, Text). Make sure each data field in the data class is displayed, and that actions are represented by buttons or other interactive elements. | ||
|
|
||
| ## Examples | ||
| ### Input Schema | ||
| ```dart | ||
| final basicCardSchema = S.object( | ||
| properties: { | ||
| 'component': S.string(enumValues: ['BasicCard']), | ||
| 'title': S.string(), | ||
| 'description': | ||
| 'action': A2uiSchemas.action(), | ||
| }, | ||
| required: ['title'], | ||
| ); | ||
| ``` | ||
|
|
||
| ### Expected Output | ||
| ```dart | ||
| class _BasicCardData { | ||
| final String title; | ||
| final JsonMap? action; | ||
|
|
||
| _BasicCardData({required this.title, this.action}); | ||
|
|
||
| factory _BasicCardData.fromJson(Map<String, Object?> json) { | ||
| try { | ||
| return _BasicCardData( | ||
| title: json['title'] as String, | ||
| action: json['action'] as JsonMap?, | ||
| ); | ||
| } catch (e) { | ||
| throw Exception('Invalid JSON for _BasicCardData: $e'); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| final basicCard = CatalogItem( | ||
| name: 'BasicCard', | ||
| dataSchema: basicCardSchema, | ||
| widgetBuilder: (itemContext) { | ||
| final json = itemContext.data as Map<String, Object?>; | ||
| final data = _BasicCardData.fromJson(json); | ||
|
|
||
| return _BasicCard( | ||
| data: data, | ||
| onTap: () async { | ||
| final action = data.action; | ||
| if (action == null) return; | ||
| final event = action['event'] as JsonMap?; | ||
| final name = (event?['name'] as String?) ?? ''; | ||
| final JsonMap contextDefinition = | ||
| (event?['context'] as JsonMap?) ?? <String, Object?>{}; | ||
| final JsonMap resolvedContext = await resolveContext( | ||
| itemContext.dataContext, | ||
| contextDefinition, | ||
| ); | ||
| itemContext.dispatchEvent( | ||
| UserActionEvent( | ||
| name: name, | ||
| sourceComponentId: itemContext.id, | ||
| context: resolvedContext, | ||
| ), | ||
| ); | ||
| } | ||
| ); | ||
| }, | ||
| ); | ||
|
|
||
| class _BasicCard extends StatelessWidget { | ||
| final _BasicCardData data; | ||
| final VoidCallback onTap; | ||
|
|
||
| const _BasicCard({super.key, required this.data, required this.onTap}); | ||
|
|
||
| @override | ||
| Widget build(BuildContext context) { | ||
| return Card( | ||
| child: ListTile( | ||
| title: Text(data.title), | ||
| onTap: onTap, | ||
| ), | ||
| ); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Constraints | ||
| - Ensure proper use of `try-catch` blocks and type casting when parsing JSON in `fromJson`. | ||
| - Make sure action resolution accurately fetches variables via `resolveContext` and uses `itemContext.dispatchEvent` when actions are present in the Schema. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,151 @@ | ||
| --- | ||
| name: integrate-genui-firebase | ||
| description: Use this skill when the user asks to integrate the genui package and get a simple conversation going with Firebase AI Logic. | ||
| --- | ||
|
|
||
| # Integrate GenUI with Firebase AI Logic | ||
|
|
||
| ## Goal | ||
| To successfully integrate the `genui` package into a Flutter app and set up a basic conversational agent using Firebase AI Logic. This skill assumes Firebase AI Logic is already set up and working in the project. | ||
|
|
||
| ## Instructions | ||
| When tasked with integrating `genui` and starting a simple conversation, follow these steps: | ||
|
|
||
| 1. **Verify Firebase Setup:** | ||
| Ensure `firebase_core` and `firebase_ai` are available in `pubspec.yaml`. | ||
| Verify that `Firebase.initializeApp` is called in the `main()` function: | ||
| ```dart | ||
| WidgetsFlutterBinding.ensureInitialized(); | ||
| await Firebase.initializeApp(options: DefaultFirebaseOptions.currentPlatform); | ||
| ``` | ||
|
|
||
| 2. **Add GenUI Package:** | ||
| Add `genui` to the `pubspec.yaml` dependencies. | ||
|
|
||
| 3. **Import Required Libraries:** | ||
| Import `genui` and hide `TextPart` so it doesn't conflict with other packages, then import it again with an alias: | ||
| ```dart | ||
| import 'package:genui/genui.dart' hide TextPart; | ||
| import 'package:genui/genui.dart' as genui; | ||
| ``` | ||
|
|
||
| 4. **Configure Basic Logging:** | ||
| At the beginning of the `main()` function, configure GenUI logging: | ||
| ```dart | ||
| configureLogging( | ||
| logCallback: (level, msg) => debugPrint('GenUI $level: $msg'), | ||
| ); | ||
| ``` | ||
|
|
||
| 5. **Create Model and Chat Session:** | ||
| Initialize the generative model and start a chat session. | ||
| ```dart | ||
| final model = FirebaseAI.googleAI().generativeModel( | ||
| model: 'gemini-3-flash-preview', | ||
| ); | ||
| final _chatSession = model.startChat(); | ||
| ``` | ||
|
|
||
| 6. **Identify Target StatefulWidget:** | ||
| **STOP AND ASK THE USER IF UNCLEAR:** This integration requires a `StatefulWidget` to hold the references to GenUI controllers (`SurfaceController`, `A2uiTransportAdapter`, and `Conversation`). Identify which `StatefulWidget` to use in the application. If you are unsure which widget should hold this state, ask the user before proceeding. | ||
|
|
||
| 7. **Wire up GenUI Controllers inside State:** | ||
| Inside your identified `State` class, instantiate `SurfaceController`, `A2uiTransportAdapter`, and `Conversation`: | ||
| ```dart | ||
| final catalog = BasicCatalogItems.asCatalog(); // Optionally inject custom CatalogItems | ||
| final _controller = SurfaceController(catalogs: [catalog]); | ||
| final _transport = A2uiTransportAdapter(onSend: _sendAndReceive); | ||
| final _conversation = Conversation( | ||
| controller: _controller, | ||
| transport: _transport, | ||
| ); | ||
| ``` | ||
|
|
||
| 8. **Implement the `_sendAndReceive` Method:** | ||
| Create a method to take messages from the transport adapter, send them to Firebase, and feed the AI's response back to the transport. | ||
| ```dart | ||
| Future<void> _sendAndReceive(ChatMessage msg) async { | ||
| final buffer = StringBuffer(); | ||
|
|
||
| for (final part in msg.parts) { | ||
| if (part.isUiInteractionPart) { | ||
| buffer.write(part.asUiInteractionPart!.interaction); | ||
| } else if (part is genui.TextPart) { | ||
| buffer.write(part.text); | ||
| } | ||
| } | ||
|
|
||
| if (buffer.isEmpty) return; | ||
|
|
||
| final text = buffer.toString(); | ||
| final response = await _chatSession.sendMessage(Content.text(text)); | ||
|
|
||
| if (response.text?.isNotEmpty ?? false) { | ||
| _transport.addChunk(response.text!); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| 9. **Listen to Conversation Events:** | ||
| Create stubbed-out methods in your State class for each event type, including DartDoc comments explaining their required behavior. Depending on the interface design, new surfaces and text coming from the agent will be handled in different ways. A conversational interface might add everything to a list that's display in a `ListView`, for example, while an interface featuring UI components in specific locations (such as headers, footers, etc.) might rely on specific surface IDs given to the agent in the system instruction to know which surfaces to display in which locations. | ||
| ```dart | ||
| /// Updates state to include the new [surfaceId] so a new `Surface` widget can be built. | ||
| void _onSurfaceAdded(String surfaceId) { | ||
| // TODO: Implement state update to add surfaceId | ||
| } | ||
|
|
||
| /// Updates state to remove the [surfaceId] so its `Surface` widget is no longer built. | ||
| void _onSurfaceRemoved(String surfaceId) { | ||
| // TODO: Implement state update to remove surfaceId | ||
| } | ||
|
|
||
| /// Handles displaying raw text content received from the AI to the user. | ||
| void _onContentReceived(String text) { | ||
| // TODO: Implement displaying the received text | ||
| } | ||
|
|
||
| /// Handles errors that occur during the conversation appropriately. | ||
| void _onError(Object error) { | ||
| // TODO: Implement error handling | ||
| } | ||
| ``` | ||
|
|
||
| Subscribe to `_conversation.events` to track when UI surfaces or chat messages arrive, dispatching them to the appropriate stubbed out methods: | ||
| ```dart | ||
| _conversation.events.listen((event) { | ||
| switch (event) { | ||
| case ConversationSurfaceAdded added: | ||
| _onSurfaceAdded(added.surfaceId); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It might be nice to add a bit more context around how to manage the conversation state here and in step 11. E.g. perhaps this could call a fictional method called Widget buildConversationTurn(MyTurn turn) {
if (turn.surfaceId != null) {
return Surface(turn.surfaceId);
} else {
// build non-UI conversation turn
}
}
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I've added some additional text in step 9 to talk about different kinds of UI patterns (conversational vs. named, positioned surfaces) and what to do when surfaces are added. I'm hoping that this skill can work for both.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Great idea. I have literally never worked with a bound value before, though, so that's something for me to learn. I can either add it to this skill or create a separate "bind to a value" skill. |
||
| case ConversationSurfaceRemoved removed: | ||
| _onSurfaceRemoved(removed.surfaceId); | ||
| case ConversationContentReceived content: | ||
| _onContentReceived(content.text); | ||
| case ConversationError error: | ||
| _onError(error.error); | ||
| default: | ||
| } | ||
| }); | ||
|
redbrogdon marked this conversation as resolved.
|
||
| ``` | ||
|
|
||
| 10. **Initialize System Prompt:** | ||
| Use `PromptBuilder` to give the AI basic instructions, then send it as a system message. | ||
| ```dart | ||
| final promptBuilder = PromptBuilder.chat( | ||
| catalog: catalog, | ||
| instructions: 'You are a helpful assistant. Respond to messages in a chatty way.', | ||
| ); | ||
| _conversation.sendRequest(ChatMessage.system(promptBuilder.systemPrompt)); | ||
| ``` | ||
|
|
||
| 11. **Display Surfaces:** | ||
| In your Flutter `build()` method, use the `Surface` widget wherever you need to render GenUI widgets using the `surfaceIds` you collected in step 9. | ||
| ```dart | ||
| Surface(surfaceContext: _controller.contextFor(surfaceId)); | ||
| ``` | ||
|
|
||
| 12. **Ask User for Input Preferences:** | ||
| **STOP AND ASK THE USER:** Ask the user for clarification on what UI elements should be used for user input. Explain that a `TextField` and `ElevatedButton` are good defaults, but you should not assume they want those exact widgets unless they clarify. | ||
|
|
||
| ## Constraints | ||
| - Do not make assumptions about user input UI elements; see step 12. | ||
| - Make sure to properly clean up GenUI controllers (`_transport.dispose()`, `_controller.dispose()`) inside the widget's `dispose()` method. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be good to have something in here about subscribing to dynamic values using the utilities like BoundString etc. E.g. if you change the title to A2uiSchemas.stringReference, and then use BoundString, similar to this:
https://github.com/flutter/genui/blob/main/packages/genui/lib/src/catalog/basic_catalog_widgets/text.dart