Add semantic/episodic/procedural memory layering to managed-memory skill

[jennsun]

Summary

Adds an optional layered-memory convention on top of the flat managed-memory store, so an agent can organize what it remembers into semantic / episodic / procedural layers.

The base skill stores entries at flat /memories/... paths with no notion of memory type. This PR adds a convention (prompt + path prefixes — no new tools, grants, or code) that has the agent distill each memory into one of three kinds and store it under a matching /memories/<layer>/ prefix.

What changed

• SKILL.md — new "Step 6 (optional) — Layered memory" section:
  • Layer table mapping semantic → /memories/semantic/..., procedural → /memories/procedural/..., episodic → Conversations API + /memories/episodic/... (every path still starts /memories/ per the API).
  • Episodic via the Conversations API — DatabricksOpenAI → conversations.create(...) → responses.create(conversation=...), reusing the same resolve_scope() scope as the five tools. Honestly scoped to the Supervisor API path (not the in-process Agents SDK / LangGraph loop), with guidance for the loop case.
  • Pointers added from Step 5 and the Notes; new trigger keywords in frontmatter.
• memory-layers.md — new companion file: layer definitions, a "decide the layer before you save" procedure, path-prefix examples, and a drop-in layered MEMORY_INSTRUCTIONS.
• Step 2 fix (USE_CATALOG / USE_SCHEMA): grant the deployed app SP USE_CATALOG on the catalog and USE_SCHEMA on the schema, plus a troubleshooting row. READ/WRITE_MEMORY_STORE alone is insufficient — UC needs the parent traversal privileges or every entries call fails with User does not have USE CATALOG. Found during end-to-end testing below.
• Synced to all conversational templates via sync-skills.py.

End-to-end testing

Validated the layering against a live Databricks app (LangGraph base template + the five memory tools + the layered MEMORY_INSTRUCTIONS from memory-layers.md, model databricks-claude-sonnet-4), on a staging workspace.

Setup: created a new UC memory store jenny.default.layered_memory_test and deployed a new app agent-memory-layers.

I gave the agent facts of each kind and confirmed via direct REST reads that they landed under the right path prefixes:

Layer	Input	Stored path (verified via REST)
Semantic	"I prefer TypeScript, 2-space indent" / "commit msgs imperative, 50-char"	/memories/semantic/coding-preferences.md · /memories/semantic/commit-message-style.md
Procedural	PR-review steps / deploy checklist	/memories/procedural/pr-review-process.md · /memories/procedural/deploy-checklist.md
Episodic	"on 2026-06-30 we standardized on pnpm" / "on 2026-06-28 migrated staging DB to Lakebase"	/memories/episodic/pnpm-standardization-decision.md · /memories/episodic/staging-database-migration.md

Results:

• ✅ Layer classification — the agent distilled each fact into the correct layer and prefix, unprompted beyond the system instructions.
• ✅ Recall — in fresh conversations the agent ran list_memories → get_memory and answered correctly from each layer.
• ✅ Per-user isolation — a different user scope saw none of the stored memories.
• ✅ Local + deployed — verified both via start-app --no-ui (scope faked with X-Forwarded-User) and the deployed app (scope resolved from the OBO token to the real user id).

The deployed app initially failed with User does not have USE CATALOG on Catalog 'jenny'; granting the SP USE_CATALOG + USE_SCHEMA fixed it immediately — which is the Step 2 change included here.

🤖 Generated with Claude Code

Episodic memory via a store-backed conversation (Beta) — verified working

The first draft of this section was wrong (responses.create(conversation=<id>) is rejected by the staging gateway with 400 "conversation: Extra inputs are not permitted"). Corrected to the integration that actually works, verified end-to-end on staging (databricks-openai 0.17.0, openai-agents 0.17.7):

• Back the OpenAI Agents SDK session with the memory store instead of Lakebase — create a conversation bound to {memory_store, scope}, wrap it in OpenAIConversationsSession, pass it as session= to Runner.run.
• The session reads/writes turns via conversations.items.list/create (supported), using one AsyncDatabricksOpenAI(use_ai_gateway=True) client + set_default_openai_api("responses").
• This replaces the AsyncDatabricksSession (Lakebase) block in agent-openai-advanced/agent_server/agent.py.

Test: stated a codeword in turn 1; recalled correctly in turn 2, and again from a fresh session object with the same conversation id (proves cross-request persistence in the store). Still flagged Beta since the Conversations APIs are beta.

[jennsun]

adding this from test run