Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
daea7ef
Prepare Agent Learning Kit v1 release
nik13 Jun 11, 2026
4cdf128
Update PR #49 to the current v1 kit + unified benchmark harness
nik13 Jun 23, 2026
19f379c
docs: bench harness docs + executable twins + cookbooks (release-chec…
nik13 Jun 23, 2026
d267606
docs: fix accuracy faults in bench docs (adversarial sweep)
nik13 Jun 23, 2026
f0fd0e8
refactor!: move public SDK import namespace agent_learning -> fi.alk
nik13 Jun 24, 2026
ee69932
feat(alk): platform-generated scenarios + canonical voice prompts + L…
azain-commits Jul 28, 2026
67033fa
feat(alk): wire scenario generation into CLI + Studio downloader upgr…
azain-commits Jul 28, 2026
c063404
feat(sim): stage-0/5 protocol layer, evidence sources, matrix runner,…
azain-commits Jul 29, 2026
e16933a
fix(sim): repair LiveKit SIP transport, wire Vapi originator, tighten…
azain-commits Jul 29, 2026
3cf8c8f
feat(simulate): add direct voice SDK workflow
azain-commits Jul 30, 2026
963a399
feat(simulate): separate voice target from FutureAGI LiveKit runtime …
azain-commits Jul 30, 2026
35d6525
fix(studio): omit scenario-only agent flag
azain-commits Jul 30, 2026
d2b1420
feat(simulate): complete provider acceptance flows
azain-commits Jul 31, 2026
b53e4a7
feat(simulate): pool-aware LiveKit engine + provider acceptance harde…
azain-commits Aug 4, 2026
1999a1e
fix issues on livekit engine, add support for posting simulate data t…
azain-commits Aug 5, 2026
cf3f3eb
feat(simulate): real platform submission in FutureAGIResultSink
azain-commits Aug 5, 2026
6484b2b
feat(simulate): submit target-agent token usage + cost to platform
azain-commits Aug 5, 2026
c239ace
fix(evals): exclude non-applicable metrics from agent-report aggregate
azain-commits Aug 6, 2026
e0d9e2e
feat(evals): live litellm/Vertex provider for eval + optimize-eval su…
azain-commits Aug 6, 2026
ed86c24
fix(opt): configurable task_model, drop hardcoded gpt-4o-mini/gpt-5-mini
azain-commits Aug 6, 2026
01f6904
feat(evals,opt): wire platform evals + all optimizers into agent-learn
azain-commits Aug 6, 2026
bcd1a76
fix(simulate): tool-call fidelity + per-turn latency in chat runs
azain-commits Aug 10, 2026
c695347
refactor(simulate): gym-model runtime — adapter registries + world_ki…
azain-commits Aug 10, 2026
d1208ce
feat(simulate): SDK hosted-runner package (child entrypoint + job + t…
azain-commits Aug 10, 2026
e573c93
feat(simulate): per-persona TTS voice selection
azain-commits Aug 11, 2026
3db32eb
fix(simulate): send internal Bearer secret alongside FI keys on submit
azain-commits Aug 11, 2026
ee26069
fix(simulate): enable preemptive generation to cut voice turn latency
azain-commits Aug 11, 2026
60a924c
feat(voice): run a scenario's cases concurrently, bounded by agent ca…
azain-commits Aug 12, 2026
3f1413d
feat(simulate): stream hosted voice cases to the platform as each fin…
azain-commits Aug 12, 2026
c95629f
fix(voice): run sims to natural end + stop transcript role reversal
azain-commits Aug 12, 2026
d13f8c2
fix(livekit): send empty dispatch metadata to external target agents …
azain-commits Aug 12, 2026
b93f2c9
fix(livekit): capture target-agent turns so webrtc sims run a real co…
azain-commits Aug 12, 2026
1fc2a40
fix(simulate): drive agent_first conversations from the target greeting
azain-commits Aug 13, 2026
0f4b5de
Revert "fix(simulate): drive agent_first conversations from the targe…
azain-commits Aug 13, 2026
fa87079
fix(simulate): capture the target's opening greeting in agent_first
azain-commits Aug 13, 2026
7f080e6
feat(simulate): stereo voice recording (customer left / assistant right)
azain-commits Aug 13, 2026
5de580c
feat(simulate): streaming Gemini TTS for the simulator voice (multili…
azain-commits Aug 13, 2026
bfc6caf
fix(simulate): capture final target turn, truncate interrupted turns,…
azain-commits Aug 13, 2026
34711b7
fix(simulate): commit target turns directly so the trailing closing i…
azain-commits Aug 13, 2026
0d75d91
fix(simulate): capture target closing via independent stream, not the…
azain-commits Aug 13, 2026
82f23a8
fix(simulate): also wait for the simulator's own final turn before sn…
azain-commits Aug 13, 2026
f429e1b
fix(simulate): delete the room on conversation end so the call actual…
azain-commits Aug 13, 2026
ecd0153
feat(simulate): per-case ONGOING status for hosted sims
azain-commits Aug 15, 2026
260e366
fix(simulate): align SDK voice results and recordings
azain-commits Aug 17, 2026
32101a1
fix(results): allow internal service authentication
azain-commits Aug 18, 2026
74f1c28
docs(simulate): add Spec + Runner quickstart, drop "gym" branding
azain-commits Aug 18, 2026
df2d379
fix(simulate): repair the livekit engine test suite + gate it in CI
azain-commits Aug 18, 2026
c719ffd
fix(simulate): make fi.simulate importable without the livekit extra
azain-commits Aug 18, 2026
a19d2cf
Merge pull request #55 from future-agi/feat/platform-scenarios-liveki…
azain-commits Aug 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
68 changes: 19 additions & 49 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
@@ -1,65 +1,35 @@
name: Bug report
description: Something is broken in a Python or TypeScript SDK package.
labels: ["bug", "needs-triage"]
description: Report a reproducible problem in Agent Learning Kit.
title: "[Bug]: "
labels: ["bug"]
body:
- type: markdown
attributes:
value: |
Before filing: check [open issues](https://github.com/future-agi/ai-evaluation/issues) and [Discussions](https://github.com/orgs/future-agi/discussions) — your question may already be answered.

- type: dropdown
id: package
attributes:
label: Affected package
options:
- ai-evaluation (Python)
- "@future-agi/ai-evaluation (TypeScript)"
- Not sure
validations:
required: true

- type: input
id: version
- type: textarea
id: summary
attributes:
label: Package version
placeholder: "e.g. ai-evaluation==1.1.0 or @future-agi/ai-evaluation@1.0.0"
label: Summary
description: What failed?
validations:
required: true

- type: textarea
id: description
id: reproduce
attributes:
label: What happened?
description: A clear description of the bug. What did you expect, and what did you get instead?
label: Reproduction
description: Commands, manifest, code, or example needed to reproduce.
placeholder: |
uv run ...
agent-learn ...
validations:
required: true

- type: textarea
id: repro
id: expected
attributes:
label: Minimal reproduction
description: The shortest self-contained code that shows the problem. Remove any real API keys.
render: python
label: Expected Behavior
validations:
required: true

- type: textarea
id: error
attributes:
label: Error output / stack trace
render: shell

- type: textarea
id: env
id: environment
attributes:
label: Environment
description: Python version, Node version, OS, gateway version or cloud vs self-hosted.
placeholder: |
Python 3.11.9 / Node 20.12.0
macOS 14.4 / Ubuntu 22.04
Gateway: Future AGI Cloud (gateway.futureagi.com) / self-hosted v0.9.1

- type: textarea
id: extra
attributes:
label: Anything else?
description: Python, Node, OS, package version, and relevant extras.
validations:
required: false
15 changes: 0 additions & 15 deletions .github/ISSUE_TEMPLATE/config.yml

This file was deleted.

63 changes: 23 additions & 40 deletions .github/ISSUE_TEMPLATE/feature_request.yml
Original file line number Diff line number Diff line change
@@ -1,58 +1,41 @@
name: Feature request
description: Propose a new metric, guardrail, or capability for ai-evaluation.
labels: ["enhancement", "needs-triage"]
description: Propose a new workflow, adapter, evaluator, optimizer, or release gate.
title: "[Feature]: "
labels: ["enhancement"]
body:
- type: markdown
attributes:
value: |
For big proposals (new framework integrations, API surface changes), consider opening a [Discussion](https://github.com/orgs/future-agi/discussions) first — it's easier to iterate on design before a PR exists.

- type: dropdown
id: package
attributes:
label: Which package(s) does this touch?
multiple: true
options:
- ai-evaluation (Python)
- "@future-agi/ai-evaluation (TypeScript)"
- Not sure
validations:
required: true

- type: textarea
id: problem
attributes:
label: What problem does this solve?
description: What are you trying to do, and what's getting in your way?
label: Problem
description: What developer workflow should become easier or more robust?
validations:
required: true

- type: textarea
id: solution
id: proposal
attributes:
label: Proposed solution
description: What should the SDK do? A rough API sketch is worth a thousand words.
label: Proposal
description: Describe the requested API, CLI, example, or gate.
validations:
required: true

- type: textarea
id: alternatives
attributes:
label: Alternatives considered
description: What have you tried? Why doesn't it work well enough?

- type: dropdown
id: contribution
id: surface
attributes:
label: Are you willing to open a PR?
label: Surface
options:
- "Yes — I'd like to implement this"
- "Maybe — I could help with parts"
- "No — hoping someone else picks it up"
- simulate
- evals
- optimize
- redteam
- suite
- TypeScript SDK
- release/readiness
- documentation
validations:
required: true

- type: textarea
id: extra
id: proof
attributes:
label: Anything else?
label: Suggested Proof
description: What test, cookbook, or release gate should prove this works?
validations:
required: false
35 changes: 13 additions & 22 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,19 @@
## What does this PR do?
## Summary

<!-- One paragraph. What problem does it solve, or what capability does it add? -->
Describe the user-facing change and why it is needed.

## Why?
## Verification

<!-- What was broken or missing? Link to any related issue: "Fixes #123" or "Closes #456" -->

## How was it tested?

<!-- Which tests cover this change? Did you add new ones? Did you run the integration tests? -->

- [ ] Unit tests added / updated (`pytest` or `vitest`)
- [ ] Integration tests pass (or N/A — no gateway behavior changed)
- [ ] `ruff check` / `mypy` / `npm run typecheck` / `npm run lint` all pass
- [ ] Public types, env vars, or SDK behavior changes are documented in the relevant README
- [ ] `uv run ruff check .`
- [ ] Focused pytest target:
- [ ] Full `uv run pytest -q` when touching shared runtime, CLI, SDK, or release gates
- [ ] TypeScript build/test when touching `typescript/`
- [ ] `agent-learn release-proof` when touching release-candidate behavior

## Checklist

- [ ] Branch is off `main`
- [ ] Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/) (`feat:`, `fix:`, `docs:`, `chore:` …)
- [ ] No TODOs or commented-out code left in
- [ ] No real API keys or secrets in the diff
- [ ] If prose was added or changed: checked against [`docs/VOCABULARY.md`](../docs/VOCABULARY.md)

## Notes for reviewers

<!-- Anything that needs extra attention, or context that isn't obvious from the diff. -->
- [ ] Public API names stay under `agent_learning.*`, `agent-learn`, or `@future-agi/agent-learning-kit`
- [ ] New user-facing workflow has docs or an example
- [ ] New release claim has executable gate coverage
- [ ] No hosted-service dependency was added to release-gated examples
- [ ] Security-sensitive output is redacted or justified
Binary file removed .github/assets/Logo.png
Binary file not shown.
Binary file removed .github/assets/logo-banner-dark.png
Binary file not shown.
Binary file removed .github/assets/logo-banner.png
Binary file not shown.
143 changes: 0 additions & 143 deletions .github/workflows/dev-to-main.yml

This file was deleted.

Loading
Loading