feat(evaluator): surface skill evaluators (TOOL_CALL level + skill placeholders) - #1
Closed
jariy17 wants to merge 1 commit into
Closed
feat(evaluator): surface skill evaluators (TOOL_CALL level + skill placeholders)#1jariy17 wants to merge 1 commit into
jariy17 wants to merge 1 commit into
Conversation
…aceholders)
- run-eval: map Builtin.SkillSelectionAccuracy / Builtin.SkillInstructionFollowing
to TOOL_CALL so they target skill-invocation spans instead of falling through to
the SESSION default (which scores the wrong spans, silently).
- evaluator TUI: expose {available_skills} / {invoked_skill} / {skill_content}
placeholders for TOOL_CALL custom LLM-judge evaluators, matching the service
prompt-template contract.
- tests for both.
Follow-up (not in this PR): replace the static BUILTIN_EVALUATOR_LEVELS map with a
cached listEvaluators() lookup so future built-ins self-resolve.
Owner
Author
|
Superseded: reopened from aws/agentcore-cli:skill-evaluators-p0 into bug_bash_aug_18th (no fork). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
bug_bash_aug_18th(PR aws#2012). Surfaces the P0 skill evaluators in the CLI. Scope: only the pieces that work today — the two built-in skill evaluators are alreadyACTIVEservice-side (verified). Ground-truthexpectedSkillsis intentionally excluded (no service field exists yet — it would no-op).Changes
src/cli/operations/eval/run-eval.ts): mapBuiltin.SkillSelectionAccuracyandBuiltin.SkillInstructionFollowingtoTOOL_CALL. Without this they fall through to theSESSIONdefault and get scored against the wrong spans — silently. Fixesrun eval/run batch-evaluationtargeting for skill evaluators.src/cli/tui/screens/evaluator/types.ts): expose{available_skills},{invoked_skill},{skill_content}forTOOL_CALLLLM-judge evaluators, matching the service prompt-template contract. Shown in the wizard's instructions step and accepted byvalidateInstructionPlaceholders.Verification
npm run typecheck— cleanresolves Builtin skill evaluators to TOOL_CALL level)Explicitly out of scope
expectedSkillsground-truth (batch + on-demand): net-new across the AgentCore Evaluation model, not merged / not in review. Ship behind a flag once the service adds the field + a GT skill-selection evaluator.Follow-up (not here)
Replace the static
BUILTIN_EVALUATOR_LEVELSmap with a cachedlistEvaluators()lookup so future built-ins self-resolve (the map already omits Harmfulness/Stereotyping/ToolParameterAccuracy/trajectory matchers).