Skip to content

feat(evaluators): update Luna client to support additional metrics - #259

Open
namrataghadi-galileo wants to merge 2 commits into
mainfrom
feature/SAO-16057-update-luna-client-for-customer-metrics
Open

feat(evaluators): update Luna client to support additional metrics#259
namrataghadi-galileo wants to merge 2 commits into
mainfrom
feature/SAO-16057-update-luna-client-for-customer-metrics

Conversation

@namrataghadi-galileo

Copy link
Copy Markdown
Contributor

Summary

  • Added structured runtime context for Galileo Luna scorer invocation while preserving backward compatibility.
  • Extended Step with optional tools and ground_truth fields.
  • Added a contextual evaluator hook that receives selector-selected data and the complete runtime Step.
  • Updated galileo.luna to dual-write legacy inputs and the expanded structured record.
  • Added reliable available-tool capture for Google ADK and Strands.

Scope

  • User-facing/API changes:
    • Added optional tools and ground_truth parameters to the Python SDK’s evaluate_controls.
    • Added optional tools and ground_truth fields to the generic OSS Step model.
    • Existing SDK calls and Step payloads remain valid without modification.
    • Existing evaluators implementing only evaluate(data) remain compatible.
  • Internal changes:
    • Added evaluate_with_context(data, step) with a default delegate to evaluate(data).
    • Updated the engine to pass the complete runtime Step without changing selector behavior.
    • Updated Luna requests to send legacy inputs.query, inputs.response, inputs.ground_truth, and inputs.tools alongside the structured record.
    • Mapped Step.ground_truth to record.dataset_output.
    • Added normalized tool-definition extraction for Google ADK and Strands.
    • Preserved scorer ID/version/configuration, local threshold/operator evaluation, error metadata, and existing response parsing.
  • Out of scope:
    • Tenant authentication or authorization changes.
    • Project, organization, target, or log-stream resolution.
    • Runners implementation changes.
    • Wizard, Protect, or obsolete API scorer-invoke integration.
    • galileo.llm and galileo.code evaluators.
    • Removal of the legacy query/response representation.

Risk and Rollout

  • Risk level: medium
  • Rollback plan:
    • Revert the AgentControl change and continue using the legacy Runners inputs.query/inputs.response contract.
    • Keep the Runners legacy adapter enabled throughout rollout.
    • Do not enable the new preset runtime path until the additive Runners contract is deployed.
    • Keep tenant-created Luna scorers disabled until trusted authorization support is available.
    • Retain dual-write until older supported Runners versions have been retired.

Testing

  • Added or updated automated tests
  • Ran make check using UV_OFFLINE=1 make check
  • Manually verified legacy and structured Luna request payloads through HTTP contract tests

Checklist

  • Linked issue/spec (if applicable)
  • Updated docs/examples for user-facing changes
  • Included any required follow-up tasks
    • Deploy the additive Runners contract before enabling the structured preset runtime path.
    • Enable tenant-created Luna scorers only after the separate authentication work is complete.
    • Remove legacy dual-write in a future change after older Runners versions leave the supported deployment window.

@codecov

codecov Bot commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant