Add callback for token usage

[saisakul]

Context

We need to provide token usage to Datadog so that it can calculate the LLM cost.

What

• Add a callback to add annotation to the LLM span
• Change from auto instrument to manual instrument. Need to remove botocore from auto otherwise it will overwrite the token usage that we annotated.
• Add tests for new callback function

Have you written unit tests?

• Yes
• [] No (add why you have not)

tested locally and in dev

Are there any specific instructions on how to test this change?

• Yes (if so provide more detail)
• No

run some prompts in dev and then check that DataDog traces calculate token costs

Relevant links

[github-actions]

Your PR has commits that are missing the Signed-off-by trailer. This is likely due to the pre-commit hook not being configured on your local machine. The usual fix for this issue is to run pre-commit install --install-hooks --overwrite -t commit-msg -t pre-commit, however for more detailed help in setting up the pre-commit hooks, follow the instructions at https://github.com/uktrade/github-standards/blob/main/README.md#usage

[github-actions]

Dependency Review

✅ No vulnerabilities or OpenSSF Scorecard issues found.

Scanned Files

None

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.04%. Comparing base (635ec71) to head (1a55f0f).

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #999      +/-   ##
==========================================
- Coverage   82.06%   82.04%   -0.02%     
==========================================
  Files         198      199       +1     
  Lines        8208     8239      +31     
==========================================
+ Hits         6736     6760      +24     
- Misses       1472     1479       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hwixley]

NIT: may be worth checking list is non-empty before checking first element, ie.

if respoonse.generations and response.generations[0]