feat(backend): add request ID propagation and structured JSON logging

[Aish-kul16]

Summary

This PR adds request ID propagation and structured JSON logging across backend API requests, audit logging, and background workflow execution.

Changes

• Added request context management using ContextVar

• Added request ID middleware that:

  • Accepts incoming X-Request-ID
  • Generates a request ID when absent
  • Returns the request ID in response headers

• Added structured JSON logging support

• Injected request IDs into log records

• Propagated request IDs into workflow/background task execution

• Added request ID tracking in audit log context

• Added integration tests validating request ID persistence and audit log propagation

Testing

• Verified successful audit log creation
• Verified request ID persistence in audit context
• Verified negative cases (404, unfinished tasks, generation failures)
• Verified request ID propagation from API request into downstream processing

Notes

No operator configuration changes were introduced, so documentation updates were not required.

Closes #219

[utksh1]

This PR currently has merge conflicts (DIRTY / conflicting), so I cannot review or merge it reliably.

Please rebase/update against current main, resolve the conflicts in the backend logging/database/middleware files, and make sure CI stays green. I will re-review after the branch is clean.

[Aish-kul16]

Hi @utksh1 ,

I've addressed the requested changes and pushed the updates. All CI checks are now passing successfully.

Could you please review the PR again ?

Thank you.

[utksh1]

Re-reviewed the latest update. This still overlaps heavily with the request-id/error-payload work in #618/#633 and carries unrelated CI/frontend baseline edits. Please rebase on latest main after #629, scope this PR to structured logging/request-id propagation only, and avoid broad formatting or unrelated test-baseline changes.

[Aish-kul16]

Re-reviewed the latest update. This still overlaps heavily with the request-id/error-payload work in #618/#633 and carries unrelated CI/frontend baseline edits. Please rebase on latest main after #629, scope this PR to structured logging/request-id propagation only, and avoid broad formatting or unrelated test-baseline changes.

Thanks for the review.

Understood. I'll rebase on the latest main after #629 and reduce the PR scope to only the structured logging/request-id propagation changes. I'll also remove the unrelated workflow, frontend test, and baseline updates that were pulled in while resolving conflicts.

I'll update the PR shortly.

[Aish-kul16]

Hi @utksh1

I rebased locally against the latest main and found that most of the original request-id and structured logging changes now appear to already exist through the newer work that landed in main. The remaining differences on my branch are primarily workflow request-id propagation and some conflict-resolution/test changes that I understand should not be included.

Before I rework the branch further, could you clarify whether you'd like me to:

1. Reduce this PR to only the remaining workflow request-id propagation changes, or
2. Close/supersede this PR if the request-id/logging functionality is already covered by the newer merged work?

I want to avoid duplicating functionality that is now present in main.