Skip to content

feat(snapshot): add artifact dimension to OSS upload observability - #168

Open
huajq wants to merge 1 commit into
kvcache-ai:mainfrom
huajq:feat/snapshot-upload-artifact-metrics
Open

feat(snapshot): add artifact dimension to OSS upload observability#168
huajq wants to merge 1 commit into
kvcache-ai:mainfrom
huajq:feat/snapshot-upload-artifact-metrics

Conversation

@huajq

@huajq huajq commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

What

  • Snapshot OSS upload metrics (agentenv_snapshot_oss_upload_bytes_total,
    and the put_file/put_bytes series of
    agentenv_snapshot_oss_operation_duration_seconds) gain an artifact
    label: rootfs_layer / attached_drive_layer / memory_layer /
    vm_state / manifest / record / alias.
  • Each completed put_file logs key, artifact, size_bytes,
    elapsed_ms at info level.

Why

Upload metrics could not distinguish memory layers from disk layers, and
canceled uploads left no per-artifact trace. Slow or canceled snapshot
publishes can now be attributed to a specific artifact.

Related issue

None.

Scope and non-goals

In scope: the OSS repository upload path and the shared MetricGuard.
Non-goals: no timeout/retry behavior changes; POSIX backend, ACR/regctl
path, and the download/resolver path are untouched.

Design and behavior changes

  • New MetricGuard::operation_artifact label variant; guards dropped
    mid-flight still record status="canceled", now with the artifact.
  • New OssUploadArtifact enum threaded through the publish flow;
    DiskImageSubject maps to rootfs/attached-drive artifact.
  • Removed the now-redundant debug!("uploaded managed layer to oss") (the
    managed-layer key already contains the digest) and its digest parameter.

Compatibility and operations

  • Public API or generated protocol: N/A — no API changes.
  • Configuration or defaults: N/A.
  • Snapshot manifest, artifact layout, or storage format: N/A.
  • Upgrade and rollback: upload metric series gain an artifact label;
    sum by (operation)-style queries remain valid. Rollback is a revert.
  • Host requirements, permissions, ports, or dependencies: N/A.

Validation

  • make fmt
  • make clippy
  • make test-unit
  • Relevant Rust integration tests
  • make -C services test (required when services/ changes)
  • Generated clients/server regenerated with the documented make target
  • Documentation updated
  • Benchmarks or performance comparison completed

Commands and results:

make fmt                                # OK
make clippy                             # OK, workspace-wide, no warnings (Linux host)
cargo test -p agentenv --lib snapshot:: # 116 passed, 0 failed

Skipped checks and reasons:

  • make test-unit: covered by CI; the unit tests of the touched module were
    run directly (see above).
  • Integration tests: require KVM/ublk hosts; this change touches no runtime
    path.
  • services/, generated code, docs, benchmarks: not affected.

Risks and reviewer notes

  • The duration histogram has mixed label sets (only put_file/put_bytes
    carry artifact); valid Prometheus, existing aggregations unaffected.
  • ~4 extra info log lines per snapshot publish.
  • Focus: src/snapshot/repository/backends/oss/client.rs,
    src/snapshot/repository/backends/oss/repository.rs,
    src/observability/prometheus.rs.

Checklist

  • The PR contains one coherent change and no unrelated formatting or refactoring.
  • New behavior is covered by tests, or I explained why testing is impractical.
    (Label threading only; no metrics-assertion harness exists — covered by
    the existing snapshot test suite and clippy.)
  • Logs and examples contain no credentials, tokens, or private registry information.
  • I did not manually edit generated code without updating its source and regenerating it.

Tag snapshot OSS upload metrics with the artifact kind
(rootfs/attached-drive/memory layer, vm_state, manifest, record,
alias) so slow or canceled publishes can be attributed to a specific
artifact. Completed file uploads are now logged at info level with
key, artifact, size, and elapsed time.

The upload duration histogram keeps recording dropped operations as
status="canceled", so canceled uploads now carry the artifact label
as well.
@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

🔍 OpenCodeReview found 1 issue(s) in this PR.

  • ✅ Successfully posted inline: 1 comment(s)

Comment on lines +248 to +249
info!(
key = %oss_key,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security · medium
This logs the fully expanded OSS object key on every successful file upload. full_key prepends the configured prefix, and the repository keys include snapshot IDs (for example artifacts/{snapshot_id}/...), so centralized logs can expose tenant/repository/snapshot metadata and create high-cardinality log fields. Please avoid logging the raw key, or log only a non-sensitive operation/artifact identifier (and, if correlation is required, a suitably redacted/hash-derived value).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant