Skip to content

client/resource_group: failed writes can over-count WRU #10985

Description

@YuhaoZhang00

Bug Report

What did you do?

A resource group write request can be reported before its response is known:

  1. OnRequestWait reserves write RU and records the write reservation locally.
  2. A periodic token request can report ConsumptionSinceLastRequest before the
    write response arrives.
  3. The write later fails, so AfterKVRequest pays back the write cost locally.

This came up while reviewing #10611: #10611 (comment)

The same failed response also produces a negative response-side payback delta
that should be applied to the local limiter.

What did you expect to see?

Failed writes should not be counted as actual WRU usage by PD. If a write fails
after a report boundary, the reported actual consumption should either be
corrected or the write reservation should be reported only after the response
confirms success.

The local limiter should also apply the response-side failed-write payback delta,
so the client-side token balance reflects the failed write according to the
existing payBackWriteCost semantics.

What did you see instead?

updateDeltaConsumption only emits monotonic positive deltas. After a failed
write payback decreases local WRU, the negative adjustment is not sent to PD.

As a result, a failed write that straddles a report boundary can leave PD-side
resource group stats, metrics, and metering over-counting WRU.

The response-side negative payback delta is also ignored by the local limiter
path, so the failed write can leave that payback unapplied locally.

What version of PD are you using (pd-server -V)?

Current master. Reproduced by code inspection on b9b3eedd8408e7e991ee6feb0243d52e3f6eab19.

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-7.5This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.contributionThis PR is from a community contributor.severity/moderatetype/bugThe issue is confirmed as a bug.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions