Added Backoff

[Effi-S]

Fixes #2105

Adds retry/backoff to the auditor validation lookup (AuditorCheck / RetrieveAuditTokens) so a quickly-chained transaction is no longer spuriously rejected when the async finality listener has not yet persisted a just-audited output. This mirrors the pending-status retry/backoff that the sibling Audit() path already has.

[Effi-S]

#2105

[Effi-S]

@adecaro,
I want to reconsider keeping it as or significantly reducing the backoff time

In what scenario Is 9 seconds (3 retries of 3 seconds) really necessary?

This is a race condition that warrants a failure in my opinion

The client can retry on his own, or at the very most have 200 millisecond backoffs

[github-actions]

📊 Token Validation Benchmark

Comparison of this PR against the base branch. 🟢 improvement · 🔴 regression · ➖ within ±1.0% noise.

Variant	Benchmark	Params	Workers	TPS (base → PR)	Δ TPS
csp	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	106 → 105	➖ -0.2%
csp	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	115 → 116	➖ +0.3%
ipa	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	78 → 78	➖ -0.2%
ipa	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	76 → 76	➖ -0.0%

[AkramBitar]

@adecaro, I want to reconsider keeping it as or significantly reducing the backoff time

In what scenario Is 9 seconds (3 retries of 3 seconds) really necessary?

This is a race condition that warrants a failure in my opinion

The client can retry on his own, or at the very most have 200 millisecond backoffs

@Effi-S what if we have defaults and give a way to customize this in a config file?

[AkramBitar]

Review of the pending-status retry/backoff added to RetrieveAuditTokens.

The premise checks out: finality.Commit writes the token rows (AppendValid) and the Confirmed status in the same store transaction, so IsPending == false really does imply the rows are visible; the driver-level qe here is token/sdk/vault.QueryEngine, not the retrying token.QueryEngine wrapper, so there is no accidental 3x3 nesting; and the added latency fits inside the initiator's 1-minute ReceiveTypedWithTimeout.

go vet, golangci-lint (0 issues) and go test ./token/core/common/ all pass on this head. Finding 1 below was reproduced empirically with a throwaway test.

Five findings inline, most severe first.

[AkramBitar]

On the two options in #2105

The backoff (this PR) is the right thing to do now. It removes a real inconsistency:
AuditorCheck had no tolerance for the pending-output race, while Audit() — which runs
after it — already had a grace window. Cheap, no wiring, no schema change.

It's a mitigation rather than a fix, though. The window is network → finality-listener
notification latency (token/services/ttx/finality/listener.go:209 is the only writer of
those rows), which is unbounded under load, so a ~6s retry window shrinks the failure
probability without removing it.

Suggestion 2 (the tokens.Service cache fallback) works alongside this, not instead of
it, and is better than the issue suggests: the cached value is byte-identical to what the
DB read returns (storage.go:189 stores owner_raw = tta.Tok.Owner; ListAuditTokens
selects exactly owner_raw, token_type, quantity), and the cache entry's lifetime lines up
exactly with the race — added at AuditApproveView.Call (ttx/auditor.go:356), removed in
AppendValid (tokens.go:126). Shape would be: DB read first, cache only on miss, retry as
the outer net for whatever the cache doesn't cover.

Its limits are why it doesn't replace the backoff: the cache is in-process and best-effort
(Ristretto's Set result is discarded), so a restart or a second auditor replica falls
straight through. It also needs the auditor = true filter replicated via Flags.Auditor,
and some plumbing — token/core/common can't import token/services/tokens directly, so it
needs a consumer-side interface plus an adapter wired in the SDK.

Proposal: merge the backoff here, and open a low-priority follow-up issue for suggestion 2
so the cache fallback is tracked but doesn't block this. Happy to open it if you agree.

[Effi-S]

@AkramBitar Thanks for the review,
I'm going to set backoff params in config file.
To keep this PR tidy,
I'll address all the suggestions in the next commit