Fail a device whose clock has drifted out of TLS validity

[KalebKE]

A pooled emulator is resumed, not recreated, so it can come back with its clock months behind. Nothing about it looks broken — adb answers device, the screen is idle, list reports it healthy — and the pool keeps handing it out.

Every TLS chain on that device then fails notBefore validation:

net::ERR_CERT_DATE_INVALID
  → Conscrypt TrustManagerImpl.verifyChain: "Chain validation failed"
  → FirebaseException: An internal error has occurred. [ Chain validation failed ]

So the failure surfaces as an auth bug in whatever signs in first, on branches that touch no auth code.

How it presented

ci-pool-4 was sitting four months behind the host — its logcat timestamps read 04-19 while the job ran on 08-31. Every E2E sign-in in the fleet failed at once, across three unrelated PRs including a docs-only change. Two things made it hard to see:

• The consumer's retry helper treats "an internal error has occurred" as transient, so it retried four times with backoff before giving up — the log looks like a flaky network, not a broken device.
• The clock had re-synced by the time anyone measured it, so the device tested clean afterwards.

The change

clockSkew sits alongside wedged, which exists for exactly the same class of problem — reachable but unusable — and composes into androidBackend.health the same way. 60s is far wider than NTP drift and far narrower than anything that breaks certificate validation.

Best-effort, like wedged: a device that will not answer date is not condemned on that basis. Reachability is already covered by healthy, and failing closed would empty the pool the first time an image drops the binary.

Reporting these unhealthy is what makes the rest work — repair restarts unhealthy virtual devices, and consumers that require a healthy device before trusting the pool now route around a skewed one instead of leasing it and failing somewhere unrelated.

Tests

Four cases: in step, 20s adrift (healthy), four hours behind, four hours ahead (both unhealthy, reason names the clock), plus a device whose date errors staying healthy. go vet ./... and the full suite pass. The cross-compiled binary was also run against the live pool — it reports all six devices healthy, which is correct right now.

I did not force a live skew to demonstrate detection: those devices were mid-CI-run at the time.