Our default settings around retry grace period and retry backoff max ms cause degenerate bevhavior. Any time we get a stream error (including the stream deadline, which is enabled by default) we have a flat backoff to defend against tight loops caused by servers responding immediately with errors (can sometimes happen due to proxies or auth misconfigurations). We use retry backoff max ms for this, which is by default greater than the retry grace period. We should adjust these so that by default the stream-reconnect delay caused by the flat backoff does not leave us disconnected longer than the grace period.
TLDR: by default, our normal stream rebuild time is greater than our stale grace-period, and that shouldn't be the case.
Seen in 3 providers, tested manually:
| Provider |
STALE |
ERROR |
READY |
| Java 0.14.0 |
3.64s |
8.64s |
15.64s |
| Go 0.6.0 |
3.02s |
8.03s |
15.03s |
| JS 0.16.0 |
3.19s |
8.19s |
15.19s |
All three emit ERROR at grace + 0s, while the server has already been healthy for ~3s, then recover ~12s after STALE.
Proposing 5s/10s for retry backoff max ms/retry grace period respectively. This is not a substantial change from the existing times and will quiet the spurious transitions; in a deadline or other transient error situation, READY will arrive before ERROR, after STALE.
Our e2e tests don't catch this because the test harness' ready -> stale -> ready scenario never asserts that ERROR didn't fire; that scenario should be updated to assert no error event and implementations should update their testbed versions.
Our default settings around
retry grace periodandretry backoff max mscause degenerate bevhavior. Any time we get a stream error (including the stream deadline, which is enabled by default) we have a flat backoff to defend against tight loops caused by servers responding immediately with errors (can sometimes happen due to proxies or auth misconfigurations). We useretry backoff max msfor this, which is by default greater than theretry grace period. We should adjust these so that by default the stream-reconnect delay caused by the flat backoff does not leave us disconnected longer than the grace period.TLDR: by default, our normal stream rebuild time is greater than our stale grace-period, and that shouldn't be the case.
Seen in 3 providers, tested manually:
All three emit ERROR at grace + 0s, while the server has already been healthy for ~3s, then recover ~12s after STALE.
Proposing
5s/10sforretry backoff max ms/retry grace periodrespectively. This is not a substantial change from the existing times and will quiet the spurious transitions; in a deadline or other transient error situation,READYwill arrive beforeERROR, afterSTALE.Our e2e tests don't catch this because the test harness'
ready -> stale -> readyscenario never asserts that ERROR didn't fire; that scenario should be updated to assert no error event and implementations should update their testbed versions.