Add global timeout?

[timwoj]

I'm running into some issues on CircleCI where some tests are taking a very long time to complete (for whatever reason) and are thus causing a timeout on the CI jobs. Would it be possible to add a globally-configured timeout where if any btest takes longer than some value, it gets automatically killed and marked as failed?

[bbannier]

The benefit of this over a simple outer timeout configured in the test or via timeout DURATION btest ... would be that it would allow configuring a global "unreasonably long" duration of any test in the suite. This seems potentially useful, but could IMO be longterm detrimental to test stability:

• it is probably hard to define in a good way what "unreasonably long" means globally unless we would set this to e.g., 120min, but then we could just use such a value as outer timeout
• in CI environments tests might take longer than expected e.g., due to concurrently running noisy tests, scheduling on "slow" runners, or noisy neighbors running on the same physical hardware
• we are historically not very good at estimating timeouts; I suspect many tests use a way too low timeout for e.g., btest-bg-wait
• if a single tests repeatedly does not terminate in reasonable time we could add a timeout to the particular @TEST-EXEC statement in that single test

[awelzel]

if a single tests repeatedly does not terminate in reasonable time we could add a timeout to the particular @TEST-EXEC statement in that single test

What's a bit annoying currently is that tests that run too long are not surfaced. E.g I don't know how to easily figure out which test didn't finish here here.

Seems fair to say that for zeek/zeek, no test is supposed to run longer than 2 minutes (minimally when running sequential). I agree that resource starvation can lead to 2+min, but that'd seems a problem in itself. Timing out tests and surfacing the affected ones seems reasonable to me

Related feature is maybe pytest-timeout which has:

Please read this README carefully and only use this plugin if you understand the consequences. This plugin is designed to catch excessively long test durations like deadlocked or hanging tests, it is not designed for precise timings or performance regressions. Remember your test suite should aim to be fast, with timeouts being a last resort, not an expected failure mode.

EDIT: Didn't mean to close :-)