TICKET-597: Fix worker orphan processes and add max test timeout cap

[Naragod]

Description

Worker processes running student tests could become orphaned when the RQ worker crashed or restarted before cleanup ran, because start_new_session=True fully detached test processes into their own session. This made them invisible to process supervision and allowed them to run indefinitely, blocking the dedicated test user slot.

Implementation

• Remove start_new_session=True from the subprocess call in order to reap test processes when the worker dies.
• Introduces max_test_timeout server-side configuration (default 600s) that caps per-test timeouts, preventing instructor-configured or missing timeouts from letting tests hang forever during normal operation.
• Removing start_new_session means os.killpg can no longer be used in the same-user (dev) path without killing the worker itself, so it is replaced with proc.kill() + proc.wait() to target the child process directly.

[donny-wong]

Thanks @Naragod. There are occasions where err could be empty because the OS kills the process immediately (like SIGKILL or OOM kills), and the script won't have time to print the traceback to stderr, and so the err will be an empty string. Please take into account of proc.returncode which ensures we detect these silent crashes reliably. Thank you.

[Naragod]

Thanks @Naragod. There are occasions where err could be empty because the OS kills the process immediately (like SIGKILL or OOM kills), and the script won't have time to print the traceback to stderr, and so the err will be an empty string. Please take into account of proc.returncode which ensures we detect these silent crashes reliably. Thank you.

Good catch. I have added an extra check. Please take another look.

[david-yz-liu]

@Naragod nice work. In addition to the inline comments I left, please also make sure to update the Changelog.

[david-yz-liu]

Thanks, @Naragod!