Skip to content

Return database connections to the pool after each qless job - #349

Merged
Holmes98 merged 1 commit into
masterfrom
qless-db-connections
Jun 22, 2026
Merged

Return database connections to the pool after each qless job#349
Holmes98 merged 1 commit into
masterfrom
qless-db-connections

Conversation

@Holmes98

Copy link
Copy Markdown
Member

This fixes an issue where workers could fail to judge submissions if the database connection was dropped (e.g. because postgresql was restarted).

https://discord.com/channels/670126531489824788/678154623571329025/1485894172748550185

This fixes an issue where workers could fail to judge submissions
if the database connection was dropped (e.g. because postgresql
was restarted).
@coveralls

Copy link
Copy Markdown

Coverage Status

coverage: 38.386%. remained the same — qless-db-connections into master

@tom93

tom93 commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

A summary of some messages from Discord for posterity:

The qless README suggests using ActiveRecord::Base.establish_connection, but that caused a PG::UnableToSend exception (hence the use of with_connection in this PR).
There is a similar stack trace for establish_connection at https://www.github.com/rails/rails/issues/12867#issuecomment-28690381, and it looks like the proper fix for that is https://www.github.com/rails/rails/pull/29785 but that's only in Rails 5.2.
Another suggestion from related issues is to tweak the TCP keepalive: https://www.github.com/rails/rails/issues/20411#issuecomment-360801422 (or https://www.github.com/rails/rails/issues/9421#issuecomment-41952270), might be worth trying if there are still problems in the future.

@tom93 tom93 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't test this, but it seems safe enough (even without getting to the bottom of why it works while establish_connection fails).
Edit: Not sure if this will work if the database is restarted while a submission is being judged...

@Holmes98

Holmes98 commented Jun 22, 2026

Copy link
Copy Markdown
Member Author

Not sure if this will work if the database is restarted while a submission is being judged

Yeah, it'll still throw an error in that case, but this at least seems to allow workers to automatically recover from the broken connection (on the next job) rather than failing on all subsequent submissions. Hopefully it's good enough for now.

Thanks for reviewing.

@Holmes98
Holmes98 merged commit 4de5564 into master Jun 22, 2026
4 checks passed
@Holmes98
Holmes98 deleted the qless-db-connections branch June 22, 2026 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants