Skip to content

fix(docker): bind uvicorn on every address family, not IPv4 only - #320

Open
HAAHIT wants to merge 7 commits into
mainfrom
fix/uvicorn-dual-stack-bind
Open

fix(docker): bind uvicorn on every address family, not IPv4 only#320
HAAHIT wants to merge 7 commits into
mainfrom
fix/uvicorn-dual-stack-bind

Conversation

@HAAHIT

@HAAHIT HAAHIT commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

User description

Found while investigating slow account creation. One-line runtime change, but the reasoning behind which one line matters — see the trap below.

Problem

The compose network runs with enable_ipv6: true, so DNS for backend answers with both an A and an AAAA record, and nginx uses both. uvicorn was bound to 0.0.0.0 — IPv4 only — so nothing was ever listening on the AAAA address:

[error] connect() failed (111: Connection refused) while connecting to upstream,
        upstream: "http://[fd00:b010::2]:4321/api/workspaces"
[warn]  upstream server temporarily disabled while connecting to upstream

Confirmed from inside the container:

IPv4 -> LISTENING
IPv6 -> REFUSED: [Errno 111] Connection refused

nginx marks the upstream down for fail_timeout and retries over IPv4, so requests do complete. The cost is a wasted connection attempt, recurring error-log noise, and an upstream repeatedly marked unhealthy for a reason that has nothing to do with the backend's health.

The trap: :: is not the fix

The obvious change is --host ::. That makes it worse in the other direction. asyncio sets IPV6_V6ONLY on any AF_INET6 server socket, so :: listens on IPv6 only and refuses IPv4 — swapping one broken family for the other. I shipped that first and the verification caught it.

Measured in-container rather than assumed:

--host sockets created IPv4 IPv6
0.0.0.0 [AF_INET] OK REFUSED
:: [AF_INET6] REFUSED OK
"" (this PR) [AF_INET, AF_INET6] OK OK

An empty host makes asyncio open one socket per family, which is real dual-stack. The kernel's bindv6only=0 is not enough on its own, because asyncio overrides it per socket.

Verification

After the change:

in-container:   IPv4 -> LISTENING     IPv6 -> LISTENING

from nginx:     172.18.0.2    -> OK
                fd00:b010::2  -> OK

Twelve proxied requests produced:

connection-refused upstream errors: 0
upstream disabled warnings:         0

Container comes up healthy, migrations run, and the app serves end-to-end — / 200, /api/health 200 through the nginx proxy, Vite dev server 200.

🤖 Generated with Claude Code


CodeAnt-AI Description

Improve backend connectivity, health checks, and request responsiveness

What Changed

  • Backend services now accept both IPv4 and IPv6 connections, avoiding failed proxy attempts and misleading unhealthy warnings
  • Password hashing and verification no longer block other requests handled by the same worker
  • Health checks reuse a client and cache Supabase reachability results for up to 60 seconds, then close the client during shutdown
  • Database pooling avoids an unnecessary connection check by default while allowing pool size, recycling, overflow, and connection checks to be configured through environment variables
  • Container and shell files retain Linux-compatible line endings across Windows checkouts

Impact

✅ Fewer IPv6 connection failures
✅ Faster concurrent account requests
✅ Lower health-check traffic to Supabase

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

The compose network runs with enable_ipv6, so DNS for `backend` answers with
both an A and an AAAA record and nginx uses both. uvicorn was bound to
0.0.0.0, which is IPv4 only, so nothing was ever listening on the AAAA
address. Those requests hit ECONNREFUSED first:

    [error] connect() failed (111: Connection refused) while connecting to
            upstream, upstream: "http://[fd00:b010::2]:4321/api/workspaces"
    [warn]  upstream server temporarily disabled while connecting to upstream

nginx then marks the upstream down for fail_timeout and retries over IPv4, so
requests do complete -- the cost is a wasted connection attempt, recurring
error-log noise, and an upstream that keeps being marked unhealthy for a
reason that has nothing to do with the backend's health.

Confirmed from inside the container before the change:

    IPv4 -> LISTENING
    IPv6 -> REFUSED: [Errno 111] Connection refused

The empty host is deliberate, and "::" is not a substitute. asyncio sets
IPV6_V6ONLY on any AF_INET6 server socket, so binding "::" listens on IPv6
*only* and refuses IPv4 -- swapping one broken family for the other. That was
measured, not assumed:

    host=0.0.0.0    sockets=[AF_INET]            IPv4=OK       IPv6=REFUSED
    host=::         sockets=[AF_INET6]           IPv4=REFUSED  IPv6=OK
    host=""         sockets=[AF_INET, AF_INET6]  IPv4=OK       IPv6=OK

An empty host makes asyncio open one socket per family, which is real
dual-stack.

After the change both families listen, nginx reaches the backend over each of
them, and twelve proxied requests produced zero connection-refused errors and
zero "upstream server temporarily disabled" warnings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@HAAHIT
HAAHIT requested a review from Entropy-rgb as a code owner August 3, 2026 15:53
@codeant-ai

codeant-ai Bot commented Aug 3, 2026

Copy link
Copy Markdown

🤖 CodeAnt AI — Review Status

Status Commit Started (UTC) Finished (UTC)
✅ Incremental review completed 5e22e77 Aug 04, 2026 · 11:51 11:51
✅ Reviewed your PR 84bf83a Aug 03, 2026 · 15:53 15:54

@coderabbitai

coderabbitai Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@Entropy-rgb, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 30 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 89dd988e-7825-4fd4-99b1-d76d4301f179

📥 Commits

Reviewing files that changed from the base of the PR and between 5aebabc and 5e22e77.

📒 Files selected for processing (9)
  • .env.example
  • .gitattributes
  • backend/DOCKERFILE
  • backend/app/controllers/auth.py
  • backend/app/controllers/system.py
  • backend/app/pgdatabase/engine.py
  • backend/app/server.py
  • tests/test_engine_pool_config.py
  • tests/test_health_jwks_probe.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codeant-ai codeant-ai Bot added the size:S This PR changes 10-29 lines, ignoring generated files label Aug 3, 2026
HAAHIT and others added 4 commits August 4, 2026 16:39
Git for Windows defaults to core.autocrlf=true, and with no .gitattributes
the repo had nothing to stop that from rewriting backend/docker-entrypoint.sh
to CRLF on checkout. Docker copies the file into the Linux image verbatim,
where the kernel reads the shebang as "/bin/sh\r" and looks for an
interpreter by that name. The result is:

    exec /app/backend/docker-entrypoint.sh: no such file or directory

on a file that is present and executable, so the backend container never
starts and compose fails with "dependency failed to start: container is
unhealthy". Nothing in that message points at line endings, and the build is
fine on Linux, macOS, and CI, so it only ever breaks Windows clones.

Pin the file types that Linux reads verbatim to LF regardless of platform.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…#317)

bcrypt is intentionally slow -- measured at ~750ms per call at the default
cost factor on our backend container. It is also CPU-bound and never yields,
so calling it directly from an async handler parks the event loop for that
whole time. The cost is not paid by the request doing the hashing; it is paid
by every request that worker is serving. One signup froze the process for
three quarters of a second.

All five call sites were synchronous: login (checkpw), signup (gensalt +
hashpw), change_password (checkpw + gensalt + hashpw), and reset_password
(gensalt + hashpw). checkpw is as expensive as hashpw, so logins blocked just
as hard as signups.

Route all of them through two helpers that wrap bcrypt in run_in_threadpool,
which this module already uses for the JWKS lookup. Hashes are unchanged --
same algorithm, same cost factor -- so existing stored passwords keep
verifying.

Measured with a 10ms heartbeat task watching for event-loop stalls:

    before   work 341 ms | worst loop stall 357 ms
    after    work 328 ms | worst loop stall  47 ms

The work takes just as long; it just no longer blocks everything else.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Entropy-rgb <som3shkamad@gmail.com>
* perf(db): stop pinging the database before every pooled checkout

pool_pre_ping issues a liveness round trip before handing out every pooled
connection. It is not amortised -- it repeats on every request, including for
a connection that is demonstrably healthy and was used moments earlier. On the
app's own engine that doubled the cost of a checkout:

    pre_ping=True  (before)   median 1089 ms
    pre_ping=False (after)    median  481 ms

Same query, same connection, 608ms saved per database-touching request. The
post-signup flow makes several of these back to back, so it compounds.

Worth being precise about what this is not: instrumenting the pool showed
zero new DBAPI connections across repeated checkouts in every configuration,
so connections were already being reused correctly. The cost was purely the
extra round trip, not reconnection.

What pre-ping buys is protection against a connection that died while idle in
the pool. pool_recycle already closes that window from the other side by
retiring connections on a timer, well inside any sane server or pooler idle
timeout, so pre-ping largely re-checks what recycling has handled. Default it
off and keep recycling at 300s.

Because that is a trade rather than a free win, every pool value is now
env-tunable. A deployment that genuinely sees stale connections can set
DB_POOL_PRE_PING=true without a code change.

statement_cache_size=0 was measured too and left alone -- it costs nothing
here (166ms vs 174ms per query) and is load-bearing for transaction pooling.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(build): pin shell scripts to LF so Windows clones can build (#316)

Git for Windows defaults to core.autocrlf=true, and with no .gitattributes
the repo had nothing to stop that from rewriting backend/docker-entrypoint.sh
to CRLF on checkout. Docker copies the file into the Linux image verbatim,
where the kernel reads the shebang as "/bin/sh\r" and looks for an
interpreter by that name. The result is:

    exec /app/backend/docker-entrypoint.sh: no such file or directory

on a file that is present and executable, so the backend container never
starts and compose fails with "dependency failed to start: container is
unhealthy". Nothing in that message points at line endings, and the build is
fine on Linux, macOS, and CI, so it only ever breaks Windows clones.

Pin the file types that Linux reads verbatim to LF regardless of platform.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* perf(auth): run bcrypt in the threadpool instead of on the event loop (#317)

bcrypt is intentionally slow -- measured at ~750ms per call at the default
cost factor on our backend container. It is also CPU-bound and never yields,
so calling it directly from an async handler parks the event loop for that
whole time. The cost is not paid by the request doing the hashing; it is paid
by every request that worker is serving. One signup froze the process for
three quarters of a second.

All five call sites were synchronous: login (checkpw), signup (gensalt +
hashpw), change_password (checkpw + gensalt + hashpw), and reset_password
(gensalt + hashpw). checkpw is as expensive as hashpw, so logins blocked just
as hard as signups.

Route all of them through two helpers that wrap bcrypt in run_in_threadpool,
which this module already uses for the JWKS lookup. Hashes are unchanged --
same algorithm, same cost factor -- so existing stored passwords keep
verifying.

Measured with a 10ms heartbeat task watching for event-loop stalls:

    before   work 341 ms | worst loop stall 357 ms
    after    work 328 ms | worst loop stall  47 ms

The work takes just as long; it just no longer blocks everything else.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Entropy-rgb <som3shkamad@gmail.com>

* perf(db): stop pinging the database before every pooled checkout

pool_pre_ping issues a liveness round trip before handing out every pooled
connection. It is not amortised -- it repeats on every request, including for
a connection that is demonstrably healthy and was used moments earlier. On the
app's own engine that doubled the cost of a checkout:

    pre_ping=True  (before)   median 1089 ms
    pre_ping=False (after)    median  481 ms

Same query, same connection, 608ms saved per database-touching request. The
post-signup flow makes several of these back to back, so it compounds.

Worth being precise about what this is not: instrumenting the pool showed
zero new DBAPI connections across repeated checkouts in every configuration,
so connections were already being reused correctly. The cost was purely the
extra round trip, not reconnection.

What pre-ping buys is protection against a connection that died while idle in
the pool. pool_recycle already closes that window from the other side by
retiring connections on a timer, well inside any sane server or pooler idle
timeout, so pre-ping largely re-checks what recycling has handled. Default it
off and keep recycling at 300s.

Because that is a trade rather than a free win, every pool value is now
env-tunable. A deployment that genuinely sees stale connections can set
DB_POOL_PRE_PING=true without a code change.

statement_cache_size=0 was measured too and left alone -- it costs nothing
here (166ms vs 174ms per query) and is load-bearing for transaction pooling.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(env): clarify when DB_POOL_PRE_PING should stay on

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Entropy-rgb <som3shkamad@gmail.com>
)

get_health built an httpx.AsyncClient per call and never closed it:

    resp = await httpx.AsyncClient(timeout=5).get(jwks_url)

The client owns a connection pool, so each probe abandoned one. /api/health is
polled by the container healthcheck every 10-30s, forever, which makes this a
slow resource leak on a fixed timer rather than a one-off.

The same line also put a third-party network round trip on the healthcheck
path. Backend logs showed a steady stream of outbound JWKS requests every ~13s
with nobody using the app, and it left /api/health taking 1.8-3.2s.

Share one client and cache the reachability answer for 60s. Whether Supabase
is reachable does not change between two polls seconds apart, so the
diagnostic keeps its value while the probe stops being a hot path. The client
is closed on shutdown through the existing lifespan teardown.

Behaviour is otherwise unchanged: the supabase_jwks field is still always
populated, non-200 still reports unexpected_status:<code>, and a network
failure still degrades to unreachable:<ExceptionName> rather than surfacing as
a 500 -- health must not go down because a third party did.

Verified against the running app: six consecutive /api/health calls plus the
container's own polling produced exactly one outbound JWKS request.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
HAAHIT and others added 2 commits August 4, 2026 17:20
The compose network runs with enable_ipv6, so DNS for `backend` answers with
both an A and an AAAA record and nginx uses both. uvicorn was bound to
0.0.0.0, which is IPv4 only, so nothing was ever listening on the AAAA
address. Those requests hit ECONNREFUSED first:

    [error] connect() failed (111: Connection refused) while connecting to
            upstream, upstream: "http://[fd00:b010::2]:4321/api/workspaces"
    [warn]  upstream server temporarily disabled while connecting to upstream

nginx then marks the upstream down for fail_timeout and retries over IPv4, so
requests do complete -- the cost is a wasted connection attempt, recurring
error-log noise, and an upstream that keeps being marked unhealthy for a
reason that has nothing to do with the backend's health.

Confirmed from inside the container before the change:

    IPv4 -> LISTENING
    IPv6 -> REFUSED: [Errno 111] Connection refused

The empty host is deliberate, and "::" is not a substitute. asyncio sets
IPV6_V6ONLY on any AF_INET6 server socket, so binding "::" listens on IPv6
*only* and refuses IPv4 -- swapping one broken family for the other. That was
measured, not assumed:

    host=0.0.0.0    sockets=[AF_INET]            IPv4=OK       IPv6=REFUSED
    host=::         sockets=[AF_INET6]           IPv4=REFUSED  IPv6=OK
    host=""         sockets=[AF_INET, AF_INET6]  IPv4=OK       IPv6=OK

An empty host makes asyncio open one socket per family, which is real
dual-stack.

After the change both families listen, nginx reaches the backend over each of
them, and twelve proxied requests produced zero connection-refused errors and
zero "upstream server temporarily disabled" warnings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codeant-ai codeant-ai Bot added size:L This PR changes 100-499 lines, ignoring generated files and removed size:S This PR changes 10-29 lines, ignoring generated files labels Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants