Skip to content

Guard password-auth hosts against concurrent SSH fallback storms #85

Description

@Ramlaoui

Summary

When connection reuse is unavailable for a password-authenticated host, ssync status/web refresh paths can fall back to spawning many independent password-backed SSH probes concurrently. This can turn a routine status/output refresh into a burst of authentication attempts against the remote service.

Why this matters

For hosts that require password authentication, each fallback probe is a real auth attempt. If several status, output, watcher, or health-check requests run at once while no healthy ControlMaster is available, ssync can rapidly accumulate failed password attempts and risk triggering remote throttling or temporary lockout.

Expected behavior

  • Password-auth fallback should be serialized or tightly concurrency-limited per host.
  • A password/auth failure should trip a shared host-level circuit breaker/backoff immediately.
  • While in auth backoff, status/web should serve cached data or return a clear host-auth-backoff error instead of opening more password-backed SSH sessions.
  • The backoff should be shared across status, output refresh, watcher checks, health checks, and launch/sync paths where practical.
  • Logs should clearly distinguish scheduler failures from authentication failures without printing secrets.

Suggested acceptance criteria

  • Add a regression test where many concurrent status/output requests target a password-auth host with no reusable control socket; only one password-backed SSH attempt is made before host backoff is set.
  • Add a regression test where an auth failure prevents subsequent password-backed probes during the cooldown window.
  • Ensure web/API background refresh cannot fan out unbounded password-backed SSH attempts for a single host.
  • Keep existing behavior for key-based hosts and healthy ControlMaster reuse.

Notes

This issue intentionally avoids environment-specific details. The core problem is the generic safety behavior for password-authenticated hosts when SSH connection reuse is missing or unhealthy.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions