feat: stabilize runtime lifecycle and add optional web authentication

[p-keminer]

Summary

This change bounds scanner concurrency, makes service and hardware shutdown
deterministic, and adds opt-in password protection to the complete Bjorn web
surface.

Maintainer note: why this is one combined pull request

Hi, and thank you for maintaining Bjorn. I realize that this is a substantial
change set, spanning 31 files, so I want to explain the integration boundary
up front.

The two user-facing goals—runtime stability on resource-constrained Raspberry
Pi devices and optional web authentication—look separable conceptually, but
they are not independent at the implementation, deployment, and validation
boundaries:

• Both change webapp.py. The lifecycle work replaces the server loop,
  shutdown behavior, socket reuse, and request timing, while authentication
  wraps every GET, HEAD, POST, API, static-file, and error path. Maintaining
  separate versions would create overlapping and order-dependent patches to
  the same request handler.
• Both change installation and service startup. Authentication setup,
  credential permissions, port readiness, service restart, and rollback must
  operate against the exact web-server implementation being installed.
• Scanner resource exhaustion previously affected the availability and clean
  shutdown of the same web process. Bounded worker pools, shutdown-aware Nmap,
  and deterministic thread cleanup are therefore part of keeping the
  authenticated interface reliably reachable on a Pi Zero 2 W.
• The safety guarantees are end-to-end. The validation suite checks a real
  scan, bounded thread use, clean service stop/start, port reuse, every
  authenticated route, malformed requests, credential restoration, installer
  rollback, and reboot persistence against one exact runtime state.

The features could theoretically be split, but not as two independent
drop-in patches without introducing temporary compatibility layers, duplicated
installers, competing versions of shared files, order-dependent rollouts, and
a larger matrix of partially integrated states. That would make each pull
request appear smaller while moving the main risk into the untested boundary
between them.

For that reason, this contribution uses one transactional installer, one
rollback snapshot, and one combined validation entry point. Authentication
remains strictly opt-in; installations without a credential file retain the
existing open behavior.

Most of the size is verification and operational safety rather than additional
runtime behavior: 3,920 of 6,088 added lines are tests (about 64%). Tests,
documentation, and transactional installation/rollback tooling together
account for 5,008 added lines (about 82%).

Scanner and lifecycle changes

• Replace unmanaged host and port threads with bounded worker pools.
• Wait for submitted workers before sorting, aggregating, or writing results.
• Consume worker results without materializing unused result lists while
  preserving worker exception propagation.
• Normalize semicolon-separated port data before live-status aggregation and
  stop the aggregation pipeline at the failing step without logging false
  success.
• Deduplicate configured ports and close sockets deterministically.
• Keep the Python main thread alive while Bjorn's primary thread runs.
• Run the existing Nmap discovery command as a shutdown-aware child process
  and parse completed XML through the installed python-nmap library.
• Disable Rich's background refresh worker and update progress synchronously.
• Make the web socket reusable, bound request waits, name the web thread, and
  handle its shutdown explicitly.
• Replace display sleeps with interruptible waits.
• Put the e-paper panel to sleep and close gpiozero devices, the shared pin
  factory, and lgpio's module-global notification worker.
• Report residual Python threads and fail shutdown explicitly when a worker
  misses its bounded deadline.

Optional web authentication

• Add a credential store using salted scrypt password verifiers.
• Add http_auth management commands for set, status, enable, and
  disable; interactive setup keeps plaintext passwords out of arguments and
  shell history.
• Protect every GET, HEAD, POST, API, static, and file route when enabled.
• Return a standards-compatible 401 challenge.
• Suppress routine GET and HEAD access logs while retaining POST and genuine
  error diagnostics.
• Keep normal headerless browser challenges out of the warning log while
  retaining audit warnings for malformed or incorrect credentials.
• Reject incomplete unauthenticated entity-bearing requests without waiting
  for unsent body bytes.
• Prevent config/web_auth.json from being served as static content.
• Fail closed for an existing unreadable or malformed credential file.
• Cache only a SHA-256 digest of a successful Authorization header for 30
  seconds and invalidate it when credential contents, mode, owner, or group
  change.
• Preserve current open behavior when no credential file exists.

Deployment and rollback

install_stability_web_auth.sh validates the complete source set, creates one
snapshot, stops bjorn.service once, installs the exact tested files, verifies
Python compilation, restarts the service, and accepts only a healthy HTTP 200
or 401 response on port 8000.

Any validation, installation, or startup failure restores the entire previous
state. Explicit --restore also creates a recovery snapshot before applying
the selected rollback. The normal fresh installer asks whether the optional
global /usr/local/sbin/http_auth management command should be installed and,
independently, whether credentials should be configured immediately.
Credential validation failures repeat the credential prompt without advancing
to ownership changes, while failures in required installer steps abort instead
of presenting a retry option that cannot replay the failed command. Fresh
installation starts the service and reports success only when the unit is
active and port 8000 returns HTTP 200 or 401.

Backwards compatibility

• Authentication is opt-in.
• No configuration keys or output formats change.
• Normal Nmap discovery is not shortened.
• Scan-result cleanup and retention behavior are unchanged.
• No new runtime dependency is introduced for authentication.

Automated validation

./tests/run_stability_web_auth_validation.sh --unit

sudo ./tests/run_stability_web_auth_validation.sh --all \
  --target /home/bjorn/Bjorn \
  --duration 30

git diff --check

sudo ./tests/run_reboot_persistence_validation.sh --prepare
sudo reboot
sudo ./tests/run_reboot_persistence_validation.sh --verify

The combined checkout currently has 64 unique unit and integration tests
covering scanner completion, worker limits, Nmap shutdown, process lifecycle,
display/GPIO cleanup, installer recovery ordering and port readiness,
credential parsing, configuration commands, every known web route, audit
behavior, incomplete POST handling, port reuse, web-thread shutdown, and the
isolated fresh-installer decline/install/configure/manage workflow. Dedicated
regressions also verify that manual web-server shutdown never calls the
serve_forever()-only shutdown primitive, fatal installer validation cannot be
swallowed by a conditional Bash function, and credential access-metadata
changes invalidate cached authorization.

Raspberry Pi Zero 2 W validation

• Complete scans finished without executor, thread-creation, pandas, or
  live-status aggregation errors.
• Live thread use peaked at 47 and returned to the 15-thread idle baseline.
• Repeated clean service stops completed in 4 to 15 seconds and restarted on
  port 8000.
• Live authentication command, route, challenge-stress, malformed-request, and
  credential-restoration checks passed.
• The isolated fresh-installer flow passed on Raspberry Pi OS: declining the
  optional tool left no command behind; accepting it created a real symbolic
  link; credential setup, status, disable, enable, and password verification
  passed; and the credential file mode was exactly 0600.
• A second physical clean-card install started from a verified empty target on
  Raspberry Pi OS Lite 64-bit. Three deliberately invalid short-password
  attempts each returned to the credential prompt. A valid fourth attempt
  created the mode-0600 credential file, started bjorn.service, returned
  HTTP 401 on port 8000, and completed without any false-success warning.
• The final post-review combined suite on that clean installation ran 64 tests
  in 10.483 seconds, completed a live scan, peaked at 47 threads, stopped
  cleanly in 7 seconds, restarted on port 8000, exercised 30 browser
  challenges, and restored the original credential state.
• A real reboot changed the kernel boot ID, auto-started bjorn.service,
  preserved HTTP 401 access control, completed a scan, and remained at or below
  47 threads.
• The clean-card reboot validation observed the first completed scan after 35
  seconds with PID 363, a 15-thread idle state, a 47-thread peak, unchanged
  credential hash and management-command target, and no current-boot runtime
  errors.
• No Python-finalization, scanner, or lifecycle errors remained in the
  validated current-boot journal.

Tested with Debian 12 Bookworm arm64, Python 3.11, python-nmap 0.7.1, and a
Waveshare 2.13-inch V4 display.

[gitguardian]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

---

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

[Copilot]

Pull request overview

This PR delivers a combined runtime-stability upgrade (bounded scanner concurrency + deterministic shutdown) and an opt-in HTTP Basic authentication layer for Bjorn’s web interface, including transactional installation/rollback and validation tooling.

Changes:

• Add optional HTTP Basic auth across the full web surface, backed by a salted scrypt credential store and management CLI.
• Stabilize lifecycle behavior (web server port reuse, bounded request loop, thread naming/joins) and bound scanner concurrency via worker pools.
• Introduce transactional deployment + rollback installer and a comprehensive unit/runtime validation suite for the combined change set.

Reviewed changes

Copilot reviewed 30 out of 31 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
webapp.py	Adds optional auth gate on all routes; makes server socket reusable and web thread lifecycle more deterministic.
web_auth.py	Implements credential store + Basic auth parsing and short-lived authorization caching.
configure_web_auth.py	Adds CLI for setting/enabling/disabling/status of web auth without plaintext args.
shared.py	Adds web_auth_file path to shared configuration.
Bjorn.py	Reworks shutdown coordination to join owned threads deterministically and report residual threads.
actions/scanning.py	Replaces ad-hoc threading with bounded ThreadPoolExecutor pools; adds shutdown-aware Nmap discovery.
display.py	Adds interruptible waits and explicit stop signaling to avoid shutdown hangs.
epd_helper.py	Adds explicit hardware/GPIO resource shutdown (gpiozero/lgpio) for deterministic cleanup.
install_bjorn.sh	Adds --show-plan, optional web-auth install/config step, and stricter verification behavior.
install_stability_web_auth.sh	New transactional installer with snapshot/restore and HTTP readiness verification.
README.md	Documents optional access control and management command usage.
INSTALL.md	Adds detailed stability+web-auth upgrade and validation instructions.
DEVELOPMENT.md	Adds developer documentation for auth behavior and combined validation runners.
.gitignore	Ignores config/web_auth.json so credentials aren’t committed.
tests/validate_web_auth_runtime.sh	New live-runtime validation for auth commands/routes, malformed requests, and restoration.
tests/validate_scan_runtime.sh	New runtime stability validation (PID stability, thread cap, scan completion, journal health).
tests/validate_scan_lifecycle.sh	New lifecycle validation for clean stop + restart and port readiness.
tests/run_web_auth_validation.sh	Runner for unit/runtime auth validation.
tests/run_stability_web_auth_validation.sh	Combined unit+runtime runner for stability + auth validation with summaries.
tests/run_scan_worker_validation.sh	Runner for scanner worker pool + lifecycle validation.
tests/run_reboot_persistence_validation.sh	Pre/post reboot persistence validation for combined feature state.
tests/run_fresh_installer_integration.sh	Isolated fresh-install flow test for optional auth tool + credential lifecycle.
tests/test_webapp_auth_integration.py	Integration tests asserting every known route is protected and lifecycle behaviors hold.
tests/test_web_auth.py	Unit tests for credential hashing, header parsing, caching, and fail-closed behavior.
tests/test_configure_web_auth.py	Unit tests for CLI behavior and stdin password handling.
tests/test_scanning_workers.py	Unit tests for worker pool behavior, deduped ports, and shutdown-aware discovery.
tests/test_install_bjorn_integration.py	Integration tests asserting installer ordering and safe credential prompting.
tests/test_combined_installer.py	Integration tests for transactional installer sequencing and readiness checks.
tests/test_epd_helper_shutdown.py	Unit tests for idempotent hardware shutdown/cleanup behavior.
tests/test_display_shutdown.py	Unit tests for interruptible waits and thread enumeration in display shutdown.
tests/test_bjorn_lifecycle.py	Unit tests for deterministic shutdown joins and residual thread reporting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated no new comments.

Suppressed comments (3)

actions/scanning.py:492

• scan_network_and_write_to_csv() similarly wraps executor.map() in list(...), which materializes an unused results list. Iterating the map results is sufficient to wait for completion without allocating extra memory.

                worker_count = min(MAX_HOST_SCAN_WORKERS, len(hosts))
                with ThreadPoolExecutor(max_workers=worker_count) as executor:
                    list(executor.map(self.scan_host, hosts))

webapp.py:47

• log_message() only suppresses GET request logs. With the new do_HEAD() handler, frequent HEAD-based health checks or browser probes can now spam the info log. Consider suppressing HEAD logs alongside GET to keep request logging behavior consistent and avoid noise.
  actions/scanning.py:444
• PortScanner.start() forces evaluation of executor.map() via list(...), which allocates a list of one entry per scanned port (potentially very large) just to wait for completion. This increases memory pressure unnecessarily during large scans.

This issue also appears on line 490 of the same file.

            worker_count = min(MAX_PORT_SCAN_WORKERS, len(ports))
            with ThreadPoolExecutor(max_workers=worker_count) as executor:
                list(executor.map(self.scan, ports))

[p-keminer]

Follow-up validation for fb5baf4

The follow-up commit addresses the suppressed review suggestions and the
live-status regression discovered during physical-device testing:

• Avoid unused worker-result list materialization while preserving exception
  propagation.
• Suppress routine GET/HEAD access logs while retaining POST and genuine error
  diagnostics.
• Normalize live-status port data and stop failed aggregation pipelines without
  false success logs.
• Extend runtime validation to reject live-status aggregation/write errors.

Validated on a Raspberry Pi Zero 2 W:

• 64 unit and integration tests passed.
• A real scan completed without executor, thread-pool, pandas, or live-status
  errors.
• Thread use peaked at 47.
• The service stopped cleanly in 7 seconds and returned with HTTP 401.
• 30 authentication challenges passed.
• The original credential state was restored successfully.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated no new comments.

Suppressed comments (2)

webapp.py:27

• favicon_path is defined but never used in this module, and the current os.path.join(shared_data.webdir, '/images/favicon.ico') would ignore shared_data.webdir anyway because of the leading /. Remove the unused variable (favicon serving is handled via WebUtils).
  Bjorn.py:161
• wait_for_shutdown_threads() aims to enforce a bounded shutdown deadline, but join_slice = min(0.25, remaining) is applied per-thread, so with many threads the loop can block for 0.25 * len(alive_threads) seconds per pass—overshooting the intended deadline. Use a per-thread slice derived from remaining / len(alive_threads) to keep the total wait bounded.

        join_slice = min(0.25, remaining)