Multi-threaded downloads & parallel SFTP OBB transfers with live speeds - #209
Open
xAstroBoy wants to merge 6 commits into
Open
Multi-threaded downloads & parallel SFTP OBB transfers with live speeds#209xAstroBoy wants to merge 6 commits into
xAstroBoy wants to merge 6 commits into
Conversation
SFTP fast transfers (rooted headsets running an SSH server): - SFTP.cs: detects root (su -> uid=0), probes for an SSH server on the headset (ports 22/8022/2222), authenticates via key or password, and verifies the OBB dir is visible before use. Uploads OBBs over SFTP with the same progress/ETA UI as adb push, and transparently falls back to adb push when SFTP is unavailable or fails. - Uses /storage/emulated/0/Android/obb (canonical) with /sdcard fallback. - Titlebar shows Root and SFTP status after connect. - SftpCredentialsForm: prompts for username + password and/or key file when a server is found but stored credentials do not authenticate. - SFTP settings persisted in SettingsManager (host/port/user/pass/key). ADB device detection: - GetDevicesResilient() retries across an adb daemon restart so a version-mismatch server bounce (e.g. a different-version adb in PATH killing/restarting the server) no longer flashes "No Device Connected". Measured on wireless (Quest 3, same link): SFTP ~50 MB/s vs adb push ~28 MB/s for a 200 MB payload, verified byte-identical (md5).
Downloads: - ChunkedDownloader concurrency is now configurable via a new DownloadThreads setting (default 6) instead of a hardcoded cap of 4, and SingleThreadMode now defaults to off so multithreading is on by default. SingleThreadMode still forces a single stream when enabled. SFTP OBB transfers: - CopyObbCore now uploads multiple OBB files in parallel, one SFTP connection per worker (SSH.NET clients are not safe to share). Remote directories are pre-created once up front to avoid races, and files are distributed across workers greedily by size for balanced connections. Transfer speed display: - OBB copies now report live throughput (MB/s) in the status line for both the SFTP fast path and the adb push fallback, matching downloads. Reliability: - UpdateQuestInfoPanel ran its adb shell calls (getprop/df) on the UI thread; a slow or mid-restart adb daemon would freeze the whole window. Those calls now run off the UI thread. Build: - build.cmd's vswhere lookup adds -products * so it also finds MSBuild in a Visual Studio BuildTools install, not just full VS editions.
…s, non-hiding dialogs ADB: - RunAdbCommandToString now applies a safety timeout to shell commands so a wedged device command (e.g. Android's `df` blocking on an unresponsive mount) can no longer hang the app indefinitely. Large transfers and on-device backup/restore/install prompts are exempt so they are never force-killed. Timed reads use async stdout/stderr, which also removes a latent sequential-ReadToEnd deadlock. A timeoutMs override was added. Startup: - Stop hard-killing every adb.exe on launch. That tore down a running adb server and dropped any active wireless (adb connect) session, forcing a reconnect that often failed silently. start-server reuses a compatible running server, preserving the connected device. SFTP OBB: - After a transfer, best-effort chmod -R 0777 + restorecon on the OBB folder so the game can read files an sshd wrote as root (no-op on FUSE storage, needed where the sshd writes through to a real filesystem). UI: - Message boxes are now TopMost, shown in the taskbar, and brought to front on open, so a prompt can never hide behind another window/monitor and leave the disabled main window looking frozen while it waits for an answer.
- Add a FileZilla-style Transfer window (TransferWindow / TransferQueue): a non-modal popup that appears when a transfer starts and shows a row per file with a live progress bar, speed and status (Queued/Active/Done/Failed). It hides on close and re-shows on new activity, and has a "Clear finished" button. Both SFTP uploads and adb pushes feed into it. - SFTP OBB upload is now a bounded work-stealing queue instead of firing every file at once: N workers (default 3, new SftpThreads setting) each hold one connection and pull the next file from a shared queue, so at most N files transfer at any moment regardless of OBB file count. Gentler on the on-device sshd/FUSE storage. SingleThreadMode still forces a single stream.
Transfer UI: - Replace the popup transfer window with an integrated, collapsible strip that overlays the bottom of the games list only while files are moving, themed to match the list, and hides when idle. Successful rows auto-clear; failed rows stay with a right-click Retry / Remove / Clear finished menu. Never hang (the softlock fix): - Each SFTP upload now uses a 60s OperationTimeout, so a wedged channel throws instead of blocking UploadFile forever. A single stuck file could previously freeze an entire multi-hundred-file OBB upload; it can't anymore. Auto-retry + verify: - Each file is retried up to 3x on a fresh reconnection with backoff. - After every upload the remote path is stat'd and checked for exact size; a file that isn't actually there (or is truncated) is treated as a failure and retried, never reported as done. Failure isolation: - One bad file no longer aborts the batch. Files that still fail after retries are left Failed in the strip with a per-file Retry action (SFTP.RetryUpload), instead of re-pushing the whole set. Only a total failure falls back to adb push.
AdvancedSharpAdbClient throws "An unknown error occurred." whenever it cannot parse pm install's result, which frequently happens even when the install actually succeeded (a connection blip right after install, unusual pm output). Rookie treated that as a hard failure. Now, on an install exception, VerifyInstalled reads the APK's package name and version code with aapt, confirms the package is present on the device, and requires the installed version code to match before reporting success. A real failed upgrade (old version still present) still fails, so it is not a blind pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Overhauls how large game/OBB data moves in and out of the app: downloads and SFTP OBB uploads are concurrent, every transfer is shown live in an integrated strip, and the whole path is hardened so it can't softlock, silently drop files, or report a bogus failure. Tested end to end on a wireless Quest 3 (metadata sync, multi-part download, multi-GB OBB set over SFTP, install, reconnect across restarts).
Why SFTP instead of
adb push?For multi-GB OBBs, a direct SFTP session to an on-device
sshd(rooted headsets) is several times faster thanadb push, and the difference grows with size:adb pushis throttled by the adb sync protocol. It streams a single channel in small chunks, each acknowledged, multiplexed through the adb daemon on both ends. Over wireless adb especially, that overhead caps throughput far below the link's real capacity.sshdwith large windows and far less per-chunk handshaking, so it saturates the Wi-Fi/USB link instead of the protocol.Measured here: ~62.8 MB/s for a 9.5 GB OBB over 3 SFTP connections;
adb pushover wireless is a fraction of that.Trade-off / fallback: SFTP needs root + a reachable on-device
sshd. Detection is automatic and everything degrades gracefully — no root, no server, or a total SFTP failure falls straight back toadb push.Multi-threaded downloads
ChunkedDownloaderfetches a game's split archive (.7z.001,.7z.002, …) over a localrclone serve httpproxy. Concurrency was hardcoded toMin(4, parts)and gated behindSingleThreadMode, which defaulted on — so downloads were single-threaded out of the box.DownloadThreadssetting (default 6), clamped to the part count.SingleThreadModedefaults to off; turning it on still forces a single stream.--tpslimit 1.0 --tpslimit-burst 3, so rclone throttles opening new files; extra threads still overlap in-flight streams but won't scale linearly there. Private mirrors and SFTP have no such cap.SFTP OBB transfers — concurrent, bounded, and self-healing
SftpThreadsworkers (default 3) each hold one connection and pull the next file from a shared queue — at most N files move at once regardless of OBB count (the FileZilla model). Gentle on the devicesshd/FUSE storage.OperationTimeout, so a wedged channel throws instead of blocking forever. Previously a single stuck file could freeze an entire multi-hundred-file OBB upload.stat'd for exact size; a file that isn't there or is truncated is a failure and is retried, never reported as done.SFTP.RetryUpload) instead of re-pushing the whole set. Only a total failure falls back toadb push.chmod -R 0777+restoreconso the game can read files ansshdwrote as root.Integrated transfer strip
Replaces the earlier popup with a themed, collapsible strip that overlays the bottom of the games list only while files move, then hides. One row per file (File · Size · live progress bar · Speed · Status). Successful rows auto-clear; failed rows stay with a right-click Retry / Remove / Clear finished menu. Fed by both SFTP and
adb push.Install verification
AdvancedSharpAdbClientthrows "An unknown error occurred." whenever it can't parsepm install's result — which happens even when the APK installed fine. On an install error the app now verifies against the device (APK package + version code viaaapt, matched against the installed version) and reports success when it really landed. A failed upgrade still fails.Reliability fixes (reported as "freezes")
UpdateQuestInfoPanelranadb shell getprop/dfon the UI threadRunAdbCommandToStringhad no timeout; Androiddfcan wedge on a bad mountReadToEnddeadlockadb.exe, tearing down the running server + itsadb connectsessionstart-server) instead of killing itShowInTaskbar = falseTopMost, shown in taskbar, brought to frontBuild
build.cmd'svswherenow passes-products *, so it finds MSBuild in a VS BuildTools install, not just full editions.New settings
DownloadThreadsSftpThreadsSingleThreadModefalseMissing settings fall back to these defaults, so existing
settings.jsonfiles need no migration. No new NuGet dependencies.Supersedes #208 — resubmitted after an account migration; branch and commits are identical.