Deleted songs come back: server delete and deletion tombstone both fail silently, and orphan recovery re-adopts the directory

[thcp]

Reported on macOS: deleting a song -- by clearing the trash, or purging the library -- brings it back later.

Reproduced by code inspection. Deletion has two independent halves and both swallow their failures, so several distinct mechanisms produce this symptom.

The delete path

static/js/catalog.js:1961-1970 (the clearBinBtn handler):

markJobsDeleted(toDelete); // persist before purge so reload can't re-import
purgeTrash();
saveState();
render();
for (const id of toDelete) {
  fetch(`/api/jobs/${id}`, { method: "DELETE" }).catch(() => {});
}

syncWithServer (catalog.js:2708-2723) re-adds every job from GET /api/jobs that is not in tracks, not in the trash folder, and not in the deleted-ids tombstone.

Four confirmed mechanisms

1. The server DELETE is fire-and-forget and its errors are discarded.
.catch(() => {}). A failed delete leaves the job in the registry, and nothing retries or reports it.

2. The tombstone can silently fail to persist, and it is the only remaining defence.
markJobsDeleted (catalog.js:133-138) calls storeSet(DELETED_JOBS_KEY, ...) without awaiting, and its .catch only logs a warning. If the write fails, or the app closes before it lands, the tombstone is lost. On next launch loadState reads the older list and syncWithServer re-imports every affected job. The inline comment says "persist before purge so reload can't re-import" -- the intent is right, but nothing waits for the persist.

3. A successful server delete can still leave the directory, which is then re-adopted.
_rmtree_job (app/api/jobs.py:113-121) catches every exception and only logs a warning:

try:
    shutil.rmtree(job_dir)
except Exception:
    logger.warning("failed to remove job dir %s", job_dir, exc_info=True)

delete_job (jobs.py:762-764) then calls registry_remove and registry_persist regardless of whether the files went away. On the next start, restore() (app/core/registry.py:183-190) walks jobs_dir and adopts any JOB_ID_RE-shaped directory not already known, via _recover_done_job. The job is recreated from disk.

4. Some jobs can never be deleted, and the UI never says so.
delete_job returns 409 unless job.status in ("done", "error", "cancelled"). A job stuck in queued -- for instance the phantom queued job in #520 -- 409s forever. Because the fetch is fire-and-forget, the user sees the row disappear and reappear with no error.

Also found

There is no "purge library" code path. markJobsDeleted has exactly one caller (catalog.js:1964). Any other UI route to permanent deletion does not tombstone, so syncWithServer will always re-import those tracks. catalog.js:1631-1634 (dropping a cancelled job's placeholder) is one such path: delete tracks[jobId] with no tombstone and no server delete.

Not established

No platform-gated cause was found -- nothing in these paths is macOS-specific. Mechanism 3 is the most plausible explanation for it being noticed on macOS (Finder/Spotlight touching files mid-delete makes rmtree fail more often there), but that is a hypothesis, not a verified finding.

Fix

• Await the tombstone write before purging, and treat its failure as a failed delete.
• Await the server DELETE; surface failures and retry, rather than .catch(() => {}).
• _rmtree_job should report failure to its caller; delete_job should not remove the registry entry when the files are still on disk, and should return an error the UI shows.
• Allow deleting a job in a non-terminal state, or give the UI a way to report the 409.
• Have restore()'s orphan adoption respect a server-side deletion record, so a directory that failed to delete is not silently resurrected.

Test

• Delete with shutil.rmtree raising: the job must not vanish from the registry, and the user must see an error.
• Clear the bin, kill the app before the store write completes, relaunch: the tracks must stay deleted.
• Delete a job in queued state: it must succeed or report a visible error.

[thcp]

The reporter confirmed the path used: Settings -> Reset app data, not the Clear Bin button. That is a fifth mechanism, and it is a cleaner explanation than the four above -- plus it suggests a genuinely macOS-specific trigger.

Mechanism 5: reset silently tolerates undeleted directories, then destroys the tombstone

app/core/registry.py reset_all:

for entry in jobs_dir.iterdir():
    try:
        if entry.is_dir():
            shutil.rmtree(entry)
        else:
            entry.unlink()
    except OSError:
        logger.warning("reset: could not remove %s", entry, exc_info=True)

Per-entry except OSError, log only, keep going. POST /api/reset then returns {"ok": True} whether or not anything actually got deleted.

The frontend (static/js/catalog.js, the openResetConfirm handler) treats that as success and:

1. await invoke("reset_user_data") -- clears the Tauri store, including stemdeck.deleted_jobs
2. localStorage.clear()
3. window.location.reload()

Immediately after the reload everything looks correct: reset_all cleared the in-memory _jobs, so GET /api/jobs is empty and syncWithServer has nothing to re-import.

The damage shows up on the next backend start. restore() (registry.py:183-190) walks jobs_dir, finds every directory that survived the failed rmtree, and re-adopts it via _recover_done_job. The tombstone that would have suppressed them was deleted in step 1. Every song whose directory failed to delete comes back.

So reset is strictly worse than Clear Bin: Clear Bin at least writes a tombstone, while reset deliberately destroys it while trusting a delete that reports success unconditionally.

Why this plausibly shows up on macOS specifically

shutil.rmtree scans a directory and then removes it. On macOS, Finder writes .DS_Store into a directory when it is viewed, and Spotlight (mds) touches files while indexing. A file appearing between scandir and rmdir makes the final rmdir fail with OSError: [Errno 66] Directory not empty.

That is a well-known macOS-specific rmtree race and it fits: same code, far more likely to fail on macOS than on Linux or in Docker.

Stated as a hypothesis -- I have not reproduced it. Confirming it is cheap: after a reset that "worked", check whether jobs/ still contains directories, and grep the backend log for reset: could not remove.

Additional fixes for this path

• reset_all must report which entries it failed to remove; /api/reset must not return {"ok": true} when any survived.
• The frontend must not clear the tombstone store when the reset was partial.
• Retry rmtree with onerror, or sweep the directory a second time, so a transient .DS_Store appearing mid-walk does not defeat it.
• restore()'s orphan adoption should honour a server-side deletion record so a directory that failed to delete is not silently resurrected.

[thcp]

Fixed in 0.16.1, which is now on main via #541.

Four things had to fail together for a deleted song to come back, and they did. reset_all swallowed per-directory failures, /api/reset reported success either way, the frontend then wiped its own tombstone on the strength of that, and restore() re-adopted every surviving directory on the next start.

The registry keeps a server-side deletion record now, so a directory that outlives its delete is not adopted again. The client tombstone is still written, but it is a second line of defence rather than the only one.

The fix landed in #540. I ran it on a packaged Windows build: a hard-deleted job stayed gone across a restart, and the deletion record prunes itself once the directory is actually gone.

While testing this I found a second route back that this fix did not cover, when two jobs share a source URL and one is in the Trash. That is #542, fixed separately.