Clean up import_processing (#928)

[davmlaw]

Issue #928

The bug

schedule_pipeline_stage_steps builds the FINISH stage as a chain of the FINISH-dependent steps followed by pipeline_success_task. AbstractVCFImportTaskFactory.get_finish_task_classes() returned [UploadPipelineFinishedTask], which set status = SUCCESS - so pipeline_success_task's status == PROCESSING guard then made it a no-op, and UploadPipeline.success() never ran.

That is the only place that records processing_seconds_*, fires the import_*_success event and calls remove_processing_files(). On this dev box, 156 of 178 SUCCESS pipelines have processing_seconds_wall_time IS NULL - all of them factories inheriting that finish list (Liftover, Manual Variant Entry, insert-variants-only, Variant Tags, TSO500). The 22 with timings are the three factories that don't (ClinVar, Patient Records, genotype VCF), which is also the proof that pipeline_success_task closes a pipeline correctly on its own.

So yes to the question in the issue comment: the two were duplicates and UploadPipelineFinishedTask was the redundant one. It is deleted; pipeline_success_task is now the single closer and keeps its guard (FINISH can legitimately be scheduled twice, and the guard is what makes the second run a no-op).

Everything else that leaked

• Deleting a pipeline row left its directory. New post_delete receiver on UploadPipeline, following the AnnotationRun pattern from Variant Annotation and disk space #1670 - unconditional, since with the row gone nothing can name the directory. Registering a receiver also disables Django's fast-delete path, so it fires on FileUpload/VCF cascades too.
• Generated input VCFs. Liftover, manual variant entry and classification import write a VCF into a scratch dir of its own and point FileUpload.path at it; nothing removed it. New FileUpload.is_import_processing_scratch + UploadPipeline.remove_generated_input_file(), called from success() (under the setting) and the receiver (unconditional). retry_upload_pipeline still only removes the pipeline's own dir, so a retry keeps its input.
• get_import_processing_dir() created the dir on every call. Split into import_processing_dir_path (pure) and get_import_processing_dir (creates), plus remove_import_processing_dir. ClinGenAlleleRegistryAPI.__init__ computed a failure-dump path eagerly, minting an empty clingen_allele_registry_<uuid> dir per instance - now worked out inside the except branch, into one shared clingen_allele_registry_failures/.
• Management commands. gene_annotation and human_protein_atlas_import now remove their COPY CSV; gene coverage cleanup moves off not settings.DEBUG onto IMPORT_PROCESSING_DELETE_TEMP_FILES_ON_SUCCESS.
• Tests wrote into the real tree. IMPORT_PROCESSING_DIR points at a temp dir under UNIT_TEST, removed by VariantGridTestRunner.teardown_test_environment; fake_annotation builds its per-test dir inside it.

Sweep for existing deployments

manage.py import_processing_cleanup [--dry-run] [--min-age-days N] walks the top level of IMPORT_PROCESSING_DIR and removes only entries whose owner is finished with them (row gone, or SUCCESS/FINISHED). A PROCESSING pipeline, an ERROR annotation run, a non-empty ClinGen dump dir, anything modified within --min-age-days, and any unrecognised name are reported and left alone. Registered as a ManualOperation in upload/migrations/0042_one_off_import_processing_cleanup.py, gated on the directory being non-empty.

Sample dry run on this box:

Would remove:
  annotation_run: 55 (1634.0 MB)
  clingen_allele_registry: 506 (0.0 MB)
  liftover: 10 (0.0 MB)
  pipeline: 29 (915.5 MB)
Kept:
  clingen_allele_registry (holds ClinGen API failure dumps): 7
  gene_coverage (gene coverage collections clean up their own dirs): 1
  liftover (owner not finished): 5

Notes

• No data migration for the removed task: UploadStep rows naming UploadPipelineFinishedTask belong to finished pipelines and are never re-launched (schedule_pipeline_stage_steps filters start_date__isnull=True); a retry rebuilds its steps from the factory.
• Trade-off accepted: "Retry import" on an already successful liftover / manual entry / classification import is no longer available, since its generated input is gone. ERROR pipelines keep everything, which is the case retry exists for, and view_upload_pipeline already handles a missing file.
• Full suite passes (3172 tests). New tests: upload/tests/test_pipeline_cleanup.py, upload/tests/test_import_processing_cleanup.py.
• Plan claude/plans/928_import_processing_cleanup_plan.md deleted; its knowledge is now in claude/research/upload.md and upload/CLAUDE.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F1Rk7aXMC6RFc6oouK6Hck