1.3.3

[dosorio]

This pull request introduces several performance, memory, and usability improvements across the SCORPION package, especially targeting parallel processing, memory usage, and normalization routines. It also adds support for the optional mori package to optimize parallel data sharing, improves resource cleanup, and refines core statistical calculations.

Parallelization and memory management improvements:

• In runSCORPION.R, gene expression matrices are pre-split by group before parallel processing, reducing memory footprint and serialization overhead. The optional mori package is supported for OS-backed shared memory, allowing efficient sharing of large data frames (tfMotifs, ppiNet) across worker processes. The future.globals.maxSize option is set to Inf during parallel execution to allow large objects to be exported. [1] [2] [3]
• In testEdges.R, the future.globals.maxSize option is also set to Inf during parallel execution, and is properly restored after processing. [1] [2]
• Memory is aggressively freed throughout the codebase by removing large intermediate objects as soon as they're no longer needed, e.g., in runSCORPION.R, makeSuperCells.R, and runPANDA.R. [1] [2] [3] [4] [5] [6] [7]

Performance and algorithmic improvements:

• The normalization routines in normalizeNetwork.R, fastCorrelation.R, and tanimotoSimilarity.R have been refactored to avoid creating large temporary matrices, replacing full-matrix operations with column-wise loops. This reduces peak memory usage and improves speed, especially for large datasets. [1] [2] [3] [4] [5]
• The iterative update steps in runPANDA.R have been streamlined for clarity and efficiency, and now use direct updates rather than storing intermediate variables.

Batch correction and usability:

• The batch correction function in removeBatch.R now robustly handles singleton and collinear batches by using the Moore-Penrose pseudo-inverse when necessary, preventing errors due to singular design matrices. It also skips correction if there is only one batch level.

Statistical calculation refinements:

• In testEdges.R, the calculation of log2 fold change for two-sample tests now follows the limma convention, using the difference of group means on log2-scale input, and the output data frame omits the redundant diffMean column. [1] [2] [3]

Other improvements:

• The package version is bumped to 1.3.3, and the mori package is added to Suggests in the DESCRIPTION file. The .Rbuildignore is updated to ignore the tests/ directory. [1] [2] [3]
• BLAS threading is explicitly controlled via RhpcBLASctl in both runSCORPION.R and scorpion.R to respect the nCores parameter. [1] [2]

These changes collectively improve the speed, robustness, and scalability of SCORPION for large single-cell datasets.

---

References:
[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25]

[Copilot]

Pull request overview

This PR bumps SCORPION to v1.3.3 and focuses on reducing memory overhead and improving performance, primarily by refining parallel execution paths and avoiding large temporary matrix allocations in core math routines.

Changes:

• Optimize parallel network construction and memory usage in runSCORPION() (pre-splitting expression matrices, optional mori shared-memory broadcasting, more aggressive cleanup).
• Refactor normalization/similarity/correlation routines to reduce peak memory by avoiding full-size outer() temporaries.
• Refine testEdges() behavior/documentation (log2FoldChange semantics, parallel maxSize handling) and add tests around mori.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/testthat/test_mori_sharing.R	Adds tests validating optional mori shared-memory behavior and parity with non-mori execution.
tests/testthat/test_edge_statistics.R	Updates expectations to use log2FoldChange instead of diffMean.
README.md	Updates documented output columns for testEdges() results.
R/testEdges.R	Adjusts parallel execution options cleanup and updates log2 fold-change calculation/output columns.
R/tanimotoSimilarity.R	Reworks normalization to avoid allocating large outer() temporaries.
R/scorpion.R	Controls BLAS threading via RhpcBLASctl to respect nCores.
R/runSCORPION.R	Pre-splits GEX by group for parallelism, adds optional mori sharing, and streams weight extraction to reduce memory pressure.
R/runPANDA.R	Streamlines iterative updates and frees intermediates to reduce memory.
R/removeBatch.R	Adds pseudo-inverse fallback for rank-deficient batch design matrices.
R/normalizeNetwork.R	Refactors normalization to avoid full-matrix broadcast temporaries.
R/makeSuperCells.R	Frees large intermediate objects earlier to reduce memory footprint.
R/fastCorrelation.R	Avoids full-size outer product in denominator by doing column-wise normalization.
man/testEdges.Rd	Updates Rd docs to match testEdges() output changes.
DESCRIPTION	Bumps version and adds mori to Suggests.
.Rbuildignore	Adds tests/* ignore pattern.

Files not reviewed (1)

• man/testEdges.Rd: Generated file

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.