Add Logistic PCA (LPCA) analysis

[Jeebjean]

Summary

Adds Logistic PCA (LPCA) as a new SPRAS analysis method. LPCA is the binary-data counterpart to the existing PCA analysis. It is designed for binary matrices (0/1) like the edge-by-run matrix built from pathway reconstruction outputs, whereas standard PCA assumes continuous data.

Changes

New config block

analysis.lpca with five options:

• include: enable/disable LPCA (default false)
• k: number of principal components (default 2)
• m: logisticPCA tuning parameter, used when cv is false (default 6)
• cv: if true, determine m by cross-validation; if false, use fixed m (default false)
• transpose: if false, run LPCA on edges x runs matrix (one embedding per edge);
  if true, run on runs x edges matrix (one embedding per pathway run,
  mirrors the classic PCA analysis) (default false)

New Snakemake rule lpca_analysis

• Restricted to algorithms with multiple parameter combinations (same guard as per-algorithm PCA)
• Outputs per algorithm: {algorithm}-lpca-scores.csv, {algorithm}-lpca.png, {algorithm}-lpca-coordinates.txt

New module spras/analysis/lpca.py

• Builds the binary matrix via summarize_networks
• Optionally transposes the matrix
• Runs the logisticPCA container via run_container_and_log
• Generates a PC1 vs PC2 scatterplot and saves coordinates

New Docker wrapper docker-wrappers/lpca/

• Pinned rocker/r-base:4.4.2 Dockerfile installing logisticPCA and ggplot2
• run_lpca.R: runs LPCA and saves scores + deviance explained
• run_cv.R: cross-validation to find optimal m
• README documenting config options and Docker image publication

Unit tests test/analysis/test_lpca.py

• 3 tests: output file existence, score shape (edges x k), score shape (runs x k with transpose)

Docker timeout

• Increased from 60s to 600s in containers.py to handle larger binary matrices

Testing locally

Build the image and run the pipeline:

docker build -t reedcompbio/lpca:v1 docker-wrappers/lpca/

Enable analysis.lpca.include: true in your config and run as usual.
Tested end-to-end with OmicsIntegrator2 (216 combos) and MincostFlow (14 combos).

Before / at merge

The image must be published to reedcompbio/lpca:v1 on Docker Hub by a maintainer:

docker build -t reedcompbio/lpca:v1 docker-wrappers/lpca/
docker push reedcompbio/lpca:v1

cc @agitter @ntalluri

[read-the-docs-community]

Documentation build overview

📚 spras | 🛠️ Build #33840946 | 📁 Comparing 5b64162 against latest (1cb9d03)

  🔍 Preview build  

3 files changed

± genindex.html
± fordevs/spras.config.html
± fordevs/spras.html

[agitter]

Note that this will close #271 when merged.

I pushed the image to DockerHub: https://hub.docker.com/r/reedcompbio/lpca

We should add this to our Docker image building test: https://github.com/Reed-CompBio/spras/blob/main/.github/workflows/build-containers.yml

This comment should be modified to note the alternative LPCA

spras/spras/analysis/ml.py

Line 159 in 07707bf

# TODO: replace PCA https://github.com/Reed-CompBio/spras/issues/271

I didn't comment on every detail yet, but in general there are some small things we can to do make this more like the existing PCA.

[agitter]

Additional comments because I accidentally submitted the first batch before finishing.

[agitter]

I am testing locally, and my workflow is stuck at the Cytoscape step. I wonder if that is related to the longer Docker timeout and it would have quickly failed otherwise? I eventually killed Snakemake.

After disabling Cytoscape, I ran into errors at the ml_analysis step. I didn't get a specific error message though, and I don't see what changes in this pull request would have affected that.

As a reminder for myself, I should rebuild and the Docker image when I merge this.

[agitter]

Now that I can run everything locally, I have more observations about the output. Anything we can't finish today we can log as a new issue so we can merge this as-is.

I am running into errors when aggregating by algorithm. The error message indicates this is because the input binary matrix is too small:

RuleException:
ContainerError in file "spras\Snakefile", line 456:
(Command formatted as list: `['Rscript', '/app/run_lpca.R', '/app/YZQWZK5/rwr_72021389-lpca-binary-matrix.csv', '/app/VBUONQG/rwr_72021389-lpca-scores.csv', '2', '6.0']`)
An unexpected non-zero exit status (1) inside the docker image docker.io/reedcompbio/lpca:v1 occurred:
    Error in fun(A, k, nu, nv, opts, mattype = "matrix") :
      nrow(A) and ncol(A) should be at least 3
    Calls: logisticPCA -> <Anonymous> -> <Anonymous> -> svds.matrix -> fun
    Execution halted

[agitter]

Noting that the package documentations suggests that it can slow down decomposition for small datasets. I think it is still fair to say that is harmless, so I don't recommend a change. I'm noting it in case we have future problems.

Since logistic PCA requires iteratively computing the eigendecomposition, it can be slow for large data. When k is small relative to the number of columns, however, the eigendecomposition can be sped up by only solving for the first k eigenvectors. This can be done for all three formulations using the partial_decomp argument. We have implemented this using the RSpectra package. Be careful when using this argument, because it can substantially slow down computation is the number of columns is small (e.g. ~20) or if k is of comparable size to the number of columns. An example of the speed-up is below.

[agitter]

Extra linebreak here

[agitter]

I'm now able to inspect the outputs of LPCA. Does it only write a scalar deviance explained unlike PCA, which provides variance per dimension? Does this account for the k requested or is it the full dimensionality?

LPCA deviance file:

0.996362653717224

PCA variance file:

PC1: 86.36363636
PC2: 13.63636364

[agitter]

What information does the coordinates file provide that is not already in the scores file? It looks redundant, but I may be missing something.

[agitter]

This can be Inf, such as data0 pathlinker_72021389-lpca-scores_deviance.txt in the example config. PCA gives nan PCs on that input.

We may want to create an issue noting this to fix both of them later.

[agitter]

We should probably do some value checking for these arguments like PCA does

    if components < 2:
        raise ValueError(f"components={components} must be greater than or equal to 2 in the config file.")

[agitter]

The LPCA annotations overlap when the coordinates are identical. PCA attempts to spread out the labels.