DArray: Add sparse matrix support and solvers and preconditioners

[jpsamaroo]

Adds a DSparseMatrix type for sparse DArray tiles, that allows in-place modifications, necessary for Datadeps algorithms. Then this builds algorithms like GEMM, GEMV, iterative solvers, and preconditioners on top that allow the DArray to work with sparse matrix and vector data effectively.

Written by Claude Opus

[github-actions]

Dagger benchmarks: dirty vs master

	master	dirty	master / dirty
array/dagger/N=1024 (block 512)/add (X + X)	3.94 ± 0.4 ms	4.5 ± 0.56 ms	0.876 ± 0.14
array/dagger/N=1024 (block 512)/alloc (rand)	3.33 ± 0.11 ms	3.6 ± 0.14 ms	0.925 ± 0.046
array/dagger/N=1024 (block 512)/broadcast (X .+ 1)	2.66 ± 0.24 ms	2.84 ± 0.31 ms	0.939 ± 0.13
array/dagger/N=1024 (block 512)/map (sin.(X))	7.09 ± 1.8 ms	7.85 ± 0.45 ms	0.904 ± 0.24
array/dagger/N=1024 (block 512)/norm	1.14 ± 0.077 ms	1.09 ± 0.064 ms	1.05 ± 0.093
array/dagger/N=1024 (block 512)/reduce (sum)	3.02 ± 1.6 ms	2.6 ± 1.8 ms	1.16 ± 1
array/dagger/N=1024 (block 512)/transpose (permutedims)	7.5 ± 0.4 ms	5.65 ± 0.46 ms	1.33 ± 0.13
array/dagger/N=256 (block 256)/add (X + X)	0.985 ± 0.31 ms	1.04 ± 0.069 ms	0.951 ± 0.31
array/dagger/N=256 (block 256)/alloc (rand)	0.935 ± 0.28 ms	1.24 ± 0.031 ms	0.752 ± 0.22
array/dagger/N=256 (block 256)/broadcast (X .+ 1)	1.36 ± 2.5 ms	0.746 ± 0.11 ms	1.83 ± 3.3
array/dagger/N=256 (block 256)/map (sin.(X))	1.1 ± 0.037 ms	1.36 ± 0.2 ms	0.808 ± 0.12
array/dagger/N=256 (block 256)/norm	0.527 ± 0.016 ms	0.649 ± 0.079 ms	0.812 ± 0.1
array/dagger/N=256 (block 256)/reduce (sum)	1.22 ± 1.6 ms
array/dagger/N=256 (block 256)/transpose (permutedims)	1.08 ± 0.099 ms	1.16 ± 0.091 ms	0.932 ± 0.11
linalg/dagger/N=1024 (block 512)/cholesky	19.7 ± 1.4 ms	21.5 ± 4.8 ms	0.917 ± 0.21
linalg/dagger/N=1024 (block 512)/lu	0.0498 ± 0.0017 s	0.048 ± 0.00069 s	1.04 ± 0.038
linalg/dagger/N=1024 (block 512)/matmul (A*A)	0.0552 ± 0.013 s	0.0741 ± 0.00064 s	0.745 ± 0.17
linalg/dagger/N=1024 (block 512)/matvec (A*x)	2.78 ± 0.28 ms	2.92 ± 0.27 ms	0.952 ± 0.13
linalg/dagger/N=1024 (block 512)/qr	0.107 ± 0.0043 s	0.11 ± 0.0028 s	0.971 ± 0.046
linalg/dagger/N=1024 (block 512)/solve (A\b via lu)	0.0505 ± 0.0042 s	0.0543 ± 0.0083 s	0.93 ± 0.16
linalg/dagger/N=1024 (block 512)/svd	0.0395 h	0.0407 h	0.971
linalg/dagger/N=1024 (block 512)/syrk (A'*A)	0.0439 ± 0.0091 s	0.0642 ± 0.0037 s	0.683 ± 0.15
linalg/dagger/N=256 (block 256)/cholesky	3.16 ± 0.25 ms	2.69 ± 1.8 ms	1.18 ± 0.78
linalg/dagger/N=256 (block 256)/lu	4.44 ± 0.86 ms	5.14 ± 0.38 ms	0.864 ± 0.18
linalg/dagger/N=256 (block 256)/matmul (A*A)	2.03 ± 0.68 ms	3.25 ± 0.093 ms	0.626 ± 0.21
linalg/dagger/N=256 (block 256)/matvec (A*x)	1.29 ± 0.047 ms	1.35 ± 0.81 ms	0.953 ± 0.57
linalg/dagger/N=256 (block 256)/qr	5.01 ± 0.25 ms	5.58 ± 0.64 ms	0.899 ± 0.11
linalg/dagger/N=256 (block 256)/solve (A\b via lu)	9.23 ± 2.4 ms	10.1 ± 4.9 ms	0.911 ± 0.5
linalg/dagger/N=256 (block 256)/svd	0.332 ± 0.09 s	0.393 ± 0.17 s	0.846 ± 0.44
linalg/dagger/N=256 (block 256)/syrk (A'*A)	3.21 ± 0.22 ms	4.9 ± 0.32 ms	0.654 ± 0.062
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap)	9.76 ± 1.1 ms	9.68 ± 0.6 ms	1.01 ± 0.13
stencil/dagger/N=1024 (block 512)/assign (const)	1.26 ± 0.035 ms	1.58 ± 0.88 ms	0.797 ± 0.44
stencil/dagger/N=1024 (block 512)/multi-expr	5.17 ± 1.7 ms	3.17 ± 1.1 ms	1.63 ± 0.78
stencil/dagger/N=1024 (block 512)/neighbors (Clamp)	6.45 ± 0.99 ms	6.84 ± 0.23 ms	0.942 ± 0.15
stencil/dagger/N=1024 (block 512)/neighbors (Pad)	8.55 ± 2.7 ms	7.48 ± 0.38 ms	1.14 ± 0.37
stencil/dagger/N=1024 (block 512)/neighbors (Reflect)	7.28 ± 0.62 ms	7.55 ± 0.8 ms	0.965 ± 0.13
stencil/dagger/N=1024 (block 512)/neighbors (Wrap)	7.49 ± 0.89 ms	7.45 ± 0.65 ms	1 ± 0.15
stencil/dagger/N=1024 (block 512)/update (+)	1.79 ± 0.39 ms	2.19 ± 0.72 ms	0.817 ± 0.32
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap)	1.96 ± 0.11 ms	1.89 ± 0.049 ms	1.04 ± 0.064
stencil/dagger/N=256 (block 256)/assign (const)	0.737 ± 0.22 ms	0.728 ± 0.66 ms	1.01 ± 0.96
stencil/dagger/N=256 (block 256)/multi-expr	1.31 ± 0.24 ms	1.32 ± 0.24 ms	0.998 ± 0.26
stencil/dagger/N=256 (block 256)/neighbors (Clamp)	1.53 ± 0.049 ms	1.51 ± 0.064 ms	1.02 ± 0.054
stencil/dagger/N=256 (block 256)/neighbors (Pad)	1.66 ± 0.3 ms	1.61 ± 0.25 ms	1.03 ± 0.25
stencil/dagger/N=256 (block 256)/neighbors (Reflect)	1.61 ± 0.073 ms	1.61 ± 0.5 ms	0.996 ± 0.31
stencil/dagger/N=256 (block 256)/neighbors (Wrap)	1.59 ± 0.14 ms	1.99 ± 1.2 ms	0.8 ± 0.47
stencil/dagger/N=256 (block 256)/update (+)	0.741 ± 0.039 ms	0.86 ± 0.19 ms	0.862 ± 0.2
sparse/dagger/N=1024 (block 64)/cg solve (laplacian)		1.33 ± 0.013 s
sparse/dagger/N=256 (block 16)/cg solve (laplacian)		1.35 ± 0.04 s
sparse/dagger/N=256 (block 16)/spmv (S*x)		0.0949 ± 0.0042 s
sparse/dagger/N=1024 (block 64)/spgemm (S*S)		1.18 ± 0.028 s
sparse/dagger/N=256 (block 16)/spgemm (S*S)		1.19 ± 0.01 s
sparse/dagger/N=1024 (block 64)/spmv (S*x)		0.0886 ± 0.0044 s
time_to_load	1.05 ± 0.0017 s	1.09 ± 0.0031 s	0.962 ± 0.0032

⚠️ Regressions (> 25.0%)

• linalg/dagger/N=256 (block 256)/matmul (A*A): +59.7%
• linalg/dagger/N=256 (block 256)/syrk (A'*A): +52.8%
• linalg/dagger/N=1024 (block 512)/syrk (A'*A): +46.4%
• linalg/dagger/N=1024 (block 512)/matmul (A*A): +34.1%
• array/dagger/N=256 (block 256)/alloc (rand): +33.0%
• stencil/dagger/N=1024 (block 512)/assign (const): +25.5%

Improvements (> 25.0% faster)

• array/dagger/N=256 (block 256)/broadcast (X .+ 1): -45.3%
• stencil/dagger/N=1024 (block 512)/multi-expr: -38.8%

Full results and plots (download the benchmark-results artifact).