Skip to content

Fix loo_compare() for subsampling results #391

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
florence-bockting wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Fix loo_compare() for subsampling results #391

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c0155fc
fix: correct elpd_diff sign and return a data.frame in loo_compare() …
Sep 2, 2026
0aad462
fix: correct elpd_diff sign in loo_compare() for subsampled loo objects
Sep 2, 2026
92c2a92
resolve merge conflics
Sep 2, 2026
798f266
tests: update tests for matrix output structure
Sep 2, 2026
8635a88
style: remove empty line in NEWS.md
Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions NEWS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# loo (development version)

* Fix `loo_compare()` when used with subsampling: compute model comparison by comparison-model minus reference-model by @florence-bockting in #391
* Update user messages in `print()` by @ishaan-arora-1, @florence-bockting in
#328.

Expand Down
15 changes: 7 additions & 8 deletions R/loo_compare.psis_loo_ss_list.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,13 +86,13 @@ loo_compare_ss_naive <- function(ref_loo, compare_loo){
checkmate::assert_class(ref_loo[[1]], "psis_loo_ss")
checkmate::assert_class(compare_loo[[1]], "psis_loo_ss")

elpd_loo_diff <- ref_loo[[1]]$estimates["elpd_loo","Estimate"] - compare_loo[[1]]$estimates["elpd_loo","Estimate"]
elpd_loo_diff <- compare_loo[[1]]$estimates["elpd_loo", "Estimate"] - ref_loo[[1]]$estimates["elpd_loo", "Estimate"]
elpd_loo_diff_se <- sqrt(
(ref_loo[[1]]$estimates["elpd_loo","SE"])^2 +
(compare_loo[[1]]$estimates["elpd_loo","SE"])^2)
(ref_loo[[1]]$estimates["elpd_loo", "SE"])^2 +
(compare_loo[[1]]$estimates["elpd_loo", "SE"])^2)
elpd_loo_diff_subsampling_se <- sqrt(
(ref_loo[[1]]$estimates["elpd_loo","subsampling SE"])^2 +
(compare_loo[[1]]$estimates["elpd_loo","subsampling SE"])^2)
(ref_loo[[1]]$estimates["elpd_loo", "subsampling SE"])^2 +
(compare_loo[[1]]$estimates["elpd_loo", "subsampling SE"])^2)

c(elpd_loo_diff, elpd_loo_diff_se, elpd_loo_diff_subsampling_se)
}
Expand All @@ -112,8 +112,8 @@ loo_compare_ss_diff <- function(ref_loo, compare_loo){
checkmate::assert_true(ref_loo[[1]]$loo_subsampling$loo_approximation != "none")
checkmate::assert_true(compare_loo[[1]]$loo_subsampling$loo_approximation != "none")

diff_approx <- ref_loo[[1]]$loo_subsampling$elpd_loo_approx - compare_loo[[1]]$loo_subsampling$elpd_loo_approx
diff_sample <- ref_loo[[1]]$pointwise[,"elpd_loo"] - compare_loo[[1]]$pointwise[,"elpd_loo"]
diff_approx <- compare_loo[[1]]$loo_subsampling$elpd_loo_approx - ref_loo[[1]]$loo_subsampling$elpd_loo_approx
diff_sample <- compare_loo[[1]]$pointwise[,"elpd_loo"] - ref_loo[[1]]$pointwise[,"elpd_loo"]
est <- srs_diff_est(diff_approx, y = diff_sample, y_idx = ref_loo[[1]]$pointwise[,"idx"])

elpd_loo_diff <- est$y_hat
Expand Down Expand Up @@ -182,7 +182,6 @@ print.compare.loo_ss <- function(x, ..., digits = 1) {
invisible(x)
}


#' Compute comparison matrix for `psis_loo_ss` objects
#' @noRd
#' @keywords internal
Expand Down
16 changes: 8 additions & 8 deletions tests/testthat/_snaps/loo_subsampling_cases.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,33 +105,33 @@
print(comp)
Output
elpd_diff se_diff subsampling_se_diff
model2 0.0 0.0 0.0
model1 16.5 22.5 0.4
model2 0.0 0.0 0.0
model1 -16.5 22.5 0.4

---

Code
print(comp)
Output
elpd_diff se_diff subsampling_se_diff
model2 0.0 0.0 0.0
model1 16.1 4.4 0.1
model2 0.0 0.0 0.0
model1 -16.1 4.4 0.1

---

Code
print(comp2)
Output
elpd_diff se_diff subsampling_se_diff
model2 0.0 0.0 0.0
model1 16.3 4.4 0.1
model2 0.0 0.0 0.0
model1 -16.3 4.4 0.1

---

Code
print(comp3)
Output
elpd_diff se_diff subsampling_se_diff
model2 0.0 0.0 0.0
model1 16.5 4.4 0.3
model2 0.0 0.0 0.0
model1 -16.5 4.4 0.3

5 changes: 5 additions & 0 deletions tests/testthat/test_loo_subsampling.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1071,6 +1071,11 @@ test_that("loo_compare_subsample", {
expect_equal(lcssohhapi, lcssohh)
expect_silent(lcss2mapi <- loo_compare(lss2o1, lss3o1))
expect_equal(lcss2mapi, lcss2m)
# check that comparison is comp - ref model (i.e., elpd_diff is neg.)
for (m in list(lcss, lcss2, lcssohh)) {
expect_lt(m[2, "elpd_diff"], 0)
expect_lt(m[3, "elpd_diff"], 0)
}

@jgabry jgabry Sep 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not strictly necessary, but maybe we should also add a test like this, since I don't think we have anything like it? It tests both the sign and the magnitude at the same time:

# subsampling should agree with non-subsampling when nothing
# is actually subsampled (pins the direction *and* the magnitude)
l1ss <- loo:::as.psis_loo_ss.psis_loo(l1)
l2ss <- loo:::as.psis_loo_ss.psis_loo(l2)
l3ss <- loo:::as.psis_loo_ss.psis_loo(l3)
expect_equal(
  unname(loo_compare(l1ss, l2ss, l3ss)[, "elpd_diff"]),
  loo_compare(l1, l2, l3)$elpd_diff
)

The first argument to expect_equal() could then be simplified with the data.frame conversion in the other PR.

})

test_that("Test 'tis' and 'sis'", {
Expand Down
34 changes: 17 additions & 17 deletions vignettes/loo2-large-data.Rmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -481,9 +481,9 @@ print(comp)
```
Warning: Different subsamples in 'model2' and 'model1'. Naive diff SE is used.

elpd_diff se_diff subsampling_se_diff
model2 0.0 0.0 0.0
model1 16.5 22.5 0.4
elpd_diff se_diff subsampling_se_diff
model2 0.0 0.0 0.0
model1 -16.5 22.5 0.4
```

This new object `comp` contains the estimated difference of expected
Expand Down Expand Up @@ -538,23 +538,23 @@ subsampled observations.

```{r, eval=FALSE}
comp <- loo_compare(loo_ss_1, loo_ss_2)
print(comp)
print(comp)
```

```
elpd_diff se_diff subsampling_se_diff
model2 0.0 0.0 0.0
model1 16.1 4.4 0.1
elpd_diff se_diff subsampling_se_diff
model2 0.0 0.0 0.0
model1 -16.1 4.4 0.1
```

First, notice that now the `se_diff` is now around 4 (as opposed to 20 when using
different subsamples). The first column shows the difference in ELPD relative to
the model with the largest ELPD. In this case, the difference in `elpd` and its
scale relative to the approximate standard error of the difference) indicates a
preference for the second model (`model2`). Since the subsampling uncertainty is
so small in this case it can effectively be ignored. If we need larger
subsamples we can simply add samples using the `update()` method demonstrated
earlier.
different subsamples). The `elpd_diff` column shows the difference in ELPD
relative to the model with the largest ELPD. In this case, the difference in
`elpd` and its scale relative to the approximate standard error of the
difference) indicates a preference for the second model (`model2`).
Since the subsampling uncertainty is so small in this case it can effectively
be ignored. If we need larger subsamples we can simply add samples using the
`update()` method demonstrated earlier.

It is also possible to compare a subsampled loo computation with a full loo object.

Expand All @@ -580,9 +580,9 @@ the loo calculations for both `model1` and `model2` are included in the
computations for the comparison.

```
elpd_diff se_diff subsampling_se_diff
model2 0.0 0.0 0.0
model1 16.3 4.4 0.3
elpd_diff se_diff subsampling_se_diff
model2 0.0 0.0 0.0
model1 -16.3 4.4 0.3
```

Here we actually see an increase in `subsampling_se_diff`, but this is due to a
Expand Down
Loading