What happened
loo_compare() returns elpd_diff with the opposite sign for subsampled
psis_loo_ss objects. It also returns a matrix, not a data.frame.
What I expected
In loo_compare() the difference is comparison-model minus reference-model,
and the output is a data.frame.
Minimal reproducible example
library(loo)
set.seed(4711)
N <- 500
S <- 500
data <- data.frame(y = rnorm(N, mean = 0, sd = 1))
# two models: model1 is well centred, model2 is biased by 0.3
draws1 <- matrix(rnorm(S, 0.0, 0.05), ncol = 1, dimnames = list(NULL, "mu"))
draws2 <- matrix(rnorm(S, 0.3, 0.05), ncol = 1, dimnames = list(NULL, "mu"))
llfun <- function(data_i, draws, ...) {
dnorm(data_i$y, mean = draws[, "mu"], sd = 1, log = TRUE)
}
l1 <- loo(llfun, data = data, draws = draws1, r_eff = rep(1, N))
l2 <- loo(llfun, data = data, draws = draws2, r_eff = rep(1, N))
loo_compare(list(model1 = l1, model2 = l2))
set.seed(4711)
s1 <- loo_subsample(llfun, data = data, draws = draws1,
observations = 200, loo_approximation = "plpd",
r_eff = rep(1, N))
set.seed(4711)
s2 <- loo_subsample(llfun, data = data, draws = draws2,
observations = s1, loo_approximation = "plpd",
r_eff = rep(1, N))
loo_compare(list(model1 = s1, model2 = s2))Actual output
> loo_compare(list(model1 = l1, model2 = l2))
model elpd_diff se_diff p_worse diag_diff diag_elpd
model1 0.0 0.0 NA
model2 -25.8 6.6 1.00
> loo_compare(list(model1 = s1, model2 = s2))
elpd_diff se_diff subsampling_se_diff
model1 0.0 0.0 0.0
model2 25.7 6.6 0.0
The return types also differ:
full: class compare.loo/data.frame | rownames 1,2
ss : class compare.loo_ss/compare.loo/matrix/array | rownames model1,model2
Environment
- OS: Ubuntu 24.04.4 LTS
- Package version: loo 2.10.0
- R version: 4.6.1 (2026-06-24)
What happened
loo_compare()returnselpd_diffwith the opposite sign for subsampledpsis_loo_ssobjects. It also returns a matrix, not a data.frame.What I expected
In
loo_compare()the difference is comparison-model minus reference-model,and the output is a data.frame.
Minimal reproducible example
Actual output
The return types also differ:
Environment