Skip to content

diagnose: the error argument is sent as epsilon; the output directory is never removed #871

Open

Description

@atarutin

Summary

CmdStanModel.diagnose (cmdstanpy develop at the time of writing, also 1.3.0) has two bugs:

  1. The error argument is sent as epsilon. CmdStan never receives error=, and when both arguments are given, epsilon appears twice on the command line.
  2. Each call leaves a temporary output directory under _TMPDIR, removed only at interpreter exit. This is the same pattern as log_prob leaves a temporary output directory per call until interpreter exit #867, which is being fixed for log_prob in Remove log_prob's temporary output directory when the call returns #869.

Where

cmdstanpy/model.py, CmdStanModel.diagnose, on develop:

if epsilon is not None:
    cmd.append(f"epsilon={epsilon}")
if error is not None:
    cmd.append(f"epsilon={error}")      # should be f"error={error}"
...
output_dir = tempfile.mkdtemp(prefix=self.name, dir=_TMPDIR)   # never removed

CmdStan 2.40.0's argument tree accepts both sub-arguments under diagnose test=gradient (output of <model> diagnose help-all):

gradient
  Valid subarguments: epsilon, error
  epsilon=<double>   Finite difference step size     Defaults to 1e-6
  error=<double>     Error threshold                 Defaults to 1e-6

Effect

  • diagnose(error=x) silently sets the finite-difference step to x, and the error threshold stays at its default 1e-6. The gradient check then judges against a threshold the caller did not ask for, using a step size the caller did not ask for.
  • diagnose(epsilon=a, error=b) passes epsilon=a epsilon=b.
  • The output directory accumulates in long-running processes, as described in log_prob leaves a temporary output directory per call until interpreter exit #867.

Suggested fix

  • Send f"error={error}".
  • Scope the output directory with tempfile.TemporaryDirectory(prefix=self.name, dir=_TMPDIR), the approach agreed on Remove log_prob's temporary output directory when the call returns #869.
  • Add a test asserting that the command carries error= (the logged command, as the log_prob tests do) and that _TMPDIR's entry count is unchanged after calls on both the success path and the failure path.

We can fold this into #869, or open a separate PR, whichever you prefer.

Versions

cmdstanpy develop (also 1.3.0), CmdStan 2.40.0, Python 3.12, Linux x86_64.

Activity

  1. WardBrian commented on Oct 9, 2026

    @WardBrian
    Member

    This would be welcome as a second PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions