Add JET testing on Julia 1.6 onwards

[PatrickHaecker]

No description provided.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.85%. Comparing base (206e68f) to head (c0fa41e).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #324   +/-   ##
=======================================
  Coverage   96.85%   96.85%           
=======================================
  Files           7        7           
  Lines         796      796           
=======================================
  Hits          771      771           
  Misses         25       25           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kimikage]

What specific problem is this intended to solve?
To what extent does this offer advantages over running the test locally?

I'm not opposed to using JET, but providing an explanation might make it easier to gain the support of other maintainers.

[PatrickHaecker]

What specific problem is this intended to solve?
To what extent does this offer advantages over running the test locally?

JET finds errors that only show up on code paths the tests don't run, such as calls that can throw a MethodError, undefined names, or broken convert and promote paths. Running it in CI catches these on every PR and on both lts and the current release, so we don't depend on each contributor remembering to run JET locally on the right Julia version. The cost is one test dependency and a few seconds of CI time, and with no current findings it acts as a regression guard.

Note, that 100% test coverage does not mean that JET testing is unnecessary. The 100% only mean that all lines of code are covered, it does not mean that all code paths are covered. JET also does not guarantee this, but it can, for example, find problems which happen when using other types than the tested ones.

Additionally, when using agents, they can sometimes early on detect a problem in their approach with the corresponding JET results, so you don't waste time, tokens and energy by finding problems with the LLM, which can already be found deterministically by JET.

[PatrickHaecker]

Now that 0.9 is released, I rebased the branch on master (don't get confused by my old branch name). There are no JET versions which would support Julia 1.0, so I changed minimum compatibility to Julia 1.6 for the upcoming FixedPointNumbers 0.10 release (#350).

[kimikage]

Instead of integrating the JET test into runtests.jl, how about setting it up as a separate workflow (e.g., JET.yml)?

The primary concern is the risk of ruining the new PkgEval report (which replaces the Nanosoldier Report).
In fact, I have encountered cases where the main tests for nightly builds could not be verified due to Aqua test failtures in certain packages.
Aqua is essentially something like a wrapper and doesn't directly interact with the deeper internals of Julia, so I don't think the impact of nightly changes will be particularly severe.

Of course, we can switch the test content based on the julia version.
But isn't that exactly what you wanted to avoid? (Are the code snippets specific to v1.0 and those specific to nightly essentially different?)

[PatrickHaecker]

That would be possible.

Advantage:

• independent of the regular tests, so no test dependency and therefore mostly version-agnostic

Disadvantages (which might all be the same):

• The user typically won't run the JET test
• The LLM typically won't run the JET test
• You can't as easily test different Julia versions
• One more thing to maintain

While setting it up in regular tests is some trouble initially, I made good experience with just having it. That being said, if you prefer it as a separate job, we can do that. It's always possible to add it to the tests later.

[kimikage]

The user typically won't run the JET test

Running it in CI catches these on every PR and on both lts and the current release, so we don't depend on each contributor remembering to run JET locally on the right Julia version.

🤔

[PatrickHaecker]

I meant that the chances are higher that users run the test suite on their local machine than that they run an independent JET call.
On CI chances are the same, because either way it will always be run.

[PatrickHaecker]

Anyway, let's go with #358. We can reopen #324 whenever we see a benefit.