Define how to run tests

[furtib]

Why:
The requirements for running tests are not well defined.
This patch aims to define which targets should be run for testing.
This patch does not define how the user should set up their environment, only how the tests should be run. That is a todo for the next patch.

What:

• Added a script that runs all checks.
• Modified the CONTRIBUTING.md to specify which tests must be run

Addresses:
none?

[nettle]

Good start! Thanks @furtib!

[furtib]

I have investigated the possibility of using the bazel module: https://github.com/bazel-contrib/toolchains_llvm
And I found it to be surprisingly doable. I don't even think the big download size is a huge blocker, since it won't redownload it even after a bazel clean.
With that said, it introduces the complexity of whether we require all users to use it or not. (How do we test both methods if we do not require it?)

So instead, I tried this different approach, setting up the environment "manually".
I try not to set any dependency that we need to vary, like Bazel; I'm still figuring out how to handle that.
I chose this because it produces an environment that I think is more easily reproducible to end users compared to what Micromamba provides.

[nettle]

This looks to me as introduction of venv + llvm environment :)
It should go under .ci/ as yet another dev env along with mise and micromamba.
And it should be a separate review, not a part of "Define how to run tests".

[furtib]

Fair!
I will strip this back to just the script.

That raises some questions about whether the file test/test.sh is still being used in CI workflows, or if I can remove that entirely.

[nettle]

Fair! I will strip this back to just the script.

Actually I think venv or uv could be useful :)
But again as separate PRs

That raises some questions about whether the file test/test.sh is still being used in CI workflows, or if I can remove that entirely.

To be honest just adding such script does not bring much value without defining the environment I think.
Without defining reliable environment (micromamba, mise, venv, uv or something else)
we cannot say we have answer to the question "how to run tests?"

[nettle]

I have investigated the possibility of using the bazel module: https://github.com/bazel-contrib/toolchains_llvm And I found it to be surprisingly doable. I don't even think the big download size is a huge blocker, since it won't redownload it even after a bazel clean. With that said, it introduces the complexity of whether we require all users to use it or not. (How do we test both methods if we do not require it?)

This is actually interesting!
Can you please do a prototype? (as a separate PR)

So instead, I tried this different approach, setting up the environment "manually". I try not to set any dependency that we need to vary, like Bazel; I'm still figuring out how to handle that. I chose this because it produces an environment that I think is more easily reproducible to end users compared to what Micromamba provides.

Well, setting up the environment "manually" sounds extremely unreliable.
To make sure we have "common grounds" for tests the setup must be automated.
But of course it starts from describing "manual" steps and conditions.

[Szelethus]

I understand that this patch is now beyond a simple "single point of entry" test script.

Lets take a step back, I fear some misunderstandings have to be beaten into my head. The end goal (browsing through our last meeting notes) is the following. We should consider a PR ready to merge (after reviewer approval(s)) if:

• Tests pass locally
• Github CI tests pass
• Jenkins job passes

And of course it would be great if those CI environments could be replicated and run locally with relative ease, with mise/micromamba. What can we do today that isn't a hack and gets us further towards this end goal? I suggest that

1. First, lets finish the discussion on Recommended requirements #219 and land it. This PR is somewhat, but not entirely orthogonal to that.
2. We all agree that we should define the set of commands that needs to be run on each patch. We also seem to mostly agree what those commands should be (currently in this PR's run_tests.sh). Lets create a PR that either lands that script bare-bones without any environmental setup, or simply lists those commands in CONTRIBUTING.
3. Proceed with the discussion on the definition of testing environment.

Let's land and finish up the few, small steps we agreed upon, so we don't have so many threads open.

[nettle]

I understand that this patch is now beyond a simple "single point of entry" test script.

It was never been "a simple single point of entry test script" to me :)
The intention is to define how exactly we set the environment (environment manager, tools, versions etc) so that all tests pass always?

Lets take a step back, I fear some misunderstandings have to be beaten into my head. The end goal (browsing through our last meeting notes) is the following. We should consider a PR ready to merge (after reviewer approval(s)) if:

• Tests pass locally
• Github CI tests pass
• Jenkins job passes

And of course it would be great if those CI environments could be replicated and run locally with relative ease, with mise/micromamba.

Exactly! That's the goal of this task - how to setup the environment? what tests to run? how?
And all of these must be the same for any local environment and CI.
The results must be repeatable always!

What can we do today that isn't a hack and gets us further towards this end goal?
I suggest that

1. First, lets finish the discussion on Recommended requirements #219 and land it. This PR is somewhat, but not entirely orthogonal to that.

Yes, that we should do. However, #219 is not a showstopper for this task.

2. We all agree that we should define the set of commands that needs to be run on each patch. We also seem to mostly agree what those commands should be (currently in this PR's run_tests.sh). Lets create a PR that either lands that script bare-bones without any environmental setup, or simply lists those commands in CONTRIBUTING.

No, this does not help much without setting up reliable environment.
To me such scripts (run_tests.sh and similar) are completely useless if they fail when running in unpredictable environment.

3. Proceed with the discussion on the definition of testing environment.

I'm sure this is the main thing we need to do here.
We have some options - micromamba, mise, venv? uv? docker?
Unfortunately, we cannot "equalize" the environment for "standard" OSs like Ubuntu and RedHat, we discussed that.
Unfortunately, docker is also unavailable in E2C/VDI RHEL.
So... then, micromamba, mise, venv? uv? something else?

Let's land and finish up the few, small steps we agreed upon, so we don't have so many threads open.

I would suggest addressing the main problem - reliable environment.
That's the only serious problem here :)

[Szelethus]

We chatted with @furtib a bunch, and I think it'd be best to pursue micromamba! We'll come back with some more concrete ideas once time allows.

[nettle]

I think dev is actually $ENV_NAME

[nettle]

What I think is that run_tests.sh is actually micromamba dependent, so maybe we should keep it as .ci/micromamba/run_tests.sh ?

[nettle]

Hm... Are you sure this is safe? There reason might be running session. Maybe proper deactivation may help?