From dde7354a39049c7b1b89a5a538d9055f37416ae4 Mon Sep 17 00:00:00 2001 From: Maksim Davydov Date: Tue, 22 Sep 2026 23:18:42 +0300 Subject: [PATCH 1/3] IGNITE-29067 [ducktests] Make the local Docker runner usable on Windows The runner did not work out of the box on Windows: - ducker-ignite build wrote the image id into docker/build/ without creating it, so a fresh checkout failed that step; - ducker-ignite up died with "failed to find the /etc/hosts entry for ducker01" under Git Bash, where MSYS rewrites the bare /etc/hosts argument into a Windows path before docker sees it. // suppresses the rewrite and Linux resolves //etc/hosts identically; - shell scripts checked out with CRLF (Git's default on Windows) failed inside the containers, which execute them from the bind-mounted checkout. A .gitattributes now pins *.sh, mvnw and ducker-ignite to LF. Separately, a checkout on a Windows drive makes node startup take ~250s instead of ~4s, because class loading stats the classpath across the filesystem boundary. That exceeds the 60s service startup timeout, so tests fail that pass in seconds from a WSL2 checkout. It is not a code bug, so README.md gains a Windows section covering the WSL2 setup, those measurements, and the two traps that follow from moving to a Linux filesystem: the image baking the host uid, and run_tests.sh reusing an already-running cluster. Verified on Windows 11 + Docker Desktop 4.91 (WSL2): clone and build inside the distro, then smoke_test.py SmokeServicesTest passes both test_ignite_start_stop (5.5s) and test_ignite_app_start_stop (15.7s) at stock timeouts. --- .gitattributes | 11 ++++ modules/ducktests/README.md | 62 ++++++++++++++++++++ modules/ducktests/tests/docker/ducker-ignite | 7 ++- 3 files changed, 79 insertions(+), 1 deletion(-) create mode 100644 .gitattributes diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000000000..93523177423d9 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,11 @@ +# Shell scripts must keep LF endings even when checked out on Windows: the +# ducktests runner bind-mounts the checkout into Linux containers and executes +# them there, where CRLF fails with "$'\r': command not found". +*.sh text eol=lf +mvnw text eol=lf +modules/ducktests/tests/docker/ducker-ignite text eol=lf +modules/ducktests/tests/docker/clean_up.sh text eol=lf + +# Windows batch files are only ever run on Windows. +*.bat text eol=crlf +*.cmd text eol=crlf diff --git a/modules/ducktests/README.md b/modules/ducktests/README.md index 4f0ccc3dde11f..8719bcbefa5ea 100644 --- a/modules/ducktests/README.md +++ b/modules/ducktests/README.md @@ -21,6 +21,7 @@ Docker is used to emulate a distributed multi-node cluster environment where eac ### 1. Prerequisites * **Docker** installed and running on your host system. * **Python >= 3.8** installed on your host system (required only for local environment scripts and development). +* **A Linux (or WSL2) host for the checkout.** On Windows, see [Running on Windows](#running-on-windows) before you start: the checkout has to live inside a WSL2 distro, not on a Windows drive. ### 2. Prepare the Environment & Code Execute these preparation steps from the root directory of your project: @@ -101,6 +102,67 @@ Always clean up and tear down active background nodes after your test runs finis --- +## Running on Windows + +Ducktests run on Windows through Docker Desktop's WSL2 backend, with one rule that decides whether it works at all: + +> **Keep the Ignite checkout inside a WSL2 distro (e.g. `~/ignite`), not on a Windows drive (`C:\...` or `/mnt/c/...`).** + +`ducker-ignite` bind-mounts your checkout into every container as `/opt/ignite-dev`, and each node builds its classpath from it (`bin/include/build-classpath.sh`). Class loading stats thousands of files, which is slow across the Windows filesystem boundary: + +| Checkout location | Node start, to its first topology snapshot | +|---|---| +| Inside a WSL2 distro (ext4) | ~4 s | +| Windows drive, bind-mounted | ~250 s | + +Since the default service startup timeout is 60 s, a checkout on a Windows drive fails tests that the same checkout inside WSL2 passes in seconds. The node is not crashing in that case — it is still class-loading when ducktape stops waiting. + +### Setup + +1. **Install WSL2 and a distro** — in an Administrator PowerShell: `wsl --install -d Ubuntu`, then reboot if asked. +2. **Install Docker Desktop**, select the **WSL 2 backend**, and enable Settings → Resources → **WSL Integration** for your distro. Allow it at least 6 GB of RAM for small runs; the 13-container default needs considerably more. +3. **Prepare the distro:** + ```bash + sudo apt update && sudo apt install -y openjdk-17-jdk maven git python3 python-is-python3 + sudo usermod -aG docker $USER # then `wsl --shutdown` in PowerShell, and reopen the distro + docker ps # must work without sudo + python --version # run_tests.sh calls `python`, not `python3` + ``` +4. **Clone and build inside the distro**, then follow the Quick Start above from there: + ```bash + cd ~ && git clone https://github.com/apache/ignite.git && cd ignite + ./mvnw clean install -DskipTests + ./scripts/build-module.sh ducktests + ``` + +Run `run_tests.sh` from the distro's shell, not from Git Bash or PowerShell. + +### If you build on Windows + +To keep building with your Windows toolchain or IDE, treat the Windows checkout as the source of truth and sync it into the distro before each run: + +```bash +rsync -a --delete --exclude .git --exclude results --exclude .ducktape --exclude '*/src/' \ + /mnt/c/path/to/ignite/ ~/ignite/ +``` + +The sync is dominated by scanning the Windows mount rather than by copying, so it costs much the same whether one file changed or many. Java sources are excluded because nothing needs them at runtime. + +Two things to know in this setup: + +* **Build the ducker image from inside the distro.** The image bakes the host's uid (`--build-arg USER_UID=$(id -u)`), so an image built from Git Bash carries a Windows uid and cannot write into a checkout owned by your Linux user. The symptom is `PermissionError: [Errno 13] Permission denied: '.ducktape/metadata/session_id'`. It does not appear on a Windows bind mount, where ownership is synthesised. +* **After rebuilding an image, run `./docker/ducker-ignite down -f`.** `run_tests.sh` reuses an already-running cluster, so old containers otherwise keep serving the previous image. + +### Older clones and line endings + +The repository ships a `.gitattributes` that keeps `*.sh` and `ducker-ignite` LF on checkout. A clone made before it was added, on a Windows machine with `core.autocrlf=true`, still has CRLF copies, which fail inside the containers with `$'\r': command not found`. Renormalise such a clone with: + +```bash +git rm --cached -r -q . && git reset --hard +``` + +--- + ## Local Development & Code Checks See [DEV_GUIDE.md](DEV_GUIDE.md) for writing new ducktests. diff --git a/modules/ducktests/tests/docker/ducker-ignite b/modules/ducktests/tests/docker/ducker-ignite index 5c204f02bb881..e983d854bb814 100755 --- a/modules/ducktests/tests/docker/ducker-ignite +++ b/modules/ducktests/tests/docker/ducker-ignite @@ -288,6 +288,8 @@ ducker_build_image() { # Save docker image id to the file. Then could use this file to find version of docker image built last time. # It could be useful if we don't confident about necessity of stoping the cluster. + # The build directory does not exist in a fresh checkout, and only 'ducker-ignite up' creates it. + mkdir -p "${ducker_dir}/build" get_image_id "${image_name}" > "${ducker_dir}/build/image_${image_name}.build" echo "** Successfully built ${what} image in $((duration / 60))m $((duration % 60))s." @@ -426,7 +428,10 @@ attempting to start new ones." exec 3<> "${ducker_dir}/build/node_hosts" for n in $(seq -f %02g 1 ${num_nodes}); do local node="ducker${n}" - docker exec --user=root "${node}" grep "${node}" /etc/hosts >&3 + # //etc/hosts, not /etc/hosts: under MSYS (Git Bash) a bare absolute path argument is + # rewritten to a Windows path before docker sees it. The leading // suppresses that, and + # Linux resolves //etc/hosts the same as /etc/hosts. + docker exec --user=root "${node}" grep "${node}" //etc/hosts >&3 [[ $? -ne 0 ]] && die "failed to find the /etc/hosts entry for ${node}" done exec 3>&- From 28e54973960f871e1b07e5916e4751e9611b69b4 Mon Sep 17 00:00:00 2001 From: Maksim Davydov Date: Wed, 23 Sep 2026 13:07:20 +0300 Subject: [PATCH 2/3] IGNITE-29067 [ducktests] Keep .gitattributes to the LF rules the fix needs Drop the clean_up.sh entry, which *.sh already covers, and the *.bat/*.cmd CRLF block. The runner fix doesn't need them, and the CRLF rule changed working-tree bytes on Linux and macOS checkouts. --- .gitattributes | 5 ----- 1 file changed, 5 deletions(-) diff --git a/.gitattributes b/.gitattributes index 93523177423d9..2d846a04ac2f1 100644 --- a/.gitattributes +++ b/.gitattributes @@ -4,8 +4,3 @@ *.sh text eol=lf mvnw text eol=lf modules/ducktests/tests/docker/ducker-ignite text eol=lf -modules/ducktests/tests/docker/clean_up.sh text eol=lf - -# Windows batch files are only ever run on Windows. -*.bat text eol=crlf -*.cmd text eol=crlf From 59fee530be950608a5264d9fc7163c4ac6b3cb74 Mon Sep 17 00:00:00 2001 From: Maksim Davydov Date: Wed, 23 Sep 2026 13:18:32 +0300 Subject: [PATCH 3/3] IGNITE-29067 [ducktests] Address review: ssh-config LF, clearer Windows guide - .gitattributes: pin docker/ssh-config to LF, since the image copies it in as the SSH client configuration. Drop the mvnw rule, which the runner does not need. Explain in plain words why these files must stay LF. - README: rewrite "Running on Windows" as a step-by-step guide covering installing WSL2 and Docker Desktop, preparing the distro, building and running the smoke test. Add a troubleshooting table. The renormalize command now warns that it discards uncommitted work. - README: the prerequisites no longer suggest that a Linux host is required; macOS is unaffected. - Use American spelling and mvn, matching the Quick Start. --- .gitattributes | 10 +- modules/ducktests/README.md | 107 ++++++++++++------- modules/ducktests/tests/docker/ducker-ignite | 4 - 3 files changed, 76 insertions(+), 45 deletions(-) diff --git a/.gitattributes b/.gitattributes index 2d846a04ac2f1..d0a1dfe541f39 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,6 +1,8 @@ -# Shell scripts must keep LF endings even when checked out on Windows: the -# ducktests runner bind-mounts the checkout into Linux containers and executes -# them there, where CRLF fails with "$'\r': command not found". +# Files that the ducktests Docker runner uses inside its Linux containers must keep +# LF line endings, even in a Windows checkout with core.autocrlf=true. The containers +# see the checkout through a bind mount, and a CRLF copy fails there: shell scripts +# stop with "$'\r': command not found", and ssh-config is copied into the image as +# the SSH client configuration. *.sh text eol=lf -mvnw text eol=lf modules/ducktests/tests/docker/ducker-ignite text eol=lf +modules/ducktests/tests/docker/ssh-config text eol=lf diff --git a/modules/ducktests/README.md b/modules/ducktests/README.md index 8719bcbefa5ea..a6cd2b0340383 100644 --- a/modules/ducktests/README.md +++ b/modules/ducktests/README.md @@ -21,7 +21,7 @@ Docker is used to emulate a distributed multi-node cluster environment where eac ### 1. Prerequisites * **Docker** installed and running on your host system. * **Python >= 3.8** installed on your host system (required only for local environment scripts and development). -* **A Linux (or WSL2) host for the checkout.** On Windows, see [Running on Windows](#running-on-windows) before you start: the checkout has to live inside a WSL2 distro, not on a Windows drive. +* **On Windows:** the checkout must live inside a WSL2 distro, not on a Windows drive. Read [Running on Windows](#running-on-windows) before you start. ### 2. Prepare the Environment & Code Execute these preparation steps from the root directory of your project: @@ -104,63 +104,96 @@ Always clean up and tear down active background nodes after your test runs finis ## Running on Windows -Ducktests run on Windows through Docker Desktop's WSL2 backend, with one rule that decides whether it works at all: +Ducktests run on Windows through Docker Desktop and WSL2. One rule decides whether tests pass: -> **Keep the Ignite checkout inside a WSL2 distro (e.g. `~/ignite`), not on a Windows drive (`C:\...` or `/mnt/c/...`).** +> **Keep the Ignite checkout inside the WSL2 filesystem (for example `~/ignite`), not on a Windows drive (`C:\...`, seen from WSL2 as `/mnt/c/...`).** -`ducker-ignite` bind-mounts your checkout into every container as `/opt/ignite-dev`, and each node builds its classpath from it (`bin/include/build-classpath.sh`). Class loading stats thousands of files, which is slow across the Windows filesystem boundary: +Every container mounts your checkout as `/opt/ignite-dev` and builds the node classpath from it, so starting a node reads thousands of files. Across the boundary between Windows and Linux those reads are slow. -| Checkout location | Node start, to its first topology snapshot | -|---|---| -| Inside a WSL2 distro (ext4) | ~4 s | -| Windows drive, bind-mounted | ~250 s | +### Step 1. Install WSL2 and Docker Desktop -Since the default service startup timeout is 60 s, a checkout on a Windows drive fails tests that the same checkout inside WSL2 passes in seconds. The node is not crashing in that case — it is still class-loading when ducktape stops waiting. +1. Open PowerShell as Administrator and run `wsl --install -d Ubuntu`. Reboot if asked, then open Ubuntu from the Start menu and create your Linux user. +2. Install [Docker Desktop](https://www.docker.com/products/docker-desktop/). In its settings: + * General: enable **Use the WSL 2 based engine**. + * Resources -> WSL Integration: enable integration for your Ubuntu distro. + * Resources: give Docker at least 6 GB of memory for small runs such as the smoke test. Running the default 13 containers needs much more. -### Setup +### Step 2. Prepare the distro -1. **Install WSL2 and a distro** — in an Administrator PowerShell: `wsl --install -d Ubuntu`, then reboot if asked. -2. **Install Docker Desktop**, select the **WSL 2 backend**, and enable Settings → Resources → **WSL Integration** for your distro. Allow it at least 6 GB of RAM for small runs; the 13-container default needs considerably more. -3. **Prepare the distro:** - ```bash - sudo apt update && sudo apt install -y openjdk-17-jdk maven git python3 python-is-python3 - sudo usermod -aG docker $USER # then `wsl --shutdown` in PowerShell, and reopen the distro - docker ps # must work without sudo - python --version # run_tests.sh calls `python`, not `python3` - ``` -4. **Clone and build inside the distro**, then follow the Quick Start above from there: - ```bash - cd ~ && git clone https://github.com/apache/ignite.git && cd ignite - ./mvnw clean install -DskipTests - ./scripts/build-module.sh ducktests - ``` +Run these commands in the Ubuntu shell: -Run `run_tests.sh` from the distro's shell, not from Git Bash or PowerShell. +```bash +# Tools for building Ignite and running the test scripts +sudo apt update && sudo apt install -y openjdk-17-jdk maven git python3 python3-venv python-is-python3 -### If you build on Windows +# Let your user run docker without sudo +sudo usermod -aG docker $USER +``` -To keep building with your Windows toolchain or IDE, treat the Windows checkout as the source of truth and sync it into the distro before each run: +Close the Ubuntu window, run `wsl --shutdown` in PowerShell, and open Ubuntu again so the group change takes effect. Then check the setup: ```bash -rsync -a --delete --exclude .git --exclude results --exclude .ducktape --exclude '*/src/' \ - /mnt/c/path/to/ignite/ ~/ignite/ +docker ps # must print a (possibly empty) container list, without sudo +python --version # must print Python 3.8 or newer; the scripts call `python`, not `python3` ``` -The sync is dominated by scanning the Windows mount rather than by copying, so it costs much the same whether one file changed or many. Java sources are excluded because nothing needs them at runtime. +### Step 3. Get the code and build it + +Clone into your Linux home directory, not into `/mnt/c`: + +```bash +cd ~ +git clone https://github.com/apache/ignite.git +cd ignite +mvn clean install -DskipTests +./scripts/build-module.sh ducktests +``` + +If you run tests that need `ignite-extensions`, clone it next to `ignite` in the same directory (see [Testing with Ignite Extensions](#testing-with-ignite-extensions)). + +### Step 4. Run the smoke test -Two things to know in this setup: +From here, follow [Quick Start](#quick-start-local-docker-run) steps 3–6 in the Ubuntu shell. In short: -* **Build the ducker image from inside the distro.** The image bakes the host's uid (`--build-arg USER_UID=$(id -u)`), so an image built from Git Bash carries a Windows uid and cannot write into a checkout owned by your Linux user. The symptom is `PermissionError: [Errno 13] Permission denied: '.ducktape/metadata/session_id'`. It does not appear on a Windows bind mount, where ownership is synthesised. -* **After rebuilding an image, run `./docker/ducker-ignite down -f`.** `run_tests.sh` reuses an already-running cluster, so old containers otherwise keep serving the previous image. +```bash +cd ~/ignite/modules/ducktests/tests +python -m venv ~/.virtualenvs/ignite-ducktests-dev +source ~/.virtualenvs/ignite-ducktests-dev/bin/activate +pip install -r docker/requirements-dev.txt +pip install -e . + +./docker/run_tests.sh -t ./ignitetest/tests/smoke_test.py::SmokeServicesTest.test_ignite_start_stop -n 3 --global-json '{"cluster_size": 2}' +``` + +The first run builds the Docker image, which takes several minutes. Later runs reuse it. -### Older clones and line endings +Run every command from the Ubuntu shell. The scripts also start from Git Bash, but Git Bash works with a checkout on a Windows drive, which is too slow for tests to pass. -The repository ships a `.gitattributes` that keeps `*.sh` and `ducker-ignite` LF on checkout. A clone made before it was added, on a Windows machine with `core.autocrlf=true`, still has CRLF copies, which fail inside the containers with `$'\r': command not found`. Renormalise such a clone with: +### Keeping your IDE on Windows + +If you prefer to edit and build in a Windows checkout, keep it as your working copy and copy it into the distro before each test run: ```bash -git rm --cached -r -q . && git reset --hard +rsync -a --delete --exclude .git --exclude results --exclude .ducktape --exclude '*/src/' \ + /mnt/c/path/to/ignite/ ~/ignite/ ``` +Keep the trailing `/` on the source path, so rsync copies the contents of the directory rather than the directory itself. Java sources (`src/`) are skipped because the tests only need the compiled classes. Most of the time goes into scanning the Windows drive, so a sync takes about as long whether you changed one file or many. + +Two rules apply in this setup: + +* **Build the Docker image from the Ubuntu shell, not from Git Bash.** The image records the uid of the user who builds it. An image built from Git Bash gets a Windows uid and cannot write to the checkout in your Linux home directory. +* **After rebuilding the image, run `./docker/ducker-ignite down -f`.** `run_tests.sh` reuses containers that are already running, so without this they keep running the old image. + +### Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `$'\r': command not found` | The shell scripts have Windows (CRLF) line endings. This happens in a clone checked out on Windows before the repository added its `.gitattributes`. | Commit or stash your changes first, because the next command discards uncommitted work. Then run `git rm --cached -r -q . && git reset --hard` in the repository root to check the files out again with the correct line endings. | +| `PermissionError: [Errno 13] Permission denied: '.ducktape/metadata/session_id'` | The Docker image was built from Git Bash and has the wrong uid. | Rebuild the image from the Ubuntu shell with `./docker/ducker-ignite build`, then run `./docker/ducker-ignite down -f`. | +| Tests fail with a timeout while waiting for a node to start | The checkout is on a Windows drive. | Move the checkout into the WSL2 filesystem ([Step 3](#step-3-get-the-code-and-build-it)). | +| Changes to the Docker image have no effect | Containers from the old image are still running. | Run `./docker/ducker-ignite down -f` and start the tests again. | + --- ## Local Development & Code Checks diff --git a/modules/ducktests/tests/docker/ducker-ignite b/modules/ducktests/tests/docker/ducker-ignite index e983d854bb814..f33efe816a05c 100755 --- a/modules/ducktests/tests/docker/ducker-ignite +++ b/modules/ducktests/tests/docker/ducker-ignite @@ -288,7 +288,6 @@ ducker_build_image() { # Save docker image id to the file. Then could use this file to find version of docker image built last time. # It could be useful if we don't confident about necessity of stoping the cluster. - # The build directory does not exist in a fresh checkout, and only 'ducker-ignite up' creates it. mkdir -p "${ducker_dir}/build" get_image_id "${image_name}" > "${ducker_dir}/build/image_${image_name}.build" @@ -428,9 +427,6 @@ attempting to start new ones." exec 3<> "${ducker_dir}/build/node_hosts" for n in $(seq -f %02g 1 ${num_nodes}); do local node="ducker${n}" - # //etc/hosts, not /etc/hosts: under MSYS (Git Bash) a bare absolute path argument is - # rewritten to a Windows path before docker sees it. The leading // suppresses that, and - # Linux resolves //etc/hosts the same as /etc/hosts. docker exec --user=root "${node}" grep "${node}" //etc/hosts >&3 [[ $? -ne 0 ]] && die "failed to find the /etc/hosts entry for ${node}" done