Skip to content

Introduce faster file refresh on Linux (x86_64 only for now)#2793

Merged
iloveeclipse merged 2 commits into
eclipse-platform:masterfrom
iloveeclipse:Linux86_64_fast
Jul 3, 2026
Merged

Introduce faster file refresh on Linux (x86_64 only for now)#2793
iloveeclipse merged 2 commits into
eclipse-platform:masterfrom
iloveeclipse:Linux86_64_fast

Conversation

@iloveeclipse

@iloveeclipse iloveeclipse commented Jul 2, 2026

Copy link
Copy Markdown
Member

The code is supposed to speed up NFS file access on Linux by an order of magnitude by fetching directory children data (IFileInfo) in one shot from JNI (via array of StructStat).

LocalFile.childInfos(int, IProgressMonitor) overrides now the FileStore.childInfos(int, IProgressMonitor) and calls (on Linux) LinuxFileHandler.listDirectoryAndGetFileInfos() which forwards to faster native implementation in
LinuxFileNatives.listDirectoryAndGetFileInfos().

  • Introduced new libfastlinuxfile_1_0_0 library for x86_64 Linux architecture (which can be also built on other Linux architectures).
  • Created dedicated Linux specific package org.eclipse.core.internal.filesystem.local.linux
  • removed Mac code paths from new code to simplify Linux port
  • Added system property to disable new binary if needed via -Declipse.filesystem.useFastLinuxNatives=false.
  • With the flag above, the behavior on Linux should be unchanged compared with the previous code state.
  • In next release we can try to remove native code and use FFM API if we will allow the bundle to use Java 25.
  • As of today, FFM use is not possible - multi-release based bundle attempt failed due missing PDE / Equinox support for debugging MR code (see Support debugging multi-release bundles eclipse-pde/eclipse.pde#2391).
Variant NFS (ms) Local (ms)
Original code 70000 - 90000 900 - 2500
Updated code 800 - 5500 800 - 2500

I've measured data with the extra patch to trace the refresh data from Eclipse : #2792 on RHEL 9.6.

To drop local FS caches, I've used sync && echo 3 | sudo tee /proc/sys/vm/drop_caches .

Unfortunately this doesn't drop NFS server file caches, but interestingly, I couldn't see any caching effect on the NFS results for the old code anyway. Only new code is affected by the NFS server cache and produces sometimes NFS times comparable to the local FS performance.

Script below was used to create test directory structure with 50 folders a 5000 files each (250.000 files) used for performance tests above. Just create shell script and run it inside a project.

#!/bin/bash

TOTAL_FILES=250000
FILES_PER_DIR=5000
BASE_DIR="testdata"

mkdir -p "$BASE_DIR"

NUM_DIRS=$(( (TOTAL_FILES + FILES_PER_DIR - 1) / FILES_PER_DIR ))

for ((d=1; d<=NUM_DIRS; d++)); do
(
    dir="$BASE_DIR/dir_$(printf "%03d" $d)"
    mkdir -p "$dir"

    start=$(( (d-1)*FILES_PER_DIR + 1 ))
    end=$(( d*FILES_PER_DIR ))
    [ $end -gt $TOTAL_FILES ] && end=$TOTAL_FILES

    for ((i=start; i<=end; i++)); do
        : > "$dir/file_$(printf "%06d" $i).dat"
    done
) &
done

wait
echo "Done"

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a Linux x86_64–specific “fast” native path for directory listing + bulk IFileInfo retrieval to significantly speed up refresh/file access on NFS, with a system property to disable the fast natives and fall back to the existing behavior.

Changes:

  • Add a new Linux-specific native implementation (fastlinuxfile_1_0_0) and corresponding Java JNI layer to fetch directory entries + file infos in one call.
  • Update native handler selection to prefer the fast Linux path on Linux x86_64 (configurable via eclipse.filesystem.useFastLinuxNatives).
  • Extend performance benchmarks and bump test/fragment versions to reflect the new functionality.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
resources/tests/org.eclipse.core.tests.resources/src/org/eclipse/core/tests/resources/perf/BenchFileStore.java Adds Linux-fast-native benchmark variants (needs platform guarding).
resources/tests/org.eclipse.core.tests.resources/pom.xml Version bump for test bundle.
resources/tests/org.eclipse.core.tests.resources/META-INF/MANIFEST.MF Version bump for test bundle.
resources/bundles/org.eclipse.core.filesystem/src/org/eclipse/core/internal/filesystem/local/unix/UnixFileNatives.java Refactors constant usage to static imports; no functional change intended.
resources/bundles/org.eclipse.core.filesystem/src/org/eclipse/core/internal/filesystem/local/unix/StructStat.java Simplifies StructStat fields and toFileInfo() for the Unix native path.
resources/bundles/org.eclipse.core.filesystem/src/org/eclipse/core/internal/filesystem/local/LocalFileNativesManager.java Adds fast-Linux-native selection + new disable property; routes to Linux handler on Linux x86_64.
resources/bundles/org.eclipse.core.filesystem/src/org/eclipse/core/internal/filesystem/local/linux/LinuxStructStat.java New Linux-only stat struct mirror used for bulk directory listing JNI.
resources/bundles/org.eclipse.core.filesystem/src/org/eclipse/core/internal/filesystem/local/linux/LinuxFileNatives.java New JNI wrapper that loads fastlinuxfile_1_0_0 and exposes bulk listing + stat operations.
resources/bundles/org.eclipse.core.filesystem/src/org/eclipse/core/internal/filesystem/local/linux/LinuxFileHandler.java New NativeHandler implementation delegating to LinuxFileNatives.
resources/bundles/org.eclipse.core.filesystem/src/org/eclipse/core/internal/filesystem/local/linux/LinuxFileFlags.java New Linux flag resolver populated via JNI getFlag.
resources/bundles/org.eclipse.core.filesystem/natives/unix/fastlinux/Makefile Build script for the fast Linux native library (install target path is currently incorrect).
resources/bundles/org.eclipse.core.filesystem/natives/unix/fastlinux/fastlinuxfile.h JNI header and helper prototypes for the fast Linux native library.
resources/bundles/org.eclipse.core.filesystem/natives/unix/fastlinux/fastlinuxfile.c Native implementation for fast directory enumeration + bulk stat collection.
resources/bundles/org.eclipse.core.filesystem.linux.x86_64/pom.xml Builds and packages the new fast Linux native library into the x86_64 fragment.
resources/bundles/org.eclipse.core.filesystem.linux.x86_64/META-INF/MANIFEST.MF Version bump and updated host-version range for the fragment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread resources/bundles/org.eclipse.core.filesystem/natives/unix/fastlinux/Makefile Outdated
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Test Results

    54 files  ±0      54 suites  ±0   56m 19s ⏱️ - 3m 11s
 4 680 tests ±0   4 658 ✅ ±0   22 💤 ±0  0 ❌ ±0 
11 934 runs  ±0  11 781 ✅ ±0  153 💤 ±0  0 ❌ ±0 

Results for commit 8f4ede3. ± Comparison against base commit 6bef442.

♻️ This comment has been updated with latest results.

The code is supposed to speed up NFS file access on Linux by an order of
magnitude by fetching directory children data (IFileInfo) in one shot
from JNI (via array of StructStat).

LocalFile.childInfos(int, IProgressMonitor) overrides now the
FileStore.childInfos(int, IProgressMonitor) and calls (on Linux)
LinuxFileHandler.listDirectoryAndGetFileInfos() which forwards to faster
native implementation in
LinuxFileNatives.listDirectoryAndGetFileInfos().

- Introduced new libfastlinuxfile_1_0_0 library for x86_64 Linux
architecture (which can be also built on other Linux architectures).
- Created dedicated Linux specific package
org.eclipse.core.internal.filesystem.local.linux
- removed Mac code paths from new code to simplify Linux port
- Added system property to disable new binary if needed via
"-Declipse.filesystem.useFastLinuxNatives=false"

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 3 comments.

@iloveeclipse iloveeclipse merged commit e777932 into eclipse-platform:master Jul 3, 2026
18 checks passed
@iloveeclipse iloveeclipse deleted the Linux86_64_fast branch July 3, 2026 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants