Skip to content

GH-50046: [CI][C++] Improve ccache reuse by saving to cache only from main #50047

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rok wants to merge 3 commits into
base: main
Choose a base branch
from
+100 −30
Open

GH-50046: [CI][C++] Improve ccache reuse by saving to cache only from main #50047

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4a893c1
store to ccache only from main
rok May 26, 2026
afb1a1f
save from main if not cancelled and before test
rok May 27, 2026
3b73f71
Cache steps now continue-on-error: true
rok May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 30 additions & 9 deletions .github/workflows/cpp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,11 +119,11 @@ jobs:
persist-credentials: false
fetch-depth: 0
submodules: recursive
- name: Cache Docker Volumes
uses: actions/cache@v5
- name: Restore Docker Volumes
uses: actions/cache/restore@v5
with:
path: .docker
key: ${{ matrix.image }}-${{ hashFiles('cpp/**') }}
key: ${{ matrix.image }}-${{ github.run_id }}
restore-keys: ${{ matrix.image }}-
- name: Setup Python on hosted runner
if: |
Expand All @@ -149,6 +149,13 @@ jobs:
sudo sysctl -w vm.mmap_rnd_bits=28
source ci/scripts/util_enable_core_dumps.sh
archery docker run ${{ matrix.image }}
- name: Save Docker Volumes
if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
continue-on-error: true
uses: actions/cache/save@v5
with:
path: .docker
key: ${{ matrix.image }}-${{ github.run_id }}
- name: Docker Push
if: >-
success() &&
Expand Down Expand Up @@ -254,11 +261,11 @@ jobs:
id: ccache-info
run: |
echo "cache-dir=$(ccache --get-config cache_dir)" >> $GITHUB_OUTPUT
- name: Cache ccache
uses: actions/cache@v5
- name: Restore ccache
uses: actions/cache/restore@v5
with:
path: ${{ steps.ccache-info.outputs.cache-dir }}
key: cpp-ccache-macos-${{ matrix.macos-version }}-${{ hashFiles('cpp/**') }}
key: cpp-ccache-macos-${{ matrix.macos-version }}-${{ github.run_id }}
restore-keys: cpp-ccache-macos-${{ matrix.macos-version }}-
- name: Build
run: |
Expand All @@ -276,6 +283,13 @@ jobs:
export BUILD_WARNING_LEVEL=PRODUCTION
fi
ci/scripts/cpp_build.sh $(pwd) $(pwd)/build
- name: Save ccache
if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
continue-on-error: true
uses: actions/cache/save@v5
with:
path: ${{ steps.ccache-info.outputs.cache-dir }}
key: cpp-ccache-macos-${{ matrix.macos-version }}-${{ github.run_id }}
- name: Test
shell: bash
run: |
Expand Down Expand Up @@ -361,17 +375,24 @@ jobs:
- name: Setup MSYS2
shell: msys2 {0}
run: ci/scripts/msys2_setup.sh cpp
- name: Cache ccache
uses: actions/cache@v5
- name: Restore ccache
uses: actions/cache/restore@v5
with:
path: ccache
key: cpp-ccache-${{ matrix.msystem_lower}}-${{ hashFiles('cpp/**') }}
key: cpp-ccache-${{ matrix.msystem_lower}}-${{ github.run_id }}
restore-keys: cpp-ccache-${{ matrix.msystem_lower}}-
- name: Build
shell: msys2 {0}
run: |
export CMAKE_BUILD_PARALLEL_LEVEL=$NUMBER_OF_PROCESSORS
ci/scripts/cpp_build.sh "$(pwd)" "$(pwd)/build"
- name: Save ccache
if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
continue-on-error: true
uses: actions/cache/save@v5
with:
path: ccache
key: cpp-ccache-${{ matrix.msystem_lower}}-${{ github.run_id }}
- name: Download Timezone Database
if: matrix.msystem_upper == 'CLANG64'
shell: bash
Expand Down
78 changes: 60 additions & 18 deletions .github/workflows/cpp_extra.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,11 +141,11 @@ jobs:
persist-credentials: false
fetch-depth: 0
submodules: recursive
- name: Cache Docker Volumes
uses: actions/cache@v5
- name: Restore Docker Volumes
uses: actions/cache/restore@v5
with:
path: .docker
key: extra-${{ matrix.image }}-${{ hashFiles('cpp/**') }}
key: extra-${{ matrix.image }}-${{ github.run_id }}
restore-keys: extra-${{ matrix.image }}-
- name: Setup Python
uses: actions/setup-python@v6
Expand All @@ -169,6 +169,13 @@ jobs:
done
fi
archery docker run ${{ matrix.run-options || '' }} ${{ matrix.image }}
- name: Save Docker Volumes
if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
continue-on-error: true
uses: actions/cache/save@v5
with:
path: .docker
key: extra-${{ matrix.image }}-${{ github.run_id }}
- name: Docker Push
if: >-
success() &&
Expand Down Expand Up @@ -227,11 +234,11 @@ jobs:
- name: Free up disk space
run: |
ci/scripts/util_free_space.sh
- name: Cache Docker Volumes
uses: actions/cache@v5
- name: Restore Docker Volumes
uses: actions/cache/restore@v5
with:
path: .docker
key: jni-${{ matrix.platform.runs-on }}-${{ hashFiles('cpp/**') }}
key: jni-${{ matrix.platform.runs-on }}-${{ github.run_id }}
restore-keys: jni-${{ matrix.platform.runs-on }}-
- name: Setup Python
uses: actions/setup-python@v6
Expand All @@ -248,6 +255,13 @@ jobs:
run: |
source ci/scripts/util_enable_core_dumps.sh
archery docker run cpp-jni
- name: Save Docker Volumes
if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
continue-on-error: true
uses: actions/cache/save@v5
with:
path: .docker
key: jni-${{ matrix.platform.runs-on }}-${{ github.run_id }}
- name: Docker Push
if: >-
success() &&
Expand Down Expand Up @@ -300,11 +314,11 @@ jobs:
- name: Prepare ccache
run: |
echo "CCACHE_DIR=${PWD}/ccache" >> ${GITHUB_ENV}
- name: Cache ccache
uses: actions/cache@v5
- name: Restore ccache
uses: actions/cache/restore@v5
with:
path: ccache
key: jni-macos-${{ hashFiles('cpp/**') }}
key: jni-macos-${{ github.run_id }}
restore-keys: jni-macos-
- name: CMake
run: |
Expand Down Expand Up @@ -347,6 +361,13 @@ jobs:
cmake --build cpp/examples/minimal_build.build
cd cpp/examples/minimal_build
../minimal_build.build/arrow-example
- name: Save ccache
if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
continue-on-error: true
uses: actions/cache/save@v5
with:
path: ccache
key: jni-macos-${{ github.run_id }}

odbc-linux:
needs: check-labels
Expand All @@ -372,11 +393,11 @@ jobs:
persist-credentials: false
fetch-depth: 0
submodules: recursive
- name: Cache Docker Volumes
uses: actions/cache@v5
- name: Restore Docker Volumes
uses: actions/cache/restore@v5
with:
path: .docker
key: ubuntu-cpp-odbc-${{ hashFiles('cpp/**') }}
key: ubuntu-cpp-odbc-${{ github.run_id }}
restore-keys: ubuntu-cpp-odbc-
- name: Setup Python on hosted runner
uses: actions/setup-python@v6
Expand All @@ -393,6 +414,13 @@ jobs:
sudo sysctl -w vm.mmap_rnd_bits=28
source ci/scripts/util_enable_core_dumps.sh
archery docker run ubuntu-cpp-odbc
- name: Save Docker Volumes
if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
continue-on-error: true
uses: actions/cache/save@v5
with:
path: .docker
key: ubuntu-cpp-odbc-${{ github.run_id }}
- name: Docker Push
if: >-
success() &&
Expand Down Expand Up @@ -464,11 +492,11 @@ jobs:
id: ccache-info
run: |
echo "cache-dir=$(ccache --get-config cache_dir)" >> $GITHUB_OUTPUT
- name: Cache ccache
uses: actions/cache@v5.0.2
- name: Restore ccache
uses: actions/cache/restore@v5
with:
path: ${{ steps.ccache-info.outputs.cache-dir }}
key: cpp-odbc-ccache-macos-${{ matrix.macos-version }}-${{ matrix.build-type }}-${{ hashFiles('cpp/**') }}
key: cpp-odbc-ccache-macos-${{ matrix.macos-version }}-${{ matrix.build-type }}-${{ github.run_id }}
restore-keys: cpp-odbc-ccache-macos-${{ matrix.macos-version }}-${{ matrix.build-type }}-
- name: Build
run: |
Expand All @@ -486,6 +514,13 @@ jobs:
export ARROW_CMAKE_ARGS="-DODBC_INCLUDE_DIR=$ODBC_INCLUDE_DIR"
export CXXFLAGS="$CXXFLAGS -I$ODBC_INCLUDE_DIR"
ci/scripts/cpp_build.sh $(pwd) $(pwd)/build
- name: Save ccache
if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
continue-on-error: true
uses: actions/cache/save@v5
with:
path: ${{ steps.ccache-info.outputs.cache-dir }}
key: cpp-odbc-ccache-macos-${{ matrix.macos-version }}-${{ matrix.build-type }}-${{ github.run_id }}
- name: Setup Python
uses: actions/setup-python@v6
with:
Expand Down Expand Up @@ -589,11 +624,11 @@ jobs:
shell: bash
run: |
echo "cache-dir=$(ccache --get-config cache_dir)" >> $GITHUB_OUTPUT
- name: Cache ccache
uses: actions/cache@v5
- name: Restore ccache
uses: actions/cache/restore@v5
with:
path: ${{ steps.ccache-info.outputs.cache-dir }}
key: cpp-odbc-ccache-windows-x64-${{ hashFiles('cpp/**') }}
key: cpp-odbc-ccache-windows-x64-${{ github.run_id }}
restore-keys: cpp-odbc-ccache-windows-x64-
- name: Checkout vcpkg
uses: actions/checkout@v6
Expand Down Expand Up @@ -630,6 +665,13 @@ jobs:
call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
set VCPKG_ROOT=%VCPKG_ROOT_KEEP%
bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build"
- name: Save ccache
if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
continue-on-error: true
uses: actions/cache/save@v5
with:
path: ${{ steps.ccache-info.outputs.cache-dir }}
key: cpp-odbc-ccache-windows-x64-${{ github.run_id }}
- name: Register Flight SQL ODBC Driver
shell: cmd
run: |
Expand Down
13 changes: 10 additions & 3 deletions .github/workflows/cpp_windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,11 +107,11 @@ jobs:
shell: bash
run: |
echo "cache-dir=$(ccache --get-config cache_dir)" >> $GITHUB_OUTPUT
- name: Cache ccache
uses: actions/cache@v5
- name: Restore ccache
uses: actions/cache/restore@v5
with:
path: ${{ steps.ccache-info.outputs.cache-dir }}
key: cpp-ccache-windows-${{ inputs.arch }}-${{ hashFiles('cpp/**') }}
key: cpp-ccache-windows-${{ inputs.arch }}-${{ github.run_id }}
restore-keys: cpp-ccache-windows-${{ inputs.arch }}-
- name: Build
shell: cmd
Expand All @@ -120,6 +120,13 @@ jobs:
run: |
call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" %VCVARS_ARCH%
bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build"
- name: Save ccache
if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
continue-on-error: true
uses: actions/cache/save@v5
with:
path: ${{ steps.ccache-info.outputs.cache-dir }}
key: cpp-ccache-windows-${{ inputs.arch }}-${{ github.run_id }}
- name: Test
shell: cmd
env:
Expand Down
Loading