Skip to content

feat: skip validation and reuse pre-converted arrow cache - #3983

Open
macandro96 wants to merge 2 commits into
super-v3.5-posttrainingfrom
amahishi/hf-arrow-dataset-cache
Open

feat: skip validation and reuse pre-converted arrow cache#3983
macandro96 wants to merge 2 commits into
super-v3.5-posttrainingfrom
amahishi/hf-arrow-dataset-cache

Conversation

@macandro96

@macandro96 macandro96 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What does this PR do ?

  • Load NeMo Gym JSONL through Hugging Face’s reusable Arrow cache while preserving raw rows and supporting pre-converted Arrow/Parquet datasets; observed dataset setup improved from ~152 seconds to 2.2 seconds.

  • Skip validation dataset loading when validation is disabled, avoiding unnecessary duplicate data initialization while preserving validation and trajectory-collection behavior.

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
@macandro96
macandro96 requested review from a team as code owners September 3, 2026 19:52
@macandro96 macandro96 added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Sep 3, 2026
@copy-pr-bot

copy-pr-bot Bot commented Sep 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
@macandro96
macandro96 force-pushed the amahishi/hf-arrow-dataset-cache branch from 06f75ea to 6aff5f9 Compare September 3, 2026 19:55
@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test 6aff5f9

@macandro96
macandro96 requested a review from yfw September 3, 2026 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant