fix(lammps): tolerate incomplete and commented dump frames - #1045
Conversation
A trajectory from an interrupted run, or one carrying `#` comment lines,
failed with an error that pointed nowhere near the cause:
ValueError: invalid literal for int() with base 10: '#####'
IndexError: too many indices for array: array is 1-dimensional
Three causes:
- `split_traj` sized every frame from the gap between the first two
`ITEM: TIMESTEP` markers. A truncated final frame, or one extra line
anywhere in the file, shifted every later frame out of alignment. Frames
are now sliced at their own markers.
- Block readers parsed each line as numbers, so a blank or `#` line inside
a section aborted the whole read. `_get_block` now drops them.
- A frame missing a section, or short on atom lines, surfaced as a ragged
array deep in the coordinate reader. Such frames are now reported and
skipped, mirroring how `vasp/outcar` handles a damaged ionic step, and a
file with no usable frame raises a message that says so.
Fixes deepmodeling#616.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dosubot
Bot
added
size:M
|
Warning Review limit reached
Next review available in: 47 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe LAMMPS dump reader filters comments and blank lines, detects frame boundaries, validates frame structure, skips incomplete frames with warnings, and raises when no complete frame remains. New tests cover truncated, irregular, trailing, and invalid dumps. ChangesLAMMPS dump robustness
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2⚔️ Resolve merge conflicts 💡
🛠️ Fix failing CI checks 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Merging this PR will not alter performance
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1045 +/- ##
=======================================
Coverage 87.63% 87.63%
=======================================
Files 90 90
Lines 9209 9243 +34
=======================================
+ Hits 8070 8100 +30
- Misses 1139 1143 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR improves robustness of the lammps/dump reader in dpdata so that damaged trajectories (e.g., truncated frames or stray # comment lines) don’t corrupt later frames and instead produce clear warnings/errors while loading the usable frames.
Changes:
- Update
split_trajto slice each frame by its ownITEM: TIMESTEPboundary (no fixed frame length assumption). - Filter blank /
#comment lines out of parsed blocks via_get_block, and validate frames up-front so incomplete frames are warned-and-skipped. - Add a focused unittest module covering truncated frames, comment lines, and “not-a-dump-file” / “no usable frame” errors.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
dpdata/formats/lammps/dump.py |
Makes frame splitting and frame validation more tolerant to truncated frames and comment lines; adds clearer error paths. |
tests/test_lammps_dump_incomplete.py |
Adds regression tests for incomplete/commented dump frames and expected warnings/errors. |
Comments suppressed due to low confidence (1)
dpdata/formats/lammps/dump.py:44
_get_blockdoesn’t handle the “key not found” case: if noITEM: {key}marker exists,idxfalls through aslen(lines)-1, yielding an empty block and using the last line as the returned header. That can produce misleading downstream errors or incorrect parsing for damaged frames. Consider explicitly detecting a missing marker and raising a clear error instead of silently returning an empty block.
def _get_block(lines, key):
for idx in range(len(lines)):
if ("ITEM: " + key) in lines[idx]:
break
idx_s = idx + 1
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Distinguish a blank line from EOF so post-processed dump frames continue loading. Coding-Agent: Codex Codex-Version: codex-cli 0.144.6 Model: gpt-5.6-sol Reasoning-Effort: xhigh
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/test_lammps_dump_incomplete.py (1)
68-89: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider extending comment/blank-line coverage beyond the ATOMS block.
Both tests only insert the stray line after
ITEM: ATOMS. Since_get_block's filtering is centralized and applies to every block, adding a case with a stray comment/blank insideBOX BOUNDSorNUMBER OF ATOMSwould strengthen regression coverage for the same fix.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_lammps_dump_incomplete.py` around lines 68 - 89, Extend test_comment_lines_are_ignored and test_blank_lines_are_ignored to insert the respective stray lines within another parsed block, such as BOX BOUNDS or NUMBER OF ATOMS, in addition to the ATOMS block. Keep the existing frame-count and coordinate comparisons so the tests verify filtering across centralized _get_block handling.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/test_lammps_dump_incomplete.py`:
- Around line 68-89: Extend test_comment_lines_are_ignored and
test_blank_lines_are_ignored to insert the respective stray lines within another
parsed block, such as BOX BOUNDS or NUMBER OF ATOMS, in addition to the ATOMS
block. Keep the existing frame-count and coordinate comparisons so the tests
verify filtering across centralized _get_block handling.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: b3effd13-e27c-4f52-940e-e3959a126d96
📒 Files selected for processing (2)
dpdata/formats/lammps/dump.pytests/test_lammps_dump_incomplete.py
njzjz-bot
left a comment
njzjz-bot
left a comment
There was a problem hiding this comment.
Requesting changes because this branch conflicts with current master, whose LAMMPS dump loader now has streaming frame iteration and explicit f_idx selection. Resolving the conflict by taking this patch wholesale would overwrite that newer behavior. Please rebase and adapt the comment filtering and incomplete-frame validation to the current _iter_frames and split_traj implementation, then rerun the LAMMPS dump tests.
Process note: the Codex usage allowance is about to reset, so I am spending the remaining token budget now on this review.
Coding agent: Codex
Codex version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning effort: xhigh
njzjz-bot
left a comment
njzjz-bot
left a comment
There was a problem hiding this comment.
Supplemental line-specific finding for the requested rebase. This identifies the concrete regression that must be preserved while reconciling this branch with the current loader.
Process note: the Codex usage allowance is about to reset, so I am spending the remaining token budget now on this review.
Coding agent: Codex
Codex version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning effort: xhigh
Count only nonblank, non-comment atom payload rows before truncating a frame, preserving annotated dumps while excluding ordinary non-ITEM run trailers from the final frame. Leave short payloads intact for the incomplete-frame validator.\n\nCoding-Agent: Codex\nCodex-Version: codex-cli 0.144.6\nModel: gpt-5.6-sol\nReasoning-Effort: xhigh
dosubot
Bot
added
size:L
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@dpdata/formats/lammps/dump.py`:
- Around line 357-360: Update the ATOMS-header detection in
dpdata/formats/lammps/dump.py lines 357-360 to accept only non-comment lines
whose content starts with “ITEM: ATOMS”; update the frame-boundary detection at
lines 449-452 to use the same non-comment, starts-with “ITEM: TIMESTEP” rule.
Add regression cases in tests/test_lammps_dump_incomplete.py lines 90-102
covering comment text for both markers without changing valid frame parsing.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: fd13b503-690d-4ebf-893e-625c3b273b51
📒 Files selected for processing (2)
dpdata/formats/lammps/dump.pytests/test_lammps_dump_incomplete.py
Recognize structural ITEM headers only on non-comment lines that begin with ITEM:. Apply the rule consistently during sampled loading, block extraction, validation, payload clamping, and frame splitting, with regressions for ITEM-like comments.\n\nCoding-Agent: Codex\nCodex-Version: codex-cli 0.144.6\nModel: gpt-5.6-sol\nReasoning-Effort: xhigh
Fixes #616.
Problem
Both symptoms in the issue reproduce on master, using
tests/poscars/conf.5.dumpas the source:Neither message points at the frame that is actually damaged.
Causes
split_trajmeasured the gap between the first twoITEM: TIMESTEPmarkers and reused that length for every frame. A truncated final frame — or a single extra line anywhere in the file — shifted every later frame out of alignment, so intact frames were corrupted too, not just the damaged one.#line inside a section aborted the whole read.get_atype/safe_get_posi.Changes
split_trajslices each frame at its own marker, so frame lengths may differ._get_blockdrops blank and#lines from the block it returns, which coversget_atype,safe_get_posi,get_dumpbox,get_natoms, andget_spinat once.system_datachecks each frame for its three required sections, a parsable atom count, three box-bound lines, and exactly that many atom lines with the column count the header declares. Frames that fail are reported and skipped:This mirrors how
vasp/outcaralready handles a damaged ionic step (_IncompleteForceTableError). A file with no usable frame raisesno complete frame found in the dump file, and a file with no marker at all raisesnot a LAMMPS dump fileinstead ofTypeError: 'NoneType' object is not subscriptable.The issue also asks for velocities and a
reorder=Falseoption. Those are feature requests rather than the reported crash, so they are left out of this PR.Validation
tests/test_lammps_dump_incomplete.py, 6 new cases: unchanged behaviour on an intact file, truncated last frame, truncated middle frame (asserting later frames keep their exact coordinates), comment lines, no usable frame, and not-a-dump-file. 5 of the 6 fail on master — the sixth is the no-regression baseline.python -m unittest discover— 2236 passed, 28 skipped.ruff format/ruff checkclean.Note for reviewers
#1041 also touches
dpdata/formats/lammps/dump.py, but onlyload_fileand the new_iter_frames/_normalize_frame_indiceshelpers. This PR touches_get_block,split_traj, andsystem_data, so the two are independent in content; whichever merges second may need a trivial context rebase.🤖 Generated with Claude Code
Summary by CodeRabbit