Skip to content

DE-MV, DE-TH, EC-LT, EC-LV, EC-SI: complete and valid output - #349

Merged
m-mohr merged 5 commits into
mainfrom
fix-invalid-collections
Oct 30, 2026
Merged

m-mohr merged 5 commits into
mainfrom
fix-invalid-collections

Conversation

@ivorbosloper

Copy link
Copy Markdown
Collaborator

Fixes the converters whose output failed fiboa validate or was incomplete, found when the catalog's single-edition collections were rebuilt.

  • DE-MV read one WFS request, and the service caps a request at 20,000 features, so 51,537 of the 71,537 field blocks were missing. It now uses WFSConverterMixin, sorted by fbid so the pages do not overlap. The count request leaves out the output format, which the service rejects with a 500.
  • DE-TH writes 0 as area_last_year for a field block that is new this year; that is now empty, and the field no longer required.
  • EC-LT, EC-LV and EC-SI declare the crop extension but lacked crop:code, crop:code_list or both. Lithuania's code list is keyed on the group name, so the name is also the code; Slovenia's code is SIFRA_KMRS; EC-SI leaves out the parcels without a crop.
  • The conversion tests run the validator without checking its result, so expected_columns now guards these fields.

🤖 Generated with Claude Code

ivorbosloper and others added 4 commits October 27, 2026 23:00
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…irst 20,000

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… rejects

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the dataset-converter fixes.

No open issues 🎉

What changed in this PR

This PR fixes incomplete or invalid output from five dataset converters, improving field coverage and compliance with fiboa schemas.

Changes:

  • Pages DE-MV’s WFS requests to retrieve all field blocks.
  • Treats missing prior-year area in DE-TH as empty and completes required crop fields for EC-LT, EC-LV, and EC-SI.
  • Adds regression checks for WFS pagination and crop-column presence.
File Description
tests/​test_converter_wfs.py Checks DE-MV pagination, sorting, and count-request parameters.
tests/​test_convert.py Requires crop code and code-list columns for the three EuroCrops converters.
fiboa_cli/​datasets/​ec_si.py Adds crop codes and code-list metadata; excludes parcels without crops.
fiboa_cli/​datasets/​ec_lv.py Adds crop code-list metadata.
fiboa_cli/​datasets/​ec_lt.py Uses the crop group name as the crop code.
fiboa_cli/​datasets/​de_th.py Converts nonpositive prior-year area to empty and makes the field optional.
fiboa_cli/​datasets/​de_mv.py Uses sorted WFS pagination and omits output format from count requests.
CHANGELOG.md Documents the converter fixes.

Review effort: Balanced

@m-mohr
m-mohr merged commit 17edba1 into main Oct 30, 2026
7 checks passed
@m-mohr
m-mohr deleted the fix-invalid-collections branch October 30, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants