Skip to content

GH-51265: [Python] Refactor and centralize duplicated reader docstrings - #51630

Open
MarkCruz911 wants to merge 3 commits into
apache:mainfrom
MarkCruz911:main
Open

MarkCruz911 wants to merge 3 commits into
apache:mainfrom
MarkCruz911:main

Conversation

@MarkCruz911

@MarkCruz911 MarkCruz911 commented Sep 29, 2026 •

Copy link
Copy Markdown

Rationale for this change

This PR resolves issue #51265 by refactoring duplicated reader docstrings in python/pyarrow/parquet/core.py to adhere to DRY (Don't Repeat Yourself) principles.

What changes are included in this PR?

  • Centralized shared documentation for thrift_string_size_limit, thrift_container_size_limit, and schema_depth_limit into a module-level constant _READER_COMMON_DOCS.
  • Replaced three duplicated docstring blocks across ParquetFile, ParquetDataset, and _read_table_docstring with {_READER_COMMON_DOCS} f-string interpolation.

Are these changes tested?

Yes, verified syntax validity and confirmed via git diff and grep checks that duplicated string blocks were eliminated without changing the end-user documentation output.

Are there any user-facing changes?

No. Rendered documentation and API signatures remain identical.

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51265 has been automatically assigned in GitHub to PR creator.

@MarkCruz911

Copy link
Copy Markdown
Author

Hi @AlenkaF @raulcd @rok, friendly ping on this! Whenever you have a moment, could you please approve the workflows to run the CI checks and review the changes? Thanks!

@raulcd raulcd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We seem to be using already _read_docstring_common on lots of them, can't this be part of that?
It seems wrong to have both _read_docstring_common and a new _READER_COMMON_DOCS

@MarkCruz911

Copy link
Copy Markdown
Author

Hi @raulcd, thanks for the feedback!

I've updated the PR in commit 7789a60 to consolidate the shared documentation in _read_docstring_common, using "dataset" and "limits" entries, and removed _READER_COMMON_DOCS. All affected readers now use that dictionary.

I also fixed the ParquetFile.doc assignment, preserved the .format() placeholders, and escaped the literal braces in the example.

With AI assistance, I checked syntax and evaluated the four affected docstrings in isolation. I haven't run the PyArrow test suite or built the documentation.

The workflows are currently awaiting maintainer approval. Could you please approve them so CI can run, and take another look when you have a chance?

Thanks!

@MarkCruz911

Copy link
Copy Markdown
Author

Hi @raulcd @AlenkaF @rok, a friendly follow-up on this PR.

I pushed commit 7789a60 on October 17 to address @raulcd’s feedback by consolidating the shared reader documentation in _read_docstring_common and removing _READER_COMMON_DOCS.

Could you take another look when you have a chance? If the CI workflows still require maintainer approval, could you also approve them so the checks can run?

I'm happy to make any further adjustments. Thanks for your time!

@raulcd raulcd changed the title GH-51265: [Python] Refactor and centralize duplicated reader docstrin… GH-51265: [Python] Refactor and centralize duplicated reader docstrings Oct 22, 2026
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51265 has been automatically assigned in GitHub to PR creator.

@raulcd raulcd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, we are pretty overwhelmed with AI PRs and it takes as a long time to tackle reviews so we appreciate your patience.
This is not exactly what I meant. If you take a look on the functions where we are adding those docstrings we end up adding all the _read_docstring_common to the final function not only dataset or limit.
For example ParquetFile contains read_dictionary, binary_type, list_type, memory_map, buffer_size, pre_buffer. The only one that is not part of the API is partitioning which could be factored out as _read_docstring_dataset.
We could then render on ParquetFile, ParquetDataset and read_table_docstring all the _read_docstring_common parameters as we are still joining the dataset on read_table.__doc__ and read_pandas.__doc__.

We could have a single variable (no dataset/limits dict split) and add them together. Factoring out the only non-common parameter.

@MarkCruz911

Copy link
Copy Markdown
Author

Thank you @raulcd, I'm working on that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants