Repository navigation
GH-51265: [Python] Refactor and centralize duplicated reader docstrings - #51630
MarkCruz911 wants to merge 3 commits into
Conversation
…cstrings in parquet/core.py
|
|
raulcd
left a comment
There was a problem hiding this comment.
We seem to be using already _read_docstring_common on lots of them, can't this be part of that?
It seems wrong to have both _read_docstring_common and a new _READER_COMMON_DOCS
|
Hi @raulcd, thanks for the feedback! I've updated the PR in commit 7789a60 to consolidate the shared documentation in _read_docstring_common, using "dataset" and "limits" entries, and removed _READER_COMMON_DOCS. All affected readers now use that dictionary. I also fixed the ParquetFile.doc assignment, preserved the .format() placeholders, and escaped the literal braces in the example. With AI assistance, I checked syntax and evaluated the four affected docstrings in isolation. I haven't run the PyArrow test suite or built the documentation. The workflows are currently awaiting maintainer approval. Could you please approve them so CI can run, and take another look when you have a chance? Thanks! |
|
Hi @raulcd @AlenkaF @rok, a friendly follow-up on this PR. I pushed commit 7789a60 on October 17 to address @raulcd’s feedback by consolidating the shared reader documentation in _read_docstring_common and removing _READER_COMMON_DOCS. Could you take another look when you have a chance? If the CI workflows still require maintainer approval, could you also approve them so the checks can run? I'm happy to make any further adjustments. Thanks for your time! |
|
|
raulcd
left a comment
There was a problem hiding this comment.
Sorry, we are pretty overwhelmed with AI PRs and it takes as a long time to tackle reviews so we appreciate your patience.
This is not exactly what I meant. If you take a look on the functions where we are adding those docstrings we end up adding all the _read_docstring_common to the final function not only dataset or limit.
For example ParquetFile contains read_dictionary, binary_type, list_type, memory_map, buffer_size, pre_buffer. The only one that is not part of the API is partitioning which could be factored out as _read_docstring_dataset.
We could then render on ParquetFile, ParquetDataset and read_table_docstring all the _read_docstring_common parameters as we are still joining the dataset on read_table.__doc__ and read_pandas.__doc__.
We could have a single variable (no dataset/limits dict split) and add them together. Factoring out the only non-common parameter.
|
Thank you @raulcd, I'm working on that. |
Rationale for this change
This PR resolves issue #51265 by refactoring duplicated reader docstrings in
python/pyarrow/parquet/core.pyto adhere to DRY (Don't Repeat Yourself) principles.What changes are included in this PR?
thrift_string_size_limit,thrift_container_size_limit, andschema_depth_limitinto a module-level constant_READER_COMMON_DOCS.ParquetFile,ParquetDataset, and_read_table_docstringwith{_READER_COMMON_DOCS}f-string interpolation.Are these changes tested?
Yes, verified syntax validity and confirmed via
git diffandgrepchecks that duplicated string blocks were eliminated without changing the end-user documentation output.Are there any user-facing changes?
No. Rendered documentation and API signatures remain identical.
Was AI used for this PR?
PR code and description written by:
Reviewed before submission by: