Skip to content

Optimize Cython LZ4 compression buffer allocation #991

Description

@dkropachev

Follow-up to #809. A 16 KiB fixed stack buffer is a reasonable short-term compromise, but every compression call still reserves that full stack frame and larger inputs require a temporary heap buffer plus a copy.

Proposed solution: allocate the result bytes at 4 + LZ4_compressBound(input_size), compress directly into it, write the four-byte header, then shrink it to the exact compressed size with _PyBytes_Resize(). This removes the scratch stack buffer, avoids malloc/free, and eliminates the final copy. Benchmark against the 16 KiB stack implementation and document the CPython-private API tradeoff.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions