Skip to content

Fix a bug that Prism.parse_stream may write too much data to internal buffer#4160

Open
kou wants to merge 1 commit into
ruby:mainfrom
kou:fgets
Open

Fix a bug that Prism.parse_stream may write too much data to internal buffer#4160
kou wants to merge 1 commit into
ruby:mainfrom
kou:fgets

Conversation

@kou

@kou kou commented Jul 7, 2026

Copy link
Copy Markdown
Member

pm_source_stream_read() has 4096 bytes internal buffer to read a line:

prism/src/source.c

Lines 368 to 369 in 39fb41f

#define LINE_SIZE 4096
char line[LINE_SIZE];

It uses gets(4095) to read the next chunk to the buffer:

VALUE line = rb_funcall((VALUE) stream, rb_intern("gets"), 1, INT2FIX(size - 1));

But gets(4095) may read 4095 or more data when the character at 4095 is a multi-byte character. In the case, parse_stream_fgets() writes 4097 or more data to the buffer.

We can avoid it by using gets(4095 - 6) not gets(4095). 6 is the max character length in all available encodings. We can reduce it by using stream.external_encoding and rb_enc_mbmaxlen() but it may slow down. So I used constant 6 in this change.

Copilot AI review requested due to automatic review settings July 7, 2026 06:27
@kou kou force-pushed the fgets branch 2 times, most recently from 13471ce to ff6095a Compare July 7, 2026 06:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a potential buffer overflow when Prism.parse_stream reads from a Ruby IO via gets(limit), where gets may exceed the byte limit to avoid splitting a multi-byte character.

Changes:

  • Reduce the gets limit used by parse_stream_fgets to leave headroom for multi-byte characters.
  • Add a regression test that exercises a long line with a multi-byte character at the boundary.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
test/prism/api/parse_stream_test.rb Adds a test covering long-line streaming with a multi-byte boundary case.
ext/prism/extension.c Adjusts the gets limit to avoid overfilling the fixed-size internal line buffer.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/prism/api/parse_stream_test.rb Outdated
# case. gets(4095) may return 4095 or more bytes if the last
# character is a multi-byte character. For example,
# StringIO.new("\u3042").gets(1).bytesize is 3 not 1.
# parse_stream_fgets() uses
Comment thread ext/prism/extension.c
Comment on lines +1084 to +1088
#define MAX_ENC_LEN 6
RUBY_ASSERT(size > MAX_ENC_LEN);

VALUE line = rb_funcall((VALUE) stream, rb_intern("gets"), 1, INT2FIX(size - 1));
VALUE line = rb_funcall((VALUE) stream, rb_intern("gets"), 1, INT2FIX(size - MAX_ENC_LEN - 1));
#undef MAX_ENC_LEN
…al buffer

`pm_source_stream_read()` has 4096 bytes internal buffer to read a
line:
https://github.com/ruby/prism/blob/39fb41f06e5f4ccbe1783122fa8a4470e09dfcfe/src/source.c#L368-L369

It uses `gets(4095)` to read the next chunk to the buffer:
https://github.com/ruby/prism/blob/39fb41f06e5f4ccbe1783122fa8a4470e09dfcfe/ext/prism/extension.c#L1076

But `gets(4095)` may read 4095 or more data when the character at 4095
is a multi-byte character. In the case, `parse_stream_fgets()` writes
4097 or more data to the buffer.

We can avoid it by using `gets(4095 - 6)` not `gets(4095)`. `6` is the
max character length in all available encodings. We can reduce it by
using `stream.external_encoding` and `rb_enc_mbmaxlen()` but it may
slow down. So I used constant `6` in this change.

@Earlopain Earlopain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is this condition

prism/src/source.c

Lines 375 to 384 in 39fb41f

if (length == LINE_SIZE) {
/*
* If we read a line that is the maximum size and it doesn't end
* with a newline, then we'll just append it to the buffer and
* continue reading.
*/
length--;
pm_buffer_append_string(buffer, line, length);
continue;
}

which will now basically never be reached.

Comment thread ext/prism/extension.c
* max character length in all available encodings by the
* following command line in the Ruby source directory:
*
* grep -Eh "max (enc|byte) length" enc/*.c |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

../../../../ext/prism/extension.c:1079:46: warning: ‘/*’ within comment [-Wcomment]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants