Infinite loop while reading file

The following code enters in an infinite loop on line

var lines = outlines.ToList();

Custom.csv file has been added.

Uncomment line with BufferSize makes the code run perfectly.

Loop has been detected so far at CSVFile Line 58

using CSVFile;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;

namespace CSVFileLoader
{
    internal class Program
    {
        static void Main(string[] args)
        {
            var filename = @"Custom.csv";
            FileInfo f = new FileInfo(filename);

            Console.WriteLine($"File size = {f.Length}");

            CSVSettings settings = new CSVSettings
            {
                LineSeparator = "\n",
                FieldDelimiter = '\t',
                //BufferSize = (int)f.Length
            };

            var csv = CSVReader.FromFile(filename, settings);
            
            var outlines = csv.Lines();
            var lines = outlines.ToList();
            Console.WriteLine($"Real number of lines = {lines.Count}");

            // Count amount fields by line
            var fieldCountDictionary = new Dictionary<int, int>();
            foreach (var line in lines)
            {
                int fieldCount = line.Length; // Amount of fields
                if (fieldCountDictionary.ContainsKey(fieldCount))
                {
                    fieldCountDictionary[fieldCount]++;
                }
                else
                {
                    fieldCountDictionary[fieldCount] = 1;
                }
            }

            // Show number of rows by number of fields
            foreach (var kvp in fieldCountDictionary)
            {
                Console.WriteLine($"There are {kvp.Value} rows with {kvp.Key} fields");
            }
        }
    }
}

For information, issue happens while reading line 1474 of Custom.csv file
Record uses 3 lines, but I don't think it's the reason why, because setting BufferSize to file Length bypass the issue.

Thanks for this report! Checking. I've seen some weirdness around buffer sizes straddling UTF-8 characters, this may be one of those problems.

Same issue hit us in prod, been using for years, then a particular csv file came in that went into infinite 100% cpu breaking it, and the other services running on that cluster since they all got cpu starved. Wasn't a character sequence, created a little console app to diagnose, parsed first 250 rows fine then copy pasted them over and over. Deleted back down and under 313 rows it was fine, but 314 it would freeze up, unless we pasted a different row for 314 then we could add dozens of more rows. But same line that was on 314 was in the file for 9 other rows since we were getting copy paste happy while diagnosing.

Fyi, narrowed down the problem. If a csv goes over the buffer size while in the middle of a Text qualifier it goes into an infinite loop.
the fix is to add

  if (_inTextQualifier)
  {
      return true;
  }

        public bool NeedsMoreText()
        {
            // If we're inside a text qualifier, we always need more text unless we've found the end
            if (_inTextQualifier)
            {
                return true;
            }

            return String.IsNullOrEmpty(_line) || _position + _settings.LineSeparator.Length >= _line.Length;            
        }

Thank you so much for these findings! Would you like to submit a PR for this or would you like me to create one? I'll be happy to test and release this fix.

Thank you so much for these findings! Would you like to submit a PR for this or would you like me to create one? I'll be happy to test and release this fix.

I 100% meant to create a pull request, but I ran out of time today, and I'm going on pto till wed 15. I'll try to remember and check in. If you haven't already resolved then I'll open a pull request then.

@james-perfectserve

Thank you so much for these findings! Would you like to submit a PR for this or would you like me to create one? I'll be happy to test and release this fix.

I 100% meant to create a pull request, but I ran out of time today, and I'm going on pto till wed 15. I'll try to remember and check in. If you haven't already resolved then I'll open a pull request then.

@james-perfectserve I have started a PR, and it looks like there's some major changes I need to make to my GitHub actions for automated analysis. I'll keep going with PR #70 but if you could glance at it I'd appreciate your feedback!

        public static IEnumerable<string[]> ParseStream(StreamReader inStream, CSVSettings settings = null)
        {
            int bufferSize = settings?.BufferSize ?? CSVSettings.DEFAULT_BUFFER_SIZE;
            var buffer = new char[bufferSize];
            var machine = new CSVStateMachine(settings);

            while (machine.State == CSVState.CanKeepGoing)
            {
                var line = string.Empty;

                if (machine.NeedsMoreText() && !inStream.EndOfStream)
                {
                    var readChars = inStream.ReadBlock(buffer, 0, bufferSize);
                    line = new string(buffer, 0, readChars);
                }

                var row = machine.ParseChunk(line, inStream.EndOfStream);

                if (row != null)
                {
                    yield return row;
                }
                else if (inStream.EndOfStream)
                {
                    break;
                }
            }
        }

for some additional context, this is where the infinite loop was happening it was because machine.NeedsMoreText() was returning false, but it wasn't at the end of the file yet so rather than pulling the next part of the stream it kept trying to parse the empty string from var line = string.Empty; over and over and over rather than reading next block.

after the tweak to NeedsMoreText() it return true and cleanly reads the steam till the end.

@james-perfectserve

@james-perfectserve are you able to confirm that release 3.2.1 addressed this issue? I added some tests as well so I can see the new code is being exercised.

@james-perfectserve are you able to confirm that release 3.2.1 addressed this issue? I added some tests as well so I can see the new code is being exercised.

I switched to 3.2.1 and didn't get any infinite loops. My same test but using 3.2.0 and I get the infinite loops.
Looks like 3.2.1 correctly addressed the issue 👍

Thank you so much! Closing this issue.

Also, if you would like to submit these tests as a PR I'd welcome the contribution!

Navigation Menu

Repository navigation

Infinite loop while reading file #68

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Infinite loop while reading file #68

Description

Activity

AlainBartmanDilaw commented on Oct 17, 2024

tspence commented on Nov 12, 2024

james-perfectserve commented on Oct 9, 2025

james-perfectserve commented on Oct 9, 2025

tspence commented on Oct 9, 2025

james-perfectserve commented on Oct 9, 2025

tspence commented on Oct 9, 2025

james-perfectserve commented on Oct 9, 2025

tspence commented on Oct 12, 2025

james-perfectserve commented on Oct 15, 2025

tspence commented on Oct 15, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions