When firstByte is > 0, detect if it's inside a quote field or not

[severo]

When parsing from a random byte position, we cannot know for sure whether we're inside a quoted field.

This means that we cannot know if we should:

• create a new field when the character is the delimiter,
• create a new line when the character is the newline,
• escape a quote character when the character is the quote character
• ignore a comment when the character is the comments character

The best way to handle this is to try the different options on the first rows and use a heuristic to decide which is best.

We could use "Dialect detection" from DuckDB as an example: https://duckdb.org/docs/stable/data/csv/auto_detection#dialect-detection. Its two criteria, by priority, are:

(1) a consistent number of columns for each row, and (2) the highest number of columns for each row.

There are multiple cases:

• in a quoted field or not
• the first character is \n (it might be the second character of \r\n)
• the first character is " (or another quote, or escape, character)

i.e., parsing is a state machine, and we have to detect in which state we start.

I think an algorithm might be to test all the possible states (inside a quoted field, etc) and on each state, count the number of columns for each row, for the first N rows, and choose the state with the best criteria. It might help to start some bytes before or after.

[severo]

Note that we might also accept some quirks, and primarily decide if we start in quoted field or not in quoted field. It's a quick win that helps in most of the cases.

[severo]

For inspiration: https://github.com/duckdb/duckdb/tree/main/src/execution/operator/csv_scanner, in particular the "sniffer" and the "state machine".

It's awfully long, but possibly we have to do something like this:

https://github.com/duckdb/duckdb/blob/main/src/execution/operator/csv_scanner/sniffer/dialect_detection.cpp

See the last part for the high-level logic: https://github.com/duckdb/duckdb/blob/main/src/execution/operator/csv_scanner/sniffer/dialect_detection.cpp#L605