Skip to content

Refactor Parser.Stream#124

Open
AmosNico wants to merge 2 commits into
fgdorais:mainfrom
AmosNico:Parser.Stream
Open

Refactor Parser.Stream#124
AmosNico wants to merge 2 commits into
fgdorais:mainfrom
AmosNico:Parser.Stream

Conversation

@AmosNico

@AmosNico AmosNico commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Currently all Parser.Stream instances (with the exception of OfList, see later) use a wrapper type like String.Slice or Subarray as the stream. Internally the instances use the underlying String or Array for Parser.Stream.setPosition. This means that in practice there are two types of position used: the one on the level of the wrapper (i.e String.Pos in the case of String.Slice) and the one of Parser.Stream (i.e. String.Pos.Raw in the case of String.Slice). The first one is used to update the position in the stream when reading forward, and the second one for storing and restoring the position.
It would make more sense to use the same type for both purposes: Instead of changing the stream when moving forward, use a position to point to the current location in the stream, and update this instead when moving forward.

This pr implements this change, and also provides a new instance for Parser.Stream String Char (which was not possible to do efficiently in the previous approach). The same could also be done for Array and ByteArray. I did not include them in the pr yet, as I am not sure whether they are needed: one can also just use SubArray and ByteSlice (the same remark holds for String).
I removed OfList completely, and replaced it by an instance for List. I think that storing a reference to a List (which is already in memory anyway) and using that list later on, is more efficient then the current approach where the list first needs to be reconstructed before it can be used again.

Note that Parser.Stream does not depend on Std.Stream any more. Instead it has its own version of next?, which uses positions instead of the stream itself.

I also tried some variants:

  • For String/String.Slice parsers, it would be nice to use String.Pos/String.Slice.Pos instead of the raw versions (which would solve the proof issues in Replace Substring.Raw with String.Slice #123). This requires the position type to depend on the specific stream, which was previously not really feasible (since it changed constantly). Given that the parsed stream now remains unchanged, I tried adding it to the parser signature (i.e. add s to the signature, where s : σ ). This worked well, but it might be confusing for users, and I am not sure what the impact at runtime would be. Let me know if you would be interested in this approach, then I could tidy up the code and make a pull request.
  • I tried hiding the token type in a similar way as the type used for positions (which was possible given that Parser.Stream does not depend on Std.Stream any more), but this caused issues in Parser.Char (since this file works for all streams where the token type is Char).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant