Let a program choose byte or character semantics for Text (byte length / byte-at-index have no counterpart today)

[logbie]

Summary

WFL should let a program choose byte-oriented or character-oriented semantics when
indexing and measuring Text. Today only character semantics exist, and there is no byte
counterpart to length of or substring of. Character semantics is the right default and
should stay the default — but a program that must reproduce another system's byte behaviour,
or that processes a byte-oriented format, currently has no way to ask for it.

This is a request for the choice. It is deliberately not a request to change what Text
already does.

Motivation — a concrete case, measured

Porting JShrink (a PHP JavaScript minifier) to WFL
produced a port that matches the original byte-for-byte on 47 of 48 differential test cases,
including the 609,574-byte jquery_ui.js. The single divergence is caused only by the
indexing model.

PHP indexes strings by byte, so the two bytes of π (CF 80) are classified separately.
WFL indexes by character, so π is one unit. Given var π = Math.PI, radians = π / 180:

PHP (byte-indexed):   ...,radians=π/ 180,degrees=180 / π;...
WFL (char-indexed):   ...,radians=π / 180,degrees=180 / π;...

One space. Both are valid, semantically identical JavaScript.

Worth being precise about the cause, because it is not the character class: the letter test
agrees in both models. preg_match('/^[\w\$\pL]$/', "\xCF") returns 1, since PCRE in non-UTF
mode reads 0xCF as Latin-1 Ï, a letter. The divergence comes purely from how the string
is divided into units.

To confirm that, I patched a copy of the PHP original to change exactly two things — index a
UTF-8 character array instead of bytes, and add /u to the letter test — and ran all 48 cases.
Exactly one output changed: the one above. That patched copy is a faithful model of WFL's
semantics, and the WFL port matches it on 48 of 48. So the port is provably correct for
character semantics; it simply cannot express the byte semantics the source had.

Current state

// "var π" is 5 characters but 6 bytes (π is CF 80 in UTF-8).
store src_text as "var π"
display "length of      : " with length of src_text
display "char 4         : " with substring of src_text and 4 and 1

length of      : 5
char 4         : π

Exit 0 — correct for character semantics. There is no byte-oriented counterpart to either
line: nothing in the stdlib reports the byte length of a Text, and nothing retrieves byte
k. The only byte-named natives are secure_random_bytes (crypto) and content_bytes (web),
neither of which applies to a Text value.

Why the existing binary path does not serve this

Binary is not an escape hatch here — see #702. It reports a correct byte length but cannot
be indexed (Cannot index Binary with Number), iterated (Cannot iterate over Binary), or
sliced. The only working decode route found was inductive — trial-write each of 256 candidate
prefixes, read back, compare — measured at roughly 1.43 s/byte and O(n²), which extrapolates
to days for the 609 KB fixture above. For contrast the character-based port processes that
whole file in 17 seconds.

Binary also only comes from file reads, so it cannot answer a question about a Text value
already in hand.

This issue is therefore related to but distinct from #702: that one asks for Binary to be
inspectable; this one asks for Text to be addressable in bytes when a program needs it.

What "either or" might look like

Sketching shapes rather than prescribing one — the maintainer's call:

• Byte-oriented counterparts alongside the existing functions, e.g. byte length of <text>,
  byte at <text> and <n>, byte substring of <text> and <start> and <count>. Additive, no
  effect on existing programs, and reads in the existing natural-language style.
• A conversion, e.g. bytes of <text> returning an indexable list of numbers, plus a
  text from bytes inverse. This also closes most of No way to express byte-oriented processing: Text is char-indexed and Binary is opaque (length only, cannot index/iterate/slice) #702 if Binary gained the same
  conversion.
• ord/chr equivalents (code of <char>, character from <code>), which would separately
  remove the need for the lexicographic ch is less than " " idiom currently used to detect
  control characters, and would make \x-style escapes expressible.

Whatever the shape, the properties that matter for this use case are: get the byte length of a
Text; get the numeric value of byte k in O(1); and rebuild a Text from bytes losslessly.

Why character-by-default is right, and should not change

Character indexing is the correct default for a language whose first principle is
natural-language readability — length of "café" should be 4. Nothing here argues otherwise.
The gap is that byte semantics are currently unreachable, so a whole class of programs
(faithful ports, wire protocols, byte-exact tooling, anything reproducing a byte-oriented
system's observable behaviour) cannot be written correctly at all.

There is a No-Unlearning angle too: a beginner correctly learns that length of counts
characters, and today a production user who needs bytes has nowhere to grow into. An additive
byte vocabulary would give them that path without changing the beginner's form.

Environment

• wfl --version: WebFirst Language (WFL) version 26.8.4
• binary: system install C:\Program Files\wfl\bin\wfl.exe
• commit: c277d8f
• OS: Windows 11 Pro 10.0.26200
• build: release

Context

Found while porting G:/repos/JShrink/src/JShrink/Minifier.php (738 lines of PHP) to WFL. The
port is complete, passes 77/77 behaviour tests and 48/48 differential parity, and was not
worked around — the divergence is documented in the port rather than papered over, and this
issue is the honest reason it exists. Related: #702 (Binary opaque).

[logbie]

Warden assessment: valid enhancement, unusually well-evidenced — and the design framing in the report is the right one

Verified against main @ 36de4fa (v26.8.8) to the extent a static read allows: there is no byte-length or byte-index counterpart to length of / substring of in the stdlib, and the only byte-named natives are the crypto and web ones you identify, neither of which applies to a Text in hand.

The 47-of-48 → 48-of-48 experiment is the strongest part of this report and it should not get lost. Patching the PHP original to use character indexing and re-running the full corpus, and getting exact parity, converts "the port has a bug" into "the port is provably correct for character semantics and the semantics are the variable." That is a much stronger claim than a differential test alone would support, and it is what makes this an enhancement request rather than a defect.

On design fit, since that is where this will live or die. The report's insistence that this is a request for the choice, with character semantics staying the default, is exactly the framing that keeps it viable. Concretely, that argues for additive vocabulary — a byte length of and a byte-at-index operation as separate named operations — and against a mode switch that changes what length of means. A mode switch would be a direct no-unlearning violation: the same expression would mean different things in different files, and a reader could no longer tell what length of does without hunting for a pragma. Additive vocabulary costs a beginner nothing, because they never encounter it until they need it.

On sequencing, one suggestion. This issue and #702 are close to the same request seen from two ends, and #702 looks like the cheaper and more general first slice. Binary already reports a correct byte length but cannot be indexed, iterated, or sliced — fix that, and a Text→Binary conversion gives byte semantics without introducing any new Text vocabulary at all. That route also retires the inductive 1.43 s/byte decode workaround, which is the kind of measurement that should not survive in a repo's issue history for long. If #702 lands first, this issue may reduce to a single bytes of <text> conversion rather than a parallel indexing model.

Not proposing that as a decision — just noting the dependency, since the two were filed together and the ordering materially changes the size of this one.

Posted by the WFL repo warden (automated triage pass).