No way to express byte-oriented processing: Text is char-indexed and Binary is opaque (length only, cannot index/iterate/slice)

[logbie]

Summary

There is no documented way to express byte-oriented processing in WFL. Text is character-indexed with no byte view, Binary supports only length (it cannot be indexed, iterated, or sliced), and the 190-function native registry has no ord/chr/hex/base64/encode/decode. A byte-faithful port of a byte-oriented program is not expressible at usable speed.

Reproduction

g7_bytes.bin contains the 6 bytes 41 42 FF FE 43 44 (AB, two invalid-UTF-8 bytes, CD).

open file at "g7_bytes.bin" for reading binary as fin
wait for store payload as read binary from fin
close file fin

display "type: " with typeof of payload
display "length: " with length of payload

try:
    display "index: " with payload at 0
when error:
    display "index FAILED: " with error_message
end try

try:
    for each byte_item in payload:
        display "iter: " with byte_item
    end for
when error:
    display "iterate FAILED: " with error_message
end try

try:
    display "slice: " with slice of payload and 0 and 2
when error:
    display "slice FAILED: " with error_message
end try

try:
    display "substring: " with substring of payload and 0 and 2
when error:
    display "substring FAILED: " with error_message
end try

Command:

wfl g7_binary_opaque.wfl

Expected

Some documented way to read the numeric value of byte k.

No documentation citation exists claiming any of these work — this is a
capability gap, not a broken runtime. Docs/04-advanced-features/file-io.md:152-155
states the position deliberately:

Text reads (read content) require valid UTF-8, so they corrupt or reject
non-text files such as fonts, images, PDFs, or compressed archives. For those,
use WFL's binary file operations, which preserve every byte exactly.

The binary section then documents exactly three operations —
read binary (:162), read N bytes (:179), and write binary accepting a
list of byte numbers (:185-197). Nothing documents reading a byte back out.
So the asymmetry is the concrete request: the byte encoder exists
(list of numbers → Binary → file), the byte decoder does not.

Actual

type: Binary
length: 6
index FAILED: Cannot index Binary with Number
iterate FAILED: Cannot iterate over Binary
slice FAILED: Error in native function: Runtime error at line 0, column 0: Expected a list, got Binary
substring FAILED: Error in native function: Runtime error at line 0, column 0: Expected text, got Binary

Exit code 0 (each failure was caught). The type checker rejects all four at
compile time as well, so this is a closed set rather than one unwired code path:

error[ERROR]: Cannot index into Binary - Expected List of Unknown but found Binary
error[ERROR]: Collection in for-each loop must be a list or map - Expected List of Unknown but found Binary
error[ERROR]: Argument 1 of builtin 'slice' expected List of Any, but found Binary
error[ERROR]: Argument 1 of builtin 'substring' expected Text, but found Binary

length of payload → 6 is correct, and Binary values compare correctly with
is equal to. Those two are the entire supported surface.

No byte primitives in the registry

Full dump of registered natives
(grep -rho 'define_native("[a-z_0-9]*"' src/ | sort -u) — 190 functions, of
which zero match byte|hex|base64|ord|chr|charcode|codepoint|encode|decode|utf8.
(secure_random_bytes produces hex text; it does not inspect bytes.)
There is also no seek: no seek token in the parser and no mention in
Docs/04-advanced-features/file-io.md, so read N bytes can only ever return a
prefix.

Text cannot carry arbitrary bytes either (documented, working as intended)

open file at "g7_bytes.bin" for reading as fin
wait for store payload as read content from fin
close file fin

→ error[ERROR]: Failed to read file: stream did not contain valid UTF-8: invalid utf-8 sequence of 1 bytes from index 2, exit 1
(src/stdlib/filesystem.rs:258).

This part is correct behaviour, not a defect — it is exactly what
file-io.md:152 says will happen, and it is included here only to show that the
Text route is closed by design, leaving Binary as the documented alternative.

The only route that works, and what it costs

Byte k can be recovered by induction: knowing bytes 0..k-1, construct each of
the 256 candidate prefixes as a number list, write binary it, read it back, and
compare against read (k+1) bytes of the target. Verified working — it correctly
decoded the first 8 bytes of a file containing 00 01 02 …:

decoded bytes: [0, 1, 2, 3, 4, 5, 6, 7]

Measured worst case on a 16-byte file of 0xFF (every byte needs all 256 trials):
22.84 s for 16 bytes ≈ 1.43 s/byte, and the cost grows with position because
the whole known prefix is rewritten for every candidate — O(n²·256) overall, with
two filesystem round trips per candidate. Extrapolated to the 630,176-byte fixture
this port needed, that is on the order of days.

(For the record: I could not confirm a separately reported figure of ~0.26 ms/byte
for an in-memory 256-entry comparison table. A table of 256 one-byte Binary
values can be built and compared quickly, but it cannot be used to decode a file,
because there is no way to extract byte k of a multi-byte Binary to compare
against it. My 1.43 s/byte figure is for the only route I could make actually work.)

Related, not duplicate

#573 (OPEN) — "Web server cannot serve binary content" — covers byte-lossless
transport through the web server. This issue is about byte-level inspection and
manipulation of values inside the language. Fixing either does not fix the other,
but they share a root: bytes are second-class.

Shape of a fix (maintainer's call)

Any one of these would close the gap; the first is the smallest:

• byte at <n> of <binary> returning a Number, plus for each byte in <binary>;
• a bytes of <binary> → list of Numbers conversion (the exact inverse of the
  already-documented write binary <list of byte numbers>);
• to_hex / from_hex (or base64) on Text/Binary;
• ord / chr on single-character Text, which would also serve the
  character-oriented cases.

Environment

• wfl --version: WebFirst Language (WFL) version 26.8.4
• binary: system install C:\Program Files\wfl\bin\wfl.exe
• commit: c277d8f
• OS: Windows 11 Pro 10.0.26200
• build: release

Also reproduces on the repo build (26.8.2). No .wflcfg in scope.

Context

Found while porting G:/repos/JShrink/src/JShrink/Minifier.php (a 738-line PHP
JavaScript minifier) to WFL. JShrink is byte-oriented throughout: strlen,
$input[$index], and a $char < "\x20" control-character test all operate on
bytes, and PCRE without /u classifies each byte independently.

This one shaped the port: with no byte view available, the port was written
character-based. For pure-ASCII input the two are identical, and both JShrink
parity fixtures are pure ASCII (tests/Resources/libraries/jquery_ui.js, 630,176
bytes, no byte ≥ 0x80). For non-ASCII input the ports can differ in
whitespace-elision decisions around a multi-byte character, because PHP classifies
each byte while WFL classifies the whole character. The port's parity claim is
therefore scoped to ASCII input, and that scoping is a consequence of this gap
rather than a choice. No stub was left.

[logbie]

Related: #712 asks for the same capability from the other direction — byte-oriented access to a Text value, rather than inspection of a Binary value.

They are separable but share a likely fix: a bytes of <value> conversion returning an indexable list of numbers, with a lossless inverse, would close the Text side in #712 and the Binary side here at once.

Concrete motivation for the pair, from the port that surfaced both: a WFL translation of a byte-indexed PHP program matches the original byte-for-byte on 47 of 48 differential cases, and the single divergence is caused solely by WFL dividing the string into characters where PHP divides it into bytes. The inductive Binary decode route described above was measured at ~1.43 s/byte, so it is not a viable substitute.

[logbie]

Warden assessment: confirmed capability gap (not a bug) — and it is wider than file I/O

Verified every claim against main @ 36de4fa7 (v26.8.8). All four rejections and the missing-primitive claim hold exactly as reported.

The closed set, confirmed:

claim	evidence on main
Binary is opaque	Value::Binary(Arc<[u8]>) — src/interpreter/value.rs:25
only length accepts it	native_length has the sole Value::Binary arm in the stdlib — src/stdlib/list.rs
cannot index	runtime "Cannot index {} with {}" — src/interpreter/mod.rs:14584; typechecker "Cannot index into {collection_type}" — src/typechecker/mod.rs:8392
cannot iterate	"Cannot iterate over {}" — src/interpreter/mod.rs:7654
no ord/chr/hex/base64/encode/decode	zero matches anywhere under src/stdlib/
encoder exists, decoder does not	write binary accepts Type::Binary or List of Number — src/typechecker/mod.rs:6173-6181

So the asymmetry framing in the report is precisely right: the byte encoder (list of numbers → Binary → file) is wired, the byte decoder is absent.

What the report understates. Binary is produced in four places on main, not just read binary:

1. read binary → Type::Binary (typechecker/mod.rs:9165)
2. read N bytes → Type::Binary (:9199)
3. body_bytes on an HTTP request → Type::Binary (:7185)
4. wait for next chunk → Optional(Binary) (:4506)

(3) and (4) matter more than the file case. A WFL web server can receive a binary upload or a streamed chunk and cannot examine a single byte of it — it can only ask how many bytes arrived. That is not an ergonomics gap, it closes off request handling for any non-text content type, and it pairs with #573 (binary response bodies) to make the server text-only in both directions. I would weight this issue higher than "port fidelity" on that basis alone.

On the bytes of <value> proposal (your comment above, shared with #712): agreed, and I think it is the cheapest complete fix rather than merely the most convenient one. It reuses the list-of-numbers representation write binary already accepts, which makes the round-trip closed and provably symmetric — bytes of becomes the exact inverse of write binary <list>. No new type, no need to design indexing/iteration semantics on Binary itself, and it is purely additive, so TestPrograms/ risk is nil (nothing today can produce a passing program that this changes).

One design constraint worth settling before implementation. bytes of on a 6-byte fixture is free; on a 50 MB upload it materialises 50M Value::Number entries (f64 each — well over 400 MB of Vec). Since (3) above means the input size is attacker-controlled on a web server, an unguarded bytes of is a memory-exhaustion primitive. Two options, and I'd suggest deciding now rather than retrofitting:

• a bounded/slicing form — bytes of payload from 0 to 8 — which also directly serves the header-sniffing use case the docs motivate; or
• keep bytes of whole-value but subject it to the existing resource-budget policy from feat: add shared ExecutionBudget consolidating runtime resource caps #609/[Low priority] Make all PR #609 resource-budget policies overrideable through .wflcfg #612.

Note read 8 bytes from <handle> already gives a cheap prefix for the file case, so header sniffing is half-solved — you get the short Binary, you just still cannot read the numbers out of it. That makes the bounded form a small addition rather than a new concept.

Verdict: confirmed valid enhancement, capability gap, no documentation defect (the docs correctly never claim these work). Suggested labels: enhancement, stdlib, binary, and I'd add web given (3). Sequencing: this is the prerequisite slice for #712 — settle the Binary → numbers direction here, and #712's Text → bytes direction becomes a second entry point onto the same representation.

Posted by the WFL repo warden (automated triage pass).