Add FATX support - #101
Open
omensight wants to merge 13 commits into
Open
Conversation
The Xbox 360 uses the same filesystem as the original Xbox with all multi-byte on-disk values stored big-endian. libfatx had no endianness handling at all, so it only ever worked on original Xbox disks. Introduce a runtime variant on struct fatx_fs and route every multi-byte value crossing the disk boundary through swap helpers in fatx_endian.h: the superblock, directory entry fields, and FAT entries. Raw file data is deliberately left untouched -- it is a byte stream with no byte order. FAT entries are swapped at the four per-entry access sites rather than at cache fill/flush, because the flush rewrites the entire window including untouched entries and has an early-out when the cache is not dirty, so a fill-time swap would have to be exactly inverted on every flush path. The variant is detected from the partition signature, which is byte-identical in both flavours: the original Xbox writes 'FATX' and reads it little-endian, the 360 writes 'XTAF' and reads it big-endian, and both yield 0x58544146. Whichever way round the raw word matches therefore identifies the disk's byte order. fatx_open_device keeps its signature and defaults to auto-detection; the new fatx_open_device_ex takes an explicit variant. This keeps the three hand-maintained downstream bindings of that prototype working unchanged. Also adds the 360's fixed partition map. It has no drive letters, so partitions are named: sysext, sysext2, compat and data. Note that 0x120eb0000, widely cited as the data partition, is the backwards compatibility partition; data begins at 0x130eb0000. fatxfs gains --variant=auto|xbox|x360 and --partition=<name>. Formatting a 360 partition is explicitly rejected for now; only reading and writing existing ones is supported. Tests: tests/ builds a synthetic big-endian image byte by byte from the format spec rather than with libfatx's own writer, so a field that is never swapped cannot cancel itself out between writer and reader. It checks auto-detection, the directory tree, byte-exact file contents, and that forcing the wrong variant is rejected rather than silently misparsed. The original Xbox path is unchanged and still passes fatxfs/test.sh.
fatx_write_dir declares its raw directory entry on the stack and fills only the first filename_len bytes of the 42-byte filename field, and fatx_attr_to_dirent does the same. Every other byte of the field was written to disk as whatever happened to be on the stack. Readers use filename_len and so are unaffected, but this writes process memory out to the filesystem and makes otherwise identical images differ byte for byte. Pad the unused tail with 0xFF, matching the fill byte fatx_write_superblock already uses.
The 360 partition offsets are not MiB-aligned, so a dd invocation with a round block-count skip lands in the wrong place. This skips in bytes, and checks for the XTAF signature before committing to a large read so a wrong device or a non-standard layout fails immediately rather than after copying gigabytes.
FUSE's -o ro stops the kernel issuing writes, but the device underneath is still opened r+b, so a bug in the driver can still reach the disk. That is not a comfortable guarantee when the thing being mounted is somebody's only copy of a console's hard drive. fatx_open_device_ex now takes flags, and FATX_OPEN_READ_ONLY opens the device "rb" and refuses every write. The check lives in fatx_dev_write, which every write in the library funnels through, so one test covers all of them. fatxfs exposes it as --read-only. The flags argument replaces nothing: fatx_open_device keeps its original signature and behaviour, so the hand-maintained bindings of it in pyfatx and gfatx are unaffected.
Mounting a retail 360 disk showed every timestamp was wrong. The 360
differs from the original Xbox in three separate ways here, not one:
- The epoch is 1980, the standard FAT base, not 2000. The disk's
factory-written files are stamped 2005-11-22, the console's launch
date, which only decodes to that from 1980.
- The time field uses the standard FAT widths, 5 bits of hour and 6 of
minute, rather than the 4 and 5 this library uses for the original
Xbox. The disk carries hours of 21 and 23 and minutes of 50 and 59,
none of which fit the narrower fields.
- The date is stored before the time in each pair, the opposite way
round to the original Xbox, so the struct member named modified_time
holds the date on a 360. Reading the pair in the original Xbox's
order yields impossible calendar dates for a fifth of the entries.
Timestamp packing and unpacking now take the filesystem so they can
resolve all three from the variant, and the slot order is handled in one
place in fatx_attr.c rather than at each call site.
Cross-checks on the same disk: the decoded dates line up with when the
files were actually written, including three separate 2026 working
sessions.
The synthetic test image now uses the 360's rules, with an hour and
minute deliberately outside what the original Xbox's fields can hold, and
the test asserts the decoded timestamp instead of ignoring it.
Also fixes an unrelated hour-sized error that this made visible on both
consoles: fatx_ts_to_time_t left struct tm uninitialized and never set
tm_isdst, so mktime applied daylight saving unpredictably. FATX stores
local wall-clock time with no DST flag of its own, so tm_isdst is now -1
and mktime works it out.
Adds variant, partition and read_only to Fatx, along with the 360 partition map, and surfaces all three on the command line. The cdef, which is a hand-maintained copy of fatx.h, gains fatx_open_device_ex and fatx_x360_partition_to_offset_size. struct fatx_fs is opaque to cffi, so its new fields needed no declaration. The test suite gains a second fixture that builds a synthetic big-endian image, reusing tests/make_x360_image.py so there is one definition of what a 360 filesystem looks like rather than two that can drift apart. Seven tests cover auto-detection, explicit and wrong variants, sizes, contents, subdirectories, and that a read-only filesystem refuses writes. The existing sixteen original Xbox tests are untouched and still pass. Note for anyone running these: the fixtures fallocate an 8 GiB image, so TMPDIR must point somewhere with room. On a machine where /tmp is a tmpfs, fallocate fails, and the suite reports OK in under a second without having really tested anything.
Mirrors the C library. The on-disk structs stay declared little-endian
and are converted at the point they are read, in Superblock::normalize
and DirectoryEntry::normalize, so nothing downstream has to know which
console it is looking at. FAT entries are swapped as they come out of
the cache, matching where the C side does it.
DirectoryEntry::normalize also swaps the two halves of each timestamp,
because the 360 stores the date before the time. The timestamp accessors
take a Variant for the epoch and the field widths, which also differ.
The variant is detected from the raw partition signature, or forced with
--variant. A mismatch between what was asked for and what is on the disk
is an error rather than a silent misparse.
Also adds, all of which the 360 work needed:
- the Xbox 360 partition map, and --partition to select from it
- the "f" partition, which was in the C partition table but missing here
- u64::MAX as "the rest of the device", resolved against the device
length; the partitions that run to the end of the disk need it, and
it previously failed the sector-alignment check
- --offset and --size, for images that do not begin at a known
partition boundary, such as a truncated dump
Verified against a retail 360 disk: this and the C driver produce
identical listings, sizes and file contents for the same partition.
One inconsistency left alone deliberately: this tree interprets FATX
timestamps as UTC while the C tree interprets them as local time, so the
two report times differing by the local UTC offset. That predates this
change and affects the original Xbox equally, so fixing it belongs in its
own commit.
CI runs clippy with -D warnings, so this would have failed the build.
Round-tripping through libfatx cannot establish that the write path is correct. A driver that byte-swaps consistently but wrongly reads back everything it wrote and looks perfectly healthy, because the same mistake is applied in both directions. So verify_x360_dirents.py decodes the raw image with no libfatx code involved, straight from the format: big-endian fields, a 1980 epoch, standard FAT hour and minute widths, and the date before the time. The write tests run it after every mutation. That is what makes the result meaningful. Had the writer used the original Xbox's epoch the timestamps would decode as 2006; its time field widths would mangle the clock; little-endian would garble the cluster and size; and time-before-date would produce impossible calendar dates. All of those would have passed a round-trip test. Covered: create, mkdir, nested files, a 100 KiB file spanning seven clusters (which exercises building and walking a FAT chain), that pre-existing data survives, rename, delete, timestamp round-tripping, and that a read-only mount refuses to write. The decoder also checks that the filename tail is padded rather than carrying leftover stack bytes. Still unverified, and deliberately so: no Xbox 360 has yet read a filesystem this library has written to. These tests establish that the bytes match the format as observed on a retail disk, which is a different and weaker claim.
The synthetic tests establish that the driver is self-consistent. They cannot establish that it reads a real disk correctly, because they only ever read images built to the same understanding of the format. This walks an actual console filesystem end to end and checks it against facts originating outside this project: file formats with their own magic numbers, and STFS packages whose display name sits at a fixed offset as UTF-16BE. A directory walk, FAT chain and byte order that are not all correct at once cannot produce a readable title out of that header. It reads only, and mounts with --read-only so the device is opened without write access rather than relying on FUSE's -o ro. Cross-checks the C and Rust implementations against each other on the same drive. It still is not a listing captured from the console itself, which remains the one check this cannot perform for itself; it writes listing.txt for diffing against one pulled over FTP.
- gate_real_drive.sh: guard python subshells with if/then instead of set -e + $?, which was killing the script silently on expected failures; skip STFS display-name check for Saved Game/Profile/Cache File content types, which don't carry that field - hw_write_test.sh / hw_write_test_cleanup.sh: create and remove an isolated test folder on real hardware, cross-checked against an independent raw decoder and a full before/after listing diff - gate_recheck.sh, diag_stfs_names.sh: supporting diagnostics used while tracking down the STFS display-name false positives - ignore gate-results/, which holds generated test artifacts
fatxfs previously only recognized its options as --long=value flags, which broke fstab/mount(8) usage: a mount helper invoked from fstab gets its options as a comma-separated `-o key=value,...` string, not as separate --flags. Match each FATXFS option both ways. Also fixes a latent bug this surfaced: --partition and --log stored a raw pointer into arg, which is safe for a real argv entry but dangles once arg comes from libfuse's temporary -o comma-split buffer. Both now strdup the value. Add a PKGBUILD so `makepkg -si` installs fatxfs plus a mount.fatxfs helper (mirroring how ntfs-3g is packaged on this system), enabling proper fstab entries with type `fatxfs`.
Summarize the endianness, timestamp, and partition differences the 360 support handles, and note the write path is hardware-verified.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds Xbox 360 support — this library already had a working FUSE driver with full read/write support, but it only ever worked on original Xbox disks, because nothing in it handled byte order. The 360 uses the exact same filesystem, just written the other way around, so it silently failed (or worse, silently corrupted things) on 360 drives.
Original Xbox support isn't touched by any of this — same behavior, same tests all still passing.
What's actually different on a 360
Turns out it's not just "flip the bytes." Once I had this reading a real console-formatted drive, I found four separate things the 360 does differently, and all four had to be fixed before anything decoded correctly:
I found these by comparing against files that were actually written by a real console (some dated to the console's 2005 launch day), not just by reading the spec. A couple of these would have quietly produced wrong-but-plausible-looking dates if I hadn't checked against real hardware.
The library figures out which kind of disk it's looking at automatically (it can tell from the partition signature), so nothing about how you use it changes if you're still on an original Xbox.
Other things that came out of this
mount -othe normal way, and added a package for installing it that way.How I know it actually works
I didn't want to just trust that the driver could read back what it wrote — that only proves it's consistent with itself, not that it's right. So:
So this isn't just "looks right on paper" — an actual console has read data this wrote.