6502: What processing should be done in aambundle?

[dstelzer]

Right now, when building a 6502 disk image, aambundle passes the .aastory through unmodified. This means that all the processing needs to happen on the 6502 itself—everything from parsing the CSS to building the font. (aambundle includes the entire font data file in the C64 disk image, and it's the interpreter's job to select and use the glyphs it needs from that.)

Some of this processing could be done far more easily in aambundle itself. The CSS could be preprocessed, for example, into the eight-byte blocks the 6502 interpreter uses. This would make the 6502 interpreter both smaller and faster.

The cost, though, would be making the 6502 interpreter less capable—it would need to be used with aambundle, not on its own. (But is anyone using it on its own as it is?)

Alternatively, aambundle could add a new chunk to the .aastory file, containing machine-readable data for the 6502 interpreter. The 6502 would use the new chunk if it was available, and parse the raw CSS (vel sim) if not. This would help with interpreter speed, but not with interpreter size—the CSS parsing code would still need to be in there. (The .aastory could be made smaller by removing the now-unneeded chunks, but I don't think that's where bytes really need to be saved, is it?)

What do people think?

[dstelzer]

Tagging in @susan-davis and @sehugg as the people with, I believe, the most 6502 expertise.

[dstelzer]

Previous discussion of the style data format

[sehugg]

I discovered that aambundle's trim_chunks() already rewrites the file -- it adds padding before the WRIT chunk and removes FILE chunks. So 6502 interpreters never could run raw files anyway.

My proposal is that aambundle replaces the LOOK chunk with a new internal format that is version-locked to the interpreter. It has three arrays:

• geo[] - 8 byte geometry records (margin/padding, width/height, float, alignment, etc)
• sty[] - 4 byte style records (style on/off, fg color, etc)
• xsty[] - sparse array with c64 palettes for SET_STYLE (and future uses)

Okay, actually two more arrays:

• geoidx[] - map class index to geo[] slot, or $00 if no style data
• styidx[] - map class index to sty[] slot, or $00 if no style data

This means that style and geometry data can be de-duplicated separately -- many classes contain one or the other. The new chunk is specific to the system+interpreter being bundled, and omits all attributes that can't be processed (e.g. apple2 will only have "reverse" bit, so it will only have 1 style record max)

As an example, "familiar" has 27 classes, which map to 15 geo records and 6 sty records on the c64.

This will also be paired with a CSS parser in aambundle similar to the one in frontend.c. It'll process Z-machine-compatible attributes and also things like "-iftf-c64-color: red". It could emit warnings, like when you try to do italics on the Apple it suggests a nice reverse instead.

[sehugg]

Actually this is simpler but same concept:

• classidx[] - map class index to slot index (0 = no style, hi bit set = div)
• slots[] - packed array of spans (4 bytes) mixed with divs (8 bytes)

So a classidx = index*4 in the slots buffer, but the hi bit tells you if the record is 4 or 8 bytes.

;;; div and span attributes
0   styon       ; style bits to set
1   styoff      ; style bits to unset
2   fg          ; foreground color, system-specific, $80 = inherit
;;; div-only attributes
3   flags       ; relw, relh, float, align
4   vmargins     ; margin-top << 4 | margin-bottom, in rows
5   width       ; width in columns
6   height      ; height in rows
7   padleft     ; left indent in columns (future)

For SET_STYLE and other rarely-used features, instead of a sparse array there could be an 'extend' bit in the record that tells you to look for addtional data in the following 4 or 8 bytes.

[dstelzer]

Is a few bytes in the story file worth more time unpacking the structure? If aambundle is the only thing creating this new chunk, I'd think it would be easier to store the data in whatever format the interpreter wants; time in aambundle is much cheaper than time in the interpreter.

[sehugg]

The 'div' structure is pretty much how the interpreter expects it right now with a few fields rearranged. The additions are a lookup table from class -> slot and having a 4-byte version for span/style-only classes.

The savings with this scheme don't really kick in until you get files with a larger # of classes. For example, one story has 37 classes, which encodes to 300 bytes with a naive array and <200 bytes in the mapped/slotted format.

In any case, replacing the LOOK chunk with even a naive array will save ~800 bytes of CSS parser and 2000+ bytes of CSS text, so it's the right move--the optimal encoding won't be clear until digging in to the code (and may change with new interpreter features)

The thing that doesn't exist yet is changing C64 palettes with SET_STYLE, so that'll be a new encoding.

[sehugg]

Actually you make a good point, it's easier to start with what exists (the flat array) and optimize later. This means the 6502 changes are trivial until new features are added.

Do you have thoughts on naming? I like the prefix "U" (for "micro") so I suggest USTY for the new chunk. When those files are written out for testing or debugging, they could have a ".ustory" suffix so they are not mistaken for the real .aastory format. aamshow would still be able to parse it and decode the current version of the USTY chunk.

The CSS parser in aambundle would be similar to the one in frontend, but issue more warnings and handle system-specific overrides. For example, background-color would be a warning, because C64 does not support it for divs -- it would suggest using (body style $) and the -iftf-sys-c64-background-color (or whatever) as an alternative.

[dstelzer]

I like the name USTY. It would be nice to tie it to LOOK, which is the chunk it's replacing, but I can't think of any way to abbreviate LOOK to 3 letters that doesn't look a bit ridiculous (LOO? LOK? OOK?). And then the U prefix can be used for other chunks that get preprocessed; the only existing chunk that starts with U is URLS.

[dstelzer]

So for the format of USTY, at the moment:

• BYTE: chunk format (currently 0, so that we don't need a new chunk name if we update the format later)
• BYTE: number of style classes (the Å-machine supports up to $3FFF of them, but I don't think the 6502 interpreter can index more than 255 as it's currently implemented?)
• BYTE[N]: width
• BYTE[N]: height
• BYTE[N]: top margin
• BYTE[N]: bottom margin
• BYTE[N]: styles to set
• BYTE[N]: styles to unset
• BYTE[N]: flags
• BYTE[N]: currently unused

So to look up (say) the set-styles for style class 5, you would start at the top of the chunk, add two (the first two bytes), add N (skip past width), add N (skip past height), add N (skip past top margin), add N (skip past bottom margin), then index with 5.

Does that seem like an efficient way to handle it on the 6502? I'm still getting used to which things are idiomatic and which aren't.

[sehugg]

The first two bytes are pretty much as you stated (though there will be a couple more header bytes for the SET_STYLE table later)

The rest is close, but instead of a set of arrays it's just one array of structs. In C it would look like this:

   typedef struct {               
       uint8_t width; 
       uint8_t height; 
       uint8_t margin_top;          
       uint8_t margin_bottom;
       uint8_t styon;
       uint8_t styoff;
       uint8_t flags;
       uint8_t fg;                 /* color 0..15, $80 = inherit */
   } usty_div;

   usty_div styles[MAX_STYLES];

This is exactly how the 6502 CSS parser stored it after parsing, and USTY keeps the same layout. Each element is 8 bytes, so finding a struct address just requires multiplying the style class by 8, adding it to the base array address, and storing it in a zero-page pointer. Then finding an attribute is pretty easy, for example:

        ldy     #STY_STYON     ; STY_STYON = 4, offset of styon field
        ora     (phydata),y
        sta     rstyle

The nice thing is nothing changes except instead of calling the CSS parser, it calls readdata to load the array from disk.

[dstelzer]

That all sounds reasonable to me!

My only concern is with having "C64 foreground color" be explicitly stored—how would that interact with the style bits? If inside a div with a default color of green, for example, what should italic do, and what should un-italic do?

[sehugg]

An explicit color attribute could either override all text colors, or just override the "normal" color. I think the latter is more flexible, especially if a future SET_STYLE feature lets you change the entire color palette.

Now, should the Apple II warn if you try to change the color? Probably, but it might get annoying. I may add some simple warning flags like Dialog has.