Move as much code as possible from book-to-pef to pipeline-modules

[bertfrees]

Hyphenation

• Produce the hyphenation files in a separate project, deploy it somewhere (e.g. in Maven Central in a ZIP) and import it in libhyphen-utils.

  Some useful work has been done in this repository: https://github.com/nlbdev/spell-no (the "norsk/patterns" subdirectory). It includes a build script for the patterns file and a UI for viewing the input words (that are used to build the patterns file) and hyphenations.

  In progress: see Pack hyphenation table in ZIP and publish to Maven spell-no#1.

Script options

• The script options that are generic enough can be moved to xml-to-pef. This is linked to the "Style sheets" section below because most of the script options are implemented using style sheets. Note that, just like the "Style sheets" section, this is not an essential item because options that are solely used to set Sass variables can now be handled with the "Style sheet parameters" option.
• The Norwegian translations can also be moved once we have support for internationalization. This is only relevant for manual productions.
• (What would be really nice, but probably out of scope, is if we would be able to analyze a provided style sheet and based on this dynamically compute the relevant options and present only these to the user.)

Translator features

• Recognizing the CSS property text-transform: uncontracted: could be made into a generic feature: done in Support "text-transform:uncontracted" in the generic translator daisy/pipeline-modules#9

• Selecting sub-translators for different languages: should be moved to liblouis-utils (Braille translator should handle multi-language documents daisy/pipeline-mod-braille#196).

• Overriding how undefined characters should be handled: could be made into a translator option. Either just via an argument in the constructor of LiblouisTranslatorImpl, or it could be configurable via a "query feature". See "dots-for-undefined-char" feature for Liblouis translator daisy/pipeline-mod-braille#206.

• Handling of @text-transform strong { system: -nlb-indicators; open: '⠠⠄'; close: '⠠⠄'; } etc.: can possibly be generalized, although I would much prefer to do this via Liblouis. For this, we need to be able to control the order of the begin/end marks (Web UI handling of em and strong #217 (comment), How to control order of emphasis indicators for multiple emphasis? liblouis/liblouis#922).

• Marking of URLs and e-mail addresses. This should be done by Liblouis. Unfortunately Liblouis doesn't have any opcodes to insert indicators to announce and close a computer braille string, but I think it should be possible to accomplish. The default Liblouis table for Norwegian uses begcomp and endcomp (which are undocumented opcodes) to insert indicators to announce and close a computer braille string, but the NLB version of the table for some reason has disabled this and it has been implemented in Java.

  This is "on hold" because Dawn and Lars are figuring out whether the requirement needs to be updated.

• px:nfc should be moved to liblouis-utils, but not as a separate XProc step. Instead Liblouis tables could contain metadata about which Unicode variants are supported (NFC and/or NFD) (Table metadata to indicate supported Unicode normal forms liblouis/liblouis#923), and liblouis-utils should then do the conversion if needed (Liblouis based translator: do Unicode normalization step when needed daisy/pipeline-mod-braille#197). Ideally this should be handled within Liblouis itfself (glyph vs diaeresis liblouis/liblouis#98). Other related issues: Full support for combining marks #4, Basic support for combining marks #21.

Style sheets

The items below are not essential because style sheets are considered just another input for Pipeline. They don't need to be included inside the pipeline-mod-nlb module. With some small modifications, the existing style sheets should still work with the latest version. The benefit of porting CSS code is that it becomes available for everyone.

• We can select bits of CSS code around certain elements, make it more configurable (less NLB specific) and move them to pipeline-modules, just like we did for the volume-breaking.scss module. Candidates are:
  • Captions
    • configurable prefix for image captions
    • configurable formatting of captions
    • option to omit captions
  • Images
    • configurable prefix/suffix for image descriptions
    • configurable formatting of images
    • option to omit images
    • option to omit image groups
  • Notes: done: see the "Notes placement" option (http://daisy.github.io/pipeline/Get-Help/User-Guide/Scripts/html-to-pef/)
    • configurable formatting of noterefs
    • configurable formatting of notes
    • configurable position of notes: bottom of page, end of volume, end of book, end of chapter, bottom of page with fallback to end of volume, beneath paragraph/table/verse
    • different positioning for different classes of notes
    • configurable footnotes page area
    • configurable title of endnotes section
    • configurable page style of endnotes section
    • configurable definition of "chapter" (for chapter notes)
    • configurable "on-volume-start" of end-of-book notes section
    • option to omit note references
  • Print page break indication
    • option to put page break marker in left margin
    • option to put page break marker in header/footer if print page break coincides with braille page break
    • option to render line across width of page
    • option to render print page number (at the position of the break)
    • option to render print page (range) in header/footer
    • configurable format of print page range
  • Tables: done: see http://daisy.github.io/pipeline/modules/braille/html-to-pef/src/main/resources/css/tables.html
    • table layout classes: matrix, simple list, nested list
    • option to transpose tables
    • option dynamically choose table layout based on certain parameters
    • configurable formatting of different table classes
    • configurable formatting of table captions
  • Definition lists: done: see http://daisy.github.io/pipeline/modules/braille/html-to-pef/src/main/resources/css/definition-lists.html
    • option to group dt element with following dd elements (this is not something that can be done with plain CSS)

[bertfrees]

analyze a provided style sheet and based on this dynamically compute the relevant options and present only these to the user

This is probably too complicated. However what would be doable is to filter script options from a "base script XML" at build time (like in the px:extends mechanism), but based on a list of CSS (or XSLT) style sheets. This could work by allowing to associate an option in the base script with a style sheet URL (and possibly a variable name). A parameter port in the "extending script" could be associated with a list of style sheets.

Obviously you'll still need your own custom script, but it will require less maintenance because we can put a lot more options and style sheet modules in the Pipeline, available to everyone. Your custom script will only include some inputs, outputs and standard options (not associated with a style sheet, or specific to NLB), one or more parameter ports, and the logic to invoke the actual conversion (load, convert, store).

The other big advantage of this is that I can remove options from the generic scripts that currently say "Not implemented". I would also make the "default.scss" style sheet, that is currently always included, public and free to include or not by custom scripts.

It will be a good idea to remove the "stylesheet" option from custom scripts because the available options will be determined at build time and therefore related to a fixed style sheet. So all formatting options that you want to present to the user should be in this fixed style sheet.

I think I would wrap the content of style sheet modules inside a big @if, so that you could make conditional imports, for example:

$chose-between-module-x-and-y: x !default;
$enable-module-x: $chose-between-module-x-and-y == x;
@import "http://www.daisy.org/pipeline/modules/braille/html-to-pef/css/module-x.scss";
$enable-module-y: $chose-between-module-x-and-y == y;
@import "http://www.daisy.org/pipeline/modules/braille/html-to-pef/css/module-y.scss";

To avoid that you would have to override option documentation, I would remove the parts that say "includes the following rule by default ...", because you're likely to override some of these default (too simple) rules in your custom style sheet, and so the documentation would be wrong.

Obviously the option documentation should also be internationalized.

[josteinaj]

Sounds good.

I think it would be nice to have a stylesheet option still though, in case we want to try out some new rules, or override some existing rules (for testing purposes or as a one-off production).

[bertfrees]

OK sure, that's still possible, as long as the user is aware that the options belong to the default style sheet.