License: MIT

efi-conv

Reference repository for creating mappings to the AVefi schema. If you consider contributing data to the AVefi project, that is, registering persistent identifiers for some audio-visual material in your collection, this may be a good place to start. Contributions are very welcome. Feel free to fork and submit pull requests or otherwise get in touch and let us jointly work on your mapping.

Usage example

The check module included in this package allows you to validate generated JSON files that are supposed to comply with the AVefi schema. Simply clone the repository and either use docker-compose or install the package in a virtual environment in editable mode. The latter can be done in the traditional way using pip but it is highly recommended to use the dependency manager UV instead, especially if you consider contributing and might need to add dependencies at some point.

For seasoned Docker users, here is how to run the checks against your data just a few simple steps:

$ git clone https://github.com/AV-EFI/efi-conv.git
## [...]
$ cd efi-conv
$ docker-compose pull
## If that does not work run:
## $ docker-compose build
## Then:
$ docker-compose run efi-conv check tests/avportal/efi_records.json
INFO efi_conv.core.check: Processing tests/avportal/efi_records.json
INFO efi_conv.core.check: All 3 records passed the checks successfully

Everyone else should rather setup a dedicated virtual environment, preferably using the dependency manager UV.

Here is a more complete usage example showing how to convert some test data:

$ git clone https://github.com/AV-EFI/efi-conv.git
## [...]
$ cd efi-conv
$ uv sync --no-python-downloads
## [...]
$ uv run efi-conv --help
Usage: efi-conv [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  check  Sanity check EFI_FILES and optionally remove invalid records.
  from   Convert files from some schema into a JSON file with AVefi records.
$ uv run efi-conv from --help
Usage: efi-conv from [OPTIONS] [INPUT_FILES]...

  Convert files from some schema into a JSON file with AVefi records.

Options:
  -f, --format [avportal|fmdu]  Source data format.  [required]
  -o, --output FILE             Output file (stdout if not specified).
  --help                        Show this message and exit.
$ uv run efi-conv from -f avportal -o efi_records.json tests/avportal/*.xml
INFO efi_conv.avportal.avportal: Replaced name 'Dore Kleindienst-Andrée' by 'Kleindienst-Andrée, Dore'
INFO efi_conv.avportal.avportal: Replaced name 'E. Fischer' by 'Fischer, E.'
$ uv run efi-conv check tests/avportal/efi_records.json
INFO efi_conv.core.check: Processing tests/avportal/efi_records.json
INFO efi_conv.core.check: All 3 records passed the checks successfully

Alternatively, use the provided docker-compose file in order to run this tool inside a container:

Developer note

If you consider hacking on this package and even making a pullrequest at some point, it is advisable to install the pre-commit hooks configured on this repository. This way, some quality checks and coding style guide lines will be enforced right from the beginning which will make merging your code much easier, eventually. The pre-commit package is not part of the package's virtual environment and needs to be installed globally instead. This is because the hooks are executed automatically on every git commit; the hooks themselves are configured for this repository only, of course. So, here is one way to set things up:

$ pipx install pre-commit
$ pre-commit install

That's all. Feel free to start hacking!

Add new converters as modules within the efi_conv package. Then, add this module as another choice to the IMPORTERS list in cli.py to make it accessible from the command line. Take care that the module provides .module_name:efi_import function similar to what the avportal module does.

Unless you already have a suitable python parser for your data, check out whether the xsData or similar projects can help you there. In fact, the avportal module relies on xsData for parsing as has been briefly documented.

The actual mapping is the tedious part. See avportal.py for the kind of work you are letting yourself in for. Also consult the AVefi schema documentation.