This tool was created for the following paper:
"The Unified Evaluation App for DNA Data Storage Codecs" by Aleksandar Anžel^, Chisom Anyabolu^, Leon Wimbes, Luca Staus, Ihsan Tri Heldian, David Sonnabend, Khawla Elhadri, Samuel Becker, Felix Klein, Raffael Schön, Marius Welzel, Michael Schwarz, Bernd Freisleben, Dominik Heider, and Georges Hattab
^ --- These authors contributed equally to this work
Please cite the paper as:
bibtex placeholderDOI badge Placeholder
Abstract:
Abstract placeholder
The code is written in Python 3.12.2 and tested on Linux with the libraries included in the following Anaconda environment: environment.yml. Some DNA coding schemes (codecs) have additional dependencies that are all included in this environment and there is no need to install them separately.
| Location | Description |
|---|---|
| Data/ | contains all data used to evaluate codecs and store intermediary data representations |
| Data/Original/ | contains the original data used to benchmark all codecs |
| Data/Encoded/ | the directory is empty and used only during benchmarking coding schemes to save the encoded original data |
| Data/Decoded/ | the directory is empty and used only during benchmarking coding schemes to save the decoded previously encoded data |
| Data/Uploaded/ | the directory is empty and used only for storing user-uploaded data |
| Data/cache_data/ | contains cached data |
| Data/cache_data/benchmark_figures/ | contains publication-ready exported figures |
| Data/cache_data/partial_results/ | contains partial per-file unmerged results created during benchmarking |
| Data/cache_data/int_detail.pkl | contains cached intermediary benchmark data of 5 runs for each DNA codec and each original file. This file eventually becomes pre_detail.pkl after a manual check has been performed |
| Data/cache_data/int_summary.pkl | contains cached benchmark data that represents the summary of Data/cache_data/pre_detail.pkl aggregated across runs. This file eventually becomes pre_summary.pkl after a manual check has been performed |
| Data/cache_data/pre_detail.pkl | contains cached benchmark data of 5 runs for each DNA codec and each original file |
| Data/cache_data/pre_summary.pkl | contains cached benchmark data that represents the summary of pre_detail.pkl aggregated across runs |
| Data/cache_data/huff3.dict | contains cached codebook for the 2D code |
The cached benchmark data can be re-calculated using the Source/benchmark_ui.py script and the following command:
# The user should be placed inside the Source/ directory
streamlit run benchmark_ui.pyIf used in an HPC headless environment, the following script should be used instead Source/benchmark_hpc.py alongside a bash wrapper Source/hpc_slurm_benchmark.py. Users should change the SLURM parameters within this script (especially the e-mail).
The data located in Data/Original/ was used to test and benchmark all DNA coding schemes. The directory contains the following files:
.jpg&oldid=889592425){kind=link}
| Location | Description |
|---|---|
| Source/ | contains all scripts necessary to run the tool |
| Source/Encodings | contains all DNA codecs, each in its own separate directory |
| Source/utilities.py | contains the code that implements main functionalities of the platform |
| Source/visualize.py | contains the code that implements visualization functionalities of the platform |
| Source/generate_publication_figures.py | contains the code that generates publication-ready figures and places them at Data/cache_data/benchmark_figures/ |
| Source/text_content.py | contains only various markdown strings that are used to build some pages or provide help tootlips |
| Source/mesa_call.py | contains the code that implements MESA API calls using Source/mesa.json configuration |
| Source/simulate_errors.py | contains the code that implements DNA error simulation which replaces MESA script if necessary |
| Source/benchmark_ui.py | contains the code that runs the benchmark that can be monitored via UI and creates files Data/cache_data/pre_detail.pkl and Data/cache_data/pre_summary.pkl |
| Source/benchmark_hpc.py | contains the HPC (high performance computing)-optimized code that runs the benchmark and creates files Data/cache_data/pre_detail.pkl and Data/cache_data/pre_summary.pkl. This script is meant to be run headless using the runner script Source/hpc_slurm_benchmark.sh |
| Source/hpc_slurm_benchmark.sh | sets up the SLURM parameters and runs the benchmark_hpc.py script. Users should change the SLURM parameters within this script (especially the e-mail) |
| Source/main.py | contains the code that builds the main application layout and connects all pages of the platform's dashboard |
| Source/1_home.py | contains the code that builds the home page |
| Source/2_benchmark.py | contains the code that builds the benchmark page |
| Source/3_upload.py | contains the code that builds the upload page |
| Source/4_about.py | contains the code that builds the about page |
| Source/5_contact.py | contains the code that builds the contact page |
We would like to thank the authors of the coding schemes (located at Source/Encodings) for helping us integrate them into our platform. We acknowledge their contributions in the table below.
| Location | Authors | Reimplemented by | Repository of the Reimplementation | Publication |
|---|---|---|---|---|
| Simple codec | George M. Church, Yuan Gao, and Sriram Kosuri | Khawla Elhadri | https://github.com/ke-pm/DNA-Encoding | 10.1126/science.1226355 |
| 2D codec | Robert N. Grass, Reinhard Heckel, Michela Puddu, Daniela Paunescu, and Wendelin J. Stark | Samuel Becker, Felix Klein, and Raffael Schön | https://github.com/raeudigerRaeffi/reedSolomonDNA | 10.1002/anie.201411378 |
| XOR-based codec | James Bornholt, Randolph Lopez, Douglas M. Carmean, Luis Ceze, Georg Seelig, and Karin Strauss | Leon Wimbes, and Luca Staus | https://github.com/StausWimbes/XOR_based_DNA_encoding | 10.1145/2872362.2872397 |
| Repetition codec | Nick Goldman, Paul Bertone, Siyuan Chen, Christophe Dessimoz, Emily M. LeProust, Botond Sipos, and Ewan Birney | David Sonnabend | https://github.com/davidabendsonn/DNAencoding-Goldman2013 | 10.1038/nature11875 |
| DNA-Aeon | Marius Welzel, Peter Michael Schwarz, Hannah F. Löchel, Tolganay Kabdullayeva, Sandra Clemens, Anke Becker, Bernd Freisleben, and Dominik Heider | Marius Welzel | https://github.com/MW55/DNA-Aeon/tree/main | 10.1038/s41467-023-36297-3 |
| NOREC4DNA Raptor-based codec | Peter Michael Schwarz & Bernd Freisleben | Peter Michael Schwarz | https://github.com/umr-ds/NOREC4DNA/tree/dev/unified | 10.1186/s12859-021-04318-x (Source codec by Amin Shokrollahi 10.1109/TIT.2006.874390) |
| NOREC4DNA LT-based codec | Peter Michael Schwarz & Bernd Freisleben | Peter Michael Schwarz | https://github.com/umr-ds/NOREC4DNA/tree/dev/unified | 10.1186/s12859-021-04318-x (Source codec by Michael Luby 10.1109/SFCS.2002.1181950) |
| NOREC4DNA Online-based codec | Peter Michael Schwarz & Bernd Freisleben | Peter Michael Schwarz | https://github.com/umr-ds/NOREC4DNA/tree/dev/unified | 10.1186/s12859-021-04318-x (Source codec by Petar Maymounkov TR2002-833.pdf) |
If you would like to simulate DNA degradation that comes with storing DNA sequences for archival purposes, UNACORM includes MESA as the main simulator. Please note that if you have firewall or a VPN activated, MESA might not work as its container has to be able to freely connect to the UNACORM's container. If you would like to use MESA, the following sequence of instructions (for Linux-based systems) should be used to build and run the MESA container:
# How to build the docker container and run it locally
# Clone the repository
git clone -b feature/adminapikey https://github.com/umr-ds/mesa_dna_sim.git
cd mesa_dna_sim/
# Set the parameters for UNACORM connection
tmpfile=$(mktemp) && \
sed -E 's/^([[:space:]]*)#[[:space:]]*ADMIN_API_KEY:[[:space:]]*CHANGE_ME_ADMIN_API_KEY/\1ADMIN_API_KEY: UNACORM_ADMIN_API_KEY/' docker-compose.yml > "$tmpfile" && \
mv "$tmpfile" docker-compose.yml
tmpfile=$(mktemp) && \
sed -E 's|^([[:space:]]*-[[:space:]]*)/srv/docker/postgresql:/var/lib/postgresql/data$|\1./postgresql:/var/lib/postgresql/data|' docker-compose.yml > "$tmpfile" && \
mv "$tmpfile" docker-compose.yml
tmpfile=$(mktemp) && \
sed -E 's|^([[:space:]]*-[[:space:]]*)/srv/docker/redis:/data$|\1./redis-data:/data|' docker-compose.yml > "$tmpfile" && \
mv "$tmpfile" docker-compose.yml
# Compose the containers
docker compose up -d
cd ..
# From here on UNACORM installation instructions should be followedThe easiest way to install the tool is to use our latest Docker image by building a docker container and running it locally. To do so, you first need to install Docker by following the official documentation found at https://docs.docker.com/engine/install/. After installing Docker, you should use a following sequence of instructions (for Linux-based systems) to build and run the container:
# How to build the docker container and run it locally
# Clone the repository
git clone https://github.com/AAnzel/UNACORM.git
cd UNACORM/
# Build the image
docker build -t unacorm-local:stable .
# Run the container (in detached mode)
docker run --publish 8501:8501 --detach --name unacorm-local --add-host=host.docker.internal:host-gateway unacorm-local:stableYou can start using the tool by opening a web browser and typing in http://localhost:8501/ as the address. If you run the docker container, you have to use the IP address or hostname instead of localhost. If you would like to remove the container and the image completely, the following sequence of commands should be used:
# ----- Stopping and removing the container -----
# Stop the running container gracefully
docker stop unacorm-local
# Remove the stopped container
docker rm unacorm-local
# ----- Removing the built image -----
# Remove the image (can refer by name or image ID)
docker rmi unacorm-local:stable
# If you want to confirm the image ID first, list images:
docker images
# Alternative forced removal (only if needed)
docker rmi -f unacorm-local:stableThe platform can also be installed without the docker container. This process requires users to install Anaconda or Miniconda package managers. Users can install Conda using the instructions on the following link https://docs.anaconda.com/miniconda/#quick-command-line-install. If Conda is installed, the following sequence of instructions (for Linux-based systems) installs all dependencies and runs the dashboard:
git clone https://github.com/AAnzel/UNACORM.git
cd UNACORM/
conda env create -f environment.yml
conda activate unacorm_env
cd Source/Encodings/XORBasedCode
python setup.py install
cd ../RaptorCode
python -m build .
pip install -e .
cd ../DNA_Aeon
python setup.py
cd ../../
streamlit run main.pyEvery subsequent run of the tool after restarting the terminal emulator session requires activating the installed conda environment and running the dashboard. The following sequence of instructions defines that process:
conda activate unacorm_env
streamlit run main.pyLicensed under the GNU General Public License, Version 3.0 (LICENSE or https://www.gnu.org/licenses/gpl-3.0.en.html)
Any contribution intentionally submitted for inclusion in the work by you, shall be licensed under the GNU GPLv3.
To see how to contribute, please check CONTRIBUTE.md.