Add run-caffluxsysts Step

[alexbooth92]

Closes #103. Implementation of the flux-weight addition step in the production workflow. The underlying software, developed by @luketpickering, has a main executable that takes a CAF and produces a new CAF.

Install testing:* Install philosophy mirrors what's done in run-cafmaker, works at NERSC.

abooth@perlmutter:login38:~/dune/Production/development_repos/ND_Production/run-caffluxsyst> ./install_caffluxsyst.sh 

The following have been reloaded with a version change:
  1) cudatoolkit/13.2 => cudatoolkit/12.4

Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
Cloning into 'duneanafluxtools'...
remote: Enumerating objects: 23, done.
remote: Counting objects: 100% (23/23), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 23 (delta 0), reused 12 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (23/23), 5.52 MiB | 9.81 MiB/s, done.
Note: switching to 'd9d02612260dc0ee5b329faef059f8f2dba34c20'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

-- The CXX compiler identification is GNU 9.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /cvmfs/larsoft.opensciencegrid.org/products/gcc/v9_3_0/Linux64bit+3.10-2.17/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found nlohmann_json: /cvmfs/larsoft.opensciencegrid.org/products/nlohmann_json/v3_11_2/share/cmake/nlohmann_json/nlohmann_jsonConfig.cmake (found suitable version "3.11.2", minimum required is "3.11.2") 
-- Configuring done (6.1s)
-- Generating done (0.0s)
-- Build files have been written to: /global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build
[ 16%] Building CXX object src/duneanafluxtools/CMakeFiles/duneanafluxtools_fluxweighter.dir/FluxWeighter.cxx.o
[ 33%] Linking CXX shared library libduneanafluxtools_fluxweighter.so
[ 33%] Built target duneanafluxtools_fluxweighter
[ 50%] Building CXX object app/CMakeFiles/caffluxweighter.dir/caffluxweighter.cxx.o
[ 66%] Linking CXX executable caffluxweighter
[ 66%] Built target caffluxweighter
[ 83%] Building CXX object examples/CMakeFiles/duneanafluxtools-dumpratios.dir/duneanafluxtools-dumpratios.cxx.o
[100%] Linking CXX executable duneanafluxtools-dumpratios
[100%] Built target duneanafluxtools-dumpratios
Install the project...
-- Install configuration: ""
-- Installing: /global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/lib/libduneanafluxtools_fluxweighter.so
-- Set runtime path of "/global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/lib/libduneanafluxtools_fluxweighter.so" to ""
-- Installing: /global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/include/duneanafluxtools/FluxWeighter.h
-- Installing: /global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/include/duneanafluxtools/CAFInterface.h
-- Installing: /global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/bin/caffluxweighter
-- Set runtime path of "/global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/bin/caffluxweighter" to ""
-- Installing: /global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/bin/duneanafluxtools-dumpratios
-- Set runtime path of "/global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/bin/duneanafluxtools-dumpratios" to ""
-- Installing: /global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/bin/setup.duneanafluxtools.sh
-- Installing: /global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/share/duneanafluxtools/inputs/flux_variations_FD_and_PRISM_2023.root
-- Installing: /global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/lib/cmake/duneanafluxtools/duneanafluxtoolsTargets.cmake
-- Installing: /global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/lib/cmake/duneanafluxtools/duneanafluxtoolsTargets-noconfig.cmake
-- Installing: /global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/lib/cmake/duneanafluxtools/duneanafluxtoolsConfig.cmake
-- Installing: /global/homes/a/abooth/dune/Production/development_repos/ND_Production/run-caffluxsyst/install/duneanafluxtools/build/lib/cmake/duneanafluxtools/duneanafluxtoolsConfigVersion.cmake
	linux-vdso.so.1 =>  (0x00007ffea7e94000)
	libduneanaobj_StandardRecord.so => /cvmfs/dune.opensciencegrid.org/products/dune/duneanaobj/v04_01_00/slf7.x86_64.e20.prof/lib/libduneanaobj_StandardRecord.so (0x00007fbcb8200000)
	libTree.so.6.28 => /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e20-p3915-prof/lib/libTree.so.6.28 (0x00007fbcb7e00000)
	libduneanafluxtools_fluxweighter.so => not found
	libHist.so.6.28 => /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e20-p3915-prof/lib/libHist.so.6.28 (0x00007fbcb7600000)
	libPhysics.so.6.28 => /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e20-p3915-prof/lib/libPhysics.so.6.28 (0x00007fbcb7200000)
	libMatrix.so.6.28 => /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e20-p3915-prof/lib/libMatrix.so.6.28 (0x00007fbcb6e00000)
	libGenVector.so.6.28 => /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e20-p3915-prof/lib/libGenVector.so.6.28 (0x00007fbcb6a00000)
	libMathCore.so.6.28 => /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e20-p3915-prof/lib/libMathCore.so.6.28 (0x00007fbcb6400000)
	libImt.so.6.28 => /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e20-p3915-prof/lib/libImt.so.6.28 (0x00007fbcb6000000)
	libMultiProc.so.6.28 => /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e20-p3915-prof/lib/libMultiProc.so.6.28 (0x00007fbcb5c00000)
	libNet.so.6.28 => /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e20-p3915-prof/lib/libNet.so.6.28 (0x00007fbcb5800000)
	libRIO.so.6.28 => /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e20-p3915-prof/lib/libRIO.so.6.28 (0x00007fbcb5200000)
	libThread.so.6.28 => /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e20-p3915-prof/lib/libThread.so.6.28 (0x00007fbcb4e00000)
	libCore.so.6.28 => /cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e20-p3915-prof/lib/libCore.so.6.28 (0x00007fbcb4600000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fbcb43e4000)
	libstdc++.so.6 => /cvmfs/larsoft.opensciencegrid.org/products/gcc/v9_3_0/Linux64bit+3.10-2.17/lib64/libstdc++.so.6 (0x00007fbcb4000000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fbcb3cfe000)
	libgcc_s.so.1 => /cvmfs/larsoft.opensciencegrid.org/products/gcc/v9_3_0/Linux64bit+3.10-2.17/lib64/libgcc_s.so.1 (0x00007fbcb3a00000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fbcb3632000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fbcb844d000)
	libtbb.so.12 => /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e20/lib/libtbb.so.12 (0x00007fbcb3200000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fbcb2ffc000)
	libssl.so.10 => /lib64/libssl.so.10 (0x00007fbcb2d8a000)
	libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fbcb2927000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fbcb26c5000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fbcb249f000)
	libxxhash.so.0 => /lib64/libxxhash.so.0 (0x00007fbcb2292000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007fbcb2083000)
	libz.so.1 => /lib64/libz.so.1 (0x00007fbcb1e6d000)
	libzstd.so.1 => /lib64/libzstd.so.1 (0x00007fbcb1bb2000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fbcb1965000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fbcb167c000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fbcb1478000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fbcb1245000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fbcb1035000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fbcb0e31000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fbcb0c17000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fbcb09f0000)

Run testing: Running philosophy mirrors what's done in run-cafmaker, the plumming at least works at NERSC. I made a MiniProdN5p1 nominal CAF with duneanaobj v04_01_00 and ran interactively by adding the following exports to the top of the run script:

export ND_PRODUCTION_IN_NAME=MiniProdN5p1_NDComplex_FHC.caf.full.sanddrift
export ND_PRODUCTION_INDEX=965
export ND_PRODUCTION_OUT_NAME=MiniProdN5p1_NDComplex_FHC.caffluxsyst.full.sanddrift
export ND_PRODUCTION_OUTDIR_BASE=/pscratch/sd/a/abooth/WC_260817/output
export ND_PRODUCTION_LOGDIR_BASE=/pscratch/sd/a/abooth/WC_260817/logs

Resulted in the following:

> ./run_caffluxsyst_interactive.sh

The following have been reloaded with a version change:
  1) cudatoolkit/13.2 => cudatoolkit/12.4

Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
Seed is 966
globalIdx is 0000965
runNo is 965
outDir is /pscratch/sd/a/abooth/WC_260817/output/run-caffluxsyst/MiniProdN5p1_NDComplex_FHC.caffluxsyst.full.sanddrift
outName is MiniProdN5p1_NDComplex_FHC.caffluxsyst.full.sanddrift.0000965
logBase is /pscratch/sd/a/abooth/WC_260817/logs/run-caffluxsyst/MiniProdN5p1_NDComplex_FHC.caffluxsyst.full.sanddrift
RUNNING caffluxweighter /pscratch/sd/a/abooth/WC_260817/output/run-cafmaker/MiniProdN5p1_NDComplex_FHC.caf.full.sanddrift/CAF/0000000/MiniProdN5p1_NDComplex_FHC.caf.full.sanddrift.0000965.CAF.root /pscratch/sd/a/abooth/WC_260817/output/tmp/run-caffluxsyst/MiniProdN5p1_NDComplex_FHC.caffluxsyst.full.sanddrift/MiniProdN5p1_NDComplex_FHC.caffluxsyst.full.sanddrift.0000965.CAF.root
Warning in <TClass::Init>: no dictionary for class genie::NtpMCEventRecord is available
Warning in <TClass::Init>: no dictionary for class genie::NtpMCRecordI is available
Warning in <TClass::Init>: no dictionary for class genie::NtpMCRecHeader is available
Warning in <TClass::Init>: no dictionary for class genie::EventRecord is available
Warning in <TClass::Init>: no dictionary for class genie::GHepRecord is available
Warning in <TClass::Init>: no dictionary for class genie::Interaction is available
Warning in <TClass::Init>: no dictionary for class genie::InitialState is available
Warning in <TClass::Init>: no dictionary for class genie::Target is available
Warning in <TClass::Init>: no dictionary for class genie::ProcessInfo is available
Warning in <TClass::Init>: no dictionary for class genie::Kinematics is available
Warning in <TClass::Init>: no dictionary for class pair<genie::EKineVar,double> is available
Warning in <TClass::Init>: no dictionary for class genie::XclsTag is available
Warning in <TClass::Init>: no dictionary for class genie::KPhaseSpace is available
Warning in <TClass::Init>: no dictionary for class genie::GHepParticle is available
Input cafTree has 13 entries.
  Couldn't find parameter directory: FluxParameters/HadronProduction/pca_21 assuming we have 20 parameters.

real	0m5.748s
user	0m2.467s
sys	0m1.307s

@luketpickering should comment on whether this is the output we expect, or whether there should be more. Either way, the install script and the run script, I believe, are doing their jobs.

duneanafluxsyst PR #1 needs to be merged first.