Enabling Distributed Processing - #61

Open

markmac99 wants to merge 312 commits into

wmpg:masterfrom

markmac99:distrib_processing

This pull request is big! We're only showing the most recent 250 commits

Commits on Feb 3, 2026

make uploaded database names unique
markmac99
committed
fd38bd5
oops, make sure phase1 dir exists even in mc mode
markmac99
committed
6aac11f

Commits on Feb 4, 2026

small bugfixes
markmac99
committed
ce2877b
remove some excess verbosity
markmac99
committed
3622d0c
indicate how many are being solved
markmac99
committed
67fefbe
bugfix to handle nans in the database
markmac99
committed
c4eecac
add log suffix
markmac99
committed
92f865a
add log suffix properly
markmac99
committed
b271037
catch SFTP fails
markmac99
committed
1d366bf
more small changes to protect against sftp fails
markmac99
committed
333b48e

Commits on Feb 11, 2026

Commits on Feb 12, 2026

make sure folders exist
markmac99
committed
6ece9ab
initialise remote data handling in the right place for phase2
markmac99
committed
c9bb9bd
improve logging
markmac99
committed
c9ede51
prevent crash in mcmode 2
markmac99
committed
cda60ba
print operation mode more frequently
markmac99
committed
fbd7899
update remote data handling to actually move trajectories!
markmac99
committed
7677acc
slight improvement in shutdown messaging
markmac99
committed
21c94b4
bugfix
markmac99
committed
1b73810
chmod upload folders as needed
markmac99
committed
8d8a325
fix perms on upload folders
markmac99
committed
cb20be5

Commits on Feb 13, 2026

add some verbosity for debug
markmac99
committed
db2f17c
bugfix in pairing routine
markmac99
committed
e753a1e

Commits on Feb 14, 2026

Merge branch 'wmpg:master' into distrib_processing
markmac99
authored
b2cf365
rename archiveoldrecords to archivemonths
markmac99
committed
b12504a
Merge branch 'distrib_processing' of github.com:markmac99/WesternMeteorPyLib into distrib_processing
markmac99
committed
379a5e9
remove unnecessasry flag
markmac99
committed
4d212af
whoops changed param name incorrectly
markmac99
committed
22dadc5
initialise obs and traj db
markmac99
committed
86721cd

Commits on Feb 16, 2026

a bit of documentation
markmac99
committed
508f40e
check status from merge functions
markmac99
committed
bd31fa0
if connection fails, close gracefully and retry later
markmac99
committed
8de1c47
write pidfile
markmac99
committed
e04f9f4
remove unnecessary saving of temp pickles in ph2
markmac99
committed
17ef66d

Commits on Feb 18, 2026

tidying up the date formats when reporting candidates, fails etc
markmac99
committed
0a102b9

Commits on Feb 19, 2026

bugfix failed to define ref_dt
markmac99
committed
b73f07d

Commits on Feb 26, 2026

add some comments
markmac99
committed
7b45db1
create function to merge broken so we can call it elsewhere if needed
markmac99
committed
91ab59e

Commits on Feb 27, 2026

commit the write-ahead-log after every candidate save op
markmac99
committed
615ea33
commit the obs database after saving candidates
markmac99
committed
43a4e8d
don't load stations or scan folders if in phase1 or phase2 only modes
markmac99
committed
1a5d592
support fractional bins
markmac99
committed
4ba68fc
small correction in correlateDB
markmac99
committed
3c0ae48
remove unnecessary cursors
markmac99
committed
f51e56c
support to purge local processed data
markmac99
committed
d898e1f
make sure dbs are closed in the maintenance section
markmac99
committed
704aa0a
add verbosity
markmac99
committed
51b637a
fix verbosity
markmac99
committed
3e7ac46
error handling in db merge process
markmac99
committed
7c16e37
remote remote WAL dbs after merger
markmac99
committed
739ae25
avoid trying to merge empty databases
markmac99
committed
dcc904a
more fixes for database or table not existing
markmac99
committed
6415260
documentation
markmac99
committed
d46b06f
add more debug
markmac99
committed
70b59ce
delete rather than update when unpairing
markmac99
committed
8071f49
handle unpairing more efficiently with multi-insert
markmac99
committed
f663308

Commits on Mar 3, 2026

rename function more sensibly
markmac99
committed
6e20cf7
switch logic round to mark paired after succeessful phase1. This reduces db writes
markmac99
committed
821bf68
remove unnecessay commit
markmac99
committed
5fe13e0

Commits on Mar 4, 2026

pass new obs to solveTrajectory so they can be marked paired if needed
markmac99
committed
cfc0b81

Commits on Mar 5, 2026

bug in phase1 processing
markmac99
committed
7db6bdf
working on ref_dt bug
markmac99
committed
ac19e59
add debug messages to help testing
markmac99
committed
ced94c2

Commits on Mar 6, 2026

tidying up
markmac99
committed
9db92f7

Commits on Mar 7, 2026

avoid reprocessing already processed candidates
markmac99
committed
e9d3c6e

Commits on Mar 8, 2026

bugfixes and performance improvements
markmac99
committed
c2572de
remove incorrect location of Obs DB
markmac99
committed
e9794a8

Commits on Mar 9, 2026

various improvements
markmac99
committed
9612432
tidying up
markmac99
committed
b9d9747

Commits on Mar 10, 2026

fixes for issues #86, #87, #88, #94
markmac99
committed
d7fabbd

Commits on Mar 11, 2026

Add candidate database and use it
markmac99
committed
88bcd09
purge older candidate data to keep database size reasonable
markmac99
committed
48a35b0
save cands using jdt_ref to make checks easier
markmac99
committed
cfd7645
small bug in cands db purge process
markmac99
committed
c820387
move cand loading to within datahandle
markmac99
committed
bcaa042
avoid slightly misleading message in log
markmac99
committed
f234e8e
make sure to close and commit databases on exit
markmac99
committed
16a3db2

Commits on Mar 12, 2026

bugfix
markmac99
committed
2416d4b
bugfixes and tidying up
markmac99
committed
1e44569

Commits on Mar 15, 2026

add obs id to traj data
markmac99
committed
87c5109

Commits on Mar 16, 2026

remove some debug
markmac99
committed
21ab115
add obs_ids to failed traj if convergence angle too small
markmac99
committed
8f6eaf6

Commits on Mar 17, 2026

add ignored obs to database
markmac99
committed
d9505a1
add ignored obs to traj object
markmac99
committed
92e42bd
simplify updating the traj database
markmac99
committed
323d4c9
bugfix in trajdb
markmac99
committed
6fe3f86

Commits on Mar 20, 2026

Commits on Mar 21, 2026

remove some debugging
markmac99
committed
e74b8d7

Commits on Mar 24, 2026

better remote data error management
markmac99
committed
830cce1
improved duplicate management
markmac99
committed
5bbf10e
be sure to remove both traj with an overlap in obs so they get reanalysed
markmac99
committed
0fbfb3e

Commits on Mar 26, 2026

possible dupes: don't delete both trajs if not in auto mode
markmac99
committed
4a0c828
less aggressive purging of processed phase1
markmac99
committed
c317cd6

Commits on Mar 27, 2026

bugfixes in process to copy data from Json file
markmac99
committed
37cb56a
bugfix in allocation of data to children
markmac99
committed
e269d94
log where phase1 files are saved
markmac99
committed
be840b8
adding remote processing config file
markmac99
committed
8aa22e4
tweaking debug
markmac99
committed
c12138c

Commits on Mar 28, 2026

support conversion of legacy trajectories as well as failed to sqlite
markmac99
committed
b592ac8

Commits on Mar 29, 2026

more work on getting buckets right
markmac99
committed
9a3a4ed
bugfix
markmac99
committed
f421533
fix error in cand/phase1 saving logic
markmac99
committed
4fb4433
reworking on obs loading
markmac99
committed
83098f3
improve exception handling
markmac99
committed
704bbb4
fixing timebucket calcs
markmac99
committed
af31def
remove some debugging
markmac99
committed
7521455

Commits on Mar 30, 2026

improvements in data selection to avoid
markmac99
committed
8b22845
performance bugfix in finding unprocessed data
markmac99
committed
a89601a

Commits on Mar 31, 2026

bugfix when iterating through testing stations to ignore
markmac99
committed
5785fed
remove some debug and add some informational messages
markmac99
committed
9433235

Commits on Apr 1, 2026

fix bug that was creating duplicates when reprocessing existing traj or doing phase2 calculations, if the new traj had a different ref_dt.
markmac99
committed
e6a8679

Commits on Apr 3, 2026

updating some inline documentation
markmac99
committed
8c8356b
Adding documentation on new capabilities
markmac99
committed
68154de
Move documentation to right place
markmac99
committed
0dfb9b4
More work on documentation
markmac99
committed
047f784
More doco tweaks
markmac99
committed
bb872b7

Commits on Apr 4, 2026

don't load json db if its not present (doh)
markmac99
committed
2807a0b

Commits on Apr 7, 2026

add merge process for canddb and fix small bug
markmac99
committed
5c09521

Commits on Apr 11, 2026

bugfix in addTrajectory
markmac99
committed
f437bca
avoid closing closed databases
markmac99
committed
bae0d29
filter out legacy traj that can't be checked for dupes as easily
markmac99
committed
8c5ce08

Commits on Apr 16, 2026

make dbdir and logdir if they dont exist
markmac99
committed
5e44e3c

Commits on Apr 20, 2026

make sure traj objects include obs_ids when persisted
markmac99
committed
b4daaa1
bugfix - set candidate id and ref_ts in UTC
markmac99
committed
300376b

Commits on Apr 21, 2026

sort trajbasics by jdt_ref
markmac99
committed
7996132
bugfix for zero-length obsids
markmac99
committed
cb989e3
correct return value from addTrajectory
markmac99
committed
58d7dea
fix issues with updateTrajectoryDatabase
markmac99
committed
2d7a586

Commits on Apr 24, 2026

missing verbose parameter
markmac99
committed
ed9137d

Commits on May 1, 2026

bugfix to avoid crash if dupe traj arlready removed
markmac99
committed
781639e

Commits on May 4, 2026

tiny tweak to avoid reprocessing the same events twice
markmac99
committed
ca3fdf7

Commits on May 5, 2026

trying to fix bug #110
markmac99
committed
610b902

Commits on May 6, 2026

add some debug
markmac99
committed
2f55828
adding a little commentary
markmac99
committed
4ce305c
Resolving issue where deduplication would crash if there were no
markmac99
committed
4bbc6cb

Commits on May 7, 2026

cosmetic tweak to remove irrelevant comment
markmac99
committed
32b9059

Commits on May 13, 2026

improvements in archiving/purging process
markmac99
committed
ba5e525

Commits on May 16, 2026

Commits on May 28, 2026

set default archivemonths to zero, meaning purge anything older than 21 days from the sqlite DBs
markmac99
committed
04ee755
improve documentation
markmac99
committed
9bf71d2
missed an f-string
markmac99
committed
f42cbca

Commits on Jun 2, 2026

update docs a little
markmac99
committed
68da3d4
minor tweak
markmac99
committed
88258d6

Commits on Jun 4, 2026

improve handling of situations where the remote 'processed' file exists
markmac99
committed
9f67b33
slight improvement in documentation
markmac99
committed
d2fc408
small doco changes
markmac99
committed
02057b9
move processed phase1 from child nodes
markmac99
committed
11a431b
more clarification of the docs
markmac99
committed
2f4fc0a
make sure we log assignment of candidates to nodes or master
markmac99
committed
b6faca9
log if files are reallocated
markmac99
committed
3a618a8
daft error when checking and redistributing data
markmac99
committed
16facf5
further small fix for capacity zero nodes
markmac99
committed
bb78616

Commits on Jun 5, 2026

small tweak to prevent overassignment to children
markmac99
committed
6e1e7a4
make sure we properly log when pickles are assigned to a child
markmac99
committed
0175ac3
typo
markmac99
committed
0959005
oops log the correct target dir
markmac99
committed
affb633
more doc changes
markmac99
committed
f677391

Commits on Jun 8, 2026

Merge pull request #115 from wmpg/master
markmac99
authored
e096cb0

Commits on Jun 12, 2026

fix for bug where the original version of remotely calculated phase-2 solutions was not cleaned up, leading to two solutions on-disk, one with and one without uncertainties
markmac99
committed
ae3cd68

Commits on Jun 16, 2026

minor tweak to use 'parent' instead of 'master'
markmac99
committed
ac23d37
replace 'master' with 'parent'
markmac99
committed
5209e1a
replace 'master' with 'parent'
markmac99
committed
7bf1f28

Commits on Jun 18, 2026

Commits on Jul 13, 2026

Commits on Jul 14, 2026

avoid double-logging
markmac99
committed
a257a17
correct a comment
markmac99
committed
eaa8b8e
remove extra param from log.info
markmac99
committed
9f63e04
change info to warning
markmac99
committed
08b1737
fix bug in aggregateAndPlot - lam_err and bet_err not properly set if uncertainties malformed, as 'traj' variable was not set properly
markmac99
committed
30e46cf
Merge branch 'wmpg:master' into distrib_processing
markmac99
authored
70c167b
make sure array is floating point
markmac99
committed
a1ee994

Commits on Jul 15, 2026

Merge branch 'wmpg:master' into distrib_processing
markmac99
authored
1ebb1cd

Commits on Jul 17, 2026

Commits on Jul 20, 2026

Commits on Jul 21, 2026

reinstate reboundx
markmac99
committed
526aed6
documentation bugs noted by Denis
markmac99
committed
46e8639
small fixes recommended by Denis
markmac99
committed
9805c9d
add support for multiple ssh key types
markmac99
committed
b445f09
archiveCandDatabase, avoid purging the newly opened/created arch database
markmac99
committed
0acdfaf
add python version constraint on rebound as version 4.3.0 isn't supported on py3.11 or later
markmac99
committed
67fe1b9
update constraint on reboundx as 4.3.0 is available for python 3.13
markmac99
committed
2e5119f
Merge pull request #125 from wmpg/master
markmac99
authored
e4c2ef8

Commits on Jul 23, 2026

apply upstream changes in rebound
markmac99
committed
378f83d
add e to sync with upstream master
markmac99
committed
c66560e
Merge pull request #128 from markmac99/master
markmac99
authored
53bab01
error trapping round insertion of observations and trajectories
markmac99
committed
165f19f
add the l-bfgs-b cutoff parameter
markmac99
committed
7c3ff7a
add l-bfgs-b cuttof as a parameter
markmac99
committed
1301016
rename the logger to wmpl_logger
markmac99
committed
85747f9
Merge pull request #129 from markmac99/quieten_rebound
markmac99
authored
ac9a0f5

Commits on Jul 24, 2026

Commits on Jul 25, 2026

Commits on Jul 28, 2026

Merge pull request #133 from wmpg/master
markmac99
authored
20c6664
remove trajectory from db by ID not date
markmac99
committed
8b05563
rework of database classes to use parameterised queries.
markmac99
committed
0d13189

Commits on Jul 29, 2026

Adding test cases for CorrelateDB
markmac99
committed
dcb35c2
Add some simple tests for Correlator
markmac99
committed
4932304
missed data
markmac99
committed
f34dfaf

Commits on Jul 30, 2026

Commits on Jul 31, 2026

manage all child data renaming etc from the child
markmac99
committed
7216992

Commits on Aug 1, 2026

Commits on Aug 2, 2026

bugfix: remove existing file when renaming a temp pickle or copied file
markmac99
committed
50b1f67

Commits on Aug 3, 2026

Commits on Aug 4, 2026

manage test data better
markmac99
committed
e92c104
ignore test artefacts
markmac99
committed
583288c
don't put coverage files in github
markmac99
committed
2f16131
improved sqlite database handling and resilience
markmac99
committed
8592a2f
Avoid distributing cands or traj that are already being processed
markmac99
committed
d075793
bugfix to avoid trying to read partial uploads
markmac99
committed
b33411b
bugfix removing write-ahead temp files
markmac99
committed
eeda63e
correct error message
markmac99
committed
ad059e0
copy/paste error
markmac99
committed
26681cc
add explanation about retries
markmac99
committed
1ac0624
typo
markmac99
committed
233ac31
on the first pass, clear the being-processed flags so that traj or cand left behind after a crash or shut down get picked up.
markmac99
committed
568379f

Commits on Aug 5, 2026

better error handling in remoteDataHandler
markmac99
committed
3e90e6c
quieten successful uploads
markmac99
committed
2b76f25
Quieten remote successful data downloads
markmac99
committed
fa3728e

Commits on Aug 6, 2026

prevent removal of traj where the ph1 got uploaded but the main traj is pending
markmac99
committed
7d66b18