Release 4.71.0#518
Merged
Merged
Conversation
… new job manager to contain all the new job management functionality, which sits in API services. FS code was updated to allow testing bucket access but simulated with just a local test directory in place of all the buckets (this has its own test too). Some logs about upload/downloading changed do DEBUG level. Promoted getModuleVersion to a wsHelper so it can be reused. This change also involved fixing some circular dependencies and changing some data types to uint to make things more robust, so more files affected. Existing PIQUANT run code and tests have NOT changed and should still pass, this will be the case until the new code takes over the role. Added Local job executor (for testing only, runs the code directly, not through docker or other steps). NOTE: jobrunner was brought back from external repo to be local, so we can more easily debug/test it because if it's in another repo, we need to build the docker container so we can re-run job tests
… actually run expressions (not just read cached values)
… end. Lots of things renamed/made public so the new jobmanager can access it, eg parts of quantification package. The new tests don't rely on s3 buckets but can test in local file system using FS mocking. Other things moved to break circular references. Executor removed as it seems no longer needed with new JobNode code (that handles starting docker containers on an EC2 for eg)
Feature/jobs
Changing env vars for job runner
Use newer github tag action
… - output structures and the ability to run job nodes on demand from API. Mostly working, need more testing and automated tests. Currently the QuantificationSummary structure still lacks the input param fields and it has yet to be tested in a new style environment on AWS. FOr testing we will ensure some word like legacy in the quant name will request an OG quant and we can run a new one and compare them, remove old code once new matches exactly in an environment
…e after combining quant node outputs
…AWSSecret config value is empty, we run it locally. Added check for timed-out job nodes in job manager. Log job run errors in jobnode where docker fails
… for example test output
…has predictable output
…s relevance to our AWS account
Allow specifying EC2 subnet id
… same fields. LogId went unused anyway.
… for given EC2 instances and cores per node settings
…sn't preceed QUantCreateResp in tests
…re quant output's for a docker container weren't yet uploaded somehow. Also checked in comments documenting EC2 runtime speed with various configurations of EC2 instance type and cores per node settings
…now, just checking
… we're checking. Unit tests were failing because output was not deterministic, now we don't print the PREPARING_NODES state, so it should work
Feature/jobs
…t how many cores/node we run, we just calculate it as how many jobs to break it into and then run that how we want in EC2
Feature/jobs
Fixing mongo db restore
Fixing DB backup
Fixing file sync for backup/restore
Fixing paths for restore of images/quant/scans
…, beam locations, diffraction peak statuses, manually entered diffraction peaks
…ce name for the reviewer account
Feature/jobs
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| dstFullPath := fs.filePath(dstRootPath, dstPath) | ||
|
|
||
| // Make the intermediate dirs as needed | ||
| err = os.MkdirAll(filepath.Dir(dstFullPath), 0777) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.