v4.13.0#1397
Open
andrewelamb wants to merge 40 commits into
Open
Conversation
* Add initial CLAUDE.md for AI-assisted development
…renty (#1337) * SYNPY-1508: add tutorial for downloading files by Synapse ID concurrently Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update download data tutorial * update tutorial doc * Update script name --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* update black to 26.3.1 and rerun pre-commit * update the tutorial line --------- Co-authored-by: Lingling Peng <lpeng@w290.local> Co-authored-by: Lingling Peng <lpeng@Mac.SageCorpWiFi>
* [SYNPY-1764] Add Trivy container vulnerability scanning to Docker build Add Trivy scanning to gate Docker image publication on GHCR. Both release and develop Docker jobs now follow a build→scan→push pattern where images are only pushed if no Critical/High unfixed vulnerabilities are found. New workflows: - trivy.yml: reusable Trivy scanning workflow with SARIF upload to GitHub Security tab - docker_build.yml: reusable build/scan/push workflow for image rebuilds - trivy_periodic_scan.yml: daily rescan of latest published image with auto-remediation * Address PR review feedback from linglp and jaymedina - Fix push jobs to load scanned tar instead of rebuilding (build.yml) - Pin trivy-action to SHA for v0.35.0 to address supply chain attack - Fix env.repo_name output using $GITHUB_OUTPUT (trivy_periodic_scan.yml) - Pin all third-party actions to commit SHAs - Remove unnecessary permissions on get-image-reference job - Use !cancelled() for SARIF upload condition (trivy.yml) - Use LOCAL_IMAGE_TAG env var instead of hardcoded string (docker_build.yml) - Fix IMAGE_REFERENCES YAML line continuation * Prevent infinite rebuild loop in periodic Trivy scan Restructure trivy_periodic_scan.yml so the git tag is only created after a successful rebuild (not before). If the rebuild still has vulnerabilities, a GitHub issue is opened for manual triage instead of looping endlessly. - Rename bump-tag → compute-next-version (dry_run: true) - Add create-tag job gated on update-image success - Add alert-on-failure job that opens a GitHub issue with duplicate prevention when remediation fails * pre-commit * Update Trivy scan workflow to use previous tag and adjust image references * Address PR review feedback - Pin codeql-action/upload-sarif to SHA and upgrade to v3.35.1 - Guard update-image job on compute-next-version success - Use absolute URL for Security tab link in auto-created issues
…artifact naming (#1357)
… API (#1352) * [SYNPY-1402]: Update reticulate tutorial for new OOP models/operations API - Add Docker setup instructions for fresh RStudio environment - Update Python version requirement to 3.10+ - Update install command to include pandas,curator extras - Replace legacy synapseclient$File/Project/etc with models$* pattern - Replace syn$store/get with entity$store() OOP pattern; keep operations$get for type-unknown retrieval - Update tables to use models$Table + store_rows with SchemaStorageStrategy - Update views to use models$EntityView with ViewTypeMask - Replace syn$tableQuery with models$query - Move R package onLoad guidance to its own section at end of doc Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Apply suggestion from @thomasyu888 * Update docs/tutorials/reticulate.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @thomasyu888 --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add Synapse MCP usage * Improve documentation * Address PR comment
* [SYNPY-1375]: Add Activity/Provenance tutorial Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Use --8<-- anchors to avoid having to maintain specific coding lines * Add activity to mkdocs * Clean up activity script * Update activity * Update tutorial to create files * Update docs --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* added new sync_to_synapse method * improved deprecation * created copy of SyncUploader in manifest.py * added tests * fixed unit tests * fix tests to work on windows * fixed typo bug * fix bug in tutorial * Linglings suggestions * turn send messge off in unit test * fix test fixture * fix docstrings * fix inconsistent default * fix typing * added notes about chnage in column name * tests now have send_messages = False * tutorial now has send_messages = False * minor fixes * fixed line numbers * fixed docs * reverted notes back to !!! style * add note for adding model methods * fix hierarchies * fix tutorial * update docs * update docs * remove creation of file hierarchy * various fixes * fix docstrings * refactor manifest functions for readability * various fixes * fix test and added two others * various fixes * made functions more readable * fix erroneus change to upload * removed uneeded test * added sumamry to docstring * added unit tests * remove comments * fix example formatting * improved file cleanup * fix typos * remove uneeded global * changed tests so that scheduling File cleanup happens right after files are crated in Synapse
* [SYNPY-1480]: Improve documentation around configuration file - Expand .synapseConfig example with improved comments for all sections - Document previously undocumented `use_boto_sts` transfer option - Improve `max_threads` comment with default, min, and max (128) values - Rewrite docs/tutorials/configuration.md with full table-driven coverage of every section including sftp, S3, cache, debug, endpoints, transfer - Remove duplication between configuration.md and authentication.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR comments * Add note block --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…ate from prior runs (#1363) * fix bug * fix test; edit conftest * fix test -cleaner * delete unnecessary import --------- Co-authored-by: Lingling Peng <lpeng@w290.local>
* add OOP DownloadList class * update docs and docstring examples * add module-scoped syn class * add comment * various fixes claude found * import methods so they cna be used without the class * replaced Typing.List with list * DownLoadListItem.version_number is now optional * allow DownloadListItem.version number to be null * removed module level synapse fixture * added back module level syn object * fix tests were mocking leaked to other tests, and update claude.md file * remove uneeded import * changed link format * funtion now raises an error when mumberOfFilesAdded is nto returned * added link to API doc * updated docstring * now using CsvTableDescriptor class * changed link format * add exception and warning * added link to API docs * PR suggestions * updated docstrings * remove warning test * fix imports * fix docstring * redo docs * undo review date change * uncommit a dir that wasnt supposed to be * fix style issues * combined all static methods as functions into one section * added destination parameter * created new api service fucntion for download of manifest * fix stale docstring * fix unit test error type * revert claude date * removed unit tests * added integration tests * added to docstring * cleaed up and renamed download_row func * made consistent use of methods * made max_concurrant more consistent * added tests * remove uneeded parens * did some clean up * added helper function * redid simpler unit tests * refactored tests * added integration test * expanded test * clarified how versions work in docstrings * light refactoring * add fix for test failing on Mac OS * fixed tests to work in parallel * linglings suggestions * linglings suggestions * ran black * changed class to operations functions * changed class to operations functions * added ability to delete null verison files if downloaded * added ability to delete null verison files if downloaded * linglings suggestions
* Supporting storage locations and migration OOP models * add EnumCoercionMixin to allow string and enum for the same field * Pass the port from the parsed SFTP URL to resolve the connection issue if server is on a specific port * Replaces StorageLocationType with a dataclass so SFTP and HTTPS can be distinguished, and simplifies to_dict to use the existing field mapping. * add checks for presence of required attributes * explicitly treat None as use default storage when setting storage location * add instruction for updating storage location * add deprecation for create_s3_storage_location * add deprecation for join_column_names
* fix circular dependencies * fix patchng in tests
* try fixing the syn.sendMessage * comment out test_boto_upload_acl and test_sts_external_storage_location * uncomment but test but added * [SYNPY-1826] fix integration test: test_download_list_manifest_with_custom_csv_descriptor (#1372) * fix test * comment out test * remove file * mention about server side bug --------- Co-authored-by: Lingling Peng <lpeng@w290.local> --------- Co-authored-by: Lingling Peng <lpeng@w290.local>
* add two async job; tested the basic of import * update protocol * add comment to explain the error * add test for preview request class * add test for GridCsvImportRequest class * correct errors; add unit test for import_csv * add integration test * edit test; fix typo; fix log * update test * add back if csv_table_descriptor: * remove slash * add docstring to indicate response class * fix GridCsvImportRequest class; make field required; fix test; use constant * use delete none keys * turn schema to required; fix delete_none_key * get a default to csv_descriptor * make upload_file_handle_id required; add default to import_csv_async * change to list * fix to_synapse_request * fix protocol * clean up grid * fix test * one liner fix * raise value error when there is no suggested columns from preview * add import_csv to curator.md * update protocol * edit test; fix consistency in docstring; make code cleaner * remove unnecessary line * update test * fix type hint * fix otel method * use path parameter; not file_handle_id * fix import_csv * no need to commit the file * fix integration test --------- Co-authored-by: Lingling Peng <lpeng@w290.local>
* Add validate-release workflow
* init commit * add unit test * add to test * add unit test for TestGridDownnloadCsv * add check of destination; add integration test and unit test * fix test * fix typo * fix comment format * fix docstring; ensure directory already exists * edit error message; add unit test * switch to use presigned url approach for download * remove unnecessary f string * remove this being a field * add tear down grid; update docstring * add and fix unit test * add docstring example * update test * remove optional; * update docstring * update docstring * update docstring * fix typo * added asyncio.gather --------- Co-authored-by: Lingling Peng <lpeng@w290.local>
…#1354) * decouple create_task with manifest generation and suppress manifest generation for recursive calls * update test cases and relocation manifest generation tests to a separate file * add unit tests for manifest related functions * fix path issue on windows * Create a manifest with just a header when all_files is empty
…ta task (#1374) * [SYNPY-1836] Support configurable view_type_mask in file-based metadata task Allows callers to create the EntityView with any view type (e.g. ViewTypeMask.FILE | ViewTypeMask.DOCKER) instead of being locked to ViewTypeMask.FILE.
Merge master -> Develop
Co-authored-by: Lingling Peng <lpeng@w290.local>
* add missing proxy key var * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * use env var * use env var * remove default for env var and remove extra start marker --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* implement synchronize endpoint; add test * add unit test * update docstring * change level to error; update doc; add unit test * cleanup test and add an integration test * restore the fixture * fix unused imports and type hints * restore decorator; use decorator pattern * update to use grid_session_id and make it required * make integration test better --------- Co-authored-by: Lingling Peng <lpeng@w290.local>
* fix bug cuasing RecordSet to be updated to v2 * updated unit tests * update docstring * fix type on test
Co-authored-by: Lingling Peng <lpeng@w290.local>
…1316) * add to_csv kwargs for store_row_async to ensure double quote and apostrophe formatted correctly when upload data from a dataframe * remove json string dumps function to let synapse decode data directly Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com>
* added generate_sync_manifest method * rename function * add docstring * refactoring to improve readability * improve readability * added unit tests * update doc * remove tests that didnt need to be integration tests * fix errors due to import name change * shortened docstrings * added semaphore * fixed deprecation docstring * fixed project links, added foilder links * fix cleanup * made function sync --------- Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com>
* Add viewtype mask * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Tom Yu <thomas.yu@sagebase.org> * Apply suggestion from @thomasyu888 --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Migrate all docs to new reference style * Fix folder tutorial * Fix tag * Move anchor * Fix anchors * Fix ordering * Update claude
…stallation (#1387) * Migrate all docs to new reference style * Fix folder tutorial * Fix tag * Move anchor * Fix anchors * Update docs * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Edit authentication docs --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* set higher requests version min * set higher requests version min
* added fields for CurationTask search * removed uneeded comment * fix docstring * fix docstring * redid some logic * combined two integration tests * combined two integration tests * moved code block into helper function * fix nits * Bryans suggesstions
* prepare function for removal of Grid creation * improved warning message * updated the warning * Apply suggestion from @BryanFauble Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> * added parameter for deprecation * various fixes * improve tests --------- Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com>
* [SYNPY-1835] Fix query() failure when a list column is all-NA
Problem: query() raises TypeError: Invalid value '[]' for dtype 'Int64'
when the result CSV has a list column whose every value is empty/NA.
read_csv().convert_dtypes() infers an all-empty column as the nullable
Int64 dtype, and the previous fillna({col: '[]'}) cannot store the
string '[]' into that typed column.
Fix: replace fillna+apply(json.loads) in csv_to_pandas_df with a single
apply that calls json.loads when the cell is a string and substitutes
[] otherwise. The downstream INTEGER_LIST/BOOLEAN_LIST/etc. coercion
blocks already guard with isinstance(x, list), so empty lists flow
through them untouched.
Test: parameterized reproducer in TestCsvToPandasDf covering every list
column type plus a mixed populated/all-NA case.
* added examples for getting data into a grid * clarified todo comment * un-number examples * remove redundant sentence * change exmaple to annotate two files * fix example script comment * fix c+p error * changed example to biospecimen, added example csv * added example csv * fixed typo * randomize entity names * added print statements for all entities created * update examples to add data directly into grid * split doc into curation and contribution * added not about upcoming assign_to_me param * various fixes * removed some examples * rearange doc * remove section * rename files * fix links * fixed consistency issues * remove grid creation * added back grid output * some cleanup * fix Grid session wording * added validation step * fixed typo * revert back to creating grids if needed * various fixes * added create_attached_grid method * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * added method to change the TaskState, and modified the docs to include it * changed grid bullet point to better reflect how guide works * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * ran pre-commit * added a note * moved file-based into its own section * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * Apply suggestion from @cconrad8 Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> * update CurationTask _set_last_persistent_instance * remvove uneeded line * fix broken links * combined similar fixtures into one * add missing methods * clarify docstring * expand docstring parameter description * fix how Task and GridExecutionDeatils work * added code and docs for handling old CurationTasks * add checks that RecordSet or EntityView exists before craeting grid * moved gird cleanup section in doc * update calls to create_record_based_metadata_task to use create_grid=False * update docs to use CurationTask filtering * removed CurationTaskState * various improvements * remove undded addition * cleanup docstring * method doesmt perform .upper on inputs any longer * remove uneeded tests * simplified tests * simplified tests * simplified tests * reverted TestGrid class * added principal owner id * Toms suggestions * address PR comment * Edit docs * Chistinas suggestion * stripped redundant dict.get(x, None) * removed uneeded imports from docstring examples --------- Co-authored-by: Christina Parry <114612268+cconrad8@users.noreply.github.com> Co-authored-by: Thomas Yu <thomas.yu@sagebase.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.