Add interactive cleanup, no‑auth, and generic lock manager#94
Merged
Conversation
Release 0.15.0
- Remove `tifffile` import and all compression‑related logic from `plantdb/commons/io.py`; always use `imageio.v3.imwrite`. - Fix comment typo and simplify extension handling. - Update docstring: delete `compress` parameter description, replace examples with random uint8/uint16 volumes, add size prints and `db.disconnect()` call. - Minor typo correction in `docs/developers/releases.md` (replace “…” with “…”).
- Remove `_test_write_file(..., compress=False)` and corresponding `_test_read_file` calls in `test_write_vol` and `test_read_vol` in `plantdb/src/commons/tests/test_io.py` to reflect the removal of compression handling.
- Updated connection success message in `core.py` to use `self.logger.info` with clearer wording - Changed scan creation, deletion, and completion logs from INFO to `self.logger.debug` - Switched scan owner and group update logs from INFO to `self.logger.debug` - Modified fileset and file creation/deletion logs from INFO to `self.logger.debug` - Adjusted import, raw write, and regular write logs from INFO to `self.logger.debug` - Updated login flow: existing session now logs a WARNING, successful login logs a DEBUG - Changed logout success log from INFO to DEBUG and logout failure to WARNING - Updated user data retrieval logs to DEBUG for token usage - Refined metadata update logs in `metadata.py` from INFO to DEBUG - Adjusted various other operational logs throughout `core.py` to DEBUG level for reduced noise.
- Remove unnecessary lock acquisition in `get_fileset` and `get_file` ( `core.py` ), directly checking existence and returning the object - Add `_has_active_lock` helper in `LockManager.acquire_lock` ( `lock.py` ) to detect active conflicting locks of a different type within the same process. - Introduce early retry logic with debug logging for: - `LockType.SHARED` blocked by an active `LockType.EXCLUSIVE` - `LockType.EXCLUSIVE` blocked by active `LockType.SHARED` locks. - Add new tests in `test_fsdb_lock.py`: - `test_exclusive_lock_blocks_shared_lock` verifies exclusive lock prevents shared lock acquisition. - `test_shared_lock_blocks_exclusive_lock` verifies shared lock prevents exclusive lock acquisition.
- Replaced "associated to" with "associated with" across multiple files for consistent phrasing.
…‑level lock support - Introduced `LockLevel` enum (`SCAN`, `FILESET`, `FILE`) and updated the public API to require a `level` argument for `acquire_lock`, `get_lock_status`, and related methods. - Renamed class `ScanLockManager` to `LockManager` and updated all references, imports, docstrings, and logger names accordingly. - Updated lock file and info path generation (`_get_lock_file_path`, `_get_lock_info_path`) to handle scan, fileset, and file levels with appropriate naming schemes. - Modified internal structures (`_active_locks`, `_lock_files`, `_warning_timestamps`) and helper methods (`_write_lock_info`, `_remove_lock_info`, `_release_lock`) to accept `resource_id` and `level`. - Adjusted lock key format to include resource ID, level, and lock type (`resource_id/level/lock_type`) for accurate tracking. - Updated warning debounce logic to reference the generic `resource_id` instead of `scan_id`. - Revised docstring examples to demonstrate usage with `LockManager`, `LockType`, and `LockLevel`. - Updated log messages to include the lock level and resource identifier for clearer debugging.
- Import `LockLevel` and core classes `Fileset` and `File` in `metadata.py`.
- Detect whether `self` is a `Scan`, `Fileset`, or `File` and construct the appropriate `obj_id` (`{scan.id}`, `{scan.id}/{fileset.id}`, or `{scan.id}/{fileset.id}/{file.id}`) and corresponding `lock_level`.
- Acquire an exclusive lock with the new signature `acquire_lock(obj_id, LockType.EXCLUSIVE, current_user.username, lock_level)`.
- Move and enhance debug logging to include the resolved `obj_id` and lock level.
- Add a `TypeError` raise for unsupported object types.
…re metadata access - Replace `ScanLockManager` imports with `LockManager` and add `LockLevel` import in `core.py` - Initialize `self.lock_manager` using `LockManager` instead of `ScanLockManager` - Update all lock acquisitions to the new signature `acquire_lock(obj_id, LockType, username, LockLevel.<LEVEL>)` - Use `LockLevel.SCAN` for scan‑level operations - Use `LockLevel.FILESET` for fileset‑level creation/deletion - Use `LockLevel.FILE` for file‑level creation, deletion, import, and write - Adjust `get_lock_status` call to include `LockLevel.SCAN` - Add authentication and permission decorators to metadata‑related methods in `Scan`, `Fileset`, and `File` (`get_metadata`, `get_measures`, etc.) and pass `current_user` to lock calls - Refactor metadata getters to acquire a shared lock with the appropriate `LockLevel` before reading - Update owner and sharing metadata updates to acquire an exclusive lock with `LockLevel.SCAN` - Fix docstring wording (“associated with”) and improve comments about locking being handled by `MetadataManager` - Minor adjustments: wording fixes in fileset description, update comments for exclusive lock usage, and correct typo in raw/write file logs.
- Replace `ScanLockManager` imports with `LockManager` and add `LockLevel` in `test_fsdb_lock.py`
- Rename test class to `TestLockManager`
- Instantiate `LockManager` instead of `ScanLockManager`
- Update all internal calls (`_get_lock_file_path`, `_get_lock_info_path`, `_write_lock_info`, `_remove_lock_info`) to include `LockLevel.SCAN`
- Change lock info field from `scan_id` to `resource_id` in assertions
- Adjust lock key format to `"{scan_id}/scan/{type}"` for shared and exclusive locks
- Update lock acquisition signatures to `acquire_lock(scan_id, LockType, user, LockLevel.SCAN, ...)` throughout the tests
- Modify `get_lock_status` calls to pass `LockLevel.SCAN`
- Update all lock release and status checks to use the new path and key conventions
- Ensure multi‑lock scenarios (shared, exclusive, timeout) use the new `LockLevel` argument consistently
- Update comments and docstrings reflecting the rename and new level‑based API.
- Update comment to reflect new format `resource_id/lock_level/lock_type`
- Change split to `lock_key.split('/')` and allow more than three parts
- Extract `lock_type` and `lock_level` with `pop()`
- Reconstruct `resource_id` using `"/".join(parts)` to handle IDs containing `/`
- Add path validation for `_get_lock_file_path` and `_get_lock_info_path` at `LockLevel.FILESET` and `LockLevel.FILE` in `test_fsdb_lock.py` - Verify `_cleanup_stale_locks` skips active lock files and removes stale ones - Test `_write_lock_info` / `_remove_lock_info` for fileset and file levels, asserting file existence and JSON content - Introduce lock status checks for complex lock combinations and for `LockLevel.FILESET` - Add file‑level lock acquisition tests, confirming lock files, active‑lock entries, and info files are created and cleaned up - Add tests for acquiring multiple file‑level locks on different resources simultaneously - Ensure all new tests cover correct handling of `resource_id`, `LockLevel`, and lock types across the manager.
- Updated `acquire_lock` docstring in `plantdb/src/commons/plantdb/commons/fsdb/lock.py` with multi‑level lock examples using `tempfile.gettempdir()`, demonstrating exclusive lock acquisition for both scan and fileset resources. - Enhanced `get_lock_status` docstring in the same file with example code that acquires a lock via a context manager and prints the resulting status for scan and fileset levels.
…ests - Store the new password in `user.password_hash` instead of the inexistant `user.password` attribute. - Add `test_update_password_success_case`, `test_update_password_invalid_current_password`, and `test_update_password_nonexistent_user` to `plantdb/src/commons/tests/test_auth.py` to verify proper password change behavior and edge cases.
- Introduced `has_logged_user(self) -> bool` in `plantdb/src/commons/plantdb/commons/auth/session.py` - Provides a simple boolean check for the presence of any active logged‑in user.
- Add safe session token extraction with `try/except IndexError` and log error when no user is logged in - Update `get_logged_username` signature to return `User | TokenUser | None` and adjust docstring accordingly - Guard login against existing sessions using new `has_logged_user` check - Change `logout` to return a `(success, username)` tuple, handling missing user with error and warning logs - Adjust docstring examples throughout to reflect new logout return format - Refine type hints for `username` and `token` parameters to `str | None` where appropriate - Minor documentation and logging improvements for consistency
…mons/fsdb/core.py` - Implement `update_user_password` with `@get_authentication` and `@require_authentication` decorators to allow authenticated users to change their password. - Method accepts `old_password`, `new_password`, and optional `current_user`, forwarding the request to `rbac_manager.users.update_password`. - Provide comprehensive docstring covering parameters, raised `PermissionError`, and usage examples.
- Implement `RBACManager.create_user` with permission check, logging, and delegation to `UserManager.create` - Update usage examples in `rbac.py` and `__init__.py` to call `rbac.create_user` instead of `rbac.users.create_user` - Refine class docstring. - Clarify `__init__` docstrings for `users_file` and `groups_file` parameters - Minor formatting clean‑up throughout the file.
- Update `create_user` signature in `plantdb/src/commons/plantdb/commons/fsdb/core.py` to include optional `current_user: User | TokenUser | None` and remove the `requires_permission` decorator. - Modify docstring to describe the new `current_user` parameter and update the `PermissionError` description to reference the `current_user`. - Forward user creation to `self.rbac_manager.create_user(current_user, ...)` and discard the previous return value. - Update related references from `RBACManager.users.create` to `RBACManager.create_user`. - Revise password‑update error description to a generic “permissions” message. - Correct typo in example code (`prin` → `print`). - Enhance session token handling in `get_user` to use `self.session_manager.validate_session` directly for non‑`SingleSessionManager` cases and extract the username for `SingleSessionManager`. - Apply minor formatting and documentation clean‑ups.
- Introduce `_get_fsdb` in `plantdb/src/commons/plantdb/commons/fsdb/core.py` to retrieve the `FSDB` instance from `FSDB`, `Scan`, `Fileset`, or `File` objects with proper type checks. - Fix permission checks by obtaining `db` via `_get_fsdb(self)` and using `db.rbac_manager` for permission validation.
- Update docstring parameter type hints to reflect the new union types. - Extend the usage example to demonstrate retrieving the guest user, checking permissions, creating a new user, and verifying the user is added to the JSON database.
… all README files.
- Extend `FSDB.__init__` to accept `log_level` via `**kwargs` (default from `DEFAULT_LOG_LEVEL`) and pass it to `get_logger` - Import `DEFAULT_LOG_LEVEL` and use it when initializing the logger - Add type hints (`Callable -> Callable`) to authentication and connection decorators - Introduce `use_guest_as_default` decorator that injects a guest user when `current_user` is missing - Replace `@require_authentication` with `@use_guest_as_default` on metadata/measures getters for `Scan`, `Fileset`, and `File` - Update example usage to show `log_level="DEBUG"` when creating an `FSDB` instance
- Switch `See Also` links from `plantdb.server.rest_api` to `plantdb.server.services.scan` and `plantdb.server.core.security` where appropriate. - Update `get_scan_info` calls to forward additional `**kwargs`. - Adjust imports in `api/base.py` to use `plantdb.server.core.security.rate_limit`. - Refactor assets API `See Also` sections to reference `plantdb.server.api.scan` resources instead of `rest_api`. - Update sanitization references to `plantdb.server.core.security.sanitize_name`.
- Added a **Production ready** section in `plantdb/README.md` describing how to enable authentication by setting `ROMI_DB_NOAUTH="0"` in a `.env` file or as an environment variable, and noted precedence rules.
- Updated the Python usage example to retrieve the database path with `os.getenv('ROMI_DB')` instead of direct indexing of `os.environ`.
…manager - Import `SessionManager` in `src/commons/plantdb/commons/test_database.py` - Document new `no_auth` boolean parameter in the function docstring - Update example usage to `test_database(no_auth=True)` - Implement logic: - Use `NoAuthSessionManager` when `no_auth=True` - Default to `SingleSessionManager` otherwise - Allow overriding via explicit `session_manager` argument if it is an instance of `SessionManager`
- Replace `dummy_db` with `test_database` in all usage examples and add explicit `db.connect()` and login steps across `core.py`. - Adjust `create_api_token` signature: `datasets` now accepts `list[Permission]` instead of a tuple. - Extend `create_api_token` docstring to describe the optional `token` parameter and provide a full example with `JWTSessionManager`. - Simplify implementation to call `self.session_manager.create_api_token` directly. - Correct example output for `get_username` and `get_user_data` to reflect proper token usage. - Update examples for user and group management (`create_user`, `create_group`, `list_groups`, etc.) to match the new `test_database` workflow.
- Import `SessionManager` and remove unused `SingleSessionManager` import - Consolidate `tqdm` import and eliminate duplicate import statement - Add `-> Path` return type annotations to `_get_archive`, `_get_extract_archive`, `get_test_dataset`, `get_models_dataset`, `get_configs`, `setup_empty_database`, and `setup_test_database` (split across lines for readability) - Update docstrings to reference `pathlib.Path` as the return type - Simplify session manager logic in `test_database`: - Extract `no_auth` flag and pass it to `FSDB` constructor - Use provided `session_manager` only if it is an instance of `SessionManager`; otherwise default to `None` - Adjust FSDB creation calls to include `no_auth=no_auth` parameter for both empty and populated database setups.
….py` - Document new `no_auth` parameter in `FSDB.__init__` docstring, explaining session manager selection. - Include API demo example showing connection, login, and usage of `NoAuthUserError` when `no_auth=True`. - Add second example illustrating authenticated access, token retrieval, and subsequent scan fetching. - Extend `get_metadata` example to show metadata access after login and as a guest user.
- Move `fid = file_info.get("id")` retrieval to after successful `_load_file` in `file_ops.py`, removing premature `FileNotFoundError` for missing IDs.
- Update `serialization.py` to raise `FileNotFoundError(fileset, fid)` with contextual arguments instead of a generic string message.
- Replace `from plantdb.commons import db` and `fsdb` imports with `from plantdb.commons.db import File` in `io.py` - Change every `isinstance(..., fsdb.core.File)` check to `isinstance(..., File)` - Adjust docstrings to reference the new `File` class path - Remove unused `fsdb` import after migration - Ensure all read/write helper functions use the updated `File` reference throughout `plantdb/src/commons/plantdb/commons/io.py`
…zation to restrict unauthorized usage.
…ger` Abort initialization when `ROMI_DB_NOAUTH` is set to `0`, raising a `RuntimeError`.
- Implement `__init__` in `FilesetNoIDError` (in `plantdb/src/commons/plantdb/commons/fsdb/exceptions.py`) to include the scan ID in the error message. - Implement `__init__` in `FileNoIDError` (in `plantdb/src/commons/plantdb/commons/fsdb/exceptions.py`) to include both the scan and fileset IDs in the error message.
…zation.py` - Updated `FilesetNoIDError` raise to `FilesetNoIDError(scan)` so the missing fileset ID includes the originating `scan`. - Updated `FileNoIDError` raise to `FileNoIDError(fileset)` providing the containing fileset as context instead of a generic message.
…uring loading - Import `yes_no_choice` and replace unused `FileNoIDError`, `FilesetNoIDError`, `FilesetNotFoundError` imports in `file_ops.py` - `_load_scan` now receives `needs_update` from `_load_scan_filesets` and calls `_store_scan` when needed - `_load_scans` gathers `bad_scans`, logs them, and prompts the user with `yes_no_choice` to remove the problematic scans using `Scan` objects - Updated `_load_scan_filesets`, `_load_fileset`, and `_load_fileset_files` to return a tuple `(obj, needs_update)` and propagate the `needs_update` flag upward - Switched to generic `Exception` handling in loading loops, logging errors and marking `needs_update` on failure - Added import of `Scan` inside `_load_scans` for deletion operations - Minor cleanup: removed unused imports, added explanatory comments, and refined logging messages.
Flexible FSDB load with interactive cleanup of bad scans
- Import `tempfile` in `plantdb/src/commons/plantdb/commons/fsdb/core.py`. - When the session manager is an instance of `NoAuthSessionManager`, create a temporary directory (`tempfile.mkdtemp`) and assign it to `manager_dir` to hold isolated RBAC files. - Use `manager_dir` for both `users_file` and `groups_file` paths, falling back to `basedir` for other session managers.
- Add `Any` import for type annotations in `manager.py` - Introduce `_load_user_file` to read the user list from the JSON persistence file - Add `_parse_user_list` to convert loaded dicts into `User` objects and populate `self.users` - Update `_load_users` to reset `self.users`, use the new helper methods, and handle parsing errors with critical logs - Log a warning for outdated user database files and suggest fixing with `scripts/fix_local_fsdb.py`
- Introduce `yes_no_choice` in `plantdb/src/commons/plantdb/commons/utils.py` to ask yes/no questions with an optional default answer. - Implements input handling, default selection on empty input, and validation loop with clear error messaging. - Includes comprehensive docstring with usage examples.
- Import `yes_no_choice` utility in `manager.py`
- Simplify exception capture in user loading logic
- Detect outdated user DB format and log a warning
- Prompt the user with `yes_no_choice("Do you want to replace it?")` to confirm replacement
- Raise the original exception if the user declines or if the data format is unexpected
- Change default option display from `[YES/no]` / `[yes/NO]` to `[Y/n]` / `[y/N]` - Adjust prompt construction to use the new `default_choice` string - Update docstring examples to reflect the revised prompts and return values
- In `plantdb/src/commons/plantdb/commons/auth/manager.py`, replace the `pass` placeholder with `self.users = {}` to ensure the method returns an empty dictionary for the `users` attribute, triggering the creation of a fresh user database after the user confirms replacement.
- Delete the import of `yes_no_choice` from `plant3dvision.utils` in `src/commons/plantdb/commons/fsdb/file_ops.py` as it is no longer used.
- Import `yes_no_choice` from `..utils` in `src/commons/plantdb/commons/fsdb/file_ops.py` to make the utility available for user prompts.
- Wrap `yes_no_choice` call in a `try/except EOFError` block to handle non‑interactive environments, defaulting the answer to **False** and logging a debug message. - Use the pre‑computed `n_bad` variable in the prompt instead of calling `len(bad_scans)` again. - Proceed with deletion of bad scans only when the prompt succeeds and the user confirms.
Updated `plantdb/src/client/plantdb/client/api_endpoints.py` to delete the `file_id` parameter documentation, clarifying that the endpoint no longer requires this argument.
…ring - Update `plantdb/src/commons/plantdb/commons/auth/rbac.py` docstring to reference `required_permissions` instead of the misspelled `required_permission`. - Reformat the `secret_key` description in `plantdb/src/commons/plantdb/commons/auth/session.py` for consistent indentation and clearer wording.
- Updated `plantdb/src/server/README.md` with sections on creating a Python virtual environment and Conda environment - Added instructions for installing pre‑built packages and developing from source - Included usage examples for testing, development, and production server runs - Provided links to REST API documentation and endpoint references - Minor formatting fixes and added a blank line in the features list for readability
- Updated `plantdb/mkdocs.yml` to nest REST API under a submenu with `rest_api/index.md`, `rest_api/rest_api_usage.md`, and `rest_api/rest_api_endpoints.md`
- Created new documentation files `rest_api/index.md`, `rest_api/rest_api_usage.md`, and moved endpoint details to `rest_api/rest_api_endpoints.md`
- Refactored `rest_api_endpoints.md` for consistent formatting, clearer server start instructions, and improved parameter documentation
- Fixed typo in `plantdb/docs/plantdb_sync.md` ("amy" → "may")
- Adjusted navigation entries and headings to align with the new documentation structure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This pull request introduces a collection of enhancements focused on improving user interaction, authentication flexibility, and file system database (FSDB) locking:
Interactive cleanup of invalid scans
yes_no_choiceutility and integrated prompts infile_ops.py.EOFErrorhandling for non‑interactive environments.User database handling
self.users.yes_no_choiceimport and updated docstrings.No‑auth support
NoAuthSessionManagerfor test and guest scenarios.ROMI_DB_NOAUTHchecks.FSDBinitialization to select the appropriate session manager.Generic multi‑level lock manager
ScanLockManagertoLockManagerwithLockLevelenum (SCAN,FILESET,FILE).Logging and utilities
log_levelhandling.IO simplifications
tifffiledependency and compression handling fromwrite_volume.Miscellaneous
pathlibpaths.These changes collectively make the FSDB more robust, user‑friendly, and easier to test in various environments.