Delta Migration Feature#1044
Open
yashin4112 wants to merge 75 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Implements “delta migration” support by introducing an iteration counter, restart flow, and per-iteration mapping/UID persistence so subsequent runs can selectively recreate content types, deduplicate assets, and update existing entries.
Changes:
- Replaces upload-api runtime config import with a JSON-backed config file and adds a helper to refresh/update config at runtime.
- Adds “Restart Migration” UX + iteration tracking in UI (stepper/header/logs) and introduces an Entry Mapper table for iteration > 1.
- Adds API support for iteration-scoped LowDB files (content types, fields, entries, uid-mapper) plus delta helpers (asset dedup + entry update script + restart endpoint).
Reviewed changes
Copilot reviewed 50 out of 55 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| upload-api/src/services/fileProcessing.ts | Reads refreshed config during file validation/processing. |
| upload-api/src/services/aws/client.ts | Switches AWS config source to JSON config file. |
| upload-api/src/routes/index.ts | Refreshes config per request and returns sanitized config via /config. |
| upload-api/src/helper/index.ts | Adds updateConfigFile helper for JSON config read/write. |
| upload-api/src/controllers/sitecore/index.ts | Invokes Sitecore extractEntries during mapper creation. |
| upload-api/src/config/index.ts | Removes TS-based config module. |
| upload-api/src/config/index.json | Adds JSON-based config template. |
| upload-api/package.json | Formatting-only change. |
| upload-api/migration-wordpress/libs/extractEntries.ts | Adds WordPress entry extraction to build entryMapping. |
| upload-api/migration-sitecore/libs/extractEntries.js | Adds Sitecore entry extraction to build entryMapping. |
| upload-api/migration-sitecore/index.js | Exposes extractEntries from migration-sitecore package entrypoint. |
| upload-api/migration-drupal/libs/extractEntries.js | Adds Drupal DB entry extraction to build entryMapping. |
| upload-api/migration-drupal/libs/createInitialMapper.js | Attaches Drupal entryMapping into initial mapper output. |
| upload-api/migration-drupal/index.js | Exposes extractEntries from migration-drupal package entrypoint. |
| upload-api/migration-contentful/libs/extractEntries.js | Adds Contentful export entry extraction grouped by content type. |
| upload-api/migration-contentful/libs/createInitialMapper.js | Attaches Contentful entryMapping into initial mapper output. |
| upload-api/migration-contentful/index.js | Exposes extractEntries from migration-contentful package entrypoint. |
| upload-api/migration-aem/package-lock.json | Lockfile updates (types/undici-types). |
| ui/src/services/api/migration.service.ts | Adds restart + entry-mapping API calls for delta migration UI. |
| ui/src/pages/Migration/index.tsx | Adds “Restart Migration” execution path and iteration increment. |
| ui/src/context/app/app.interface.ts | Adds iteration and stepValue to migration state and defaults. |
| ui/src/components/Stepper/HorizontalStepper/HorizontalStepper.tsx | Resets stepper state on restart/iteration increment. |
| ui/src/components/MigrationFlowHeader/index.tsx | Derives CTA label (“Restart Migration”, etc.) into Redux state. |
| ui/src/components/LogScreen/MigrationLogViewer.tsx | Avoids duplicate completion notifications; sets “Restart Migration” CTA. |
| ui/src/components/LegacyCms/legacyCms.scss | Adds layout styles for editable local-path display. |
| ui/src/components/LegacyCms/Actions/LoadUploadFile.tsx | Adds inline editing UX for local path in restart iterations. |
| ui/src/components/ContentMapper/index.tsx | Switches to Entry Mapper view when iteration > 1; refetches on iteration change. |
| ui/src/components/ContentMapper/index.scss | Scopes table wrapper styles under .content-mapper-container. |
| ui/src/components/ContentMapper/entryMapper.tsx | New Entry Mapper table for selecting entries to update in delta runs. |
| ui/src/components/ContentMapper/contentMapper.interface.ts | Adds EntryMapperType interface for entry-mapping table rows. |
| ui/package.json | Formatting-only change. |
| package.json | Formatting-only change. |
| api/src/utils/package.json | Adds a package.json under src/utils (appears unrelated to runtime). |
| api/src/utils/field-attacher.utils.ts | Makes content type creation conditional based on iteration and prior existence. |
| api/src/utils/entry-update.utils.ts | Adds logic to remove/update entries and build update config for CLI script. |
| api/src/utils/entry-update-script.cjs | Adds CLI migration script to update existing entries (incl. asset resolution). |
| api/src/utils/entry-duplicate.utils.ts | Flags duplicate entries in entry mapper DB. |
| api/src/utils/content-type-checker.utils.ts | Adds helper to skip content type creation if created in prior iterations. |
| api/src/utils/asset-update.utils.ts | Adds asset dedup logic across iterations and ref replacement in entry JSON. |
| api/src/services/updateEntryCli.service.ts | Adds service to run CLI “update entries” script after delta import. |
| api/src/services/runCli.service.ts | Persists uid-mapping output into iteration-scoped LowDB (uid-mapper.json). |
| api/src/services/projects.service.ts | Stores iteration on project creation; uses iteration-scoped DBs on delete. |
| api/src/services/migration.service.ts | Adds restart endpoint + delta pre/post steps (asset dedup + entry updates). |
| api/src/services/contentMapper.service.ts | Adds iteration-scoped DB access + entry-mapping endpoints + uid enrichment. |
| api/src/routes/migration.routes.ts | Adds POST /restart/:orgId/:projectId route. |
| api/src/routes/contentMapper.routes.ts | Adds entry mapping GET + entry status update PUT routes. |
| api/src/models/uidMapper.ts | Adds iteration-scoped LowDB model for combined asset/entry uid mapping. |
| api/src/models/project-lowdb.ts | Adds iteration field to Project model. |
| api/src/models/contentTypesMapper-lowdb.ts | Converts content type mapper DB to iteration-scoped per-project storage. |
| api/src/models/FieldMapper.ts | Converts field mapper DB to iteration-scoped per-project storage. |
| api/src/models/EntryMapper.ts | Adds iteration-scoped LowDB model for entry mapper rows. |
| api/src/controllers/projects.contentMapper.controller.ts | Exposes entry mapping + update status endpoints from controller. |
| api/src/controllers/migration.controller.ts | Exposes restart migration endpoint from controller. |
| api/package.json | Bumps CLI dependencies for migration/update flows. |
Files not reviewed (1)
- upload-api/migration-aem/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
… service and utils; unit test cases added
…t tests for content type creation logic
…sting data reference
…ping resolution in Contentful service
…esModal with async handling for step changes
…le package.json and package-lock.json files
…r better handling of missing and media blocks
…aveChangesModal with async handling for step changes" This reverts commit fe3bfed.
…iple package.json and package-lock.json files
…esModal with async handling for step changes
…aveChangesModal with async handling for step changes" This reverts commit fe3bfed.
…esModal with async handling for step changes
…aveChangesModal with async handling for step changes" This reverts commit fe3bfed.
- Added checks for valid destination stack ID and improved error messages for asset index loading. - Implemented asynchronous file system operations to verify asset index existence and type. - Updated tests to cover new validation logic and ensure proper handling of asset index scenarios.
- Replaced with for child block UIDs to preserve existing digit-prefixed UIDs during merging. - Added function to enhance matching logic by allowing fallback to title for blocks with diverged UIDs. - Updated merging process to align UIDs and titles with the destination blocks, preventing the creation of duplicates.
- Replaced inline REGION_CONFIG object in index.ts with import from region-config.json for better maintainability. - Updated sso.utils.js to reference the new JSON configuration. - Removed redundant region configuration code from index.ts.
- Upgraded @contentstack/marketplace-sdk to version 1.5.2. - Updated axios to version 1.16.0 and follow-redirects to version 1.16.0. - Added react-toastify for improved user notifications during SSO flow. - Implemented toast dismissal logic to enhance user experience during SSO process.
…oken handling - Introduced a new function to streamline the creation of the marketplace client with proper handling of SSO Bearer tokens. - Updated , , and functions to utilize the new client builder. - Added unit tests to ensure correct routing of SSO Bearer tokens to the SDK's option instead of .
…; moved locale backend call above mapper
- Introduced a comprehensive PR template to standardize submissions. - Included sections for Jira ticket links, PR type, description, affected areas, testing instructions, and author checklist. - Aims to enhance clarity and consistency in the review process.
…mpty dropdown on step2
- Introduced hasMeaningfulHtmlContent function to determine if HTML contains substantive content. - Added attachCoverBackgroundMediaToChildren and attachMediaTextFieldsToChildren functions for improved media handling in core blocks. - Updated createSchema and schemaMapper functions to accommodate new media-text block processing. - Refined logic in extractItems and updated MIME type handling in getMimeTypeFromExtension for better robustness.
…esModal with async handling for step changes
…le package.json and package-lock.json files
…iple package.json and package-lock.json files
…Stack function - Introduced tests for the fetchMarketplaceInstallationsForStack function to validate pagination handling, error scenarios, and logging behavior. - Ensured proper aggregation of installations based on stack UID and added fallback mechanisms for failed fetch attempts. - Enhanced test coverage for various edge cases, including handling of non-Error exceptions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.