-
Notifications
You must be signed in to change notification settings - Fork 7
PCSM-283 Skip sharding operations for replica set #67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
rasika-chivate
wants to merge
45
commits into
main
Choose a base branch
from
PCSM-283-Skip-sharding-operations-for-replica-set
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
45 commits
Select commit
Hold shift + click to select a range
47a43c4
PCSM-283 Skip sharding operations for replica set
rasika-chivate 542e25e
Create sharded-source-to-replica-set-target.md
rasika-chivate b789c70
Update sharded-source-to-replica-set-target.md
rasika-chivate b43c0bc
Update sharded-source-to-replica-set-target.md
rasika-chivate ed44f22
Update sharded-source-to-replica-set-target.md
rasika-chivate 8cadb85
Update sharded-source-to-replica-set-target.md
rasika-chivate 7a02a06
Update sharded-source-to-replica-set-target.md
rasika-chivate bccb242
examples before and after cross topology support
rasika-chivate 675429f
Update sharded-source-to-replica-set-target.md
rasika-chivate ba79a33
Update sharded-source-to-replica-set-target.md
rasika-chivate 9a972b7
Update sharded-source-to-replica-set-target.md
rasika-chivate 15dd913
Update sharded-source-to-replica-set-target.md
rasika-chivate 9266ff8
Potential fix for pull request finding
rasika-chivate e84e92c
Potential fix for pull request finding
rasika-chivate 003e8e2
Potential fix for pull request finding
rasika-chivate f67df55
Potential fix for pull request finding
rasika-chivate 5a290f0
Update sharded-source-to-replica-set-target.md
rasika-chivate d608fcc
Merge branch 'PCSM-283-Skip-sharding-operations-for-replica-set' of h…
rasika-chivate 090cd53
Potential fix for pull request finding
rasika-chivate 6cd6e66
Address documentation review feedback
Copilot 2d56962
Update sharded-source-to-replica-set-target.md
rasika-chivate a993af9
Merge branch 'PCSM-283-Skip-sharding-operations-for-replica-set' of h…
rasika-chivate 895a309
Update sharded-source-to-replica-set-target.md
rasika-chivate b63a5ff
Potential fix for pull request finding
rasika-chivate 3f8f690
Potential fix for pull request finding
rasika-chivate db72be9
Update docs/sharded-source-to-replica-set-target.md
rasika-chivate 19e103e
PCSM-335 Use RFC 3339 format for log timestamps
rasika-chivate bc7fde8
Merge branch 'PCSM-283-Skip-sharding-operations-for-replica-set' of h…
rasika-chivate 91c9e95
Update docs/sharded-source-to-replica-set-target.md
rasika-chivate 271c764
Update sharded-source-to-replica-set-target.md
rasika-chivate 4ff0fcb
Merge branch 'PCSM-283-Skip-sharding-operations-for-replica-set' of h…
rasika-chivate 5465fc3
implement feedback from dev
rasika-chivate d8a7af0
Update sharded-source-to-replica-set-target.md
rasika-chivate 3196536
Update sharded-source-to-replica-set-target.md
rasika-chivate 2b9c87e
Potential fix for pull request finding
rasika-chivate 644794e
Clarify sharded-target vs replica-set-target guidance in sharding docs
Copilot ea37bd1
Update sharded-source-to-replica-set-target.md
rasika-chivate 8589625
Merge branch 'PCSM-283-Skip-sharding-operations-for-replica-set' of h…
rasika-chivate 470c43d
Update sharded-source-to-replica-set-target.md
rasika-chivate 5ef5a00
Potential fix for pull request finding
rasika-chivate 4453888
Split sharding prerequisites by target topology
Copilot 5438446
implement feedback
rasika-chivate 929cde3
Merge branch 'PCSM-283-Skip-sharding-operations-for-replica-set' of h…
rasika-chivate 7421f62
Update sharded-source-to-replica-set-target.md
rasika-chivate d43e402
Addressed remaining review wording fixes
Copilot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| # Replicate from a sharded cluster to a replica set | ||
|
|
||
| !!! admonition "Version added: 0.10.0" | ||
|
|
||
| {{pcsm.full_name}} (PCSM) supports replication from a sharded MongoDB cluster to a replica set. This lets you migrate data from a sharded deployment without having to recreate the source sharding configuration on the target. | ||
|
|
||
| For example, you can use this topology when moving data from a sharded MongoDB Atlas or MongoDB Enterprise deployment to a Percona Server for MongoDB replica set. | ||
|
|
||
| For information about sharded cluster support, see [Sharding support in Percona ClusterSync for MongoDB](sharding.md). | ||
|
|
||
| ## Overview | ||
|
|
||
| When the PCSM server starts, it detects that the source is sharded and the target is a replica set. | ||
|
|
||
| During the initial sync, PCSM creates every source collection on the target, including the sharded ones, as a regular collection. It doesn't carry over the source shard key, because a replica set has no shards to distribute documents across and doesn't support [`shardCollection` :octicons-link-external-16:](https://www.mongodb.com/docs/manual/reference/command/shardCollection/){:target="_blank"}. | ||
|
|
||
| During change replication, PCSM skips `shardCollection` events coming from the source [change stream :octicons-link-external-16:](https://www.mongodb.com/docs/manual/changeStreams/){:target="_blank"} and keeps applying the data changes it supports. No additional configuration is required. | ||
|
|
||
| !!! note | ||
| A collection that is sharded on the source is created as a regular collection on the replica set target. The collection data is copied, but the target collection isn't sharded. | ||
|
|
||
| ## What is replicated | ||
|
|
||
| | **On the source** | **On the replica set target** | | ||
| |---|---| | ||
| | Sharded collection | Created as a regular collection. All documents are copied. The shard key isn't applied because it doesn't apply to a replica set. | | ||
|
rasika-chivate marked this conversation as resolved.
|
||
| | Unsharded collection | Created and copied as in a replica set to replica set sync. | | ||
| | Chunk distribution and primary shard | Not preserved. PCSM replicates data, not cluster metadata. | | ||
|
|
||
| ## Before you start | ||
|
|
||
| * Ensure the source and target MongoDB versions meet the version requirements. | ||
| * Configure authentication for both deployments. | ||
| * Configure the source connection string with the `mongos` hostname and port. Configure the target connection string with the replica set members. | ||
| * Verify that PCSM can connect to both the source sharded cluster and the target replica set. | ||
|
|
||
| ## Connection string format | ||
|
|
||
| Point the source URI at the `mongos` hostname and port. Point the target URI at the replica set members and name the replica set: | ||
|
|
||
| **Example** | ||
|
|
||
| ```sh | ||
| PCSM_SOURCE_URI="mongodb://source-user:password@mongos-source:27017/admin" | ||
|
|
||
| PCSM_TARGET_URI="mongodb://target-user:password@target1:27017,target2:27017,target3:27017/admin?replicaSet=rs0" | ||
| ``` | ||
|
|
||
| ## usage | ||
|
|
||
| The commands and API endpoints are the same as for any other topology. See, [Percona ClusterSync for MongoDB commands](pcsm-commands.md) for the command reference. | ||
|
|
||
| ??? example "Walkthrough: sharded source to replica set target" | ||
|
|
||
| Follow these steps: | ||
| {.power-number} | ||
|
|
||
| 1. Create two clusters: one sharded source cluster and one target replica set. | ||
|
|
||
| 2. Create two collections on the sharded cluster: | ||
|
|
||
| 1. `sharded_coll` (sharded) | ||
| 2. `plain_collection` (unsharded) | ||
|
|
||
| 3. Add documents to both the collections. | ||
|
|
||
| 4. Start replication: | ||
|
|
||
| ```sh | ||
| pcsm start | ||
| ``` | ||
|
|
||
| 5. Check the replication status. `clonedSizeBytes` matches `estimatedCloneSizeBytes`, and the state is `running`: | ||
|
|
||
| ```{.text .no-copy} | ||
| pcsm status | ||
| { | ||
| "ok": true, | ||
| "state": "running", | ||
| "info": "Replicating Changes", | ||
| "lagTimeSeconds": 0, | ||
| "eventsRead": 0, | ||
| "eventsApplied": 0, | ||
| "lastReplicatedOpTime": { | ||
| "ts": "1787645813.1", | ||
| "isoDate": "2026-08-25T08:16:53Z" | ||
| }, | ||
| "initialSync": { | ||
| "estimatedCloneSizeBytes": 7490, | ||
| "clonedSizeBytes": 7490, | ||
| "completed": true, | ||
| "cloneCompleted": true | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| 6. Run the same query against both deployments and compare the results to confirm that both collections are present and that document counts match: | ||
|
|
||
| ```javascript | ||
| db.sharded_coll.countDocuments() | ||
| db.plain_collection.countDocuments() | ||
| ``` | ||
|
|
||
| The collection that was sharded on the source appears here as a regular collection. That is expected. | ||
|
|
||
| 7. Finalize the sync: | ||
|
|
||
| ```{.bash data-prompt="$"} | ||
| $ pcsm finalize | ||
| ``` | ||
|
|
||
| 8. Check the status again until the state is `finalized`: | ||
|
|
||
| ```{.bash data-prompt="$"} | ||
| $ pcsm status | ||
| ``` | ||
|
|
||
| ```{.json .no-copy} | ||
| { | ||
| "ok": true, | ||
| "state": "finalized", | ||
| "info": "Finalized", | ||
| "lagTimeSeconds": 1, | ||
| "eventsRead": 0, | ||
| "eventsApplied": 0, | ||
| "lastReplicatedOpTime": { | ||
| "ts": "1787645817.1", | ||
| "isoDate": "2026-08-25T08:16:57Z" | ||
| }, | ||
| "initialSync": { | ||
| "estimatedCloneSizeBytes": 7490, | ||
| "clonedSizeBytes": 7490, | ||
| "completed": true, | ||
| "cloneCompleted": true | ||
| }, | ||
| "finalization": { | ||
| "completed": true, | ||
| "startedAt": "2026-08-25T08:16:57.539447026Z", | ||
| "completedAt": "2026-08-25T08:16:57.539557888Z" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| 9. Check the replication logs and confirm that no errors were recorded. For details, see [Logging in Percona ClusterSync for MongoDB](logging.md). | ||
|
|
||
| 10. Confirm that the documents for both `plain_collection` and `sharded_coll` got copied to the destination cluster. | ||
|
|
||
|
|
||
| ## Next steps | ||
|
|
||
| * [Install {{pcsm.full_name}}](installation.md) | ||
| * [Configure authentication](install/authentication.md) | ||
| * [Start replication](install/usage.md) | ||
| * [Monitor replication status](install/usage.md#check-the-replication-status) | ||
| * [Monitor PCSM performance with Percona Monitoring and Management](pmm-setup.md) | ||
|
|
||
|
|
||
| ## Learn more | ||
|
|
||
| [Shard Keys :octicons-link-external-16:](https://www.mongodb.com/docs/manual/core/sharding-shard-key/){:target="_blank"} | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.