Add Bitbucket Server to GitHub Enterprise migration tool - #1
Open
eddymhawejj wants to merge 74 commits into
Open
Add Bitbucket Server to GitHub Enterprise migration tool#1eddymhawejj wants to merge 74 commits into
eddymhawejj wants to merge 74 commits into
Conversation
Python CLI (bb2gh) with three subcommands: - migrate: bulk clone repos via SSH, push --mirror to GitHub - sync: continuous fetch+push every 60s for smooth transition - migrate-prs: recreate open Bitbucket PRs on GitHub with comments Includes Bitbucket Server REST API client, GitHub Enterprise client (PyGithub), state tracking, Docker support, and 15 passing tests. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 180c0b3495
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| and reviewer assignments. | ||
| """ | ||
| config = ctx.obj["config"] | ||
| migrated, skipped, failed = migrate_pull_requests(config, dry_run=dry_run) |
There was a problem hiding this comment.
migrate_pull_requests returns None when no migrated repos exist (early return in pr_migrator.py), but this call site always unpacks three values. On a fresh environment where bb2gh migrate has not run yet, bb2gh migrate-prs will raise TypeError instead of exiting cleanly, so this path needs a default (0, 0, 0) result or a guard before unpacking.
Useful? React with 👍 / 👎.
| def get_clone_url(self, repo_name): | ||
| """Get the HTTPS clone URL for a repo.""" | ||
| repo = self.org.get_repo(repo_name) | ||
| return repo.clone_url |
There was a problem hiding this comment.
This returns repo.clone_url (plain HTTPS), and the migrator uses that remote for git push --mirror; unless the runtime already has an external credential helper configured, pushes to GitHub Enterprise will fail even though gh_token exists in config. To make migration reliable in unattended runs (including Docker), this should use an authenticated transport (for example SSH URL or token-backed HTTPS).
Useful? React with 👍 / 👎.
| # One-time bulk migration | ||
| migrate: | ||
| build: . | ||
| command: ["migrate", "--config", "/config/config.yaml"] |
There was a problem hiding this comment.
With ENTRYPOINT ["bb2gh"], this becomes bb2gh migrate --config /config/config.yaml, but --config is declared on the top-level Click group (cli.py) and must be parsed before the subcommand. In this order Click raises No such option: --config, so the migrate service command (and the same pattern in sync/migrate-prs) fails at startup.
Useful? React with 👍 / 👎.
| repo_name=repo_slug, | ||
| title=title, | ||
| body=body, | ||
| head=head_branch, |
There was a problem hiding this comment.
GitHub create_pull needs source-repo context for cross-repo/fork PRs, but this passes only the branch name as head. For Bitbucket PRs whose fromRef points to a different repository, migration will fail (or target the wrong branch if names collide), so the source repository owner/name must be incorporated when building head.
Useful? React with 👍 / 👎.
Support mapping Bitbucket projects to different GitHub orgs and customizing repo names via templates and per-repo overrides in config. - Config.resolve_target() resolves (bb_project, bb_slug) -> (gh_org, gh_repo_name) - GithubClient now supports multi-org operations via per-method org_name param - State tracks the resolved GitHub org/repo for each migrated repo - PR migrator uses stored mapping with fallback to config resolution https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Covers prerequisites, Bitbucket access requirements, SSH setup, config with repo mapping examples, migration workflow, Docker usage, CLI reference, full configuration table, and troubleshooting. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Lets users migrate only a subset of repos from a Bitbucket project: - include_repos acts as an allowlist (exclusive) - exclude_repos acts as a denylist - include wins when both are set https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Fixes SSL handshake failure when Bitbucket Server uses an internal CA. Set bitbucket.verify_ssl: false in config to disable verification. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
- Inject PAT into HTTPS clone URL so git push doesn't prompt for credentials interactively - Suppress urllib3 InsecureRequestWarning when verify_ssl is false https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
… sync Rewrites .gitmodules on all branches using git plumbing (hash-object, mktree, commit-tree) in bare repos. Runs between fetch and push so GitHub always has the correct URLs. Uses fixed timestamps for deterministic commits to avoid unnecessary force-pushes. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Reads the symbolic-ref HEAD from the bare clone (Bitbucket's default branch, typically 'master') and sets it via the GitHub API after mirror push, preventing GitHub from picking a random default. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
SSH submodule URLs now remap to git@host:org/repo.git format instead of HTTPS. HTTP submodule URLs continue to remap to HTTPS. New config: github.ssh_host sets the GitHub SSH hostname. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Repos with files >100MB (GitHub's limit) are automatically handled by running git lfs migrate import --everything --above=<threshold> before push. LFS objects are pushed separately after the mirror push. Opt-in via lfs.enabled config. Works for both migrate and sync. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
git lfs migrate import requires a working tree. Now clones the bare repo to a temp directory, runs LFS migration there, then fetches the rewritten refs and LFS objects back into the bare repo. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
GitHub rejects descriptions containing control characters (newlines, tabs, etc.) that Bitbucket allows. Strip them and cap at 350 chars. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
LFS migration now creates local branches for all remote branches in the temp working copy before running git lfs migrate, ensuring all branches are rewritten. Also fetches rewritten tags. Error logs now redact tokens from URLs to prevent credential leaks. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Branch-already-exists and remote-not-found errors are expected during normal operation. Use quiet=True to prevent logging them as ERROR. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
…ailure - Match BB URLs by hostname (supports FQDN variants via ssh_hostnames config) - If ANY Bitbucket URL cannot be resolved, skip the entire .gitmodules rewrite to avoid mixing old and new URLs - Already-GitHub and external URLs are ignored (not counted as failures) - URLs already pointing to GitHub are left as-is https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Scans bare clones for repos that have .gitmodules, removes them from state.json so the next migrate run re-processes them with the updated submodule URL remapping. Supports --dry-run to preview. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
_migrate_lfs now returns whether it actually converted any files. git lfs push --all is only run when there are LFS objects to push, preventing hour-long hangs scanning repos with no large files. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Scans bare clones for blobs above a size threshold (default 100mb), resets those repos in state.json so the next migrate run re-processes them with the current LFS config. Supports --above and --dry-run. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Previous approach scanned for large blobs, but after LFS migration the blobs are already replaced with pointers. Now checks for lfs/ directory and .gitattributes with filter=lfs patterns instead. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Repos that already use LFS on Bitbucket have pointers referencing BB's LFS server. The temp clone for LFS migration would fail trying to download those objects. Set GIT_LFS_SKIP_SMUDGE=1 to skip. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Allows resetting specific projects, individual repos, or everything in state.json so they get re-migrated. Useful when changing org mappings or fixing specific repos. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
git clone --bare does not create a fetch refspec, so git fetch origin was a no-op — it never reset local refs to Bitbucket's originals. Now explicitly fetches +refs/heads/*:refs/heads/* and +refs/tags/*:refs/tags/* so re-runs properly overwrite previously remapped refs. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
git lfs migrate import may fail on the post-rewrite checkout when HEAD points to a non-existent branch. The rewrite itself completes successfully. Detect this case and continue instead of failing. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
os.listdir on lfs/objects/ was matching empty subdirectories created by git lfs install, causing git lfs push --all to run on repos with no real LFS objects. Now walks the directory tree to check for actual files before reporting LFS objects exist. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
When github.ssh_host is set, the push remote uses git@host:org/repo.git instead of HTTPS. This avoids HTTP 500 timeouts on large repos and keeps protocol consistent with origin. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Change from SCP-style (git@host:org/repo.git) to standard SSH URL format (ssh://git@host/org/repo.git) for compatibility. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
GHE may use a non-standard SSH user (e.g. gatehousesatcom@ instead of git@). New config github.ssh_url allows specifying the full SSH prefix (e.g. ssh://gatehousesatcom@host) for push remotes and submodule URL remapping. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Scan the bare repo's git objects for blobs above the threshold before cloning to a temp dir and running git lfs migrate import. Repos with no large files skip the entire LFS step (clone + rewrite), saving minutes on large repos like wireshark (81k commits, 10 min wasted). https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Specified repos use --shallow-since on clone and fetch, keeping only
recent commits. HEAD commit hashes are preserved (no history rewrite).
Config:
trim_history:
UPSTREAM/linux: "2y"
UPSTREAM/git: "1y"
Supports Ny (years), Nm (months), Nd (days).
https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
git push --mirror fails on shallow clones because the pack references parent objects at the shallow boundary that don't exist. Trimmed repos now push branches (--all --force) and tags separately. Tags referencing pruned history are skipped with a warning. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Pushing all branches at once creates a pack that exceeds GitHub's 2GB limit for large repos like Linux kernel forks. Now pushes each branch individually so each pack is small enough. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
…rimming Shallow clones can't be pushed to GitHub (missing parent objects). Now uses git replace --graft to make cutoff commits into true root commits, then filter-branch to rewrite history permanently. Pushes cleanly with --mirror. Commit hashes change but content is preserved. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
git push --mirror creates a single pack for all refs. For Linux kernel forks with 100+ branches this exceeds GitHub's 2GB limit. Now detects the error and automatically retries by pushing each branch and tag individually. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Repos listed in push_by_branch go straight to branch-by-branch push, skipping the --mirror attempt that would fail with the 2GB pack limit. Saves time and bandwidth. Auto-fallback still works for unlisted repos. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Respects migrate_delay_seconds between each branch/tag push to avoid SSH rate limiting from GitHub. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Repos listed in push_by_branch now use branch-by-branch push in sync cycles too, with migrate_delay between each push. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Tags are lightweight refs — pushing them all at once doesn't hit the 2GB pack limit. Falls back to individual push if batch fails. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Allows individual repos to target a different GitHub org than their project default. Example: repos.thuraya-autotest.github_org: "other-org" https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
New commands to prepare workspaces for Jenkins-to-GitHub-Actions conversion by AI agents: - prepare-jenkins: Creates sparse-checkout workspaces with only Jenkinsfiles and dependencies, creates migration branch on GitHub - jenkins-manifest: Generates YAML manifest listing repos with Jenkinsfiles, clone URLs, and file paths for remote AI agents Features: - Auto-discovers Jenkinsfiles (any case, any location) - Finds dependencies: vars/, src/, resources/, colocated .groovy - Parses load/readFile references in Jenkinsfiles - Per-repo manifest + global manifest for remote agent use - State tracking to avoid re-preparing repos - Dry-run support https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Repos with no HEAD (empty/broken) now skip with a warning instead of crashing. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
1. project_aliases config maps old Bitbucket project keys to current keys, so .gitmodules with outdated URLs get remapped correctly. 2. sync.protected_branches prevents sync from deleting specified branches on GitHub (e.g., ci/github-actions-migration). When set, sync uses --all --force instead of --mirror, then prunes only unprotected branches that don't exist in Bitbucket. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
1. Auto-resolve renamed project keys via Bitbucket API — when a
.gitmodules URL uses an old project key, the tool queries
GET /rest/api/1.0/projects/{old_key} to get the current key.
Results are cached per remap cycle. Manual project_aliases config
still works as a fallback.
2. sync.protected_branches config prevents sync from deleting
specified branches on GitHub (e.g., ci/github-actions-migration).
When set, sync uses --all --force + selective pruning instead of
--mirror.
https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Uses fnmatch for pattern matching, so ci/* matches ci/github-actions, ci/migration, etc. Exact names still work (e.g., main). https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
The previous resolver only checked project renames. Now queries
GET /projects/{key}/repos/{slug} which returns the repo's CURRENT
project key — handling repos that were moved from one project to
another (e.g., 3rdparty_LIBCodegenix moved from SYS_BGANRAN to
SYS_COM). Results are cached per remap cycle.
https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Previously the API resolver was a fallback after config lookup succeeded with the old project key. Now resolves the actual repo location first, so moved repos (e.g., SYS_BGANRAN -> SYS_COM) get the correct GitHub org mapping. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Previously the cache was recreated per branch, causing repeated API calls for the same project/slug combos. Now a single cache is shared across all branches within a repo remap — so if a repo has 700 branches with the same submodules, each unique combo is resolved once. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Fixed timestamp was deterministic but showed as 27 years old in git log. Now uses the current date/time, fixed for the entire run so all branches in one repo get the same timestamp. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Hidden refs (refs/pull/*, refs/merge-request/*) were included in the snapshot but cleaned before push. Next cycle's show-ref didn't include them, causing a mismatch every time. Now cleans before snapshotting. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
show-ref includes refs/remotes/github/* which change every push cycle (remap commits have different hashes due to timestamps). Now uses show-ref --heads --tags to only compare Bitbucket's branches and tags, ignoring internal remote tracking refs. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
The LFS push had no timeout and would hang indefinitely scanning large repos. Now respects sync_lfs_timeout_seconds for both the migration and the push steps. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
--all scans every branch and tag history for LFS pointers, taking hours on repos with many refs even with only a few LFS files. Now enumerates actual LFS object files in lfs/objects/ and pushes each by OID directly — no scanning, instant for a handful of objects. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
--include-closed flag migrates merged/declined PRs by: 1. Finding the source branch's last commit SHA from Bitbucket PR data 2. If the commit exists in the repo, recreating the branch on GitHub 3. Creating a real PR with full discussion, then closing it 4. Fallback: if commit doesn't exist, creates a GitHub Issue instead Also adds --repo filter to migrate-prs command. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
When fromRef.latestCommit is garbage-collected (common after squash merges), the migrator now falls back to: 1. properties.mergeCommit from the Bitbucket PR object 2. The MERGED activity's commit field For squash/merge commits (which are already on the target branch), creates a temp base branch at the commit's parent so the GitHub PR shows the actual squash diff rather than "nothing to compare". Also adds BitbucketClient.get_pull_request() and get_merge_commit() methods, plus tests for the SHA resolution cascade. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
--closed-only skips open PRs entirely (for users who already migrated
open PRs and now want to backfill merged/declined ones).
Throttler class enforces:
- Minimum spacing between GitHub API calls (--api-delay, default 0.5s)
- Pause between full PR migrations (--pr-delay, default 3s)
- Automatic retry with exponential backoff on 429 / secondary rate
limits, honoring Retry-After header when present
Defaults are configurable via a new pr_migration section in config.yaml:
pr_migration:
api_delay_seconds: 0.5
pr_delay_seconds: 3.0
max_retries: 5
https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
Simpler and more useful than the branch-recreation approach: every closed PR (merged or declined) becomes a closed GitHub Issue with all its comments preserved. Labeled `migrated-pr` + `merged`/`declined` for filtering. Everything is searchable in one place — the issue tracker — instead of split between PRs and issues. Removes: - Branch recreation code (git subprocess, temp branches, push logic) - SHA resolution cascade (_resolve_head_sha, get_merge_commit) - BitbucketClient.get_pull_request, get_merge_commit - All dependency on the bare clone for closed PR migration Also fixes chronological comment ordering — comments were sorted by activity.createdDate but Bitbucket puts it on comment.createdDate. https://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary\n\n- Bulk migration (
bb2gh migrate): Clones all repos from Bitbucket Server via SSH, creates them on GitHub Enterprise, and pushes--mirror(all branches, tags, full history)\n- Continuous sync (bb2gh sync): Fetches from Bitbucket and pushes to GitHub every 60 seconds, enabling a soft cutover where teams can still merge PRs in Bitbucket during transition\n- PR migration (bb2gh migrate-prs): Recreates open Bitbucket PRs as real GitHub PRs with title, description, comments, and reviewer assignments\n\nIncludes Bitbucket Server REST API client with pagination, GitHub Enterprise client (PyGithub), JSON state tracking, Docker/docker-compose support, and 15 passing unit tests.\n\n## Test plan\n\n- [x] All 15 unit tests pass (python -m pytest tests/ -v)\n- [ ] Integration test: configure against a real Bitbucket Server instance and runbb2gh migrate\n- [ ] Verify continuous sync picks up new commits within 60s\n- [ ] Test PR migration with--dry-runfirst, then live\n- [ ] Test Docker deployment withdocker compose up -d sync\n\nhttps://claude.ai/code/session_01HK99UAxTgAiWTNYAfSoVvP