Fix main#41
Open
leborchuk wants to merge 14 commits into
Open
Conversation
This import old CI job from open-gpdb/gpdb + yezzey. This merely checks that yezzey and cloudberry compiles together with no issues. As discussed privately we will redesign it later in another PR. Right now we copy existing code from open-gpdb in order to make sure all out future PR is Ok
* Allow group access for init CBDB * Allow group access for segments CBDB --------- Co-authored-by: Leonid Borchuk <xifos@qavm-f9b691f5.qemu>
Co-authored-by: Leonid Borchuk <xifos@qavm-f9b691f5.qemu>
* Add yezzey build option * Move yezey to commit 4c6b5b8 --------- Co-authored-by: Leonid Borchuk <xifos@qavm-f9b691f5.qemu>
When expanding a cluster, gpexpand copies the postgresql.conf file directly from the template segment (content 0). This causes issues for tools like wal-g which use a --content-id flag in archive_command and restore_command. Previously, new segments inherited --content-id=0 from the template. This caused them to push WAL segments to the wrong location, potentially overwriting segment 0's segments. This fix ensures the content ID in archive_command and restore_command is updated to match the new segment's ID during expansion. If the commands do not contain the --content-id flag, they remain unchanged.
To the MWP cbdb version
* Move yezzey forward to full support Cloudberry
Historically Yandex Greenplum allows non-superuser no managed resource groups. So, a regular non-superuser role allowed to run pg_resgroup_move_query(), and tune CPU/memory limits if granted with mdb_admin. Such feature was introduced as early as 6.22, see also gpdb commit 3ac99962. This commit introduces same feature for managed Cloudberry. To disallow altering predefined roles, fixed-OID hardening is used, reserving 8067 OID to be an mdb_admin role OID. We choose this (efficiently a catalog change) over complex bookkeeping what CREATEROLE can do and what is disallowed. We use Yandex managed predefined roles bootstrap util via auxiliary contrib extension, based on what Yandex Postgres fork does, see also pg-sharding/cpg repo. Co-authored-by: Andrey Borodin <x4mmm@yandex-team.ru> Co-authored-by: reshke <reshke@double.cloud>
There was a problem hiding this comment.
Pull request overview
This PR aggregates several fixes/features aimed at improving Cloudberry’s managed-service operability and extension/build ergonomics, including (1) a fixed-OID mdb_admin privilege role to allow non-superusers to manage resource groups, and (2) gpexpand-related correctness around wal-g --content-id propagation, plus Yezzey build/CI wiring.
Changes:
- Introduces
pg_aux_catalogextension to create a fixed-OIDmdb_adminrole, adds auxiliary OID-range support, and gates resource-group operations onmdb_adminmembership (with superuser-only exceptions for system groups). - Updates new-segment configuration to rewrite wal-g
--content-idinarchive_command/restore_command, and adds an isolation2 test + schedule entry to validate behavior. - Adds Yezzey build option/submodule and a dedicated CI workflow; expands build plumbing for optional contrib components.
Reviewed changes
Copilot reviewed 32 out of 35 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| src/test/isolation2/sql/gpexpand_archive_restore_conf.sql | Adds isolation2 SQL to validate wal-g --content-id rewrite on expansion. |
| src/test/isolation2/isolation2_expandshrink_schedule | Schedules the new gpexpand wal-g config test. |
| src/test/isolation2/expected/gpexpand_archive_restore_conf.out | Expected output for the new isolation2 test. |
| src/Makefile.global.in | Adds with_* knobs for optional contrib builds (diskquota/gp_stats_collector/yezzey). |
| src/include/utils/acl.h | Adds fixed OID constant for mdb_admin. |
| src/include/catalog/oid_dispatch.h | Exposes next_aux_pg_authid_oid for fixed role OID assignment. |
| src/include/access/transam.h | Defines auxiliary OID range (FirstAuxObjectId..LastAuxObjectId) and IsAuxOid. |
| src/backend/utils/resgroup/resgroup_helper.c | Changes pg_resgroup_move_query() privilege gate to mdb_admin. |
| src/backend/commands/resgroupcmds.c | Updates resource-group DDL privilege checks to require mdb_admin (superuser still passes via is_member_of_role). |
| src/backend/catalog/oid_dispatch.c | Adds fixed/preassigned OID support for auxiliary roles via next_aux_pg_authid_oid. |
| pom.xml | Adds Apache RAT excludes for new pg_aux_catalog files. |
| gpMgmt/bin/lib/gpcreateseg.sh | Passes --allow-group-access when initializing new segments. |
| gpMgmt/bin/lib/gpconfigurenewsegment | Rewrites wal-g --content-id in new segments’ archive/restore commands. |
| gpMgmt/bin/gpinitsystem | Passes --allow-group-access when initializing coordinator data dir. |
| gpcontrib/Makefile | Adds optional recurse targets for gp_stats_collector/yezzey; attempts diskquota gating. |
| contrib/pg_aux_catalog/sql/pg_aux_catalog.sql | Regression test for pg_aux_catalog role creation and permission gate. |
| contrib/pg_aux_catalog/README.md | Documents the extension, fixed OID rationale, and testing. |
| contrib/pg_aux_catalog/pg_aux_catalog.control | Extension control file. |
| contrib/pg_aux_catalog/pg_aux_catalog.c | Implements pg_create_mdb_admin_role() using fixed OID dispatch. |
| contrib/pg_aux_catalog/pg_aux_catalog--1.0.sql | Extension SQL installing the C function. |
| contrib/pg_aux_catalog/Makefile | Build + regression targets; adds installcheck-isolation2 target. |
| contrib/pg_aux_catalog/isolation2/sql/resgroup_mdb_admin.sql | Multi-session isolation2 test for mdb_admin-gated resgroup operations. |
| contrib/pg_aux_catalog/isolation2/isolation2_schedule | Schedule for the pg_aux_catalog isolation2 test. |
| contrib/pg_aux_catalog/isolation2/expected/resgroup_mdb_admin.out | Expected output for the pg_aux_catalog isolation2 test. |
| contrib/pg_aux_catalog/expected/pg_aux_catalog.out | Expected output for the regression test. |
| contrib/pg_aux_catalog/.gitignore | Ignores generated test artifacts for the new contrib module. |
| contrib/Makefile | Adds pg_aux_catalog to contrib build. |
| configure.ac | Adds --with-yezzey (and should define other with_* knobs too). |
| configure | Adds CLI vars/handlers for some new --with-* options (generated file). |
| .gitmodules | Adds Yezzey as a submodule under gpcontrib. |
| .github/workflows/yezzey-ci.yaml | Adds Yezzey CI pipeline (Cloudberry build + Yezzey build + MinIO + tests). |
| .github/workflows/build-deb-cloudberry.yml | Adds contrib/pg_aux_catalog:installcheck to CI test matrix. |
| .github/workflows/build-cloudberry.yml | Adds contrib/pg_aux_catalog:installcheck to CI test matrix. |
| .github/workflows/build-cloudberry-rocky8.yml | Adds contrib/pg_aux_catalog:installcheck to CI test matrix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+609
to
+613
| if (OidIsValid(next_aux_pg_authid_oid)) | ||
| { | ||
| if (!IsAuxOid(next_aux_pg_authid_oid)) | ||
| elog(ERROR, "pre-assigned auxiliary role OID %u is out of the auxiliary OID range", | ||
| next_aux_pg_authid_oid); |
Comment on lines
+1376
to
+1382
| # | ||
| # yezzey | ||
| # | ||
| PGAC_ARG_BOOL(with, yezzey, no, | ||
| [build with Yezzey extension]) | ||
| AC_SUBST(with_yezzey) | ||
|
|
Comment on lines
+38
to
+40
| ifeq "$(with_diskquota)" "yes" | ||
| recurse_targets += diskquota | ||
| endif |
Comment on lines
+23
to
+36
| -- --------------------------------------------------------------------- | ||
| -- Setup. The mdb_admin role is not predefined in the catalog; it is | ||
| -- created here the same way the control plane provisions it at runtime. | ||
| -- --------------------------------------------------------------------- | ||
| CREATE RESOURCE GROUP rg_perm_test WITH (concurrency=2, cpu_max_percent=10); | ||
| CREATE | ||
| CREATE ROLE mdb_admin; | ||
| CREATE | ||
| CREATE ROLE role_rg_admin RESOURCE GROUP rg_perm_test; | ||
| CREATE | ||
| CREATE ROLE role_rg_noadmin RESOURCE GROUP rg_perm_test; | ||
| CREATE | ||
| GRANT mdb_admin TO role_rg_admin; | ||
| GRANT |
Comment on lines
+1
to
+9
| /*------------------------------------------------------------------------- | ||
| * | ||
| * pg_aux_catalog.c | ||
| * Extension for auxiliary catalog management | ||
| * | ||
| * contrib/pg_aux_catalog/pg_aux_catalog.c | ||
| * | ||
| *------------------------------------------------------------------------- | ||
| */ |
| @@ -0,0 +1 @@ | |||
| test: resgroup_mdb_admin | |||
Comment on lines
+355
to
+357
| <exclude>contrib/pg_aux_catalog/pg_aux_catalog.c</exclude> | ||
| <exclude>contrib/pg_aux_catalog/isolation2/isolation2_schedule</exclude> | ||
|
|
…icense headers, expected output
… outputs - privileges.out: update expected output for terminate_nothrow test to show 4 background processes (autovacuum launcher, dtx recovery process, logical replication launcher, login monitor) instead of 0 rows, matching Cloudberry's actual pg_terminate_backend behavior - output/misc.source: change expected value from 't' to 'f' for mdb_locale_enabled() since CI builds without --with-mdblocales (ENABLE_MDBLOCALES defaults to false in configure-cloudberry.sh)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I am going to rebase main and amend all our fixes here not merged to cloudberry
So moved them to the REL_2_STABLE to not to forget them merge to original main after some time.
The list of fixes:
Yezzey test
Add group access to CBDB
Add yezzey as submodule
UseAnonymousAddress
Add yezzey build option
Fix gpexpand not changing content-id for wal-g cmd
Bump yezzey
An extra opts to configure yezzey
Move yezzey forward to full support Cloudberry
Bump yezzey for PAX fix
Fix for CDBD expand with concurrent distributed tx
Allow non-superuser role to manage resource groups.