Adds path stripping to Azure Blob destination, adds hostname to MySQLConfig#467
Adds path stripping to Azure Blob destination, adds hostname to MySQLConfig#467Jsalz2000 wants to merge 19 commits intotwindb:masterfrom
Conversation
|
This PR plans to fix this issue [Question] Hard-Coding --host=127.0.01 in mysql_source.py · Issue #184 · twindb/backup If you would rather split these changes into two separate PRs, please let me know. |
|
I would also like to discuss adding support for building on enterprise linux (in general) 7, 8 and 9. I created these branches in a personal repo that I have been building our images with. These docker images use Oracle Linux as a base, it's frequently updated and JustWorks. We could possibly create another repo in twindb, or look at merging with twindb/omnibus-centos |
|
@akuzminsky - Could I please get a review/approve for this PR? |
ernstae
left a comment
ernstae
left a comment
There was a problem hiding this comment.
I've reviewed this code internally, and we are running this successfully against Azure Blob storage. Looking to get this merged in, and a further story on RPM and other package builds. We should consider whether there's a path forward for Oracle Linux, Rocky Linux or Alma Linux build processes to get new RPMs out into the world (or at least in the Releases artifacts of this repo).
Cheers!
|
@akuzminsky - bumping this thread once more.. We have been running this version in production for almost a year now without any issues. |
Introduces auth_mode=managed_identity for the Azure Blob destination so
the backup process can obtain an access token from the VM's managed
identity instead of using connection strings or SAS tokens.
- Accepts new [az] options:
* auth_mode: connection_string | managed_identity
(default: connection_string; preserves existing behavior)
* account_url: required when auth_mode=managed_identity
* managed_identity_client_id: optional, for a user-assigned identity
* managed_identity_resource_id: optional, for a user-assigned identity
selected by its ARM resource ID
- ManagedIdentityCredential is used when client_id or resource_id is
set; DefaultAzureCredential is used otherwise (system-assigned MI or
local development via az login / environment credentials).
- client_id and resource_id are mutually exclusive and validated at
config load time.
- Adds unit coverage for the new field, the mutual-exclusivity rule,
and the three credential code paths (resource_id, client_id, neither).
- Updates docs/usage.rst and the example twindb-backup.cfg files.
- Adds docs/azure_worm_compatibility.rst capturing the follow-on work
required for Azure immutable/WORM containers, which is out of scope
for this change.
Covers the new Azure managed-identity auth_mode and the managed_identity_resource_id field added in the preceding commit.
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| Documentation | 6 minor |
| ErrorProne | 7 high |
| Security | 76 high |
| CodeStyle | 8 minor |
| Complexity | 3 medium |
🟢 Metrics 421 complexity · 16 duplication
Metric Results Complexity 421 Duplication 16
TIP This summary will be updated as you push new changes. Give us feedback
Introduces a new ``[source] server_name`` config option that overrides the per-source segment of the remote backup path (previously hardcoded to ``socket.gethostname()``). When multiple replicas of a cluster back up to the same destination, pointing every replica at the same ``server_name`` collapses the backups into a single shared tree instead of one fan-out per hostname. On Azure Blob destinations, ``BaseDestination.cluster_lock`` is also overridden to acquire an Azure blob lease before running the backup. Only one replica holding the lease proceeds; the rest skip the run cleanly. A background thread renews the lease for the duration of the backup so long-running xtrabackup streams stay protected within the Azure 60s lease maximum. On other destinations ``cluster_lock`` is a no-op and always reports the lock as acquired, preserving existing behaviour. The ``server_name`` value also drives the status file directory and the ``--hostname`` defaults in the ``status`` and ``verify mysql`` CLI commands so listing and verification use the same namespacing. Adds unit tests covering the new config parsing, the ``BaseSource`` override path, and the Azure lease behaviour (happy path, existing blob, ``LeaseAlreadyPresent`` skip, transient-error fail-open, TTL clamping, release-on-exception).
2 participants
This pull request includes a version update for TwinDB Backup from 3.3.0 to 3.4.0, along with several other enhancements and fixes. The most important changes include updating version references, adding a new hostname configuration option for MySQL, and improving the handling of remote paths in Azure Blob Storage.
Version Update:
3.3.0to3.4.0inREADME.rst,docs/installation.rst,omnibus/config/projects/twindb-backup.rb,setup.cfg,setup.py, andtwindb_backup/__init__.py. [1] [2] [3] [4] [5] [6] [7]MySQL Configuration Enhancements:
hostnameoption to MySQL configuration with a default value of127.0.0.1indocs/usage.rst,support/twindb-backup.cfg,twindb_backup/configuration/mysql.py, andvagrant/environment/puppet/modules/profile/files/twindb-backup.cfg. [1] [2] [3] [4] [5]backup_mysqlandbackup_binlogsfunctions to use the newhostnameconfiguration intwindb_backup/backup.py. [1] [2] [3] [4]Azure Blob Storage Improvements:
twindb_backup/destination/az.py,tests/unit/destination/az/test_init.py,tests/unit/destination/az/test_list_files.py, andtests/unit/destination/az/util.py. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]Unit Tests:
tests/unit/configuration/test_mysql.py.tests/unit/destination/az/test_list_files.py.Miscellaneous:
get_streammethod intwindb_backup/source/mysql_source.pyto use the newhostnameconfiguration.