diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..d52cf22 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,57 @@ +## Testing & Development + +This repository uses [`nextest`](https://nexte.st/) to perform test +env spin-up and teardown. To install it, either see their website, or +run: + +``` +cargo install cargo-nextest --locked +``` + +Tests can then be run using: + +``` +cargo nextest run [--no-fail-fast] [-E 'test()'] +``` + +Note that tests need to be executed from the repository root since we +do not currently implement anything to find files required for tests +relative to it. + +In addition, a modern docker with the `compose` subcommand is +required - importantly, this is not true for many distro docker +packages. Firewalls also need to be configured to allow container <-> +container as well as container <-> host communication. + +### E2E test architecture + +For e2e testing, we need an ldap and a zitadel instance to test +against. These are spun up using docker compose. + +After ldap and zitadel have spun up, another docker container runs, +which simply executes a script to clean up existing test data and +create the necessary pre-setup organization structures to support a +test run. + +Tests are then expected to directly communicate with ldap/zitadel to +create/delete/modify users and confirm test results. + +Importantly, since the tests run with no teardown between individual +tests, users created *must* have different LDAP ID/email addresses, so +that they can be managed independently. + +E2E tests cannot run concurrently, since this would cause +synchronization to happen concurrently. + +For LDAP-over-TLS, openldap is configured to allow connections without +client certificates, but if one is provided, it must be verified +correctly. This allows us to test scenarios with and without client +certificates. + +## Contributing + +### Pre-commit usage + +1. If not installed, install with your package manager, or `pip install --user pre-commit` +2. Run `pre-commit autoupdate` to update the pre-commit config to use the newest template +3. Run `pre-commit install` to install the pre-commit hooks to your local environment diff --git a/README.md b/README.md index 6b952b9..b33c9c1 100644 --- a/README.md +++ b/README.md @@ -7,39 +7,107 @@ Currently supported sources: - CSV - Custom endpoint provided by UKT -## Configuration - -> [!WARNING] -> -> When creating a service user, limit them to the specific project and -> organization scope that they are intended to sync. `famedly-sync` -> currently does not separately limit the scope of the sync, see #103. - -The tool expects a configuration file located at `./config.yaml`. See example configuration files in [./sample-configs/](./sample-configs/). - -The default path can be changed by setting the new path to the environment variable `FAMEDLY_SYNC_CONFIG`. - -Also, individual configuration items and the whole configuration can be set using environment variables. For example, the following YAML configuration: - -```yaml -sources: - ldap: - url: ldap://localhost:1389 -``` - -Could be set using the following environment variable: - -```bash -FAMEDLY_SYNC__SOURCES__LDAP__URL="ldap://localhost:1389" -``` - -Note that the environment variable name always starts with the prefix `FAMEDLY_SYNC` followed by keys separated by double underscores (`__`). - -Some configuration items take a list of values. In this cases the values should be separated by space. **If an empty list is desired the variable shouldn't be created.** +## Deployment -Config can have **various sources** to sync from. When a source is configured, the sync tool tries to update users in Famedly's Zitadel instance based on the data obtained from the source. +Note that famedly-sync is currently not designed to be deployed to +projects with existing users, or projects that have been synced to +with a different source before. -**Feature flags** are optional and can be used to enable or disable certain features. +> [!CAUTION] +> +> Since famedly-sync relies on metadata to perform its tasks, if users +> that were not created by famedly-sync using the same source exist in +> the instance, they may be **deleted** by a sync, or cause desync +> issues if they happen to be users with elevated permissions. + +### LDAP + +To deploy famedly-sync to sync an LDAP server to a Zitadel instance, +the following need to be collected first: + +- LDAP instance details + - URL of the LDAP/AD server + - Bind DN (=~username of the admin user) with which to authenticate + - Bind password with which to authenticate + - Base DN under which the users that should be synced are available + - An LDAP user filter to select specifically the users to sync + - The mapping of LDAP attributes to Zitadel attributes + - This may differ between instances, hence it's configurable. It + is often quite straightforward from listing the users with + `ldapsearch`, but more on this later + - TLS certificates, if TLS is used and mTLS is required and/or the + server certificate is not in the root store +- Zitadel instance details + - A service user must be created. + - The [Zitadel + documentation](https://zitadel.com/docs/guides/integrate/service-users/private-key-jwt#steps-to-authenticate-a-service-user-with-private-jwt) + covers creating such a user. + - The user should be created with *only* the `Iam User Manager` + role for the organization that should be synced to. + - The relevant Project and Org IDs + - These are the "Resource Ids" listed on their pages in the + Zitadel UI + - SSO settings + - When using SSO, an IDP ID needs to be defined. IDP configuration + is located in the Zitadel instance's default settings - the ID + of a specific IDP can be glanced from its URL. + - Currently, this is not optional, see [this + issue](https://github.com/famedly/famedly-sync/issues/102). For + now, setting the ID to `000000000000000000` when not needed is a + reasonable workaround. + +Once this has been gathered, the following steps are advised: + +1. Asserting correctness of the LDAP configuration + + This can be done using `ldapsearch`, which is part `openldap` (on + ubuntu hosts, in the `ldap-utils` package). The basic search command + can be translated from the previously acquired data to: + + ``` + ldapsearch -H "$LDAP_URL" -D "$LDAP_BIND_DN" -w "$LDAP_BIND_PASSWORD" -b "$LDAP_BASE_DN" "$LDAP_USER_FILTER" + ``` + + If this does not return all user data required for the attribute + mapping, also add the expected attributes to the end of the command. + +2. Writing the famedly-sync configuration + + To prevent this document going out of date, we will not repeat the + configuration syntax here. See + [./sample-configs/ldap-config.sample.yaml](./sample-configs/ldap-config.sample.yaml) + for details. + + Noteworthy is that the `use_attribute_filter` config setting + *should* be set to true if, and only if, attributes needed to be + added to the `ldapsearch` command. + +3. Deploying the docker container + + A relatively fool-proof way of doing this is with `docker + compose`. This composefile, and a copy of the service user's JWT + and the just-created configuration in `./opt/`, is all that should + be required: + + ```yaml + services: + famedly-sync-agent: + image: docker-oss.nexus.famedly.de/famedly-sync-agent: + volumes: + - type: bind + source: ./opt + target: /opt/famedly-sync + network_mode: host + ``` + + Assuming a recent-enough docker version, this can then be invoked + with `docker compose up`. + + Since the `./opt` directory will be bind-mounted to + `/opt/famedly-sync` in the docker container (and docker containers + do not permit access to host files by other means), any files + referenced in the configuration *should* be relative to prevent + file access issues. ## Migrations @@ -61,111 +129,42 @@ base64-encoded one. No other values should change during migration. A Zitadel instance that is already using hex-encoded IDs will not be altered (though famedly-sync will still make various HTTP calls). -## Testing & Development - -This repository uses [`nextest`](https://nexte.st/) to perform test -env spin-up and teardown. To install it, either see their website, or -run: +## Kubernetes Deployment -``` -cargo install cargo-nextest --locked -``` - -Tests can then be run using: +The provided manifest `ldap-sync-cronjob.yaml` can be used +to deploy this tool as a CronJob on a Kubernetes cluster. ``` -cargo nextest run [--no-fail-fast] [-E 'test()'] +kubectl create -f ldap-sync-deployment.yaml ``` -Note that tests need to be executed from the repository root since we -do not currently implement anything to find files required for tests -relative to it. - -In addition, a modern docker with the `compose` subcommand is -required - importantly, this is not true for many distro docker -packages. Firewalls also need to be configured to allow container <-> -container as well as container <-> host communication. - -### E2E test architecture - -For e2e testing, we need an ldap and a zitadel instance to test -against. These are spun up using docker compose. - -After ldap and zitadel have spun up, another docker container runs, -which simply executes a script to clean up existing test data and -create the necessary pre-setup organization structures to support a -test run. - -Tests are then expected to directly communicate with ldap/zitadel to -create/delete/modify users and confirm test results. - -Importantly, since the tests run with no teardown between individual -tests, users created *must* have different LDAP ID/email addresses, so -that they can be managed independently. - -E2E tests cannot run concurrently, since this would cause -synchronization to happen concurrently. - -For LDAP-over-TLS, openldap is configured to allow connections without -client certificates, but if one is provided, it must be verified -correctly. This allows us to test scenarios with and without client -certificates. - -## Contributing - -### Pre-commit usage - -1. If not installed, install with your package manager, or `pip install --user pre-commit` -2. Run `pre-commit autoupdate` to update the pre-commit config to use the newest template -3. Run `pre-commit install` to install the pre-commit hooks to your local environment - -## Deployment - -The easiest way to deploy this tool is using our published docker -container through our [composefile](./docker-compose.yaml). - -To prepare for use, we need to provide a handful of files in an `opt` -directory in the directory where `docker compose` will be -executed. This is the expected directory structure of the sample -configuration file: +It will run `docker-oss.nexus.famedly.de/famedly-sync-agent:v0.4.0` once per day +at 00:00 in the namespace `ldap-sync`. It requires a ConfigMap named `famedly-sync` +to be present in the `ldap-sync` namespace. The ConfigMap can be created using ``` -opt -├── certs -│ └── test-ldap.crt # The TLS certificate of the LDAP server -├── config.yaml # Example configs can be found in [./sample-configs](./sample-configs) -└── service-user.json # Provided by famedly +kubectl create configmap --from-file config.yaml famedly-sync --namespace ldap-sync ``` -Once this is in place, the container can be executed in the parent -directory of `opt` with: +### Configuration through env variables -``` -docker compose up -``` +Since kubernetes deployments prefer env-based configuration, famedly-sync supports this. -Or alternatively, without `docker compose`: +Individual configuration items and the whole configuration can be set using environment variables. For example, the following YAML configuration: +```yaml +sources: + ldap: + url: ldap://localhost:1389 ``` -docker run --rm -it --network host --volume ./opt:/opt/famedly-sync docker-oss.nexus.famedly.de/famedly-sync-agent:latest -``` - -### Kubernetes Deployment -The provided manifest `ldap-sync-cronjob.yaml` can be used -to deploy this tool as a CronJob on a Kubernetes cluster. +Could be set using the following environment variable: -``` -kubectl create -f ldap-sync-deployment.yaml +```bash +FAMEDLY_SYNC__SOURCES__LDAP__URL="ldap://localhost:1389" ``` -It will run `docker-oss.nexus.famedly.de/famedly-sync-agent:v0.4.0` once per day -at 00:00 in the namespace `ldap-sync`. It requires a ConfigMap named `famedly-sync` -to be present in the `ldap-sync` namespace. The ConfigMap can be created using - -``` -kubectl create configmap --from-file config.yaml famedly-sync --namespace ldap-sync -``` +Some configuration items take a list of values. In this cases the values should be separated by space. **If an empty list is desired the variable shouldn't be created.** ## Quirks & Edge Cases