Fix index.db namespace collisions and support divergent-tag builds

AGENTS.md

@@ -57,3 +57,4 @@ Key modules:
 - **Never edit existing Alembic migrations** — generate a new revision instead.
 - **API ↔ Worker task signatures must match** — renaming args silently breaks in-flight requests.
 - **Avoid making changes to the old worker architecture** - Always ask before making changes to the following functions - `handle_add_request`, `handle_rm_request`, `handle_merge_request`, `handle_create_empty_index_request`, `handle_fbc_operation_request`, `handle_regenerate_bundle_request`
+- **Divergent-tag requests never merge their MR** — they reuse the base OCP branch's Konflux Component and must stay throw-away; the guard is `overwrite_from_index and not sources.is_divergent`. Never source their index.db from ORAS.

docker/containerized/README.md

@@ -253,6 +253,39 @@ The worker configuration is in `docker/containerized/worker_config.py`. This fil
 - Includes the containerized task modules
 - Validates required configuration on startup
 
+## Git Branch and Tag Semantics
+
+The Git branch used for a request's catalog is keyed on the **image tag**, not a fixed OCP-version mapping. For the existing fleet this is a no-op, since prod index tags already equal the OCP version (e.g. `v4.19` → branch `v4.19`).
+
+### Onboarding a Non-OCP Tag
+
+To onboard a tag that isn't an OCP version (e.g. a custom or pre-release tag), a maintainer must:
+
+1. Create a Git branch in the catalog repository named exactly after the tag.
+2. Provision a Konflux Component for that branch, so pushes to it trigger a PipelineRun.
+
+Once both exist, requests against that tag build and push normally, and `overwrite_from_index` is honored like any other branch.
+
+### Divergent Tags (No Matching Branch)
+
+If a request targets a tag with no corresponding Git branch — for example a timestamped, point-in-time tag cut from an existing index — IIB treats it as a **divergent tag**:
+
+- Configs and `index.db` are extracted directly from the index image (unprivileged), not sourced from ORAS. Only the FBC configs and the hidden `index.db` are extracted; if the image carries no hidden `index.db`, the request fails ("no index.db found, onboard the image to build") — there is no labeled-db or empty-db fallback.
+- The build reuses the base OCP branch's existing Konflux Component via a throw-away merge request.
+- That MR is **never merged** — it is always closed after the pipeline completes (or on failure), regardless of outcome.
+- `overwrite_from_index` is **rejected** for divergent-tag requests; there is no direct-push path.
+
+This lets IIB build and validate a one-off tag without requiring per-tag branch/Component provisioning.
+
+## Index DB Artifact and ImageStream Tag Naming
+
+Cached `index.db` artifact tags (ORAS) and ImageStream tags are keyed on the index image's content (manifest) digest — `idb-<sha256>` — resolved via `skopeo inspect`, not on its pullspec. Because the key is the content itself, it is both namespace-safe and promotion-safe:
+
+- **Namespace-safe:** two images that share a repository name in different registry namespaces (e.g. `quay.io/redhat/my-index:v4.17` vs `quay.io/redhat-pending/my-index:v4.17`) have different content and therefore different digests, so they never collide on the same cache tag.
+- **Promotion-safe:** the same image content addressed by different pullspecs after a release or mirror (e.g. `quay.io/my-namespace/iib-pub:v4.17` → `registry.access.redhat.com/some-namespace/operator-index:v4.17`) preserves its manifest digest, so both pullspecs resolve to the *same* cache entry and share one `index.db`.
+
+Cache entries written under the previous pullspec-derived naming scheme are orphaned by this change — they are not migrated in place. On the normal path IIB never falls back to extracting `index.db` from the image: if the digest-keyed artifact is missing, the request fails with a "no index.db found for the image, onboard the image to build" error, and the image must be onboarded (which populates the artifact) before it can be built. Orphaned entries are cleaned up by the existing cache-pruning process rather than any code path in this workflow.
+
 ## Differences from Traditional Workflow
 
 | Aspect | Traditional Workflow | Containerized Workflow |

iib/exceptions.py

@@ -13,6 +13,15 @@ class IIBError(BaseException):
     """An error was encountered in IIB."""
 
 
+class FileNotFoundInImageError(IIBError):
+    """A requested path was not present in a container image.
+
+    Subclasses IIBError so existing ``except IIBError`` handlers still catch it,
+    while letting callers distinguish a genuinely absent path from a real
+    extraction failure (registry, OCI parsing, layer, or tar error).
+    """
+
+
 class ValidationError(BaseException):
     """Denote invalid input."""
 

iib/workers/config.py

@@ -71,7 +71,7 @@ class Config(object):
     iib_index_db_oras_auth_path: str = os.path.join(
         os.path.expanduser('~'), '.docker', 'oras', 'config.json'
     )
-    iib_index_db_artifact_tag_template: str = '{image_name}-{tag}'
+    iib_index_db_artifact_tag_template: str = 'idb-{digest}'
     iib_index_db_artifact_template: str = '{registry}/index-db:{tag}'
     # Whether to use OpenShift ImageStream cache for index.db artifacts
     # Requires OpenShift cluster with ImageStream configured

iib/workers/tasks/build_containerized_add.py

@@ -20,7 +20,7 @@
 )
 from iib.workers.tasks.celery import app
 from iib.workers.tasks.containerized_utils import (
-    prepare_git_repository_for_build,
+    prepare_build_sources,
     fetch_and_verify_index_db_artifact,
     write_build_metadata,
     git_commit_and_create_mr,
@@ -148,38 +148,41 @@ def handle_containerized_add_request(
     distribution_scope = prebuild_info['distribution_scope']
 
     index_to_gitlab_push_map = index_to_gitlab_push_map or {}
-    # Variables mr_details, last_commit_sha and original_index_db_digest
-    # needs to be assigned; otherwise cleanup_on_failure() fails when an exception is raised.
+    # Variables mr_details and last_commit_sha need to be assigned; otherwise
+    # cleanup_on_failure() fails when an exception is raised.
     mr_details: Optional[Dict[str, str]] = None
     last_commit_sha: Optional[str] = None
-    original_index_db_digest: Optional[str] = None
 
     Opm.set_opm_version(from_index_resolved)
 
     _update_index_image_build_state(request_id, prebuild_info)
     present_bundles: List[BundleImage] = []
     present_bundles_pull_spec: List[str] = []
     with tempfile.TemporaryDirectory(prefix=f'iib-{request_id}-') as temp_dir:
-        branch = prebuild_info['ocp_version']
-
-        # Set up and clone Git repository
-        (
-            index_git_repo,
-            local_git_repo_path,
-            localized_git_catalog_path,
-        ) = prepare_git_repository_for_build(
+        sources = prepare_build_sources(
             request_id=request_id,
             from_index=str(from_index),
+            from_index_resolved=from_index_resolved,
             temp_dir=temp_dir,
-            branch=branch,
+            ocp_version=prebuild_info['ocp_version'],
             index_to_gitlab_push_map=index_to_gitlab_push_map,
+            overwrite_from_index=overwrite_from_index,
         )
-
-        # Pull index.db artifact (uses ImageStream cache if configured, otherwise pulls directly)
-        artifact_index_db_file = fetch_and_verify_index_db_artifact(
-            from_index=str(from_index),
-            temp_dir=temp_dir,
-        )
+        index_git_repo = sources.index_git_repo
+        local_git_repo_path = sources.local_git_repo_path
+        localized_git_catalog_path = sources.localized_git_catalog_path
+        branch = sources.target_branch
+
+        # Divergent path already has index.db extracted from the image; the normal
+        # path pulls it from ORAS. NEVER fall back to ORAS on the divergent path —
+        # that would read the base OCP branch's index.db.
+        if sources.index_db_path is not None:
+            artifact_index_db_file = sources.index_db_path
+        else:
+            artifact_index_db_file = fetch_and_verify_index_db_artifact(
+                from_index=str(from_index),
+                temp_dir=temp_dir,
+            )
 
         msg = 'Checking if bundles are already present in index image'
         log.info(msg)
@@ -330,20 +333,21 @@ def handle_containerized_add_request(
 
             # Push updated index.db before merging the MR so that on failure both
             # git and the index.db artifact remain consistent (MR stays open,
-            # cleanup_on_failure closes it and rolls back the artifact).
-            original_index_db_digest = push_index_db_artifact(
+            # cleanup_on_failure closes it; the content-addressed artifact needs no rollback).
+            push_index_db_artifact(
                 request_id=request_id,
                 from_index=str(from_index),
                 index_db_path=artifact_index_db_file,
                 operators=operators,
+                output_image=image_url,
                 overwrite_from_index=overwrite_from_index,
                 request_type='add',
             )
 
             # Merge or close the MR as the final step so that all side effects
             # (replication, metadata, index.db push) have succeeded before git
             # is advanced. This prevents git/index.db divergence on partial failure.
-            if overwrite_from_index:
+            if overwrite_from_index and not sources.is_divergent:
                 merge_mr_after_build(mr_details, index_git_repo)
                 # Prevent cleanup_on_failure from trying to close an already-merged MR
                 mr_details = None
@@ -364,7 +368,6 @@ def handle_containerized_add_request(
                 request_id=request_id,
                 from_index=str(from_index),
                 index_repo_map=index_to_gitlab_push_map or {},
-                original_index_db_digest=original_index_db_digest,
                 reason=f"error: {e}",
             )
             raise IIBError(f"Failed to add bundles: {e}")

iib/workers/tasks/build_containerized_create_empty_index.py

@@ -34,7 +34,6 @@
     opm_validate,
 )
 from iib.workers.tasks.oras_utils import (
-    _get_artifact_combined_tag,
     _get_name_and_tag_from_pullspec,
     get_oras_artifact,
 )
@@ -176,7 +175,6 @@ def handle_containerized_create_empty_index_request(
     index_git_repo: Optional[str] = None
     last_commit_sha: Optional[str] = None
     output_pull_spec: Optional[str] = None
-    original_index_db_digest: Optional[str] = None
 
     with tempfile.TemporaryDirectory(prefix=f'iib-{request_id}-') as temp_dir:
         branch = ocp_version
@@ -199,11 +197,16 @@ def handle_containerized_create_empty_index_request(
         conf = get_worker_config()
         empty_tag = conf.get('iib_empty_index_db_tag', 'empty')
 
-        # Construct the pullspec for the empty index.db artifact
+        # Construct the pullspec for the empty index.db artifact.
+        # This tag intentionally does NOT use the content-digest key that
+        # _get_artifact_combined_tag derives for real indexes: the empty index.db
+        # is a shared, content-free seed artifact keyed only by image name +
+        # "empty", so it is deliberately reusable across indexes rather than tied
+        # to any single image's manifest digest.
         image_name, _ = _get_name_and_tag_from_pullspec(from_index)
         empty_artifact_ref = conf['iib_index_db_artifact_template'].format(
             registry=conf['iib_index_db_artifact_registry'],
-            tag=_get_artifact_combined_tag(image_name, empty_tag),
+            tag=f"{image_name}-{empty_tag}",
         )
 
         log.info('Fetching empty index.db from %s', empty_artifact_ref)
@@ -344,12 +347,13 @@ def handle_containerized_create_empty_index_request(
 
             # Push the empty index.db with request ID tag
             # Since overwrite_from_index is False, this will only push with request_id tag
-            # and will not overwrite the v4.x tag
-            original_index_db_digest = push_index_db_artifact(
+            # and will not overwrite the current artifact
+            push_index_db_artifact(
                 request_id=request_id,
                 from_index=from_index,
                 index_db_path=str(index_db_path),
                 operators=[],  # Empty list since we're creating an empty index
+                output_image=image_url,
                 overwrite_from_index=False,  # Always False for create_empty_index
                 request_type='create_empty_index',
             )
@@ -372,7 +376,6 @@ def handle_containerized_create_empty_index_request(
                 request_id=request_id,
                 from_index=from_index,
                 index_repo_map=index_to_gitlab_push_map or {},
-                original_index_db_digest=original_index_db_digest,
                 reason=f"error: {e}",
             )
             raise IIBError(f"Failed to create empty index: {e}")

iib/workers/tasks/build_containerized_fbc_operations.py

@@ -18,7 +18,7 @@
     fetch_and_verify_index_db_artifact,
     git_commit_and_create_mr,
     monitor_pipeline_and_extract_image,
-    prepare_git_repository_for_build,
+    prepare_build_sources,
     push_index_db_artifact,
     replicate_image_to_tagged_destinations,
     write_build_metadata,
@@ -112,11 +112,10 @@ def handle_containerized_fbc_operation_request(
     distribution_scope = prebuild_info['distribution_scope']
 
     index_to_gitlab_push_map = index_to_gitlab_push_map or {}
-    # Variables mr_details, last_commit_sha and original_index_db_digest
-    # needs to be assigned; otherwise cleanup_on_failure() fails when an exception is raised.
+    # Variables mr_details and last_commit_sha need to be assigned; otherwise
+    # cleanup_on_failure() fails when an exception is raised.
     mr_details: Optional[Dict[str, str]] = None
     last_commit_sha: Optional[str] = None
-    original_index_db_digest: Optional[str] = None
 
     Opm.set_opm_version(from_index_resolved)
 
@@ -131,26 +130,30 @@ def handle_containerized_fbc_operation_request(
     _update_index_image_build_state(request_id, prebuild_info)
 
     with tempfile.TemporaryDirectory(prefix=f'iib-{request_id}-') as temp_dir:
-        branch = prebuild_info['ocp_version']
-
-        # Set up and clone Git repository
-        (
-            index_git_repo,
-            local_git_repo_path,
-            localized_git_catalog_path,
-        ) = prepare_git_repository_for_build(
+        sources = prepare_build_sources(
             request_id=request_id,
             from_index=from_index,
+            from_index_resolved=from_index_resolved,
             temp_dir=temp_dir,
-            branch=branch,
+            ocp_version=prebuild_info['ocp_version'],
             index_to_gitlab_push_map=index_to_gitlab_push_map,
+            overwrite_from_index=overwrite_from_index,
         )
+        index_git_repo = sources.index_git_repo
+        local_git_repo_path = sources.local_git_repo_path
+        localized_git_catalog_path = sources.localized_git_catalog_path
+        branch = sources.target_branch
 
-        # Pull index.db artifact (uses ImageStream cache if configured, otherwise pulls directly)
-        artifact_index_db_file = fetch_and_verify_index_db_artifact(
-            from_index=from_index,
-            temp_dir=temp_dir,
-        )
+        # Divergent path already has index.db extracted from the image; the normal
+        # path pulls it from ORAS. NEVER fall back to ORAS on the divergent path —
+        # that would read the base OCP branch's index.db.
+        if sources.index_db_path is not None:
+            artifact_index_db_file = sources.index_db_path
+        else:
+            artifact_index_db_file = fetch_and_verify_index_db_artifact(
+                from_index=from_index,
+                temp_dir=temp_dir,
+            )
 
         set_request_state(request_id, 'in_progress', 'Adding fbc fragment')
         (
@@ -232,20 +235,21 @@ def handle_containerized_fbc_operation_request(
 
             # Push updated index.db before merging the MR so that on failure both
             # git and the index.db artifact remain consistent (MR stays open,
-            # cleanup_on_failure closes it and rolls back the artifact).
-            original_index_db_digest = push_index_db_artifact(
+            # cleanup_on_failure closes it; the content-addressed artifact needs no rollback).
+            push_index_db_artifact(
                 request_id=request_id,
                 from_index=from_index,
                 index_db_path=index_db_path,
                 operators=operators_in_db,
+                output_image=image_url,
                 overwrite_from_index=overwrite_from_index,
                 request_type='fbc_operations',
             )
 
             # Merge or close the MR as the final step so that all side effects
             # (replication, metadata, index.db push) have succeeded before git
             # is advanced. This prevents git/index.db divergence on partial failure.
-            if overwrite_from_index:
+            if overwrite_from_index and not sources.is_divergent:
                 merge_mr_after_build(mr_details, index_git_repo)
                 # Prevent cleanup_on_failure from trying to close an already-merged MR
                 mr_details = None
@@ -267,7 +271,6 @@ def handle_containerized_fbc_operation_request(
                 request_id=request_id,
                 from_index=from_index,
                 index_repo_map=index_to_gitlab_push_map or {},
-                original_index_db_digest=original_index_db_digest,
                 reason=f"error: {e}",
             )
             raise IIBError(f"Failed to add FBC fragment: {e}")

iib/workers/tasks/build_containerized_merge.py

@@ -159,7 +159,6 @@ def handle_containerized_merge_request(
     index_git_repo: Optional[str] = None
     last_commit_sha: Optional[str] = None
     output_pull_spec: Optional[str] = None
-    original_index_db_digest: Optional[str] = None
 
     with tempfile.TemporaryDirectory(prefix=f'iib-{request_id}-') as temp_dir:
         # Setup and clone Git repository
@@ -371,12 +370,13 @@ def handle_containerized_merge_request(
 
             # Push updated index.db before merging the MR so that on failure both
             # git and the index.db artifact remain consistent (MR stays open,
-            # cleanup_on_failure closes it and rolls back the artifact).
-            original_index_db_digest = push_index_db_artifact(
+            # cleanup_on_failure closes it; the content-addressed artifact needs no rollback).
+            push_index_db_artifact(
                 request_id=request_id,
                 from_index=effective_index_image,
                 index_db_path=source_index_db_path,
                 operators=operators_in_db,
+                output_image=image_url,
                 overwrite_from_index=overwrite_target_index,
                 request_type='merge',
             )
@@ -407,7 +407,6 @@ def handle_containerized_merge_request(
                 request_id=request_id,
                 from_index=effective_index_image,
                 index_repo_map={},
-                original_index_db_digest=original_index_db_digest,
                 reason=f"error: {e}",
             )
             # Reset Docker config for the next request. This is a fail safe.

iib/workers/tasks/build_containerized_regenerate_bundle.py

@@ -274,7 +274,6 @@ def handle_containerized_regenerate_bundle_request(
                     request_id=request_id,
                     from_index='',  # No from_index for bundle regeneration
                     index_repo_map={},
-                    original_index_db_digest=None,  # No index.db for bundle regeneration
                     reason=f"error: {e}",
                 )
                 raise IIBError(f"Failed to regenerate bundle: {e}")