Skip to content

Bug: Any source table that overrides identifier: raises a compilation error and no template is produced. #272

Description

@krutoileshii

Describe the bug

generate_unit_test_template generates source() references using the source node's identifier, which is the physical relation name, instead of its name, which is the logical source name declared in YAML.

Because dbt resolves source() using the declared name, the macro fails whenever a source overrides identifier:. The operation raises a compilation error and does not generate a unit test template.

This affects the primary use case for identifier:: allowing the physical warehouse table name to differ from the name used by dbt.

The issue is easy to miss because identifier defaults to name. For sources that do not explicitly override identifier, both values are the same and the macro works as expected.

Steps to reproduce

1. Declare a source whose logical name differs from its physical table name:

# models/staging/epicor/sources/_epicor_part.yml
version: 2

sources:
  - name: epicor_erp
    database: data_warehouse_sources
    schema: epicor_prod
    tables:
      - name: part             # Logical name used by source()
        identifier: erp__part  # Physical relation in the warehouse

2. Create a model that selects from the source:

-- models/staging/epicor/base_epicor__part.sql
with source as (

    select * from {{ source('epicor_erp', 'part') }}

),

renamed as (

    select
        nullif(trim(lower(company)), '')  as company,
        nullif(trim(lower(part_num)), '') as part_number

    from source

)

select * from renamed

3. Run the generator:

dbt run-operation generate_unit_test_template \
  --args '{"model_name": "base_epicor__part", "inline_columns": true}'

Expected results

The generated unit test should reference the source using its declared logical name:

unit_tests:
  - name: unit_test_base_epicor__part
    model: base_epicor__part
    given:
      - input: source("epicor_erp", "part")
        rows:
          - {company: , part_num: }
    expect:
      rows:
        - {company: , part_number: }

Actual results

The operation fails with a compilation error, and no template is generated.

The macro attempts to resolve:

source("epicor_erp", "erp__part")

However, the source is declared in dbt as:

source("epicor_erp", "part")

In other words, the macro uses the physical identifier instead of the logical source name.

Screenshots and log output

22:13:34  Running with dbt=1.12.0
22:13:34  Registered adapter: trino=1.10.3
22:13:35  Found 1 operation, 49 models, 1 seed, 22 data tests, 2 sources, 1172 macros
22:13:35  [ERROR]: Encountered an error while running operation: Compilation Error
  Macro 'macro.codegen.generate_unit_test_template' (macros/generate_unit_test_template.sql)
  depends on a source named 'epicor_erp.erp__part' which was not found

  > in macro default__generate_unit_test_template (macros/generate_unit_test_template.sql)
  > called by macro generate_unit_test_template (macros/generate_unit_test_template.sql)
  > called by <Unknown>

System information

The contents of your packages.yml file:

This project uses dependencies.yml:

packages:
  - package: dbt-labs/dbt_utils
    version: 1.4.1

  - package: dbt-labs/codegen
    version: 0.14.1

  - package: godatadriven/dbt_date
    version: 0.19.0

  - package: dbt-labs/dbt_project_evaluator
    version: 1.3.2

  - package: brooklyn-data/dbt_artifacts
    version: 2.10.1

  - package: metaplane/dbt_expectations
    version: 0.10.10

  - package: starburstdata/trino_utils
    version: 0.6.0

  - package: Matts52/dbt_orphan
    version: 0.1.3

Which database are you using dbt with?

  • postgres
  • redshift
  • bigquery
  • snowflake
  • other — Trino

The failure occurs during dbt name resolution, before any SQL is submitted to the database, so the issue does not appear to be adapter-specific.

The output of dbt --version:

Core:
  - installed: 1.12.0
  - latest:    1.12.0 - Up to date!

Plugins:
  - trino: 1.10.3 - Up to date!

The operating system you're using:

Ubuntu 24.04.4 LTS running under WSL2:

6.18.33.2-microsoft-standard-WSL2

The output of python --version:

Python 3.12.3

Additional context

There are two places in macros/generate_unit_test_template.sql where the macro uses item_dict.identifier.

Source-column lookup

The first occurrence causes the compilation error:

{%- if item_dict.resource_type == 'source' %}
    {%- set columns = adapter.get_columns_in_relation(source(item_dict.source_name, item_dict.identifier)) -%}

Generated unit test input

The second occurrence writes an invalid source reference into the generated YAML:

{% if item_dict.resource_type == 'source' %}
  - input: source("{{item_dict.source_name}}", "{{item_dict.identifier}}")

The second occurrence is independently important. Even if the column lookup were fixed, the generated unit test would still contain:

source("epicor_erp", "erp__part")

The unit test would then fail when it was executed.

codegen.get_resource_from_unique_id returns the source object from graph.sources, which already contains both name and identifier:

{% macro get_resource_from_unique_id(resource_unique_id) %}
    {% set resource_type = resource_unique_id.split('.')[0] %}
    {% if resource_type == 'source' %}
        {% set resource = graph.sources[resource_unique_id] %}

This means item_dict.name is already available at both affected call sites.

Suggested fix

Replace identifier with name in both locations:

--- macros/generate_unit_test_template.sql
+++ macros/generate_unit_test_template.sql
@@ -24,7 +24,7 @@
         {%- set input_columns_list = [] -%}
         {%- set item_dict = codegen.get_resource_from_unique_id(item) -%}
         {%- if item_dict.resource_type == 'source' %}
-            {%- set columns = adapter.get_columns_in_relation(source(item_dict.source_name, item_dict.identifier)) -%}
+            {%- set columns = adapter.get_columns_in_relation(source(item_dict.source_name, item_dict.name)) -%}
         {%- else -%}
             {%- set columns = adapter.get_columns_in_relation(ref(item_dict.alias)) -%}
         {%- endif -%}
@@ -59,7 +59,7 @@
     {%- for i in range(ns.depends_on_list|length) -%}
         {%- set item_dict = codegen.get_resource_from_unique_id(ns.depends_on_list[i]) -%}
         {% if item_dict.resource_type == 'source' %}
-      - input: source("{{item_dict.source_name}}", "{{item_dict.identifier}}")
+      - input: source("{{item_dict.source_name}}", "{{item_dict.name}}")
         rows:
         {%- else %}
       - input: ref("{{item_dict.alias}}")

This change does not affect sources that do not override identifier, because name and identifier are already equal for those sources.

Local verification

I tested this exact patch locally by overriding default__generate_unit_test_template within the project.

Apart from a header comment, the override was byte-for-byte identical to the upstream macro with only the two lines above changed.

After applying the patch, the same command that previously failed completed successfully and generated the following output:

unit_tests:
  - name: unit_test_base_epicor__part
    model: base_epicor__part

    given:
      - input: source("epicor_erp", "part")
        rows:
          - company:
            part_num:
            search_word:
            ...

The generated given block now correctly references:

source("epicor_erp", "part")

The operation also populated all 300 source columns and 240 model columns as expected.

Workaround

Both affected lines are still present on main as of this report, so upgrading the package does not currently resolve the issue.

The available workaround is to override the macro in the project using dispatch in dbt_project.yml:

dispatch:
  - macro_namespace: codegen
    search_order: ['<your_project>', 'codegen']

Then copy default__generate_unit_test_template into the project and replace the two uses of item_dict.identifier with item_dict.name.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtriage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions