Datasource panel with `IbisEngine` shows empty database and schema

[zilto]

Describe the bug

When creating a connection to an ibis backend and viewing its content in the DATASOURCES panel, it shows all empty database/catalogs/schemas.

Unsure if it's a desirable behavior. It starts to clutter the UI when using DuckLake

I believe it wasn't the case in the initial implementation. So it must have been added afterwards.

Will you submit a PR?

• Yes

Environment

Details

{
  "marimo": "0.17.0",
  "editable": false,
  "location": "/home/tjean/projects/{redacted}/marimo",
  "OS": "Linux",
  "OS Version": "6.14.0-33-generic",
  "Processor": "x86_64",
  "Python Version": "3.11.11",
  "Locale": "en_US",
  "Binaries": {
    "Browser": "--",
    "Node": "v20.19.4"
  },
  "Dependencies": {
    "click": "8.3.0",
    "docutils": "0.22.2",
    "itsdangerous": "2.2.0",
    "jedi": "0.19.2",
    "markdown": "3.9",
    "narwhals": "2.8.0",
    "packaging": "25.0",
    "psutil": "7.1.0",
    "pygments": "2.19.2",
    "pymdown-extensions": "10.16.1",
    "pyyaml": "6.0.3",
    "starlette": "0.48.0",
    "tomlkit": "0.13.3",
    "typing-extensions": "4.15.0",
    "uvicorn": "0.37.0",
    "websockets": "15.0.1"
  },
  "Optional Dependencies": {
    "altair": "5.5.0",
    "duckdb": "1.4.1",
    "ibis-framework": "11.0.0",
    "openai": "2.3.0",
    "pandas": "2.3.3",
    "pyarrow": "21.0.0",
    "loro": "1.8.1",
    "ruff": "0.14.0",
    "sqlglot": "27.27.0"
  },
  "Experimental Flags": {
    "chat_sidebar": true,
    "multi_column": true,
    "chat_modes": true,
    "performant_table_charts": true,
    "inline_ai_tooltip": false
  }
}

Code to reproduce

No response

[mscolnick]

I think knowing empty databases or schemas can be quite useful. I agree it can also clutter the UI.

It could make sense to have a checkbox to hide them on the front end or something in user settings.

[zilto]

Bump for this feature. When using dlt, the majority of cases will have empty schemas and tables because dlt never writes to default namespaces

[zilto]

Bump. This seems specific to IbisEngine. I don't have these empty databases and schemas when I read the same DuckDB instance directly

[Light2Dark]

Have a fix @zilto . This filters out empty databases and schemas. I have an inkling that you may want to filter out all meta tables / leave empty databases in. So do let me know.

[zilto]

Full repro

You will have to install `ibis-framework[duckdb]`, marimo, dlt

import marimo

__generated_with = "0.23.5"
app = marimo.App(width="medium")


@app.cell
def _():
    import marimo as mo
    import dlt
    import duckdb

    return dlt, duckdb


@app.cell
def _(dlt):
    pipeline = dlt.pipeline("foo", destination="duckdb")
    pipeline.run([dict(id=1, name="Alice"), dict(id=2, name="Bob")], table_name="users")
    return (pipeline,)


@app.cell
def _(pipeline):
    ibis_con = pipeline.dataset().ibis()
    return


@app.cell
def _(duckdb):
    duckdb_con = duckdb.connect(".dlt/data/agent/foo.duckdb")
    return


if __name__ == "__main__":
    app.run()

I have an inkling that you may want to filter out all meta tables / leave empty databases in. So do let me know.

It would be nice to have the same UI as DuckDB.

IIRC the marimo code, each data source backend implement if they retrieve database, schema, tables, columns, etc. eagerly or lazily. When using the IbisEngine, the internal tables will have different names across backends (e.g., BigQuery vs DuckDB)

proposed backend solution

The "greedy solution" would be to have the IbisEngine scan up to table depth all schemas under a database.

• If a schema finds at least a table, keep it (can stop scanning for more tables and lazily load them later).
• If a database finds at least a schema with a table, keep the database
  This search should complete pretty quickly in real-world scenarios

frontend solution

OTOH, I looked at the UI change in the draft PR and looks like a good solution!

[Light2Dark]

Awesome. I thought of the backend change to skip over empty, but we don't enforce this consistently across backends (eg. sqlaclhemy, redshift). So, I think the frontend toggle is probably better.