Skip to content

Feat/glue athena enhancements#4

Open
gettingstuffdone wants to merge 5 commits into
endemics:mainfrom
gettingstuffdone:feat/glue-athena-enhancements
Open

Feat/glue athena enhancements#4
gettingstuffdone wants to merge 5 commits into
endemics:mainfrom
gettingstuffdone:feat/glue-athena-enhancements

Conversation

@gettingstuffdone

Copy link
Copy Markdown

Description:
This PR completes the Athena fallback capability by automating Glue catalog setup for Parquet tables and improving SQL compatibility:

  • Glue Provisioning for Parquet:
    • Extended GlueProvisioner to support the Parquet format with appropriate ParquetHiveSerDe, MapredParquetInputFormat, and
      MapredParquetOutputFormat.
    • Added serialization.format=1 parameter to ensure standard Hive compatibility.
  • Athena SQL Translation:
    • Implemented _translate_to_athena to dynamically convert DuckDB-specific S3 scan functions (like read_parquet, iceberg_scan,
      read_csv, read_json) into qualified Glue table references (e.g., "database"."table_name") during the Athena execution path.
  • Advanced Cost Estimation:
    • Added a sophisticated partition pruning detector (_has_partition_filter) using sqlglot to identify when a query can benefit from
      S3 prefix filtering.
    • Applied a partition_fraction (0.1) to cost estimates when valid partition filters are detected, providing more accurate "Dry Run"
      previews.
  • Testing:
    • Added server/tests/test_cost_estimator_partitioning.py to verify partition pruning detection across basic, range, and complex
      IN/OR clauses.
    • Added Parquet-specific SerDe verification to server/tests/test_glue_provisioner.py.
    • Expanded server/tests/test_query_helpers.py with comprehensive SQL translation test cases.
  • Dependency: This PR is based on feat/spend-tracking.

Note: Since these branches are sequential, this PR will show the changes from the refactor as well unless feat/server-refactor is merged first.

@endemics endemics left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, much appreciated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants