← 返回首页
added support for iceberg tables on S3 table buckets using athena engine by vishnuprasad3842 · Pull Request #5774 · SQLMesh/sqlmesh · GitHub
Skip to content

Navigation Menu

Toggle navigation
Sign in
Appearance settings
Search or jump to...

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Saved searches

Use saved searches to filter your results more quickly

Appearance settings
Resetting focus

added support for iceberg tables on S3 table buckets using athena engine#5774

Open
vishnuprasad3842 wants to merge 5 commits into
SQLMesh:mainfrom
vishnuprasad3842:dev/athena-s3tables-engine-adapter
Open

added support for iceberg tables on S3 table buckets using athena engine#5774
vishnuprasad3842 wants to merge 5 commits into
SQLMesh:mainfrom
vishnuprasad3842:dev/athena-s3tables-engine-adapter

Conversation

Copy link
Copy Markdown

Description

Added support for Amazon S3 Tables (managed Iceberg tables) within the Athena Engine Adapter.

Specific changes include:

Catalog Support: Updated AthenaEngineAdapter.catalog_support to CatalogSupport.FULL_SUPPORT to allow cross-catalog queries without issuing SET CATALOG commands. This prevents SQLMesh from stripping out custom catalog names (e.g., s3tablescatalog/...) from model names.
S3 Table Partitioning: For S3 tables, partition properties are now correctly generated as partitioning=ARRAY['...'] instead of the standard partitioned_by property format.
S3 Table Properties: S3 Table CTAS queries in Athena implicitly default to table_type='ICEBERG' and do not allow explicit specification of most table properties (except format='PARQUET'). The adapter now filters out unsupported TBLPROPERTIES and table_type when generating CREATE TABLE ... AS statements for S3 tables to prevent pyathena.error.OperationalError.
Dictionary Mutation Bug: Fixed a subtle bug in _build_table_properties_exp where table_properties was being mutated during AST parsing, causing properties like s3_table to drop out during intermediate schema table creation.

Test Plan

Tested creating S3 Iceberg tables via sqlmesh plan utilizing custom MODEL properties (s3_table true / catalog defined).

Verified partitioning is properly formulated as an unquoted ARRAY structure for S3 tables.

Verified that temporary intermediate schema generation properly inherits S3 properties without failure.

Ran make fast-test to ensure no regressions with standard Hive/Iceberg tables in the Athena adapter

Checklist

I have run make style and fixed any issues

I have added tests for my changes (if applicable)

All existing tests pass (make fast-test)

My commits are signed off (git commit -s) per the DCO

vishnu added 5 commits April 23, 2026 15:21
- Updated OpenAPI schema to reflect changes in API endpoints. - Renamed parameters in API calls for consistency, changing `column_name` to `columns` in lineage-related endpoints. - Adjusted the handling of optional properties in the `ModelSQLMeshModel` class to ensure default values are set. - Changed route parameter types from `str` to `path` for better URL handling in several endpoints. These changes improve the clarity and functionality of the API.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Footer

© 2026 GitHub, Inc.