Iceberg Connection Setup

This page details the manual setup steps required to configure an Iceberg connection in Etleap.

Etleap creates and manages Iceberg tables by writing files to S3 and interacting with your AWS Glue catalog. In order to setup an Iceberg connection in Etleap you will need to provide the following properties:

• IAM Role: An AWS role that provides Etleap permission to read and write from S3 and manage tables in AWS Glue.
• S3 Bucket Name and Path Prefix: The location in S3 that Iceberg metadata and data files will be written to.
• AWS Glue Catalog: The region of the Glue catalog that will store the definitions of your Iceberg tables.

Step 1. Create an S3 Bucket

Go here  to create a new S3 bucket to store the Iceberg tables. Give it a memorable name and leave the other fields with their default values.

If there is an existing bucket you would like to use, you may skip this step.

Step 2. Create a Glue Database

Go here  to create a Glue database. If there is an existing Glue database you would like to use, you may skip this step.

1. Click Add database, specify a name and, optionally, a description. Leave the other fields blank.
2. Click Create database.

Step 3. Create an IAM Role

1. Create an IAM role for Etleap to assume in the AWS console here .
2. Select Trusted Entity Type AWS account and then choose Another AWS account.
3. Check the box that says Require External ID.
4. For the Account ID and External ID, enter the IDs provided in the instructions dropdown within the Role section of the Etleap Iceberg connection setup page .
5. Skip adding permissions for now. Click Next until you reach the Review page.
6. Enter a name for the role and click Create role.
7. Find the role you created. In the Permissions tab, under Permissions policies, click Add permissions and pick Create inline policy.
8. Click JSON and copy-paste the policy below, and make the following replacements:
   • ICEBERG_BUCKET_NAME is the bucket created in step 1
   • AWS_ACCOUNT_ID is your 12-digit account ID
   • AWS_REGION is the region the resources have been created in

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::ICEBERG_BUCKET_NAME",
        "arn:aws:s3:::ICEBERG_BUCKET_NAME/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetTable",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:CreateTable",
        "glue:UpdateTable",
        "glue:DeleteTable"
      ],
      "Resource": [
        "arn:aws:glue:AWS_REGION:AWS_ACCOUNT_ID:catalog",
        "arn:aws:glue:AWS_REGION:AWS_ACCOUNT_ID:database/*",
        "arn:aws:glue:AWS_REGION:AWS_ACCOUNT_ID:table/*"
      ]
    }
  ]
}

9. Click Next and give the policy a meaningful name, e.g. etleap_iceberg.
10. Click Create policy.

Step 4. Grant Lake Formation Permissions

Skip this step if your Glue Data Catalog is not managed by AWS Lake Formation.

When Lake Formation manages your catalog, it authorizes every Glue API call, so the IAM policy from step 3 is not enough on its own. Until the role has Lake Formation permissions, Etleap can reach the catalog but sees no databases or tables in it.

Grant the role permissions on the Glue database:

1. Go here  and navigate to Permissions → Data lake permissions.
2. Click Grant.
3. Under Principals, pick IAM users and roles and select the role you created in step 3.
4. Under LF-Tags or catalog resources, pick Named Data Catalog resources and select the Glue database from step 2. Leave Tables empty.
5. Under Database permissions, check Create table and Describe.
6. Click Grant.

Then grant the role permissions on the tables in that database:

1. Click Grant again and select the same principal and database as above.
2. In the Tables dropdown, select All tables.
3. Under Table permissions, check Select, Describe, Alter, and Drop.
4. Click Grant.

Repeat both grants for each Glue database you need Etleap to load into.

Step 5. Create an Iceberg Connection in Etleap

Create a new connection in Etleap and select Iceberg as the type. Complete the connection setup with the following properties:

1. The IAM role created in step 3
2. The S3 bucket you created in step 1

During pipeline creation, you will be able to pick which Glue Database your pipelines will write to.

Access Iceberg Tables in Snowflake

To query your Iceberg tables in Snowflake, create a Catalog Linked Database . The Catalog Linked Database will allow you to read and/or write to your Iceberg tables that live in S3 & Glue as if they were native Snowflake tables.

If Lake Formation manages your catalog, the IAM role that Snowflake’s catalog integration assumes needs its own Lake Formation grants. These are separate from the grants you made for the Etleap role in step 4, and Snowflake will not see a table until they are in place. Repeat step 4 for the Snowflake catalog integration role, including the grant on All tables.

To see what Snowflake can currently link, run the following query, replacing <database_name> with the name of your Catalog Linked Database:

SELECT SYSTEM$CATALOG_LINK_STATUS('<database_name>');

A Catalog table <database>.<table> does not exist in catalog integration failure for a table that does exist in Glue means the catalog integration role is missing Lake Formation permissions on that table.