Iceberg Connection Setup

This page details the manual setup steps required to configure an Iceberg connection in Etleap.

Etleap creates and manages Iceberg tables by writing files to S3 and interacting with your AWS Glue catalog. In order to setup an Iceberg connection in Etleap you will need to provide the following properties:

• IAM Role: An AWS role that provides Etleap permission to read and write from S3 and manage tables in AWS Glue.
• S3 Bucket Name and Path Prefix: The location in S3 that Iceberg metadata and data files will be written to.
• AWS Glue Catalog: The region of the Glue catalog that will store the definitions of your Iceberg tables.

Step 1. Create an S3 Bucket and Glue Database

1. Go here  to create a new S3 bucket to store the Iceberg tables. If there is an existing bucket you would like to use, you may skip this step.
   • Give it a memorable name and leave the other fields with their default values.
2. Go here  to create a Glue database. If there is an existing Glue database you would like to use, you may skip this step.
   • Click Add database, specify a name and, optionally, a description. Leave the other fields blank.
   • Click Create database.

Step 2. Create an IAM Role

1. Create an IAM role for Etleap to assume in the AWS console here .
2. Select Trusted Entity Type AWS account and then choose Another AWS account.
3. Check the box that says Require External ID.
4. For the Account ID and External ID, enter the IDs provided in the instructions dropdown within the Role section of the Etleap Iceberg connection setup page .
5. Skip adding permissions for now. Click Next until you reach the Review page.
6. Enter a name for the role and click Create role.
7. Find the role you created. In the Permissions tab, under Permissions policies, click Add permissions and pick Create inline policy.
8. Click JSON and copy-paste the policy below, and make the following replacements:
   • ICEBERG_BUCKET_NAME is the bucket created in step 1.1
   • AWS_ACCOUNT_ID is your 12-digit account ID
   • AWS_REGION is the region the resources have been created in

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::ICEBERG_BUCKET_NAME",
        "arn:aws:s3:::ICEBERG_BUCKET_NAME/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetTable",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:CreateTable",
        "glue:UpdateTable",
        "glue:DeleteTable"
      ],
      "Resource": [
        "arn:aws:glue:AWS_REGION:AWS_ACCOUNT_ID:catalog",
        "arn:aws:glue:AWS_REGION:AWS_ACCOUNT_ID:database/*",
        "arn:aws:glue:AWS_REGION:AWS_ACCOUNT_ID:table/*"
      ]
    }
  ]
}

9. Click Next and give the policy a meaningful name, e.g. etleap_iceberg.
10. Click Create policy.

Step 3. Create an Iceberg Connection in Etleap

Create a new connection in Etleap and select Iceberg as the type. Complete the connection setup with the following properties:

1. The IAM role created in step 2
2. The S3 bucket you created in step 1.1

During pipeline creation, you will be able to pick which Glue Database your pipelines will write to.

Access Iceberg Tables in Snowflake

To query your Iceberg tables in Snowflake, create a Catalog Linked Database . The Catalog Linked Database will allow you to read and/or write to your Iceberg tables that live in S3 & Glue as if they were native Snowflake tables.