Access Iceberg Tables in Databricks

Databricks can query the Iceberg tables Etleap writes by federating your AWS Glue Data Catalog into Unity Catalog. This page assumes you have completed the Iceberg connection setup. The steps below create a dedicated IAM role for Databricks, register it in Unity Catalog, and create a foreign catalog in which each Glue database appears as a schema.

You will need:

• A Databricks workspace on AWS with Unity Catalog enabled.
• The CREATE SERVICE CREDENTIAL, CREATE STORAGE CREDENTIAL, CREATE EXTERNAL LOCATION, CREATE CONNECTION, and CREATE CATALOG privileges on the Unity Catalog metastore. A metastore admin has all of these. To check your privileges, run the following in Databricks, replacing <user> with your Databricks user name:

  SHOW GRANTS `<user>` ON METASTORE;

  Metastore admins may see no rows, because the admin role is assigned rather than granted.

Step 1. Create an IAM Role for Databricks

Databricks uses one IAM role for both its service credential (Glue access) and its storage credential (S3 access). This role is separate from the Etleap role you created during connection setup.

1. Follow Step 1 of Databricks: Create service credentials , using the AWS instructions and the permissions policy below. Use the placeholder trust policy from the Databricks guide for now. You will replace it with the final trust policy in step 3.
2. Make the following replacements in the permissions policy:
   • <DATABRICKS_ROLE_NAME> is the name of the IAM role you are creating in this step
   • <ICEBERG_BUCKET_NAME> is the bucket used by your Iceberg connection, created in step 1 of the connection setup
   • <AWS_ACCOUNT_ID> and <AWS_REGION> are the account and region that hold the Glue Data Catalog and the Databricks IAM role
   • <GLUE_DATABASE_1> and <GLUE_DATABASE_2> are examples of Glue databases Etleap loads into, created in step 2 of the connection setup. Include a database/ and table/ line for each Glue database you use, whether that is one or several.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::<ICEBERG_BUCKET_NAME>",
        "arn:aws:s3:::<ICEBERG_BUCKET_NAME>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:GetTable",
        "glue:GetTables",
        "glue:GetPartition",
        "glue:GetPartitions"
      ],
      "Resource": [
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:database/default",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:database/<GLUE_DATABASE_1>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/<GLUE_DATABASE_1>/*",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:database/<GLUE_DATABASE_2>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/<GLUE_DATABASE_2>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<DATABRICKS_ROLE_NAME>"
    }
  ]
}

Step 2. Create the Service Credential and Storage Credential

Both credentials use the IAM role from step 1. After you create each credential, Databricks shows the Unity Catalog role ARN and an external ID. Keep both values for step 3.

1. Create a service credential  using the role. Leave Limit to read-only use disabled.
2. Create a storage credential  using the same role. Leave Limit to read-only use disabled.

Step 3. Update the Trust Policy and Validate the Credentials

Replace the placeholder trust policy on the Databricks IAM role with the one below, and make the following replacements:

• <UNITY_CATALOG_ROLE_ARN> is the Unity Catalog role ARN shown by Databricks when you created the credentials in step 2
• <SERVICE_CREDENTIAL_EXTERNAL_ID> and <STORAGE_CREDENTIAL_EXTERNAL_ID> are the external IDs of the two credentials from step 2. These may be the same value. If they match, list it once.
• <AWS_ACCOUNT_ID> and <DATABRICKS_ROLE_NAME> are the same values as in the permissions policy from step 1

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "<UNITY_CATALOG_ROLE_ARN>",
          "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<DATABRICKS_ROLE_NAME>"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": [
            "<SERVICE_CREDENTIAL_EXTERNAL_ID>",
            "<STORAGE_CREDENTIAL_EXTERNAL_ID>"
          ]
        }
      }
    }
  ]
}

Then open each credential in Catalog Explorer and click Validate Configuration. All checks must pass before you continue.

Step 4. Grant Lake Formation Permissions

Skip this step if your Glue Data Catalog is not managed by AWS Lake Formation.

When Lake Formation manages your catalog, the Databricks role needs its own Lake Formation grants. These are separate from the grants you made for the Etleap role during connection setup.

1. Grant the Databricks role Describe on each Glue database Etleap loads into, and on the default database. This lets Databricks list the databases.
2. Grant the Databricks role Describe and Select on All tables in each Glue database Etleap loads into. This lets Databricks list and read the tables. A grant on the tables does not give Describe on the database that contains them, so both grants are needed.

Step 5. Create an External Location

1. Create an external location  for the S3 path Etleap writes Iceberg tables to, for example s3://<ICEBERG_BUCKET_NAME>/, using the storage credential from step 2. One path covers every Glue database in the bucket. Do not select Read only.
2. The file events check fails with File Events Permissions Not Verified, because the permissions policy above does not include the file events permissions. File events are not required for this setup. If all other checks pass, click Force create.
3. Edit the external location and turn off file events, so that Databricks stops checking for those permissions.

Step 6. Create the Glue Connection and Foreign Catalog

Follow Create the connection  using Catalog Explorer. The wizard creates the connection and the foreign catalog together. Use these settings:

• Connection details: set Credential to the service credential from step 2.
• Catalog basics: use underscores, not hyphens, in the catalog name, and do not make the catalog read-only. Set Authorized paths to the external location path from step 5. Set Storage location to the S3 path of the Iceberg tables, the same path as the external location. A storage location is required to read Iceberg tables.
• Access and Metadata: leave the defaults.

Then sync the catalog and check the schemas. Run the following in Databricks, replacing <catalog_name> with the name of your foreign catalog:

REFRESH FOREIGN CATALOG <catalog_name>;
SHOW SCHEMAS IN <catalog_name>;

Each Glue database Etleap loads into appears as a schema in the catalog.

If queries against the tables fail, check that the catalog has a storage location. If it is missing, set it and refresh:

ALTER CATALOG <catalog_name> SET MANAGED LOCATION 's3://<ICEBERG_BUCKET_NAME>/';
REFRESH FOREIGN CATALOG <catalog_name>;

Step 7. Query Iceberg Tables

In Databricks, query tables with a three-part name:

SELECT * FROM <catalog_name>.<glue_database>.<table_name>;