Skip to Content
DocumentationIntermediate Bucket

Intermediate Bucket

Etleap needs an S3 bucket in your AWS account where it can stage the data it extracts and transforms, along with metadata about your pipelines.

Setup

To set up the intermediate bucket, you create the bucket and an IAM role that Etleap can assume to read and write files in it.

Step 1. Create a Bucket

  1. Go to the S3 section of the AWS console .
  2. Click Create bucket.
  3. Enter a Bucket name, e.g. etleap-intermediate-<org-name>.
  4. Leave all other settings as their defaults and click Create bucket.

Step 2. Create a Policy

This policy lets Etleap list, read, write, and delete files in the intermediate bucket.

  1. Go to the policies page in the IAM section of the AWS console .

  2. Click the JSON tab.

  3. Copy the policy below and replace <BUCKET> with the name of the bucket you created in Step 1.

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::<BUCKET>",
        "arn:aws:s3:::<BUCKET>/*"
      ]
    }
  ]
}

  4. Paste the policy into the JSON editor, replacing its contents, and click Next.

  5. Enter a Policy name, e.g. etleap_intermediate_access_policy, and click Create policy.

Step 3. Create a Role

Etleap uses this role to access the bucket. The role trusts Etleap’s AWS account when Etleap presents your organization’s external ID.

  1. Go to the roles page in the IAM section of the AWS console .

  2. Under Trusted entity type, select AWS account, then select Another AWS account.

  3. In Account ID, enter Etleap’s AWS account ID for your deployment:

    • Hosted - US: 223848809711
    • Hosted - EU: 167870479612
    • Private deployment: the ID of the AWS account that Etleap is deployed in.

    Etleap also shows this ID in the S3 Intermediate connection setup, under Role > Click to show instructions.

  4. Select Require external ID and enter the external ID shown in the same place in Etleap. Each Etleap organization has its own external ID. If you do not know the external ID of your organization, contact support@etleap.com.

  5. Click Next.

  6. Select the policy you created in Step 2 and click Next.

  7. Enter a Role name, e.g. etleap-intermediate-role, and click Create role.

  8. Open the role you just created and copy its ARN. You will need it in Step 4.

Step 4. Connect the Bucket in Etleap

  1. In the S3 Intermediate connection setup in Etleap, paste the role ARN from Step 3 into IAM Role ARN.
  2. Enter the bucket name from Step 1 in Data Bucket.
  3. Save the connection. Etleap then checks that the role has the permissions it needs on the bucket.
OverviewLogs and Metrics