Skip to Content
DocumentationEventsKinesis

Kinesis

Kinesis connections give you access to your organization’s Amazon Kinesis streaming data.

Note

Kinesis pipelines can only load to Iceberg destinations.

Source Setup

To allow Etleap to access data in your streams, you need to set up a role that Etleap can assume and attach a policy that grants access to Kinesis.

Step 1. Create a New Role

  1. Go to the IAM section of the AWS console .
  2. Click Create role, select Trusted Entity Type AWS account and then choose Another AWS account.
  3. The Account ID for your deployment can be found in the Kinesis connection setup under Role → Click to show instructions.
  4. Check the box that says Require External ID.
  • Enter your organization ID: this is provided in the instructions dropdown within the Role section of the Etleap connection setup page for Kinesis.
  1. Skip adding permissions for now. Click Next until you reach the Review page.
  2. Enter a name for the role.
  3. Click Create role.
Warning

If you are running Etleap in a VPC and you are restricting the roles that Etleap can assume via the roles_allowed_to_be_assumed input variable , then you will need to add the role created in this step to the list before proceeding.

Step 2. Attach a Policy to the Role

Etleap will list all the streams in the same AWS account as the role it will assume. You can choose to grant data access to all streams or restrict to specific ones.

  1. Find the role in the IAM role list in the AWS console  and click it.
  2. In the Permissions tab, click on Add permissions and then select Add inline policy.
  3. Click the JSON tab and paste the following policy.
    {
        "Version": "2012-10-17",
        "Statement": [
    {
        "Effect": "Allow",
        "Action": [
        "kinesis:ListStreams",
        "kinesis:DescribeStream",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords",
        "kinesis:ListShards"
        ],
        "Resource": "*"
    }
        ]
    }
  1. Then click Review policy.
  2. Enter a descriptive Name of your choice, e.g. etleap_kinesis_access_policy.
  3. Click Create policy

Kinesis Pipelines

  • Kinesis pipelines operate in Update Mode. You must select a primary key at the end of the pipeline creation wizard to create a Kinesis pipeline. If your iceberg destination has a warehouse with row-level deletes support, you’ll be able to query your data with the type-1 view, otherwise, data will be output as changelog (updates to records show up as new records).
  • Only data within the stream’s data retention policy will be processed by the pipeline. Refreshing a pipeline will remove any data that is outside of the retention policy.
KafkaSQS