Skip to Content
DocumentationEventsSQS

SQS

Etleap supports ingesting data from SQS FIFO queues.

Source Setup

To allow Etleap to access data in your queues, you need to set up a role that Etleap can assume, and attach a policy that grants access to SQS.

Step 1. Create a New Role

  1. Go to the IAM section of the AWS console .
  2. Click Create role, select Trusted Entity Type AWS account and then choose Another AWS account.
  3. The Account ID for your deployment can be found in the SQS connection setup under Role → Click to show instructions.
  4. Check the box that says Require External ID.
    • Enter the your organization ID: this is provided in the instructions dropdown within the Role section of the Etleap connection setup page for SQS.
  5. Skip adding permissions for now. Click Next until you reach the Review page.
  6. Enter a name for the role.
  7. Click Create role.
Warning

If you are running Etleap in a VPC and you are restricting the roles that Etleap can assume via the roles_allowed_to_be_assumed input variable , then you have to add the role created in this step to the list before proceeding.

Step 2. Attach a Policy to the Role

  1. Find the role in the IAM role list in the AWS console  and click it.
  2. In the Permissions tab, click on Add permissions and then select Add inline policy.
  3. Click the JSON tab. 
    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:GetQueueUrl",
        "sqs:ListQueues"
      ],
      "Resource": "*"
    }
  ]
}
  4. Then click Review policy.
  5. Enter a descriptive Name of your choice, e.g. etleap_minimal_access_policy.
  6. Click Create policy

After creating a role with the above policy which allows Etleap to list and extract data from your SQS queues, you can proceed with creation of a SQS connection in Etleap connection setup page.

What Data is Available?

Tip

Update-Mode Deletes Support

Deletes  are supported for update-mode entities with this connector.

Etleap SQS pipelines operate in Update Mode, and support any FIFO SQS queue. By default:

  • messageId is the record’s primary key.
  • sentTimestamp is the LUTS column.

These default attributes can be overridden on the Destination Settings screen during pipeline creation.

Delete Capturing

Etleap can capture deletions by inspecting a boolean flag named deleted, and a string value named deletedRecordId within the Message Attributes. If a record has this flag set to true, Etleap will remove the corresponding record (matching deletedRecordId with the primary key) from the destination.

Key Considerations

  • FIFO queue constraints require that messages be deleted after they are read before new messages with the same group ID are delivered.
    • To maintain data integrity, Etleap must be the sole consumer of the queue.
  • As part of standard FIFO behavior, Etleap deletes messages from the queue after processing them.
KafkaFile Types