Use Events to Integrate your Workflows

What is an Event-Driven Architecture?

An event-driven architecture uses events to trigger and communicate between decoupled services. It’s a common architecture for modern applications built with microservices. An event is a change in state, or an update. The event can either carry the state (e.g., the item purchased, its price, and a delivery address) or identifiers (e.g., a notification that an order was shipped). Event producer and consumer services are decoupled, which allows them to be scaled, updated, and deployed independently. For more information on how AWS defines event-driven architecture, go here .

This guide explains how you can integrate Etleap into your event-driven architecture to orchestrate your data pipelines and integrate workflows.

Event Architecture with Etleap

You can use events with Etleap to trigger and communicate with other services. You can do this by sending and receiving events to/from Etleap via Amazon’s Simple Notification Service (SNS). SNS is AWS’s fully managed pub/sub messaging and event-driven service. At a high level, Etleap or the customer produces/publishes an event to an SNS topic each time there is a change in state or an update about a pipeline (see Published Events section below). Once the event is published, the subscription policies configured determine where to route the information (e.g., Ops Monitoring team) or which task to execute next (e.g., transform this batch of data). For more information on SNS, please go to AWS’s documentation here .

SNS eventing with Etleap can be beneficial when trying to build and maintain your data lake/warehouse at scale if your team wants to integrate tools and processes into the end-to-end data pipelines. Examples include data quality checks, data delivery processes, monitoring and on-call tools, and external schedulers. If you’re interested in more specific examples before considering utilizing events with your Etleap pipelines and integrating your workflows, check out the Use Cases section at the end of this guide.

You can start sending/receiving events to Etleap once you’ve added an AWS SNS Topic in your Admin Console under the Notifications tab. This setting can only be changed by Org Admins.

Published Events

Check out the list of published events that you can subscribe to under the Events panel on the left-hand side of the page here . This includes information on the purpose of each event and the event attributes that you can utilize to implement more fine-grained filtering.

Note

Note: Notifications, as outlined in the Expand Notification reach guide here , are one type of a published event.

Login to Etleap.
Click the drop-down next to your name at the top right and select Admin Console.
Go to the Notifications tab.
Under the Outbound Notification Settings pane, Click Edit next to AWS SNS Topic

Note
If you’d like to set up SNS and don’t see the AWS SNS Topic setting available, please contact support@etleap.com to enable it for your organization.
Click the link titled Topics section of the SNS console to obtain your ARN.
Login to your AWS account and and either: (a) obtain the ARN of an existing topic or (b) click create topic, create a new topic, and copy the ARN.
Enter the ARN in the AWS SNS Topic field in Etleap and click Next.
Copy the JSON object from Etleap and paste it into your AWS SNS Topic’s Access Policy.
Click Save changes in AWS.
Click Verify & Save SNS Topic.

Events about pipelines, models, dbt schedules, and connections can now be sent/received to this SNS topic.

Use Cases

Create a JIRA ticket when there is a schema change

Tip

Giga wants to maximize robustness for data consumers

Giga is focused on keeping her data processes more in sync. She wants to automatically create JIRA tickets for her data team to review potential impact to downstream processes when Etleap detects new columns. This allows her data team to proactively keep reporting processes up to date and get new data in the hands of stakeholders quicker. Here’s how she integrated SNS notifications for new column notifications.

Giga created a new SNS topic dedicated for Etleap alerts and added the topic information in her Etleap SNS setting.
She subscribed to this topic in SNS and wrote a subscription policy rule that read every Etleap notification posted to the channel.
When Etleap detects a new column in a pipeline, it publishes the PIPELINE_SCHEMA_CHANGE event notification to SNS (in addition to sending the pipeline owner, shared users, and address in notification email setting).
Giga uses AWS Lambda to automatically create a JIRA ticket every time Etleap publishes the schema change notification.
When a software developer logs into JIRA, they see the ticket and review the reports that depend on this data source.
The new column is an important field for a monthly deadline so her data team updates the affected queries.

Now anytime Etleap detects a new column, her team can take action by checking their JIRA queue.

Trigger quality checks after data is transformed

Tip

Peta wants to automate handoffs in data quality check process

Peta, an Etleap power user, implemented an event-driven architecture to manage his team’s data lake. They send events to Etleap to create/transform files in the lake and receives events from Etleap to trigger data quality events on pipelines. This enables a more streamlined way to build his team’s data lake at scale, while maintaining strong data quality principles. Here’s how Peta integrated SNS notifications to trigger events after pipeline transformations completed:

Peta created a new SNS topic dedicated for Etleap events and added the topic information in his Etleap SNS setting.
He developed an AWS Lambda function which automatically triggers data quality checks on transfomred data.
Etleap publsihes the TRANFORMATION_COMPLETE after transforming a batch of data, waiting to be approved to load to the destination table.
The TRANSFORMATION_COMPLETE event triggers the subscribing AWS Lambda function that runs the data quality checks on the pending batch.
If the quality checks fails, Peta and his team diagnose the data issue and resolve.
If the quality checks passes, the AWS Lambda function publishes the QUALITY_CHECK_COMPLETE event to Etleap’s inbound SNS topic.
Upon receiving the QUALITY_CHECK_COMPLETE event, Etleap loads the batch to the destination table.
Note
Note: Steps 6 and 7 rely on inbound (in addition to outbound) notifications. Inbound Notification Settings are events that you can send to Etleap (opposed to receive). Inbound Notifications are not covered in detail in this article, so if you’re interested in setting them up, please contact support@etleap.com for more information.

This process runs 24/7, allowing Peta’s team to automatically load quality-approved batches through various automated checkpoints, powered by SNS notifications.