Skip to Content
DocumentationEventsEvent Stream

Event Stream

Etleap can process streaming data through the Event Stream connection. This can be achieved by sending events to customer-specific webhook endpoints.

Source Setup

When setting up an Event Stream connection, Etleap will create a unique webhook endpoint to receive your events. The generated unique URL depends on your deployment type with Etleap.

For the US multi-tenant environment (e.g., app.etleap.com), the unique URL structure is:

	https://<organization_code>.in.etleap.com/<unique_identifier>/

For VPC deployments and the EU environment, the unique URL structure is:

	https://<hostname>/in/<unique_identifier>/

Please contact support@etleap.com if you have any questions on your company’s deployment type.

Any events sent to this endpoint will be forwarded and stored in your organization’s intermediate S3 bucket. The events can then be processed by setting up a pipeline for this Event Stream connection.

Warning

After creating a new Event Stream connection it might take a few minutes for the connection to be active. The Etleap UI will update once the connection is ready to be used. Sending events to the newly created endpoint before the connection is active might cause some events to be lost.

What Data is Available?

An Event Stream connection stores all events sent to the unique endpoint in a separate directory in your intermediate S3 bucket. The following data is stored for each event received:

  • Remote address
  • URL query
  • Request method
  • Body of the event
  • Headers
  • Host address
  • URL path
  • Event time
  • Stream name

Event data is stored in GZIP files (.gz), which are organized in different sub-directories representing the date and time the data was received. The overall structure looks like this: year/month/day/hour_of_day.

The lowest sub-directory (representing the hour of day) can contain multiple GZIP files. The file names show the exact minute the data was received in. There might be multiple files for the same minute as different events can be processed by different hosts.

To reduce the number of files stored in your intermediate S3 bucket, Etleap merges all separate files within a “hour_of_day” sub-directory into a single combined file. This happens after a full hour is completed and either all or none of the files within that sub-directory are processed by a pipeline. The combined file contains all the events received during that hour.

Key Considerations

  • Refreshing an Event Stream pipeline can take a while for large pipelines since a pipeline refresh causes all data to be reprocessed.
  • Refreshing an Event Stream pipeline does not require re-sending all the data to the specific webhook endpoint as the streaming data is stored in your intermediate bucket.
Confluent CloudKafka