Skip to Content
DocumentationLoad Throttling

Load Throttling

It is sometimes useful to restrict pipelines to only load at certain times of the day or at a certain frequency. For example, you can use load throttling with Amazon Redshift Serverless, Snowflake or Databricks Delta Lake to reduce the time the cluster is active.

How Load Throttling Works

Load throttling works by defining a window, within which, pipelines perform their activities.

Load windows exist on destination connections, this achieves:

  1. Synchronized loads across all pipelines that load to that destination.
  2. Throttled loads that occur only once per load window.
  3. Optimized latency by scheduling extractions and transformations shortly before the loads.

Load Windows

Load throttling allows you to specify a window in which each pipeline for a destination performs at most one exhaustive load.

For each destination connection, there is only ever one load window open at a time. A load window may close before the next one begins, but, it will never overlap with the next window. When the schedule triggers the current load window closes and the next simultaneously opens.

An illustration showing how a load window behaves when all pipelines are ready to load early
An example of load window behavior when all pipelines are ready to load before the start of the next window.

Load window open

Load windows open when the schedule specified on the destination connection triggers.

When the window opens, each pipeline that loads to the throttled destination will:

  1. Begin extracting or transforming a new batch, if the pipeline’s update schedule permits it. Pipelines will not access the source any more frequently than the update schedule allows, and will skip load windows if necessary.
  2. Once the batch is ready to be loaded, the pipeline will wait until the window closes before executing the load on the destination.

This process synchronizes the loads for each throttled destination and reduces the time the cluster is in use by Etleap.

Load window close

When the window closes, each pipeline that is ready to load will start loading.

The load window will close when one of the following occurs:

  1. All pipelines that load to the throttled connection are ready to execute their loads. Paused and stopped pipelines will not prevent the load window from closing, nor will pipelines that were not permitted to start batches by the update schedule.
  2. Or, the next load window opens.

This means that the pipelines will load when everything is ready, or when the load schedule next triggers, whichever comes first.

Any pipelines that aren’t ready when the window closes, will wait for the following load window to close. This creates at most one period of load activity per load window.

An illustration showing that loads only wait as long as the maximum length of the window and late pipelines wait until the next window closes
An example of load window behavior when one pipeline does not finish in time. Other pipelines will load no later than the start of the next window, while the late pipeline waits until the next window closes to load.

Configuring a Load Throttle

Load throttles are set on the destination connection that your pipelines load to.

Edit your destination connection and go to the Schedules tab. You can configure the window, as shown in the screenshot below, in the Load Throttling section.

A screenshot of the load throttle setting modal in the Etleap UI

Default Load Schedule

This option disables load throttling. Pipelines extract and transform according to their update schedule, and loads occur as soon as the transformed data is ready.

Custom Period

This specifies the maximum duration of the load window only. New windows will regularly open at this interval, and may close early if all pipelines are ready, but will be open no longer than the duration specified.

For example

  • If the period is set to 6 hours, then there will be exactly 4 load windows per day, and each window may be up to 6 hours long.
  • If the period is set to 9 hours, then there will be exactly 8 load windows every three days, and each window may be up to 9 hours long.

Custom Schedule

This specifies the start time of each load window. The maximum duration of the load window will be the time between each occurance of the specified time.

For example

  • If the schedule is set to daily at 01:00 UTC, then the window will start at 01:00 UTC each day and maximum duration of the window would be 24 hours.

Best Practices

The synchronization that load windows provide is effective within each destination connection.

Therefore, Etleap recommends that you have one connection in Etleap per cluster. Otherwise, there may be more than one period of cluster activity for each load window.

Key Considerations

  1. Load throttling does not affect pipelines that are dependencies for dbt schedules.
  2. Load throttling does not affect how often a refresh of a pipeline will load.
  3. Pipeline may load more than once in a load window if the first batch was non-exhaustive.
  4. Failed loads are immediately retried until they succeed.
Pipeline RefreshesEnabling dbt Features