Skip to Content
DocumentationRedshiftDestination Settings

Redshift Destination Settings

This page describes all of the destination options available when using Redshift as a pipeline destination in Etleap.

These options are shown at the last step of creating a pipeline.

loading 90-Redshift-StartEtling.png...

Primary Key Mapping

Choose the column(s) that represent the unique identifier for a record in the destination. You may need to change this if you added transformations in the Wrangler that rename or alter the primary key column(s). Etleap uses the primary key for deduplicating data when loading. This option is shown if Etleap detects that there are primary keys in the source. Specifying primary keys is required to create pipelines in Append, Update and Update with History Retention mode.

Last Updated Mapping

Choose the column that represents when a record was last updated in the source. You may need to change this if you added transformations in the Wrangler that rename or alter the last-updated column. Etleap uses the last-updated mapping for ensuring that it doesn’t load outdated data and for retaining history if Retain History is checked. This option is not available if Etleap does not detect any timestamp columns or if the source entity does not support Update Mode.

Retain history

If checked, Etleap retains rather than overwrites updated rows. See Retain History

Compress columns

If checked, activates automatic compression  on the destination table.

Truncate strings

If checked, Etleap automatically truncates long strings to Redshift’s maximum string size of 64KB. If unchecked, records that exceed 64KB will treated as a parsing error.

Checking this box is convenient if you have many large fields you don’t try to parse. Leave this unchecked if you want to be alerted more visibility when there are large fields. For example, when the source has large JSON fields that could be broken into smaller fields by Etleap.

Automatic schema changes

If checked, Etleap automatically adds and removes columns in the destination table as the schema changes in the source. If unchecked, new or removed columns trigger a notification, and you manually confirm whether to apply those changes.

Distribution Style

Set the distribution style  of the table. You can leave this as the default and change it later if you want improved performance for particular queries.

If you choose key distribution style, you must also select the Distribution Column.

Sort Columns

Add the sort column(s)  if desired. You can leave this as the default and change it later if you want improved performance for particular queries.

loading Redshift-sort-columns.png...

Quality Check

When quality checks are enabled for a pipeline, Etleap will wait to receive a QUALITY_CHECK_COMPLETE event before loading data to the destination.