Append Mode

Database and Application Sources

This is for sources where new records are created over time, but existing records never change or get deleted. New rows are extracted from the source, transformed, and appended to the source table.

In order for this mode to be applicable, the source must have a method for extracting new records. In the case of a database table where Etleap uses SQL queries to extract data this means the primary key must be monotonically increasing, e.g. an auto-incrementing integer, because the SQL query fetches data whose primary key is greater than the maximum primary key of the previous batch.

Files Change (S3 Sources Only)

For S3 sources, append mode pipelines behave differently depending on whether the source files are ever modified. You choose this option on the last page of the pipeline setup wizard.