Data Partitioning

In data lakes it’s often useful to partition incoming data based on attributes in the data itself. Here’s how to do that in Etleap:

1. In the wrangler, click ‘Add Step’, and pick ‘Misc’.
2. In the ‘Operation’ dropdown pick ‘Partition Output’.
3. In the ‘Column’ input, enter the column(s) you want to partition by. Multiple columns can be specified as a comma-separated list.

4. Click ‘Add’ and ‘Next’.
5. Pick your S3 Data Lake destination.
6. On the following page, you will see that in the S3 path template there’s a placeholder for the partition column in the S3 path, which will be replaced by the partition value. Note that this placeholder comes just after the pipeline version indicator and before the load time partitioning.

By default, Etleap adds up to 2 extra partitions:

1. _loaddate which is the date when data was loaded, in ISO8061 basic format

2. _id_deleted, which is enabled only for Parquet Output, and is true when the primary keys are deleted from source, when we support extracting deletes (e.g. Salesforce or Database CDC).