Optional Connection Settings

This page describes additional Redshift connection settings that enable optional Etleap features.

Include Query Tags

You can add query tags to your Redshift destination. If this option is enabled, each executed load query is prefixed with a comment block containing JSON metadata. The metadata includes information about the objects involved (the pipeline, model, source, destination, etc.) and data recency.

Within the JSON metadata object, the property type refers to the Etleap object that the load query belongs to (e.g., pipeline or model). The sourceHighWatermark property provides the date and time of the last pipeline extraction or incremental model update being loaded.

The query metadata has the following schema:

{
  "type": "object",
  "properties": {
    "app": {
      "type": "string",  // Always "Etleap"
      "required": true
    },
    "version": {
      "type": "string"
      "required": true
    },
    "object": {
      "type": "object",
      "required": true,
      "properties": {
        "uuid": {
          "type": "string",
          "required": true
        },
        "name": {
          "type": "string",
          "required": true
        },
        "type": {
          "type": "string",
          "required": true,
          "enum": [ "PIPELINE", "MODEL" ]
        },
        "createDate": {
          "type": "string",  // UTC ISO8601-formatted, i.e. yyyy-mm-dd'T'hh:mm:ss'Z'
          "Required": true
        },
        "owner": {
          "type": "string", // Full name of the owner of the pipeline/model
          "required": true
        }
    },
    "source": {
      "type": "object",
      "required": false,  // Only present if properties.type == "PIPELINE"
      "properties": {
        "uuid": {
          "type": "string",
          "required": true
        },
        "name" : {
          "type": "string"
          "required": true
        }
      }
    },
    "destination": {
      "type": "object",
      "required": true,
      "properties": {
        "uuid": {
          "type": "string",
          "required": true
        },
        "name" : {
          "type": "string"
          "required": true
        }
      }
    },
    "loadType": {
      "type": "string",
      "required": true,
      "enum": [ "FULL", "INCREMENTAL_APPEND", "INCREMENTAL_UPDATE" ]
    },
    "sourceHighWatermark": {
      "type": "string",
      "required": false // Not present when loadType == "FULL"
    },
    "sourceHighWatermarkType": {
      "type": "string",
      "required": false,
      "enum": [ "STRING", "TIMESTAMP" ]
    },
}

Enable Dynamic Varchar Widths

When string columns do not have the width explicitly set, Etleap defaults the string width to VARCHAR(65535). With the dynamic varchar widths feature enabled, Etleap will instead load VARCHAR columns with the minimal required width based on the data it’s loading, and expand the column width as required. This can improve your query performance. If you’re considering turning this option on, there are a few important items for you to consider below.

Key Considerations

There are some important aspects to consider before enabling this feature.

• Redshift’s ALTER COLUMN ... TYPE does not support varchar width changes when tables have dependent views. As a result, connections with this option enabled do not support dependent (non-late-binding) views, which will block pipeline loads. As an alternative, we recommend you use late-binding views  or Etleap materialized view models.
• This option cannot be disabled. If you’d like to stop using it, we recommend creating a new Redshift connection.
• If turned on for an existing Redshift connection, the dynamic widths feature will only apply to new pipelines and existing pipelines (only after refreshes).
• Dynamic varchar widths are not supported for subtable string columns.
• Primary keys without the explicit string width will still be set to VARCHAR(65535).