User-Defined APIs

User-Defined API connections enable you to ingest data from any API you have access to that meets certain criteria.

Each connection specifies one or more entities, each typically representing an API resource, from which pipelines can be created. You can create pipelines in Replace or Update mode depending on whether you specify that there’s a way to fetch new data during the connection setup.

API Requirements

APIs are designed in a wide variety of ways. To use an API with Etleap’s User-Defined API source connector, it must meet the following criteria:

A server that returns JSON data via an HTTP(S) GET or POST request, with the desired records as a JSON array.
Authentication via one of the supported methods.
If the API requires paging through results, it uses one of the supported pagination strategies.
If Etleap pipelines should extract new and updated data incrementally (i.e. pipelines should be in Update mode) there are additional requirements.

Note

Do you have an API that doesn’t meet these requirements? Contact us and we’ll try to help.

Source Setup

The following subsections outline the parameters you will need when going through the wizard that guides you through the connection setup. To access the setup wizard, go to Connections → User-Defined API.

Replace mode summary — The last step of the wizard for a Replace mode entity, showing the configured parameters.

URL and Parameters

The basis of a User-Defined API connection entity is a URL that Etleap will make requests to in order to retrieve data records. You can specify additional query parameters and headers that will be included with every request in the Parameters section. Additional query parameters and headers, specified in later steps, will be added to these requests depending on the type of request that’s being made, e.g. retrieving a sample, fetching subsequent pages, or fetching the high watermark.

If you have selected POST as your request method, you will have the option to add your body parameters in the Parameters section.

Authentication

The following authentication methods are currently supported:

Basic: A username and password are combined, encoded, and passed in an ‘Authorization’ header value prefixed by ‘Basic’. More details here .
Bearer: An access token is passed in an ‘Authorization’ header value prefixed by ‘Bearer’. More details here .
Header: Authenticates using a header key and value that you specify.

Pagination

Cursor and Offset strategies are supported for APIs that require pagination.

Cursor

Each page returns a pointer to the next page. The pointer could either be a partial or a full URL. This pagination strategy takes the following parameters:

JMESPath to the Paging Cursor

This value points to the cursor field in the response. For example: if your response includes a paging.next attribute, select that here.

Next Page URL

This is the URL of the next page of data that will be retrieved. If your Paging Cursor does not include a full URL, you will need to include the portion of the URL that precedes the Paging Cursor in the Prefix field.

The full URL will be defined as a concatenation of the Prefix and the value in the Paging Cursor. For example, with the following response you would use a Prefix of https://api.example.com/marketing/v12/.


  "paging": {
    "next": {
      "after": "forms?queryParam1=8&after=Mw%3D%3D",
      "link": "https://api.example.com/marketing/v12/forms?queryParam1=8&after=Mw%3D%3D"
    }
  }

Pipeline Mode

New data or not — Choosing whether to create a Replace or Update mode entity.

By default, pipelines created from your entity will operate in Replace mode. If you would like Etleap to extract new and updated data incrementally (i.e. in Update mode) there are two strategies available: Extracted Data Update Mode and Wall Clock Update Mode.

Note

If your selected update mode uses required parameter(s) that were also set on the Query Paramters page, the values set on that page will only be used during sampling in the entity creation wizard. During operation, any pipeline created from this entity will override these parameters to reflect the values needed for your pipeline to function in the selected update mode.

Extracted Data Update Mode

The Extracted Data Update Mode allows for obtaining updated records based off of values, such as last_updated_timestamp, that exist in the data obtained directly from the API.

To enable an entity to use Extracted Data Update Mode, you will need the following:

Record ID and update timestamp fields: The names of the fields in the data records that represent the unique record identifier and the update timestamp (in ISO 8601 format), respectively.
Watermark query parameters: The query parameters that must be provided as part of the URL in order to return data in descending order of record update timestamp. This will be used to fetch the data high watermark.
Updated data query parameters: The query parameter that must be provided as part of the URL in order return data that has been created or updated since a given timestamp. You will get to choose the date format of the parameter - see screenshot below.

Begin and end parameters — Specifying the query parameters and values, as well as the date format, for querying new and updated data in a time range.

Additional Entities

Once you have created your connection and first entity, you can add additional entities, which will share the connection’s authentication parameters only. You can specify additional entities’ parameters from scratch, or you can pre-fill the parameters from an existing entity in order to make the setup quicker.

Additional entity — Using an existing entity (Accounts) as a template for a new entity (Subscriptions)