Skip to Content
DocumentationEventsKafka

Kafka

Kafka connections give you access to your company’s streaming event data in order to build analytics-ready tables.

Source Setup

You will need a valid user and password to set up an Etleap connection. Etleap will configure the authentication with SASL using JAAS .

1. Authentication Mechanism

Etleap supports authentication with SASL using JAAS : SASL/SSL, and SASL/SCRAM. All communication uses only TLS . Please follow this link  for more information.

2. Public CA Certificate

You will need to obtain a public CA certificate that you use to sign all broker’s certificates when setting up a connection. Please follow this link  for more information.

3. Schema Registry Setup

The Schema Registry is optional and should only be configured if you use the Confluent Schema Registry  to store your metadata. The following schema formats are supported: JSON, Protobuf, and Avro.

Note

Prefer using an API? Go here  and select Kafka under the Body header to create a connection via API.

What Data is Available?

You can create pipelines using the Kafka topics available on your server. Etleap automatically detects and lists these topics for you.

Note

Etleap uses Append mode for all pipelines.

Schema Formats

When using a Schema Registry, Etleap parses events according to the provided schema.

JSON

  1. Given the following JSON schema :

    {
  "type": "object",
  "properties": {
    "f1": {
      "type": "string"
    },
    "f2": {
      "type": "object",
      "properties": {
        "field": {
          "type": "integer"
        }
      }
    }
  }
}

  2. Etleap produces the following relational structure:

    topic VARCHAR,
key VARCHAR,
timestamp TIMESTAMP,
f1 VARCHAR,
f2_field: INTEGER

Protobuf

  1. Given the following Protobuf  schema:

    syntax = "proto3";
package com.acme;
 
message MyRecord {
  string f1 = 1;
}

  2. Etleap will produce the following relational structure:

    topic VARCHAR,
key VARCHAR,
timestamp TIMESTAMP,
f1 VARCHAR

Avro

  1. Given the following Avro  schema:

    {
  "type": "record",
  "namespace": "com.acme",
  "name": "Record",
  "fields": [
    { "name": "f1", "type": "string" },
    { "name": "f2", "type": "boolean" }
  ]
}

  2. Etleap will produce the following relational structure:

    topic VARCHAR,
key VARCHAR,
timestamp TIMESTAMP,
f1 VARCHAR,
f2 BOOLEAN

Key Considerations

  • Etleap supports only one topic per connection.
  • Etleap supports parallel consumption of messages.
  • Etleap supports communication over secure channels only.
Event StreamSQS