Databases Overview
Etleap can extract data from the most popular database types. This is either done through the database’s native change data capture (CDC) mechanism or through SQL queries executed against the source tables. The table below shows the databases Etleap can extract from, and the available extraction methods.
| Database Type | Query-Based Support | Change Data Capture (CDC) Support |
|---|---|---|
| Azure SQL DB | Yes | No |
| Azure SQL Managed Instance | Yes | Yes |
| BigQuery | Yes | No |
| DB2 | Yes | No |
| MariaDB | Yes | Yes |
| Microsoft SQL Server | Yes | Yes |
| MongoDB | Yes | Yes |
| MySQL | Yes | Yes |
| Oracle | Yes | Yes |
| PostgreSQL | Yes | Yes |
| Redshift | Yes | No |
| SAP HANA | Yes | Yes |
| Snowflake | Yes | No |
Auto-Replicate Destination
When Auto-Replicate Destination is enabled for a source database connection Etleap will automatically create pipelines in the specified destination for each table in the source. Any source table that has a pipeline attached to it already will be skipped in this automatic creation.
Each automatically created pipeline will be named according to its schema and tablename: <schema-name> - <table-name>. If a pipeline with this name already exists, a number will be appended to the name, e.g. Schema - Table 2.
If Auto-Replicate Destination is enabled and a schema is specified for the source database connection, tables in that schema will be created within the schema defined on the selected destination database connection. In the example below, this is public.
If Auto-Replicate Destination is enabled and a schema is not specified for the source database connection, tables in all schemas of the source database connection will be created in corresponding schemas in the destination database connection. If corresponding schemas do not already exist in the destination database connection, these will be created.
What Data Does Etleap Store for Databases?
All data extracted by Etleap is stored in your S3 Intermediate Bucket. This data may be used by future refreshes of your pipelines.
Data from tables that are not being replicated by Etleap is never written to the S3 Intermediate Bucket or otherwise held by Etleap. This includes CDC data that is read from the source database’s binary/transaction log. Although the source database log may contain data for all tables, Etleap never reads or stores data for tables that do not have Etleap pipelines.
Limitations
LOB Truncation
All LOB values are truncated to a maximum size of 512KB for both CDC and query-based extraction methods.