Snowflake Data Connector

The BigPanda Unified Data Connector (UDC) can extract tables from a private Snowflake listing. The connector runs read-only queries against the tables you select and syncs the results into BigPanda.

The Snowflake connector has no source-specific logic and should be customized to fit the listing mounted for your organization. Use the Snowflake data connector for teams that already replicate operational tables into Snowflake and share them as a private listing, and want to schedule extracts from those tables instead of calling the original source API.

Read-only connector

This connector only reads data. It does not mount listings, and it does not change grants or privileges in Snowflake. The listing must already be mounted for your organization, and the role the connection uses must already be able to query it.

Prerequisites

Before you configure the connector, make sure the following are in place:

A Snowflake private listing is already mounted for your organization.
A Snowflake role that can query the mounted listing. This connector does not create or modify grants.
Key-pair credentials for the Snowflake connection.

Authentication

The Snowflake connection uses key-pair authentication. Provide the following connection details.

Field	Description
Connection URL	The Snowflake account URL. The account identifier is read from this URL.
`user`	The Snowflake user for the connection.
`private_key`	The private key used for key-pair authentication.
`key_passphrase`	Optional. The passphrase for the private key. Provide it only when the key is encrypted.
`auth_role`	The Snowflake role the connection uses. This role must already be able to query the mounted listing.

Connection configuration

Set the warehouse, database, and schema on the connection. The connector uses this schema for every table in the connection.

Field	Description
`warehouse`	The Snowflake virtual warehouse the connector uses to run queries.
`database`	The Snowflake database that contains the tables to extract.
`db_schema`	The schema within the database. The connector uses this schema for every table.

One connection maps to one database and one schema. To extract from a second schema, create a second connection.

Table configuration

Set tables with at least one entry. Each table requires the cursor_column, and several optional settings.

Setting	Description
`cursor_column`	Required. A datetime column the connector uses to track sync progress.
`included_columns`	Optional. The list of columns to extract.
`excluded_columns`	Optional. The list of columns to omit.
`filter`	Optional. A static SQL boolean expression that is added to the query with `AND`. For example, `active = TRUE`.
`related_tables`	Optional. A list of `<child_table>:<join_column>` strings that attaches related child rows to each parent row. See the Related tables section below.

Column name matching

For standard unquoted Snowflake columns, the cursor_column, included_columns, and excluded_columns settings are case-insensitive. For example, SYS_UPDATED_ON and sys_updated_on refer to the same column. Columns created as quoted mixed-case or lowercase identifiers in Snowflake are not matched by these settings. The filter setting is the SQL text you write and is not adjusted.

Sync behavior

Choose how the connector pulls data with the mode setting, and control the sync window and batch size with the settings below.

Option	Description
`mode`	Set to `incremental` (default) or `historical`.
`start_date`	Format `YYYY-MM-DD`. Required for historical mode. In incremental mode, used as the starting point on the first run if you set it.
`end_date`	Format `YYYY-MM-DD`. Historical mode only. The sync includes the whole calendar day named by `end_date`.
`chunk_size`	Optional. The number of rows per batch. Defaults to `50,000`.

Incremental mode resumes from the last recorded cursor value. On the first run, when there is no recorded cursor yet, it starts from start_date if you set one, and otherwise from 1970-01-01.

Both start_date and end_date must use the YYYY-MM-DD format. A full timestamp is not accepted.

When you configure related_tables, related rows are attached after the parent rows are read. This does not change the incremental cursor, which continues to track the parent table.

Output schema

The connector produces one output table for each entry in tables, named after the source Snowflake table. Each output table adds one column:

_dlt_processing_time: The UTC timestamp of the extraction.

By default, files land in the snowflake_<connector_id> folder. When servicenow_data is true, files land in servicenow_<connector_id> instead.

When you configure related_tables, each parent row also includes one rel_<child_table> array per configured relationship, where the child table name is lowercased.

Each child row in the array includes every column from that child table, with lowercase field names. Related rows stay on the parent record and are not written as their own output table. There is no limit on how many child rows can appear in an array. A parent row with no sys_id value returns an empty array. When servicenow_data is true, each child field uses the ServiceNow field shape.

Rows are appended, and there is no primary key. A row that falls exactly on the cursor boundary may reappear on the next run, so plan to deduplicate downstream.

Data redaction

Redaction is optional and uses the same connection-level redaction fields as other data connectors. For detailed options on protecting sensitive data, see the Data Redaction with UDC documentation.

Troubleshooting

You may see the following errors or warnings.

Condition	How to resolve
Historical mode is selected without a `start_date`	Provide a `start_date`.
`end_date` is before `start_date`	Set `end_date` on or after `start_date`.
`tables` is empty	Add at least one table entry.
`chunk_size` is not a positive number	Set `chunk_size` to a positive integer.
A date is not in `YYYY-MM-DD` format	Use the `YYYY-MM-DD` format. A full timestamp is not accepted.
`warehouse`, `database`, or `db_schema` is missing	Set all three on the connection.
The run reports that a column cannot be found	For a standard unquoted column, letter case does not matter. Columns created as quoted mixed-case or lowercase identifiers in Snowflake are not matched.
A related-table entry is skipped with a warning	The entry is malformed, for example, it is missing the :, or has a blank table or column name. Correct the entry. The parent extract and any other valid relationships continue to run.
A table's run fails with an error that the object does not exist	A well-formed related_tables entry names a child table or join column that does not exist in Snowflake. Correct the name. This failure stops the entire run for that table entry, even when the parent window returns zero rows.
The run fails before extraction because two relationships collide	Two `related_tables` entries produce the same `rel_<child_table>` field name after lowercasing. Use child tables whose lowercased names are distinct.
The run fails before extraction because of a missing `sys_id`	You set `related_tables`, but the parent table has no `sys_id` column, `included_columns` omits `sys_id`, or `excluded_columns` includes `sys_id`. Make sure the parent table exposes `sys_id`.
Files are missing from the `snowflake_<connector_id>` folder	If servicenow_data is true, files land in `servicenow_<connector_id>`
An update fails with The ServiceNow-data setting cannot change after the connector has started.	The setting is locked because a table on the connector has saved a read cursor.

In this section: