Warehouse Planning

Overview

Between syncs, Hightouch will automatically identify the incremental changes in your data model, so that only updates (added, changed, or removed records) are sent to destinations. By default, Hightouch figures out what rows need to be synced by pulling all the rows within your data model, and then diffing on Hightouch's infrastructure. See more info on this here

This can be slow for large Models that take a long time to query.

With Warehouse Planning, however, Hightouch does this diff directly in your warehouse by storing previously synced data into a special schema managed by Hightouch. Our larger enterprise customers are typically using our in-warehouse diffing, for performance reasons.

This functionality currently supports the following warehouses:

Snowflake
BigQuery
Redshift
Postgres
Databricks

Granting Required Warehouse Permissions

Warehouse Planning stores metadata about syncs in your warehouse within a schema called hightouch_planner. The user that Hightouch uses to connect to your warehouse must have permissions to create this schema, and create tables within it.

Snowflake

Run the following SQL snippet

GRANT CREATE SCHEMA ON DATABASE <your_database> TO <hightouch_role>

BigQuery

Grant the bigquery.datasets.create permission to the Hightouch user.

Redshift / Postgres

Run the following SQL snippet

GRANT CREATE ON DATABASE <your_database> TO <hightouch_user>

Precreating the Warehouse Schema

If you don't want to grant Hightouch access to create schemas in your warehouse, you may also precreate the schema and just give Hightouch access to create and delete tables within it.

BigQuery

To enable this for BigQuery:

Create the schema by running the following SQL
```
CREATE SCHEMA hightouch_planner
```
Give your GCP service account access to run jobs and list the schema by granting it the following roles:

roles/bigquery.jobUser
roles/bigquery.metadataViewer

Give your GCP service account access to manage tables within the hightouch_planner schema by running the following SQL

GRANT `roles/bigquery.dataViewer`, `roles/bigquery.dataEditor`
ON SCHEMA hightouch_planner
TO "serviceAccount:$USER@$PROJECT.iam.gserviceaccount.com"

Enabling Warehouse Planning For Your Syncs

To enable Warehouse Planning in Hightouch, make sure that you've granted the proper permissions for one of our supported sources (see above), then navigate to your respective source and click the "Enable" button within the Warehouse Planning section:

Need help?

Our team is relentlessly focused on your success. We're ready to jump on a call to help unblock you.

Connection issues with your data warehouse?
Confusing API responses from destination systems?
Unsupported destination objects or modes?
Help with complex SQL queries?

Feature Requests?

If you see something that's missing from our app, let us know and we'll work with you to build it!

We want to hear your suggestions for new sources, destinations, and other features that would help you activate your data.