Git Sync

There are two distinct git repositories that Hightouch communicates with. Git sync (this feature) is used to version-control Hightouch config, which is entirely independent from the Git repository used to leverage dbt models.

Overview

Git Sync is a bi-directional integration between your Hightouch workspace and a git repository. This brings all the great features of git to your Data Activation workflows: commit logs of incremental changes, the ability to roll back to a previous state, and the ability to use code to create and edit Hightouch syncs and models

When enabled, any changes made to Models or Syncs within the Hightouch app will collapse to .yaml files and commit changes to your designated repository (in Github, Bitbucket, or Gitlab).

This also integrates the opposite direction, where any changes made directly to the .yaml files (or the creation of new .yaml files altogether) will update in the app as if they were completed in the UI.

This is an automated and continuous integration, so within about 60 seconds any changes made in Hightouch will push to Git automatically (marked as OUTBOUND), and any changes made in git will push directly back into Hightouch (marked as INBOUND).

Setup

To get started, you need a few prequisites:

Create a Git Repository, if you don't already have one to store your schema files
Authenticate to Git
Enable Git Sync for your workspace

Authenticate to Git

If you don't already have a git repository created, the first step is to create one. If you create an empty repository, make sure you follow the instructions to also create your first example commit, e.g. a README file, so that a branch exists.

Once you have a repository, fill in your Git Credentials on the integrations tab in settings.

For Github, Github OAuth is the easiest way to get started. You'll also need to configure permissions to the repository, selecting the particular repo Hightouch should have access to.

Enable Git Sync

Next, you'll need to enable Git Sync for your workspace. Head to the Git Sync tab on the settings page, pick the repo and branch where you'd like changes saved to and read from.

Optionally, select a Path where the sync and models folders should be created, for example, here we're saving our settings to the hightouch subfolder.

Hightouch will create folders for syncs and models within this subfolder, or in the root of your Git repo if no path is selected.

After clicking save, Hightouch will create the subfolder if needed and start writing your models and syncs to your repo.

From here, you're all set. Future changes in the UI will be written in the repo, and if you create any new models or syncs in Git, they will be also be created in Hightouch.

As a note, deletions from Git are currently not supported. If you delete a file in Git, the corresponding model or sync will still remain in Hightouch. The file won't be rewritten to Git until the next change in Hightouch, unless a full resync is triggered. We are currently working on supporting soft deletes in order to make deletions in Git more safe!

Schemas

There are two schema files: models and syncs.

Models have a consistent schema.

name: > the model name/slug. unique per model
source: > the name/slug of the source. the slug is visible on the source page
type: > one of: raw_sql, table, or dbt_model
dbtModel: > null or the name of the dbt model to select, written as `model.<package_name>.<model_name>`
rawSql: > null or the sql to use, e.g. select * from albums
tableName: > null or the table name to use
isSchema: > used internally by Hightouch, this should always be false
primaryKey: > primary key column for this model, e.g. album_id

The schema for syncs is destination dependent. For this reason, we recommend creating syncs within the UI initially, and then using the written schema to create new syncs or modify existing ones. The name of the yaml is taken as the name of the sync.

model: > model name/slug for this sync.
destination: > destination name/slug for this sync
config:
    > destination dependent config for this sync.
schedule: > type of schedule to use. the format here varies based on the
            schedule selected, we recommend using the UI as a starting point.
schedulePaused: > boolean, where this sync is paused, or enabled.

Changelog

Your git repository can be used as a changelog to track which user has created/modified which resource. It will back-populate any create/edit event from backend to your git repository, if you configure the correct email(For examples your user's hightouch email matches your github email), Github will show up the correct user who has created/edited the resources.

For example:

Tunnel

If you want to use git repository that is hosted on your on-premise server without public internet access, you can use tunnel to connect to your git server, and Hightouch will use the tunnel to connect to your git repository.

Go to Settings -> Tunnels, and create a tunnel or reverse tunnel that connect to your git server. Follow docs here.
Go to Settings -> Integration -> Git Credentials. Add your credential and select the right tunnel to use. Note: It is recommended to choose SSH protocol and ssh_privatekey to connect to git repository, as basic auth has some problems forwarding auth header through ssh tunnel and HTTP redirects. You can find more detail here on how to use ssh protocol to authenticate. Currently we don't support ssh_privatekey with passphase.

Next steps

From here, you have the full power of Git in your hands. Here's some potential ideas to try next:

Create a new model by copying an existing model and changing the name and query
Create a new model and sync at the same time

Need help?

Our team is relentlessly focused on your success. We're ready to jump on a call to help unblock you.

Connection issues with your data warehouse?
Confusing API responses from destination systems?
Unsupported destination objects or modes?
Help with complex SQL queries?

Feature Requests?

If you see something that's missing from our app, let us know and we'll work with you to build it!

We want to hear your suggestions for new sources, destinations, and other features that would help you activate your data.