Search documentation...

K

Learn more about what data Hightouch stores to power your syncs.

What data does Hightouch need to store?

Hightouch needs access to a record of your query results in order to perform Change Data Capture, and to power features like the Debugger. If you're on the Free, Starter, or Pro tier, your data is stored in a secure, encrypted bucket managed by Hightouch. If you're a Business Tier customer, Hightouch can use a storage bucket under your control.
Hightouch stores query results and execution plans (to power Change Data Capture), as well as failed row data and request logs (to power the Debugger).
We currently support Amazon S3 and Google Cloud Storage.
If you've already run a sync after setting up a custom storage bucket, you will be unable to make further changes to your storage config. This is because changing your external storage configuration is disruptive to Hightouch syncs. If you need to make such a change, please reach out to customer support.

Managed by Hightouch

If you're on the Free, Starter, or Pro tier, your query data is stored in a bucket managed by Hightouch. Hightouch-managed buckets are hosted in Amazon S3 and are secured and encrypted at-rest.

Expiration

Data is automatically expired from Hightouch-managed buckets after 30 days.
As a result, syncs that have not run in over 30 days will need to be recreated, because Hightouch depends on files from your last sync run in order to perform Change Data Capture and to power our debugger. If you encounter this issue, you'll need to trigger your syncs with Full Resync so Hightouch can start syncing again from a known state.

Managed by customer

Business Tier customers unlock the ability to use a custom external storage bucket, hosted in your Amazon S3 or Google Cloud Storage account. Hightouch will use this bucket for all storage needs, and your data will never be stored by Hightouch at-rest.
When using a customer-managed storage bucket, Hightouch places full control over object lifecycle, security, and expiration into your hands. We will not expire objects automatically, or modify your object encryption settings. Ensure that you've configured object expiration, encryption, and access control settings according to your needs.

Amazon S3

Create your S3 bucket

In Amazon S3, create your bucket. We recommend the name <company>-hightouch
Make sure to:
  • Block all public access to the bucket.
  • Enable Amazon S3 key encryption (SSE-S3).
  • Disable bucket versioning.
  • Configure your bucket object lifecycle, to enhance security and cut down on costs.

Authenticate Hightouch with AWS

Hightouch supports authenticating with AWS using Cross-account roles (via STS AssumeRole), or with an Access Key ID / Secret Access Key that you provide. We strongly encourage you to use Cross-account roles, as it does not require Hightouch to hold any of your secrets.
To set up your Hightouch AWS credential, follow the documentation here.
Hightouch needs the following IAM actions to store and retrieve items from your bucket:
  • s3:GetObject
  • s3:PutObject
  • s3:ListObjects

Configure your bucket in Hightouch

Access the external bucket settings under Settings > Storage.
Select your AWS region, enter your bucket name, and select the AWS credentials you set up on step 2.
Once you save your settings, your new syncs will automatically start using your bucket.
Run a sync to test it out!

Google Cloud Storage

Create a bucket

We recommend the name <company>-hightouch-bucket. Copy the bucket name and save it for later. Configure your bucket object lifecycle, to enhance security and cut down on costs.

Authenticate Hightouch with Google Cloud

Hightouch supports authenticating with GCP using Hightouch-managed service accounts, or by using a service account that you control.
To set up your Hightouch GCP credential, follow the documentation here.
Hightouch needs the following IAM policy grants to store and retrieve items from your bucket:
  • storage.objects.list
  • storage.objects.create
  • storage.objects.get

Enter configuration details in Hightouch

Back in Hightouch, under Settings > Storage, enter the project name and bucket name. Select the GCP credentials you set up in Step 2.
Don't forget to click 'save'.

Testing

After you've saved your Google Cloud bucket settings in the external storage area in Hightouch, run a few syncs and visit your Google Cloud bucket to see the files that are saved there. Please contact us if you have any trouble.

    Need help?

    Our team is relentlessly focused on your success. We're ready to jump on a call to help unblock you.

    • Connection issues with your data warehouse?
    • Confusing API responses from destination systems?
    • Unsupported destination objects or modes?
    • Help with complex SQL queries?

    or

    Feature Requests?

    If you see something that's missing from our app, let us know and we'll work with you to build it!

    We want to hear your suggestions for new sources, destinations, and other features that would help you activate your data.

On this page

What data does Hightouch need to store?Managed by HightouchExpirationManaged by customerAmazon S3Google Cloud Storage

Was this page helpful?