Search documentation...

K
ChangelogBook a demoSign up

Google Cloud Storage

Transfer data at scale from your warehouse to Google Cloud Storage

Supported syncing

Object TypeDescriptionSupported Sync Modes
Any data setSync data from a source to Google Drive as JSON, CSV, or Parquet filesInsert, All, Diff
  • All: All mode creates one file with all the rows in the query results, every time the sync runs.
  • Insert: Insert mode creates one file with the rows that were added since the last sync.
  • Diff: Diff mode creates three files, one for rows added, one for rows changed, and another for rows removed since the last sync.

For more information about sync modes, refer to the sync modes docs.

Connect to Google

Go to the Destinations overview page and click the Add destination button. Select Google Cloud Storage and click Continue. You can then authenticate Hightouch to Google Cloud Storage by entering your:

  • GCP Credentials: if you need to create a cloud credential, consult our documentation
  • GCP Project ID: you can follow these steps to find it
  • GCS Bucket Name: this be should just be the name of the bucket, not a URL

Sync configuration

Once you've set up your Google Cloud Storage destination and have a model to pull data from, you can set up your sync configuration to begin syncing data. Go to the Syncs overview page and click the Add sync button to begin. Then, select the relevant model and the Google Cloud Storage destination you want to sync to.

Select file format

Hightouch supports syncing JSON, CSV, and Parquet files to Google Cloud Storage.

Enter filename

The filename field lets you specify the parent directory and the name of the file you want to use for your results. You can include timestamp variables in the filename, surrounding each with {}. Hightouch supports these timestamp variables:

  • YYYY: Represents the full year in four digits.
  • YY: The last two digits of the year.
  • MM: Two-digit month format (01-12).
  • DD: Two-digit day format (01-31).
  • HH: Two-digit hour format in 24-hour clock (00-23).
  • mm: Two-digit minute format (00-59).
  • ss: Two-digit second format (00-59).
  • ms: Three-digit millisecond format.
  • X: Unix timestamp in seconds.
  • x: Unix timestamp in milliseconds.

All dates and times are UTC.

For example, you could enter upload/{YYYY}-{MM}-{DD}-{HH}-{mm}-result.json to dynamically include the year, month, date, hour, and minute in each uploaded file. Hightouch would insert each file in the upload directory, which would need to already exist in your bucket.

You can also use other variable values to include sync metadata in the filename:

  • {model.id}
  • {model.name}
  • {sync.id}
  • {sync.run.id}

If a file already exists at the path you entered at the time of a sync, Hightouch overwrites it.

If you are using an audience and would like to include the audience name, you will still use {model.name}.

Set filename offset

By default, Hightouch uses the timestamp of the sync run to fill in timestamp variables. You can optionally include an offset in seconds. For example, if you want the filename's date to be 24 hours before the sync takes place, enter '-86400' (24 hours * 60 minutes * 60 seconds).

Columns to sync

You can export all columns exactly as your model returns them or choose to export specific ones.

If you need to rename or transform any column values you're syncing, you can use the advanced mapper to do so. If you choose this option, Hightouch only syncs the fields you explicitly map.

Exporting and remapping select columns in your model

The preceding example shows how to selectively export the company, email, name, and price fields. Hightouch exports these fields to new fields in the file and ignores all other columns from your results.

CSV options

If you're syncing to a CSV file, you have additional configuration options:

  • Delimiter: Your options are comma (,), semicolon (;), pipe (|), tilde (~), and tab
  • (Optional) Whether to include a CSV header row in the exported files
  • (Optional) Whether to include a byte order mark (BOM) in the exported files, the BOM is <U+FEFF>

Empty file results

You can select how Hightouch should handle empty results files. Empty result files can occur if your model's query results haven't changed since the last sync.

You can select whether to skip empty files. If you skip empty files, it means Hightouch won't export any files if your model's query results haven't changed since your last sync.

Tips and troubleshooting

Common errors

To date, our customers haven't experienced any errors while using this destination. If you run into any issues, please don't hesitate to . We're here to help.

Sync alerts

Hightouch can alert you of sync issues via Slack, PagerDuty, SMS, or email. For details, please visit our article on alerting.

Ready to get started?

Jump right in or a book a demo. Your first destination is always free.

Book a demoSign upBook a demo

Need help?

Our team is relentlessly focused on your success. Don't hesitate to reach out!

Feature requests?

We'd love to hear your suggestions for integrations and other features.

Last updated: Jan 19, 2024

On this page

Supported syncingConnect to GoogleSync configurationSelect file formatEnter filenameSet filename offsetColumns to syncCSV optionsEmpty file resultsTips and troubleshootingCommon errorsSync alerts

Was this page helpful?