Search documentation...

K
ChangelogBook a demoSign up

Embedded Destination

Build a custom connector that looks and feels just like a native destination

If you need to connect to a destination Hightouch doesn't support yet, we recommend using the HTTP Request destination. Please also —we're always gauging interest in new destinations.

Overview

The Embedded Destination framework allows Hightouch users to develop their own custom data connector and use it as an extension of Hightouch's existing library.

If you are looking to send data to a private API or a SaaS tool that Hightouch doesn't support, you can use Hightouch to send data to your own connector in a specific format, while continuing to use all of our alerting, reliability, and error handling systems.

To implement the destination, Hightouch will need users to provide information on what objects, modes, and fields are available in the destination, as well as how data is synced.

Example destinations

Our team has produced some example destinations for reference.

  • Slack: Sends messages formatted with Liquid to a Slack channel as rows are added to the query results
  • Mailchimp: Adds and updates contacts in a Mailchimp audience list

Setup

Embedded destinations in Hightouch are implemented by providing information of two steps:

  • Sync configuration: Hightouch needs information on what objects, modes, and fields are available, as well as custom forms
  • Sync execution: Hightouch sends data that needs to be synced to a method on your connector, which should format and send the data to the destination and return a success or error response

Sync configuration

Hightouch uses the following methods to render the configuration form.

These methods can be optionally implemented and used in any combination:

  • list_objects: Retrieves a list of objects available in the destination (for example, users, organizations, etc.)
  • list_modes: Retrieves a list of modes available in your implementation (for example, upsert, update, mirror, etc.)
  • list_fields: Retrieves a list of object fields, including identifiers and custom fields
  • list_options: Retrieves a JSON schema of custom configuration fields, which will be rendered into a form based on react-jsonschema-form
  • validate: This method is called when a user attempts to save the configuration form, and is used to display errors that aren't handled automatically by default or by JSON schema

Example 1: Objects, modes, fields, and options

When all the methods are implemented and all the concepts are used in conjunction, there is a hierarchy where objects must be selected first before modes are shown, and modes must be selected before fields and options are shown.

This is because the mode options shown are conditional on the object selected, for example, "upsert" and "update" can be enabled for the "user" object, but only "update" is enabled for the "organizations" object.

This will render a form similar to the example above.

Example 2: Options only

When only list_options is implemented, Hightouch will render a form based on the JSON schema. Note that all formats such as dropdowns, arrays, text areas are available. In this case, only the list_options method is implemented.

The other methods such as list_objects aren't implemented, and therefore selecting an object is not required.

Sync execution

The Embedded Destination acts as a transformation and parsing layer between Hightouch and any API. Hightouch calls the following methods when a sync is run, either by schedule or manual run.

When a sync is run, Hightouch first calls for information on sync behavior, then sends sync requests of each operation (add, change, remove) with the data and configuration in batches of a specified size:

  • behavior: Retrieves behavior settings such as batch size, which operations to send, etc. for the sync run
  • add: This method is called with a batch of raw data from rows that have been added since the last sync run, as well as configuration details
  • change: This method is called with a batch of raw data from rows that have changed since the last sync run, as well as configuration details
  • remove: This method is called with a batch of raw data from rows that have been removed since the last sync run, as well as configuration details

Each of the add, change, remove methods should return a response with any errors encountered when sending the data to the destination API.

API reference

Implement the following methods in a JSON-RPC server available via a public URL.

The server can be implemented with packages such as:

  • Jayson: A JSON-RPC server for node.js
  • json-rpc: A JSON-RPC implementation written on Python

list_objects

This method should return the objects that are available to sync to in the destination API.

Request body

list_objects has no request body since it's the highest level method.

Response sample

{
  "objects": [
    { "label": "User", "id": "user", "description": "A user." },
    {
      "label": "Organization",
      "id": "organization",
      "description": "An organization."
    }
  ]
}

Properties of an object

NameDescription
idID of object option will be referenced in the configuration
labelLabel of object option shown in the dropdown
description (optional)Description of object option (not shown in the dropdown currently)

list_modes

This method should return the sync modes that are available for a specific object (or no object).

Request body

NameDescription
objectID of object selected in the configuration (can be undefined if list_object not implemented or returned an empty object array)

Response sample

{
  "modes": [
    {
      "id": "upsert",
      "label": "Upsert",
      "description": "Pushes new records and updates records that change in your source."
    }
  ]
}

Properties of a mode

NameDescription
idID of mode option will be referenced in the configuration
labelLabel of mode option shown in the radio selector
description (optional)Description of mode option shown in the radio selector

list_fields

This method should return the mapping fields that are available for a specific object and mode (or no object and mode).

Request body

NameDescription
objectID of object selected in the configuration (can be undefined if list_object not implemented or returned an empty object array)
modeID of mode selected in the configuration (can be undefined if list_mode not implemented or returned an empty mode array)

Response sample

{
  "fields": [
    {
      "id": "email",
      "label": "Email Address",
      "type": "STRING",
      "identifier": true
    },
    {
      "id": "first_name",
      "label": "First Name",
      "type": "STRING",
      "required": true
    },
    {
      "id": "last_name",
      "label": "Last Name",
      "type": "STRING",
      "required": true
    }
  ]
}

Properties of a field

NameDescription
idID of the field option will be referenced in the configuration
labelLabel of field option shown in the dropdown
description (optional)Description of field option (not shown in the dropdown currently)
required (optional)Whether the field is required to be mapped
type (optional)The type of the field (check the field types section for accepted values)

Field types

Hightouch supports the following field types. Hightouch will not automatically cast the value to this type, nor will it validate whether the data in the row is of that type. it's generally just for suggestion and display purposes.

STRING;
ENUM;
NUMBER;
BOOLEAN;
DATE;
DATETIME;
EMAIL;
OBJECT;
ARRAY;
REFERENCE;
UNKNOWN;

list_options

This method should return the schema and uiSchema of the form that you would like to render. Hightouch uses React JSON Schema Form to build a form out of a semi-standard JSON Schema.

If your schema at the highest level is an object, we will spread the properties into the configuration, how

NameDescription
objectID of object selected in the configuration (can be undefined if list_object not implemented or returned an empty object array)
modeID of mode selected in the configuration (can be undefined if list_mode not implemented or returned an empty mode array)

Response sample

{
  "options": {
    "schema": {
      "type": "object",
      "properties": {
        "channel": {
          "title": "Channel",
          "description": "Channel that messages are sent to.",
          "type": "string",
          "enum": [1, 2, 3],
          "enumNames": ["#general", "#engineering", "#marketing"]
        },
        "message": {
          "label": "Message",
          "description": "Template for messages in Liquid format.",
          "type": "string"
        }
      }
    },
    "uiSchema": {
      "message": {
        "ui:widget": "textarea"
      }
    }
  }

Properties of options

NameDescription
schemaThis property defines which fields will appear in the custom configuration form.
uiSchemaThis property defines the appearance of fields, such as converting text inputs to text areas.

validate

This method receives a JSON of the configuration and should return an array of errors. If there is no error, it should return an empty array.

Request body

NameDescription
configurationThe sync configuration as a JSON, including properties such as object, mode, mappings

Request sample

{
  "configuration": {
    "object": "user",
    "mode": "upsert",
    "identifierMapping": [{ "from": "user_email", "to": "email" }],
    "mappings": [
      { "from": "user_fn", "to": "first_name" },
      { "from": "user_ln", "to": "last_name" }
    ],
    "customMappings": [{ "from": "user_ltv", "to": "ltv" }],
    // the following properties come from custom configuration from list_options
    "archiveOnRemove": true
  }
}

Response sample

{
  "errors": ["If first name is mapped, last name must also be mapped."]
}

behavior

This method should return the operations that Hightouch should send, as well as the batch size Hightouch should send at.

Request body

behavior has no request body.

Response sample

{
  "batchSize": 100,
  "operations": {
    "add": true,
    "change": true,
    "remove": false
  },
  "skipInitialRun": false
}

add change remove

This method should send the batch of data forward to the destination API and return an array of rejected rows if any errors occurred.

Request body

NameDescription
idColumnThe column name of the ID selected as the primary key
configurationThe sync configuration as a JSON, including properties such as object, mode, mappings
rowsAn array of rows from the query results

Request sample

{
  "idColumn": "user_email",
  "configuration": {
    "object": "user",
    "mode": "upsert",
    "identifierMapping": [{ "from": "user_email", "to": "email" }],
    "mappings": [
      { "from": "user_fn", "to": "first_name" },
      { "from": "user_ln", "to": "last_name" }
    ],
    "customMappings": [{ "from": "user_ltv", "to": "ltv" }],
    // the following properties come from custom configuration from list_options
    "archiveOnRemove": true
  },
  "rows": [
    {
      "user_email": "alice@hightouch.com",
      "user_fn": "Alice",
      "user_ln": "Doe",
      "user_ltv": 0
    },
    {
      "user_email": "bob@hightouch.com",
      "user_fn": "Bob",
      "user_ln": "Doe",
      "user_ltv": 10
    }
  ]
}

Response sample

{
  "rejectedRows": [
    { "id": "alice@hightouch.com", "reason": "Couldn't insert user" }
  ]
}

Properties of a rejectedRow

NameDescription
idThe ID of the row referenced by idColumn
reasonReason for failure

Ready to get started?

Jump right in or a book a demo. Your first destination is always free.

Book a demoSign upBook a demo

Need help?

Our team is relentlessly focused on your success. Don't hesitate to reach out!

Feature requests?

We'd love to hear your suggestions for integrations and other features.

Last updated: Mar 21, 2023

On this page

OverviewExample destinationsSetupSync configurationSync executionAPI reference

Was this page helpful?