Integrating Data and Automation

Three directions

Every integration question resolves to one of three directions. Knowing which one you are in tells you which mechanism to reach for.

The single most common mistake is reaching for the wrong direction: Kafka, Amazon SQS and Amazon SNS are outbound destinations only. They are not ways to send data in. Ingress is HTTPS.

Between: how Data and Automation connect

If you have both products, this wiring already exists — you do not build it.

Every record type you publish arrives in Automation as actions. Create, read, update, delete and list are registered automatically for each published record type. There is no mapping step to get started and nothing to keep in sync by hand; publishing is what registers them.

A record change can trigger a workflow. Bind a workflow to created, updated, deleted or version_promoted, and add a filter so only the records you care about start a run.

A running workflow can read the record fresh. This is the part worth designing around: a step reads the record's current state mid-run, not the snapshot that triggered it. A claim approved three days after submission is evaluated against what the claim says now.

Results write back. Action outputs map onto record fields, and the record shows the workflow state it is in, its run history, and a funnel you can click through.

Not using SynaptaGrid Data? Everything above still works against your own systems — you register them as external systems and map fields the same way. You lose only the automatic action registration.

In: getting data and triggers into the products

The REST API

Every published record type has its own REST surface — see The generated API. Use POST /flat/ for one record and /flat/bulk for many. A bulk insert charges the whole batch against your row quota, so it is not a way around a limit.

Signed inbound webhooks

For an external system that should start a workflow, register it and post to its inbound endpoint. The call is signed with a per-source secret you can rotate, and is idempotent on the event identifier — a retry on your side does not start a second run.

This is the right choice when the other system is the one that knows something happened.

The client API

A tenant API key scoped to particular workflows can start runs and read execution status directly. Use this when your own code is orchestrating.

Schedules and manual starts

A workflow can run on cron, at a fixed interval, once at a set time, or when someone starts it by hand from the portal.

Out: telling your systems what happened

There are two separate event streams, and conflating them is a common source of confusion.

Record events (from Data)

Enabled per schema. Subscribe to:

entity.created · entity.updated · entity.deleted · entity.version_promoted

Updates carry a field-level difference, so a consumer can apply exactly what changed rather than re-fetching and diffing.

Workflow events (from Automation)

Subscribe to what happens inside a process:

• Runs — started, completed, failed, cancelled, timed out, transitioned
• Steps — entered, completed, failed, and the branch taken
• Tasks — created, assigned, completed, signed
• External calls — completed

Where both go

Four destinations, so you rarely need to stand up a receiver:

Delivery you can trust

• Payloads are signed. Verify before acting — an endpoint that accepts anything posted to it is an endpoint anyone can drive.
• Failures retry on a backoff, then dead-letter, with bulk redrive from the console once your receiver is healthy.
• A delivery log records every attempt with per-attempt detail and aggregate statistics.
• Alerts fire on dead-lettering and on a failure rate crossing a threshold.

Make your handler idempotent. Delivery is at least once. Key on the event identifier and ignore one you have already processed.

One guarantee worth relying on

Record events are written in the same transaction as the data change, then dispatched. The event and the data cannot drift: if the write succeeded, the event exists. You do not need a reconciliation job to catch events that "didn't fire".

API keys

Keys are how everything outside your signed-in users authenticates. Both products issue them, and they are scoped differently.

In Data

Mint, list, rotate, disable and revoke keys yourself. A key can be:

• scoped to particular record types, so an integration sees only what it needs
• given its own per-minute and per-hour request limits, within your organization's overall budget

Key holders are always served the published version, never your working draft. You cannot accidentally expose in-progress modelling to an integration.

In Automation

Keys can be scoped to particular workflows, so a system that should start one process cannot start another. Separately, each registered external system has its own inbound secret for signed webhooks, revealed and rotated independently of your API keys.

Choosing the right credential

Rotating without downtime

Rotate rather than delete. Rotation gives you a window where both the old and new key work, so you can deploy the new one before retiring the old. Deleting a key strands whatever was using it.

A worked example

Claims arrive from a broker's system, get reviewed, and the outcome goes back to finance.

1. In — the broker posts to the signed inbound endpoint. Idempotent on the event id, so their retries are safe.
2. Store — the workflow's first step creates a claim record through the generated create action.
3. Between — a later step reads the claim fresh before deciding, so an adjustment made while the run was waiting is picked up.
4. Human — the approval step pauses; over a threshold it requires re-authentication and a stated reason, producing a chained signature record.
5. Out — finance subscribes to task.signed and workflow.completed on their SQS queue; their handler is idempotent and verifies the signature.
6. Also out — the data warehouse subscribes to entity.updated and applies the field-level difference.

Two events, two audiences, one process — and the audit trail answers who approved it and when.

Where to go next

• The generated API — the REST surface in detail.
• Getting changes out — record events specifically.
• Connecting a system to a workflow — turning an API into steps.