Delivery & Retries
How Orqestra delivers each event, what counts as success, how failures are retried, and what your receiver needs to handle.
Responding to a delivery
Answer with any 2xx status within 10 seconds. That time covers the
connection, the request and your response together. Your response body doesn't mean anything to
Orqestra, and redirects are not followed.
If your processing might take longer, store the event first (a database row or a queue message),
answer 200, and do the work afterwards. Answering first and processing in memory loses
the event if your process restarts in between.
The first 16 KiB of your response body, along with its Content-Type,
Retry-After, X-Request-Id, traceparent and CF-Ray
headers, is kept in the delivery history so you can debug. Don't put anything sensitive in a
webhook response.
What counts as success
| Your response | Outcome |
|---|---|
2xx | Delivered. |
408, 425, 429, any 5xx | Retried. |
| Timeout, connection refused, DNS failure, TLS error, connection reset | Retried. |
Any 3xx | Failed, not retried. Point the endpoint at the final URL. |
Any other 4xx (400, 401, 404, 419, …) | Failed, not retried. |
| The URL is no longer allowed (see Destination rules) | Failed, not retried. |
Use a 4xx when retrying can't help, such as a bad signature. Use 503 (or let the
request time out) when you are temporarily unable to accept events.
Retry schedule
A delivery gets up to 7 attempts over 24 hours. Each time is measured from the first attempt, not from the previous one:
| Attempt | When |
|---|---|
| 1 | Immediately |
| 2 | 1 minute after the first attempt |
| 3 | 5 minutes after the first attempt |
| 4 | 30 minutes after the first attempt |
| 5 | 2 hours after the first attempt |
| 6 | 8 hours after the first attempt |
| 7 | 24 hours after the first attempt |
On a 429, a valid Retry-After header (seconds or an HTTP date) is honoured, as
long as it is later than the next scheduled attempt and before the 24-hour mark. If the last
attempt fails, the delivery is marked Failed. You can still
redeliver it by hand within 14 days.
Duplicates: delivery is at-least-once
Orqestra would rather send an event twice than lose it, so the same event can reach you more
than once. For example, your 200 might be lost on the way back and the attempt
retried, a worker might restart mid-delivery, or someone might redeliver by hand.
Every copy has the same id (also in X-Orqestra-Event-Id) and a byte-identical
body. Record the IDs you have processed and skip repeats. Keep them at least 15 days, which covers the
14-day window for manual redelivery.
-- one way to do it: a unique key does the deduplication
INSERT INTO orqestra_events (id, type, received_at, body)
VALUES ($1, $2, now(), $3)
ON CONFLICT (id) DO NOTHING;
Ordering is not guaranteed
Events are sent roughly in the order they happen, but a delivery being retried can arrive
after a newer event. A chat.status_changed to OPEN that failed once
might land after the later change to RESOLVED.
- Compare the resource's
updatedAt(or the event'screatedAt) with what you stored, and ignore anything older. - When you need the true current state, read it from the API. The webhook tells you which record changed.
Delivery history
Open an endpoint's delivery history from its row on Developers → Webhooks. Each delivery shows its event, status (Delivered, Pending, Retrying, Failed or Cancelled) and every attempt: response code, duration, error, and the start of your response. History is kept for 14 days.
Orqestra doesn't switch off an endpoint that keeps failing. Every event still gets its full retry schedule, so check the history (or your own monitoring) when your receiver has been down.
Redelivering
Redeliver on any delivery sends the event again. That is useful after you fix a bug or restore an outage. A redelivery:
- keeps the original event
idand exactly the same body, so your deduplication still works; - gets a new
X-Orqestra-Delivery-Id, a fresh timestamp and signature, and a fresh retry schedule; - goes to the endpoint's current URL and is signed with its current secret(s);
- needs the endpoint to be Active and the event to be less than 14 days old.
Pausing and changing an endpoint
- Deactivate stops delivery immediately and cancels every delivery still pending or retrying. Cancelled deliveries don't resume. Events that happen while the endpoint is inactive are never sent, and delivery restarts from the moment you activate it again.
- Changing the URL does the same, and the endpoint needs a new successful test.
- Changing events or channels applies to events from then on. Nothing already scheduled is affected.
Destination rules
To keep webhooks from being aimed at internal systems, the URL is checked when you save it, when you test it, and again right before every attempt:
- It must use
https://. It can't include a username or password, or a#fragment. - Its hostname must resolve only to public addresses. Loopback (
127.0.0.0/8,::1), private networks (10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,fc00::/7), carrier-grade NAT (100.64.0.0/10), link-local, including the cloud metadata address169.254.169.254, multicast and documentation ranges are all refused. - Orqestra connects to the address it checked, so the DNS record can't be switched in between.
- Redirects are not followed.
Payload size
A text field longer than 64 KiB (a very long message, for example) is cut at 64 KiB. Its path is
listed in data.truncatedFields, e.g. ["data.message.text"]. A whole event is
never larger than 256 KiB. If it would be, optional text fields are removed in a fixed order, starting
with ad copy and then captions, subjects and message text, and each is listed in
truncatedFields. IDs and status fields are never removed.
If truncatedFields is present and you need the full content, read the record from the
API.