Alert when a run fails

Say the nightly sync fails now and then, and nobody notices until the morning. Here’s how to post an alert to Slack the moment it fails, with the error and a link to the run, using a trigger.

1. Save the Slack token

A workspace owner or admin saves a Slack bot token that may post messages (chat:write), sendable only to slack.com:

printf %s "$SLACK_BOT_TOKEN" | reminix secrets set SLACK_TOKEN --hosts slack.com

2. Write the alert script

{
  "slug": "alert-on-failure",
  "name": "Alert on failure",
  "command": "sh alert.sh",
  "hosts": ["slack.com"],
  "secrets": {
    "SLACK_TOKEN": {
      "host": "slack.com",
      "header": "Authorization",
      "value": "Bearer {secret}"
    }
  },
  "inputSchema": {
    "type": "object",
    "required": ["runId"],
    "properties": {
      "runId": { "type": "string", "title": "Failed run" },
      "error": { "type": "string", "title": "Error" }
    }
  }
}

A short shell script does it. It reads its inputs from the JSON file Reminix names in $REMINIX_INPUTS:

set -e
text=$(jq -r '"The nightly sync failed.\nError: " + (.error // "none given") +
  "\nhttps://app.reminix.com/runs/" + .runId' "$REMINIX_INPUTS")
answer=$(jq -n --arg text "$text" '{channel: "#alerts", text: $text}' |
  curl -s https://slack.com/api/chat.postMessage -H "Content-Type: application/json" -d @-)
if [ "$(echo "$answer" | jq .ok)" != true ]; then
  echo "Slack: $(echo "$answer" | jq -r .error)" >&2
  exit 1
fi
echo '{"posted": true}'

Publish it:

reminix publish ./alert-on-failure

3. Add the trigger

On the alert script’s page, choose Triggers → Add trigger: the event run.failed, only when script.slug equals nightly-sync, with inputs taken from the event. Through the API:

curl -X POST https://api.reminix.com/v1/scripts/alert-on-failure/triggers \
  -H "Authorization: Bearer $REMINIX_TOKEN" -H "X-Workspace: acme" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Nightly sync failed",
    "eventType": "run.failed",
    "filter": { "script.slug": "nightly-sync" },
    "inputs": {
      "runId": "{{event.data.runId}}",
      "error": "{{event.data.error}}"
    }
  }'

The alert runs within about a minute of the failure, as you, and Reminix records it on its page, marked Trigger.

Good to know

Chains stop after 3 triggered runs, so an alert that fails can’t set off an endless loop.
On the sync’s page, under who uses it, the alert shows as a script that runs after it. Check there before changing or retiring the sync.