Skip to content

Notifications guide

SQLMesh can send notifications via Slack or email when certain events occur. This page describes how to configure notifications and specify recipients.

Notification targets

Notifications are configured with notification targets. Targets are specified in a project's configuration file (config.yml or config.py), and multiple targets can be specified for a project.

A project may specify both global and user-specific notifications. Each target's notifications will be sent for all instances of each event type (e.g., notifications for run will be sent for all of the project's environments), with exceptions for audit failures and when an override is configured for development.

Audit failure notifications can be sent for specific models if five conditions are met:

  1. A model's owner field is populated
  2. The model executes one or more audits
  3. The owner has a user-specific notification target configured
  4. The owner's notification target notify_on key includes audit failure events
  5. The audit fails in the prod environment

When those conditions are met, the audit owner will be notified if their audit failed in the prod environment.

There are three types of notification target, corresponding to the two Slack notification methods and email notification. They are specified in either a specific user's notification_targets key or the top-level notification_targets configuration key.

This example shows the location of both user-specific and global notification targets:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# User notification targets
users:
  - username: User1
    ...
    notification_targets:
      - notification_target_1
        ...
      - notification_target_2
        ...
  - username: User2
    ...
    notification_targets:
      - notification_target_1
        ...
      - notification_target_2
        ...

# Global notification targets
notification_targets:
  - notification_target_1
    ...
  - notification_target_2
    ...

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
config = Config(
    ...,
    # User notification targets
    users=[
        User(
            username="User1",
            notification_targets=[
                notification_target_1(...),
                notification_target_2(...),
            ],
        ),
        User(
            username="User2",
            notification_targets=[
                notification_target_1(...),
                notification_target_2(...),
            ],
        )
    ],

    # Global notification targets
    notification_targets=[
        notification_target_1(...),
        notification_target_2(...),
    ],
    ...
)

Notifications During Development

Events triggering notifications may be executed repeatedly during code development. To prevent excessive notification, SQLMesh can stop all but one user's notification targets.

Specify the top-level username configuration key with a value also present in a user-specific notification target's username key to only notify that user. This key can be specified in either the project configuration file or a machine-specific configuration file located in ~/.sqlmesh. The latter may be useful if a specific machine is always used for development.

This example stops all notifications other than those for User1:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Top-level `username` key: only notify User1
username: User1
# User1 notification targets
users:
  - username: User1
    ...
    notification_targets:
      - notification_target_1
        ...
      - notification_target_2
        ...

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
config = Config(
    ...,
    # Top-level `username` key: only notify User1
    username="User1",
    users=[
        User(
            # User1 notification targets
            username="User1",
            notification_targets=[
                notification_target_1(...),
                notification_target_2(...),
            ],
        ),
    ]
)

SQLMesh Event Types

SQLMesh notifications are triggered by events. The events that should trigger a notification are specified in the notification target's notify_on field.

Notifications are support for plan application start/end/failure, run start/end/failure, and audit failures.

For plan and run start/end, the target environment name is included in the notification message. For failures, the Python exception or error text is included in the notification message.

This table lists each event, its associated notify_on value, and its notification message:

Event notify_on Key Value Notification message
Plan application start apply_start "Plan apply started for environment {environment}."
Plan application end apply_end "Plan apply finished for environment {environment}."
Plan application failure apply_failure "Failed to apply plan.\n{exception}"
SQLMesh run start run_start "SQLMesh run started for environment {environment}."
SQLMesh run end run_end "SQLMesh run finished for environment {environment}."
SQLMesh run failure run_failure "Failed to run SQLMesh.\n{exception}"
Audit failure audit_failure "{audit_error}"

Any combination of these events can be specified in a notification target's notify_on field.

Slack Notifications

SQLMesh supports two types of Slack notification. Slack webhooks can notify a Slack channel, but they cannot message specific users. The Slack Web API can notify channels or users.

Webhook Configuration

SQLMesh uses Slack's "Incoming Webhooks" for webhook notifications. When you create an incoming webhook in Slack, you will receive a unique URL associated with a specific Slack channel. SQLMesh transmits the notification message by submitting a JSON payload to that URL.

This example shows a Slack webhook notification target. Notifications are triggered by plan application start, plan application failure, or SQLMesh run start. The specification uses an environment variable SLACK_WEBHOOK_URL instead of hard-coding the URL directly into the configuration file:

1
2
3
4
5
6
7
notification_targets:
  - type: slack_webhook
    notify_on:
      - apply_start
      - apply_failure
      - run_start
    url: "{{ env_var('SLACK_WEBHOOK_URL') }}"

1
2
3
4
5
6
notification_targets=[
    SlackWebhookNotificationTarget(
        notify_on=["apply_start", "apply_failure", "run_start"],
        url=os.getenv("SLACK_WEBHOOK_URL"),
    )
]

API Configuration

If you want to notify users, you can use the Slack API notification target. This requires a Slack API token, which can be used for multiple notification targets with different channels or users. See Slack's official documentation for information on getting an API token.

This example shows a Slack API notification target. Notifications are triggered by plan application start, plan application end, or audit failure. The specification uses an environment variable SLACK_API_TOKEN instead of hard-coding the token directly into the configuration file:

1
2
3
4
5
6
7
8
notification_targets:
  - type: slack_api
    notify_on:
      - apply_start
      - apply_end
      - audit_failure
    token: "{{ env_var('SLACK_API_TOKEN') }}"
    channel: "UXXXXXXXXX"  # Channel or a user's Slack member ID

1
2
3
4
5
6
7
notification_targets=[
    SlackApiNotificationTarget(
        notify_on=["apply_start", "apply_end", "audit_failure"],
        token=os.getenv("SLACK_API_TOKEN"),
        channel="UXXXXXXXXX",  # Channel or a user's Slack member ID
    )
]

Email Notifications

SQLMesh supports notifications via email. The notification target specifies the SMTP host, user, password, and sender address. A target may notify multiple recipient email addresses.

This example shows an email notification target, where sushi@example.com emails data-team@example.com on SQLMesh run failure. The specification uses environment variables SMTP_HOST, SMTP_USER, and SMTP_PASSWORD instead of hard-coding the values directly into the configuration file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
notification_targets:
  - type: smtp
    notify_on:
      - run_failure
    host: "{{ env_var('SMTP_HOST') }}"
    user: "{{ env_var('SMTP_USER') }}"
    password: "{{ env_var('SMTP_PASSWORD') }}"
    sender: sushi@example.com
    recipients:
      - data-team@example.com

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
notification_targets=[
    BasicSMTPNotificationTarget(
        notify_on=["run_failure"],
        host=os.getenv("SMTP_HOST"),
        user=os.getenv("SMTP_USER"),
        password=os.getenv("SMTP_PASSWORD"),
        sender="notifications@example.com",
        recipients=[
            "data-team@example.com",
        ],
    )
]

Advanced Usage

Overriding Notification Targets

In Python configuration files, new notification targets can be configured to send custom messages.

To customize a notification, create a new notification target class as a subclass of one of the three target classes described above (SlackWebhookNotificationTarget, SlackApiNotificationTarget, or BasicSMTPNotificationTarget). See the definitions of these classes on Github here.

Each of those notification target classes is a subclass of BaseNotificationTarget, which contains a notify function corresponding to each event type. This table lists the notification functions, along with the contextual information available to them at calling time (e.g., the environment name for start/end events):

Function name Contextual information
notify_apply_start Environment name: env
notify_apply_end Environment name: env
notify_apply_failure Exception stack trace: exc
notify_run_start Environment name: env
notify_run_end Environment name: env
notify_run_failure Exception stack trace: exc
notify_audit_failure Audit error trace: audit_error

This example creates a new notification target class CustomSMTPNotificationTarget.

It overrides the default notify_run_failure function to read a log file "/home/sqlmesh/sqlmesh.log" and append its contents to the exception stack trace exc:

from sqlmesh.core.notification_target import BasicSMTPNotificationTarget

class CustomSMTPNotificationTarget(BasicSMTPNotificationTarget):
    def notify_run_failure(self, exc: str) -> None:
        with open("/home/sqlmesh/sqlmesh.log", "r", encoding="utf-8") as f:
            msg = f"{exc}\n\nLogs:\n{f.read()}"
        super().notify_run_failure(msg)

Use this new class by specifying it as a notification target in the configuration file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
notification_targets=[
    CustomSMTPNotificationTarget(
        notify_on=["run_failure"],
        host=os.getenv("SMTP_HOST"),
        user=os.getenv("SMTP_USER"),
        password=os.getenv("SMTP_PASSWORD"),
        sender="notifications@example.com",
        recipients=[
            "data-team@example.com",
        ],
    )
]