Connections guide
Overview
Note: The following guide only applies when using the built-in scheduler. Connections are configured differently when using an external scheduler such as Airflow. See the Scheduling guide for more details.
In order to deploy models and to apply changes to them, you must configure a connection to your Data Warehouse and, optionally, connection to the database where the SQLMesh state is stored. This can be done in either the config.yaml
file in your project folder, or the one in ~/.sqlmesh
.
Each connection is configured as part of a gateway which has a unique name associated with it. The gateway name can be used to select a specific combination of connection settings when using the CLI. For example:
Now the defined connection can be selected in the sqlmesh plan
CLI command as follows:
State connection
By default, the data warehouse connection is also used to store the SQLMesh state, unless the configuration uses an Airflow or Google Cloud Composer scheduler. If using one of those schedulers, the state connection defaults to the scheduler's database.
The state connection can be changed by providing different connection settings in the state_connection
key of the gateway configuration:
NOTE: Spark and Trino engines may not be used for the state connection.
Default connection
Additionally, you can set a default connection by defining its configuration in the default_connection
key:
This connection configuration will be used if one is not provided in the target gateway.
Test connection
By default, when running tests, SQLMesh uses an in-memory DuckDB database connection. You can override this behavior by providing connection settings in the test_connection
key of the gateway configuration:
Default test connection
To configure a default test connection for all gateways use the default_test_connection
key:
Default gateway
To change the default gateway used by the CLI when no gateway name is provided, set the desired name in the default_gateway
key: