What you need
Service users and their API keys are created in the console only.
dip has no command that creates
either one. Creating one is a Members action, which the built-in data_engineer role does not
include, so a data engineer usually asks an administrator to prepare the owner service user and its
key, then attaches it. A custom role with the Members privileges is still limited to what it can
reach itself: it can assign the service user a role only when its own grants cover everything that
role grants, and mint a key only for a service user whose roles it covers, with capabilities it could
use.
Set up an owner
1
Create the service user
Go to Settings → Service users and create one for the pipeline, for example
svc-dbt-finance.
Give each pipeline, or each group of pipelines that touch the same data, its own service user, so
a key you revoke stops only the runs it belongs to. See
Service users and API keys.2
Give it the least access the project needs
Assign roles that reach the catalogs and schemas the dbt project reads, and the ones it builds
tables in. What that is depends on your project: a project that only builds views in one schema
needs much less than one that rebuilds a whole catalog. A
custom role with
data grants on exactly
those schemas is the narrowest choice. The built-in
data_engineer role can read and write every
catalog, so use it only when the project really does.3
Mint its API key
Select New API key on the service user’s row and tick
query_engines:execute_sql. That is
the one capability a run uses, so leave every other box empty. Copy the key: it is shown once.4
Attach the owner to the pipeline
Pipe the key into
dip pipeline set-owner. The key is read from standard input only, never from a
flag, so it stays out of your shell history and process list:dip prints dip: set owner of "finance-dbt" to service user "svc-dbt-finance", and the
pipeline’s Overview tab names the owner.owner(service_user="...") in pipeline.py does not attach an owner. It records who should own the
pipeline, and dip pipeline push never assigns or changes one. When pipeline.py names an owner and
the pipeline has none, push says so and prints the set-owner command to run, with the workspace you
pushed to and the key piped in from an OWNER_KEY variable. When the pipeline already has a different
owner, push says that owner stays and prints the command that would switch it. An external pipeline (register --external) is never run by Dippr, so it
needs no owner.Keep credentials out of the pipeline
A pipeline declaration (pipeline.py, and the pipeline.json it writes) must not carry
credentials, in any dbt var, whatever the var is called. The declaration is baked into the pipeline
image, so anyone who can pull that image can read it.
Where the run’s credential comes from depends on who runs the pipeline:
-
A pipeline Dippr runs (
dip pipeline push) runs as its owner. Attach the owner’s key withdip pipeline set-owner, which reads it from standard input only: -
An external pipeline (
dip pipeline register --external) is run by your own orchestrator and needs no owner. Give its runs their credentials in that orchestrator, for example as environment variables your dbt profile reads, never in the declaration. Reporting an external pipeline’s runs to Dippr is not available yet, so its runs do not appear in Dippr; seepipelines:report.
dip pipeline check, dip pipeline build and dip pipeline push (both before Docker runs), and
registration, all refuse a declaration whose dbt vars, or the arguments of its run-operation
macro, have a key that looks like a secret: one
containing
password, passwd, secret, token, credential, api_key, private_key or access_key, in
any case. The last three also match written with - or with no separator (apikey). The check
names the key:
dip pipeline build and dip pipeline push check the declaration pipeline.py produces on your
machine. When pipeline.py cannot run there (no python3, or it imports a package only the image
installs), they check the pipeline.json already on disk, and the image build checks the declaration
itself. That works because dip pipeline check, dip pipeline build and a dip pipeline push that
builds an image all require your project’s copy of the pipeline SDK (altimate_pipeline/ and
altimate_app/) to be the one your dip ships. push --re-register builds nothing, so it does not
check it. A project
created with an older dip keeps its older copy, so they stop, name the files that differ, and tell
you to run dip pipeline sdk-sync, which refreshes only those two directories.
This check looks at key names only. A secret under a name it does not match, such as db_pwd, is
not refused, so do not rely on it: keep credentials out of every var.
Your credential and the owner key are different
Two credentials are involved, and they do different jobs:- Your credential is what
dip pipeline set-ownersigns in with: yourdip login, or in CI an API key withpipelines:manage. It needs permission to change the pipeline. It is never stored on the pipeline, and runs never use it. - The owner key is the service user’s
alt_key you pipe in. Dippr stores it encrypted for your default data plane and never shows it again. A run uses it to get a short-lived engine token; the dbt run itself only ever receives that token, never the key.
pipelines:manage; the owner needs
only query_engines:execute_sql.
Who can attach an owner
Anyone whose role allows changing pipelines can attach or replace an owner. That covers the built-indata_engineer, data_admin and account_admin roles, and any custom role granted the Pipelines
update privilege. From CI, use an API key holding pipelines:manage whose service user has such a role.
You can attach an owner only if you could assign every role that service user holds yourself. A run
queries your engine with the owner’s roles, so attaching an owner hands those roles to the pipeline,
and Dippr checks them the same way it checks a role assignment. If the owner holds a role you could not
assign, dip pipeline set-owner is refused, names the roles, and changes nothing:
data_admin and account_admin can attach any owner.
The check counts every role you hold, unless your session is
acting as a single one of your roles. Then only that role counts, so
an owner holding more than it is refused even when your other roles, an admin role included, would
cover it.
Runs act as the service user the key belongs to, so the key you pipe in must be one minted for the
service user you name with --service-user. Dippr checks this when you attach the owner: a key that
belongs to a different service user or to another tenant, that is unknown, revoked or expired, or that
does not hold query_engines:execute_sql, is refused and nothing is stored. Keep owner keys where only the people who deploy that pipeline can read
them, and give owner service users no roles beyond what their pipelines need.
Rotate the owner key
Rotate without a gap by attaching the new key before revoking the old one:- On the service user’s row, select New API key and tick
query_engines:execute_sql. - Attach it:
printf %s "$NEW_OWNER_KEY" | dip pipeline set-owner finance-dbt --service-user svc-dbt-finance --key-stdin. Runs that start from now on use the new key. - Expand Show keys and select Revoke on the old key.
Revoke the owner key
Revoking the key under Show keys stops new runs straight away. The next run, scheduled or manual, fails before dbt starts, with the errorowner_credential_refused. A run already under way keeps
the short-lived engine token it started with until that token expires.
To stop the schedule while you sort out a new owner, Pause the pipeline first.
When a run will not start
Related
Service users and API keys
Create the service user and mint its key.
Roles and permissions
Build a custom role with grants on the schemas a pipeline touches.
dip pipeline commands
Every
dip pipeline command, including set-owner.A pipeline's runs and results
Read a run’s error when it cannot start.