Overview
The Overview tab shows the pipeline as it was registered:- Declaration — kind, mode, owner, the engine it runs on, version, schedule, the maximum run time, and the CPU and memory a run gets.
- dbt — the project, the project directory, profile and target, the dbt commands, any model
selection, the thread count, and whether runs default to a full refresh. A pipeline that runs
run-operationalso shows Operation: the macro it calls and its arguments. Project is thenamefrom yourdbt_project.yml, read from the dbt manifest baked into the image when you register; it shows—when the registration carried no manifest. Project directory is theprojectDirpath frompipeline.json. These dbt settings are part of the published image: a run uses exactly what the image declares. To change them, editpipeline.pyand push a new version (an external pipeline registers one withdip pipeline register --external); the platform refuses an edit to them on a registered pipeline. A run can still take a one-off--selector--full-refreshfromdip pipeline run. - Schedule — the cron expression, when the next run is due, when the scheduler last tried, the Failed starts in a row (scheduled runs in a row that could not be started; a run that starts and then fails does not count), and the blocked reason when there is one.
Runs
The Runs tab lists every run, newest first:
Select a run, or its View button, to open the run detail.
Run detail
The run detail opens over the page and shows everything the run reported.- Models — one row per dbt model, with its status, run time, rows affected, and its cost: bytes scanned, CPU time and query time, summed from the engine’s query log. A — means the model ran no engine queries, or those numbers have not landed yet.
- Checks — one row per dbt test, with its status, what it is attached to, and how many rows failed. Hover a status for the test’s message.
- Source freshness — one row per source, with its freshness status, when it was last loaded, and how old that is.
Why did my run fail?
A failed run shows a Run error in the run detail, and the same text indip pipeline runs. The
error starts with a code, then says what to do:
Watch a run live
While a run is Queued or Running, the detail tails its events live under Live events, newest last, and the model, check and freshness tables refresh as results arrive. A live indicator shows next to the heading while the run is active. From the next Dippr release, a run’s events are kept for 30 days after the run ends; until then they are kept. Once a run’s event log has been removed, the detail says when, in place of Live events. The run’s state, counts and error stay, and so do its model and check results (180 days) and source freshness (365 days).Cancel a run
While a run is active, Cancel run stops it. Models already built are kept; models not yet run are skipped. Cancelling needs permission to change pipelines. From the CLI,dip pipeline cancel <name|id> <run-id> does the same; dip pipeline runs <name|id>
lists the run ids. See Pipelines.
Assets
The Assets tab lists the models, seeds, snapshots and sources the pipeline’s manifest declares:
A pipeline is registered before it has run. When
pipeline.py declares the catalog and schema
(dbt_project(catalog=..., schema=...), or dip pipeline new --catalog --schema), the image build
reads your dbt project the way a run of pipeline.py does (for example dip pipeline dev): the same
project directory, under the pipeline’s own dbt target and vars, with that catalog and schema. The
Relation is then the catalog, schema and table that run builds, including anything your
generate_schema_name, generate_database_name or generate_alias_name macros derive from the
target or from var().
A run the platform starts, from a schedule, Run now or dip pipeline run, does not start dbt
through pipeline.py yet, so it does not use the declared catalog and schema. For those runs, the
Relation is where a run of pipeline.py builds, which may not be where that run wrote.
Without a declared catalog and schema, a run cannot start, and the build does not know where models
will land. The Relation then shows only the parts your project sets itself: a source’s own database
and schema, a model’s database config, or a schema your macro returns unchanged. Any part dbt
would build from the engine’s default catalog or schema is left out rather than guessed, and the
Relation often shows just the table. An asset whose relation is incomplete is not drawn in declared
lineage.
A macro that reads anything else at run time, such as the target’s user or host, or an environment
variable, can still give a different result than at registration.
Search the list by name, and use the column control to hide columns you do not need.
Related
Pipelines overview
The list, its statuses, schedules and the pause and run controls.
Query engines
The engines a pipeline runs on.