Use dip db load when your data is in files: a CSV export, a folder of Parquet files, a JSON feed, an Excel workbook. dip carries its own DuckDB, the build your databases run, so you install nothing. It reads the files on your machine and streams the rows into your database. The files don’t pass through any other storage, and the credential it uses is never shown. With no database name, every command acts on your personal database. Add a name to load into a shared one.

Load a file

1

Look at the file

dip db sql runs SQL with your database attached, and it can read local files:
To look at a file without attaching your database at all (no sign-in, no browser), add --local: dip db sql --local --sql "SELECT * FROM read_csv('exports/orders.csv') LIMIT 20".
2

Preview the load

--dry-run reads every value, so a file the load would reject fails here too. It shows the columns, the row count and the size on disk, and changes nothing. It stays offline, so it does not check the database or the table name; its output says so:
3

Load it

dip prints how many rows it loaded and the table’s columns. dip db sql reads those rows now; SQL through a query engine sees them once it has mounted the catalog, which the platform normally does by itself. For a pending mount, the next step points to dip engine list for the engine’s sync state and prints the exact dip db status command that shows the mounted catalog.
4

Check it

dip db status lists the tables in the database with their row counts and the reference to query each one by in Trino, quoted, such as "<catalog>"."main"."orders". Use it verbatim.If it says could not read the table list (tables_unavailable: true and tables_error in --json), the table list is unknown. This does not mean the load failed. Count the rows from your terminal with dip db sql --sql "SELECT count(*) FROM orders" (add the database name for a shared database). Or check through your engine in the console or with your agent, using the database’s catalog name. The schema is always main:
Replace <catalog> with the mounted catalog shown by dip db status, and query the engine named beside it (catalog and engine in --json). If the mount state is unknown (catalog_pending: null), run SHOW CATALOGS on the engine you will query, or ask your agent to list that engine’s catalogs, and use the listed name. Databases and dip db list show the expected name, which can differ from the mounted catalog.

Prepare an import recipe

dip db import checks a recipe for data saved by a connected MCP tool. A recipe is a local JSON description of which rows and columns to use and how to check pagination and counts. Use it when your agent has saved source pages and needs to verify their shape before loading data. Start by preparing a verified recipe. Then collect each saved page and publish its typed columns with the steps below. To have your agent run the whole loop from an app you connected over MCP, see Import from connected apps. For an ordinary file export, use dip db load above.
  1. Use Claude Code with the source tool already connected. Keep the session transcript and its saved-result files. Choose a source-account label that you will reuse for this account. The label records your declaration; dip does not authenticate the source account for you.
  2. Ask for a small page that still has another page after it. Inspect the saved call:
    The report shows structure, stable field IDs and sanitized field names, omitting scalar values. Field names can contain source data, such as an email address used as an object key. Sanitized names and stable IDs do not anonymize those keys. Keep the report private, and use sources with fixed field names when keys must not reveal source data. Have your agent use those IDs to propose a recipe with explicit output column names and types, row identity, cursor rules and either an exact total or an unavailable count.
  3. Check the proposed recipe against that page:
    Use the reported next request to fetch the next page through the same connection. The report repeats your base request arguments and adds the cursor; it does not print the page’s rows.
  4. Verify both saved pages:
    dip checks that the cursor advances and that both pages match the same tool, request and source-account label. It checks duplicate row identities, column conversions and any exact total, then saves the verified recipe privately for your local user. The recipe is identified by a fingerprint. Raw pages and source details remain local.
A learned report confirms the observed cursor transition. It does not mean that all pages have been fetched or that the import is complete. A missing exact count is reported as unavailable.

Which saved results can I use?

Use the normal saved-result format from Claude Code. A saved result can contain JSON text, text blocks or a structured JSON result. Inline results, errors, previews and unsupported content are refused. This refusal applies to dip db import; a small complete inline result can use the bounded MCP copy instead. The Claude Code version is not checked, but the saved-result notice must match a format dip recognizes. If the result reports an error, stop and correct the source request; do not ask the source to print its rows as a workaround. If inspection reports that the structure is too large, fetch a smaller page and try again. If even one record exceeds the limit, request fewer fields or a simpler source result. Setting MAX_MCP_OUTPUT_TOKENS in Claude Code’s settings before starting a session can send large results to files. A source tool can override that limit with anthropic/maxResultSizeChars: some results stay inline, and others show a preview before being saved. dip cannot remove rows that the client has already shown to the model. Check the tool’s behavior before claiming that its data stays outside the conversation. Recipes select explicit typed columns. They do not preserve extra raw fields in a catch-all column, accept SQL expressions or let an app read the saved source files. inspect and learn prepare the recipe; finish publishes its typed columns. See the import command reference for the flags.

Publish saved pages

An import is a local collection of saved source pages checked against one verified recipe. Choose an import ID and collect every page with the fingerprint returned by learn:
While pagination is unknown, each page and status report includes next_request: the source tool and the arguments to call next, never row values. Ask your agent to make that call through the same connection and pass the new call ID to page. When pagination is exhausted or broken, the report has no next_request. If the first page has not been collected yet, next_request is that first request. Fetching a page twice does not add its rows twice. The saved content must be identical for that page; changed content requires a new import. Continue until the source reports no next page. Duplicate row identities, source errors, broken pagination and a mismatched total prevent publication. A nullable conversion follows your recipe’s policy and reports failed-conversion counts without printing the rejected values. Use an empty personal database you own. Run dip db status --json to get its UUID, then publish to a new table:
If the count is unavailable but pagination is exhausted, review the collected pages and repeat with --confirm-count-unavailable. Your confirmation is recorded. It cannot override an unknown next page, duplicates, source errors or a mismatched count. Shared databases are refused, including one with no current invited writers. An unrelated table or view anywhere in your personal database also prevents publication. Existing output tables require their matching local import records; arbitrary tables cannot be adopted or overwritten. You can collect pages before choosing an eligible database. See Choose an import destination. The publication contains only the recipe’s explicit typed columns. Raw pages and preparation files remain under your local dip configuration directory in imports/<ID>, with directories accessible only to your OS user and files readable only by that user. Keep this directory outside app source and build folders. Never upload it as an integration or include it in an app image. Readers can also see main._dip_import_control, containing output table names and opaque publication identifiers. It contains no raw pages, source-account labels or rejected values.

Recover or clean up an import

Finishing seals the import. Its pages, recipe and destination cannot change afterward; later source rows belong to a new import. If the connection drops during publication, outcome_unknown means the table may already have been published. Keep the local files and check the recorded publication:
resume does not automatically publish again. If it cannot determine the result, the import stays unresolved so you can check again when the database is reachable. A newer competing publication is refused without changing that table. Use dip db import abort --import ID to cancel unfinished local work. It cannot undo a published table or discard an unresolved publication. dip db import sweep deletes aborted and conflicting imports older than 30 days. For a published import of that age it deletes the pages and staging and keeps the local record, which a later finish needs to replace that table. Unresolved imports are never touched. Cleanup is file deletion, not secure erasure. Check the final table before sharing read access. The import verifies the saved pages; it does not create a consistent snapshot of a source that changes while you fetch pages. Use the source’s snapshot or stable-query controls when needed.

What you can load

dip reads CSV (.csv, .tsv), Parquet (.parquet), JSON (.json, .jsonl) and Excel (.xlsx) from the file name, or you name the format with --format csv|parquet|json|xlsx. A compressed file such as .csv.gz or .jsonl.gz needs --format. Do not use ** with --from: dip checks it like *, one folder level, so it refuses files two or more levels down and does not count deeper files the reader then loads. Name each level, for example --from 'exports/*/*.csv', or use --sql with the reader, where ** does recurse.

How do I load files whose columns differ?

One --from pattern needs every file to have the same columns. For a folder of exports that drift, load a query and set union_by_name = true. A column a file lacks is NULL, and filename = true adds the file each row came from:
read_csv also takes delim, decimal_separator, nullstr and encoding, and a query can drop a trailing total row. This reads a European export saved as Latin-1:
For JSON lines, read_json('events.jsonl', ignore_errors = true) turns a malformed line into a row of all NULLs instead of failing. Filter on a key every real line has, for example WHERE order_id IS NOT NULL, or the NULL rows load. ignore_errors can also lose a valid line next to a broken one, so after loading compare the row count with the file’s line count, and fix or remove the bad lines in the file when they differ.

How do I load an Excel workbook?

dip reads the workbook’s first sheet from its first cell. When title rows sit above the table, the load comes back as one column. dip then warns, names the sheet it read, and lists the workbook’s sheets. Pick the sheet and the cells yourself: --range starts at the header row and ends at the last cell of the table.
The report names the sheet that was read, in sheet. --sheet and --range apply to a single .xlsx file given with --from; with --sql, pass sheet and range to read_xlsx yourself. --strict also refuses a one-column Excel read. The table name defaults to the file name without its extension, case kept. Name it with --table, which is one name in the schema main: --table sales.q1 makes a table named sales.q1, with the dot in the name. Prefer lower-case names with underscores. A name with a space or a dot must be double-quoted in every later query, such as "sales.q1". SQL finds a table by its name in any case, in dip db sql and through a query engine alike: a load of Sales.csv makes a table Sales, and SELECT * FROM sales reads it.

What do CSV warnings mean?

For CSV files loaded with --from, dip warns when it finds a single column whose name contains a comma, semicolon, tab or pipe. Check that the file’s delimiter and quoting are correct: malformed rows can cause the whole line to be read as one column. It also warns about column names such as column7 that may have been assigned automatically; check for empty header cells. An intentional header with that name also triggers the warning. When a CSV text column mostly contains date shapes, dip warns, for example, column "order_date" looks like dates but loaded as text (mixed or ambiguous date formats); dip did not guess the day/month order. It reads the first 1,000 rows once and warns, for each text column, when at least two non-null values, and at least half of them, look like dates. This check also runs with --dry-run and --format csv. Sampling errors skip the check. Values stay as text; dip does not guess whether 01/05/2024 means January 5 or May 1. Confirm the intended format before converting. For day/month/year text, use strptime with --sql and the required --table:
Replace FILE with your path. This recipe requires every non-null value to use that format; normalize mixed formats first, or explicitly handle each confirmed format in SQL. The warning appears as a warning: line and in JSON warnings; --strict refuses it before writing rows. These date checks apply to CSV --from loads, not Excel, JSON, Parquet or caller-supplied --sql. Warnings appear in the preview and load output. With --json, read the warnings array before trusting the inferred columns. A warning alone keeps exit status 0. Add --strict to stop with exit status 1 before writing any rows when a warning is found:
With --strict --json, a refusal keeps the warnings array and reports error.class as usage; error.message explains why the load was refused. The checks also apply when you select CSV with --format csv. They do not validate queries passed with --sql or provide a schema override.

Replacing and appending

append matches the file’s columns to the table’s by name, so a different header order is fine. A column the table does not have is refused, naming it; a table column the file lacks is filled with its default, NULL unless the table sets one. With --sql, every SELECT expression needs AS <table column>: an unaliased expression has no name to match and the append is refused. A table column the source lacks is reported in warnings (append leaves the table column(s) ... at their default (NULL unless the column declares one)), and --strict refuses the append before any row is written. Use --mode replace only with a source containing the full intended table, including all pre-existing rows. Never replace an append with only the appended source: it would discard earlier rows. A second append adds the rows a second time, so verify the original outcome before retrying.

Querying the rows in Trino

Trino reads your database live, so the rows are in the next query’s result as soon as the load finishes. The only wait is the first time a database is used: its catalog appears once the database is serving, and the engines on its data plane mount it with one catalog apply. See When your database becomes queryable. A load writes the rows and nothing else; queries read the database live, so there is nothing to export. dip db status shows the name to query the database by, and whether an engine has mounted it yet.

Verify a load when the table list is unavailable

Run dip db status --json after loading, adding the database name for a shared database. If tables_error is non-empty, an empty tables array means the list could not be read. It does not mean the database is empty or the load failed. Ask your agent to use MCP browse with level: "catalogs" on the chosen engine, then browse the catalog’s schemas and tables. Pass the same chosen engine to every browse call and both run_sql checks. Use the returned table sql_ref in run_sql to run SELECT count(*) FROM <sql_ref> and SELECT * FROM <sql_ref> LIMIT 5. Compare the count with the load’s rows_loaded; for an append, compare the increase with the count before loading. Inspect the sample too. Use the engine’s actual catalog name, which can differ from a display name. If the engine read fails, verification is incomplete. Do not repeat an uncertain append. If a successful, complete catalog listing shows the catalog is not mounted, follow the first-mount guidance.

When a load fails

A mistake in the SQL, or a file that isn’t the format its name says, ends with exit status 1 and DuckDB’s own message. If you stop a load, with Ctrl-C or by killing the process, after it prints loading …, the rows may still land in full. By then the database has the statement and can commit it whether or not dip is still running. When you press Ctrl-C, dip cancels the statement and closes its connection, waiting up to ten seconds, then says the rows may still have landed and exits. A killed dip can do neither. A load that was killed, or that lost its connection, can leave its statement open in the database. While it is open, another load into the same table, or a DROP TABLE on it, fails with Catalog write-write conflict. The database ends a connection whose client has gone and rolls back what it had not finished, usually about two minutes after the client stopped. It can take longer when the database is busy, or when the stopped load had already sent a lot of data that the database must first discard. Then the table accepts a load or a drop again and the space the stopped load used is returned. Nothing needs restarting, and other people working in the same database are not disconnected. If the table still refuses writes with that message five minutes after the load stopped, that is longer than the database normally needs. First check that no other load into the same table is still running. If none is, ask your platform administrator. They can see in the workspace database’s log whether abandoned connections are being ended or the check that ends them is failing, and they can list the database’s open connections to see whether the abandoned one is still there. Before running the load again, check dip db status; if its table list is unavailable, verify through the engine as above. Verify that the original load has finished and whether it landed before retrying. If an append landed, do not replay it. Retry an append only after confirming it finished without adding rows. An unchanged count while the original load may still be running is not proof of failure; if its outcome is still uncertain, stop. Use --mode replace only with a source containing the full intended table, including all pre-existing rows, as described in Replacing and appending. Each command connects with a new token, and the database accepts it a few seconds after dip creates it. While that happens, dip prints waiting for "<name>" to accept the token dip just minted and retries. If the database still refuses the token after five minutes, dip stops with exit status 1. Run the command again. If it keeps happening, the database is restarting: try again in a few minutes.
On Windows on Arm, dip doesn’t include DuckDB, so dip db load and dip db sql say so. Use dip db connect with the DuckDB that dip db duckdb downloads instead. See Connect and write.

Why is an ordinary column name refused?

When using dip db load or dip db sql, identifiers or statement text containing _auth, such as pr_author or last_author, are reserved by the workspace database. Rename the column, for example to author. Quoting it does not bypass the rule. The policy also refuses _snapshot_export, the words ATTACH, DETACH, INSTALL, LOAD, COPY, EXPORT, IMPORT, PRAGMA, CALL, CHECKPOINT, CREATE SECRET, and statement-leading SET, RESET or USE. Literals and comments in the statement are screened too. dip db exec uses a different statement classifier; ordinary names such as pr_author are allowed, while references to the _auth catalog are refused. When the database write step returns Authorization failed, dip db load and dip db sql report exit 1, class refused. Errors while describing the source can retain their original source error. The current Quack protocol uses that same ambiguous refusal for quota restrictions. dip db exec receives an explicit quota signal and reports exit 5, quota_exceeded. Inspect dip db status when quota is a possibility; its size is a snapshot, so it cannot establish why a statement was refused.

What happens when a database is full?

A full database is one whose stored data has grown past its storage quota. dip db status shows its size against the quota. A full database stays readable, and it keeps accepting the statements that free space:
  • DROP TABLE
  • DELETE and TRUNCATE
Everything that adds data is refused: INSERT, UPDATE, CREATE, ALTER and loads. Run the statements that free space with dip db sql --sql "DROP TABLE t" (replace t with a table you intend to remove and add the database name for a shared database), your own DuckDB, or the workspace_db_write_sql MCP tool. A load is not refused up front. It fails when the database refuses the write. Writes come back once the database is below 80% of its quota. The gap stops a database from switching between full and writable on every small write. After a DROP TABLE or DELETE through MCP, the database checks its size right away. After one from dip db sql or your own DuckDB, it checks within a few seconds. If you can’t free enough space, ask your platform operator to raise the database’s quota. There is no command to change it yourself.

Keep timestamp types consistent

Before joining CSV exports, compare date/time columns with DESCRIBE. Offset-bearing ISO values can infer as TIMESTAMPTZ; values without offsets infer as TIMESTAMP. Pin offset-bearing columns with read_csv('orders.csv', types = {'ordered_at': 'TIMESTAMPTZ'}) in --sql. For values without offsets, keep TIMESTAMP and confirm their timezone before converting to instants. If the person confirms America/New_York, use timezone('America/New_York', ordered_at) after reading ordered_at as TIMESTAMP. Do not cast offset-free text directly to TIMESTAMPTZ: that uses your session timezone. Use the same query for preview and load.