Skill: Connect Data

Purpose

Guided wizard to connect a new dataset. Walks the user through selecting a connection type, configuring credentials, validating the connection, profiling the schema, and setting up the knowledge brain.

When to Use

• •User says /connect-data or "connect my database" or "add a new dataset"
• •First-run welcome suggests connecting data
• •After /switch-dataset when the target dataset doesn't exist yet

Invocation

/connect-data — start the connection wizard /connect-data type=postgres — skip type selection

Instructions

Step 1: Choose Connection Type

Present options:

1. •CSV files — "I have CSV files in a local directory"
2. •DuckDB — "I have a local DuckDB database file"
3. •MotherDuck — "I have a MotherDuck cloud database"
4. •PostgreSQL — "I have a PostgreSQL database"
5. •BigQuery — "I have a Google BigQuery dataset"
6. •Snowflake — "I have a Snowflake warehouse"

Step 2: Collect Connection Details

For CSV:

• •Ask: "What's the path to your CSV directory? (relative to this repo)"
• •Verify the directory exists and contains .csv files
• •List found files and ask to confirm

For DuckDB:

• •Ask: "Path to your .duckdb file?"
• •Verify file exists
• •Test connection with SELECT 1

For MotherDuck:

• •Ask: "Database name and schema?"
• •Note: "MotherDuck connects via MCP. Make sure your token is configured."

For PostgreSQL / BigQuery / Snowflake:

• •Copy the appropriate template from connection_templates/
• •Ask user to fill in required fields
• •IMPORTANT: Never ask for or store passwords directly. Guide the user to use environment variables (e.g., $PG_PASSWORD).

Step 3: Create Dataset Brain

1. •Generate a dataset_id from the display name (lowercase, hyphens)
2. •Create .knowledge/datasets/{id}/ directory
3. •Write manifest.yaml from the connection template + user inputs
4. •Create empty quirks.md with section headers
5. •Create empty metrics/index.yaml

Step 4: Test Connection

Use ConnectionManager from helpers/connection_manager.py:

1. •Instantiate with the new config
2. •Call test_connection()
3. •If fails: show error, offer to retry or edit config
4. •If passes: proceed

Step 5: Profile Schema

1. •Call list_tables() to enumerate tables
2. •For each table: get column names and types via get_table_schema()
3. •Generate schema.md using schema_to_markdown() from helpers/data_helpers.py
4. •Write to .knowledge/datasets/{id}/schema.md
5. •Offer to run full data profiling: "Want me to deep-profile this dataset?"

Step 6: Set Active

1. •Update .knowledge/active.yaml to point to the new dataset
2. •Confirm: "Connected! {display_name} is now your active dataset."
3. •Show: table count, estimated row count, date range (if detected)
4. •Suggest next steps: /explore to browse, /metrics to define metrics, or just ask a question

Rules

1. •Never store credentials in plain text in manifest files
2. •Always test the connection before declaring success
3. •Always generate a schema.md — it's required for analysis
4. •Create the full .knowledge/datasets/{id}/ tree even if profiling fails
5. •If the user already has this dataset, ask before overwriting

Edge Cases

• •Directory doesn't exist: Offer to create it
• •No CSV files found: Check for other formats (.parquet, .json)
• •Connection fails repeatedly: Suggest checking credentials, firewall, VPN
• •Schema too large (>100 tables): Profile only, skip per-table details
• •Dataset name collision: Append a number (e.g., "mydata-2")