feat(copilot): add copilot_chat_messages table with dual-write rollout

[waleedlatif1]

Summary

• Introduces dedicated copilot_chat_messages table to replace the JSONB messages array on copilot_chats. The JSONB column is preserved indefinitely as the source of truth during rollout — no breaking schema change ever.
• Migration 0212_cynical_jack_murdock.sql runs an inline jsonb_array_elements ... WITH ORDINALITY backfill, idempotent via ON CONFLICT (chat_id, message_id) DO NOTHING. Self-hosters need no scripts — bun run db:migrate does everything.
• Schema follows existing precedent (0091_backfill_user_stats.sql, 0192_invitation_unification.sql): multi-statement migration, gen_random_uuid(), FK with CASCADE.
• Dual-write helper (lib/copilot/chat/messages-dual-write.ts) wired into every JSONB-mutation callsite. Best-effort — errors logged but not thrown so the legacy JSONB write remains canonical during rollout.
• Catch-up sweep on server boot (7-day window, idempotent) closes the rolling-deploy gap where old code may briefly write JSONB-only before the dual-write deploy reaches every replica.

Callsites wired

• lib/copilot/chat/post.ts — user message append
• lib/copilot/chat/terminal-state.ts — assistant message append (after FOR UPDATE tx commits)
• lib/mothership/inbox/executor.ts — inbox-derived persistence
• app/api/mothership/chats/[chatId]/fork/route.ts — forked transcript
• app/api/copilot/chat/update-messages/route.ts — snapshot replace

Indexes

• UNIQUE (chat_id, message_id) — idempotent appends, dedup streaming retries
• (chat_id, created_at, id) WHERE deleted_at IS NULL — transcript load, last-N, DISTINCT ON
• (chat_id, stream_id) WHERE stream_id IS NOT NULL — stream replay/resume

Rollout plan (this PR = R+0)

• R+0 (this): migration + dual-write + catch-up sweep. Reads still use JSONB. Zero behavior change visible to users.
• R+1 (~1 week later): switch reads to the new table, feature-flag gated for instant rollback.
• R+2 (~2-4 weeks later): stop dual-writing once R+1 is stable.
• Future: JSONB column stays in schema indefinitely.

Type of Change

• New feature (database table + infrastructure for future read cutover)

Testing

Migration verified to apply cleanly to a copy of production schema. Dual-write helper is best-effort with try/catch so any runtime failure logs but doesn't break the request. Lint passes.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		May 22, 2026 4:02pm

[cursor]

PR Summary

Medium Risk
Adds a new persisted message store and wires multiple write paths to dual-write into it, plus a background catch-up job on boot; main risk is data consistency/perf impact during rollout and migration execution on large datasets.

Overview
Introduces a new copilot_chat_messages table (with FK, uniqueness, and query indexes) and backfills it from the legacy copilot_chats.messages JSONB array via an idempotent migration.

Adds best-effort dual-write helpers (appendCopilotChatMessages, replaceCopilotChatMessages) and wires them into all existing chat message mutation paths (user/assistant appends, snapshot replace, chat fork, and inbox persistence) while keeping JSONB as the source of truth.

Schedules a fire-and-forget, 7-day bounded boot-time catch-up sweep to fill any gaps during rolling deploys where older servers may have written JSONB-only.

^{Reviewed by Cursor Bugbot for commit 069f8bd. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 069f8bd. Configure here.}

[greptile-apps]

Greptile Summary

This PR introduces a dedicated copilot_chat_messages table to replace the JSONB messages array on copilot_chats, implementing a dual-write strategy during rollout where the JSONB column remains the source of truth and the new table is populated in a best-effort, background manner.

• Schema + migration: 0212_cynical_jack_murdock.sql creates the new table with appropriate indexes and backfills existing data via an inline INSERT … SELECT jsonb_array_elements WITH ORDINALITY query with ON CONFLICT DO NOTHING.
• Dual-write helper: messages-dual-write.ts exposes appendCopilotChatMessages (upsert) and replaceCopilotChatMessages (delete-then-insert transaction), both best-effort with internal try/catch, wired into five callsites.
• Catch-up sweep: messages-catchup.ts runs at server boot via instrumentation-node.ts, re-syncing chats active in the last 7 days to close the rolling-deploy gap between old and new replicas.

Confidence Score: 3/5

Safe to merge for R+0 since reads still use JSONB, but two correctness problems in the new table population logic will need to be fixed before the R+1 read cutover.

The catch-up sweep inserts a fresh duplicate row for every message that lacks an id field in the JSONB blob on each server boot — ON CONFLICT DO NOTHING can't deduplicate rows whose message_id differs. Additionally, replaceCopilotChatMessages will silently delete all rows for a chat if validation causes rows to be empty while messages is non-empty. Neither bug is user-visible at R+0 because no reads hit the new table yet, but both would corrupt conversations when the R+1 read cutover lands.

messages-dual-write.ts (replaceCopilotChatMessages delete-without-insert risk) and messages-catchup.ts (gen_random_uuid non-idempotency) need the most attention before the R+1 read switch.

Important Files Changed

Filename	Overview
apps/sim/lib/copilot/chat/messages-dual-write.ts	New dual-write helper with two bugs: replaceCopilotChatMessages deletes all rows for a chat even when all input messages fail validation (rows is empty but messages is not), and appendCopilotChatMessages uses onConflictDoUpdate while the PR description promises ON CONFLICT DO NOTHING semantics.
apps/sim/lib/copilot/chat/messages-catchup.ts	Catch-up sweep is not truly idempotent: the gen_random_uuid() fallback for messages without an id field generates a different UUID on every boot, causing a new row to be inserted each time (ON CONFLICT DO NOTHING cannot deduplicate rows whose message_id differs).
packages/db/migrations/0212_cynical_jack_murdock.sql	Migration creates the table, indexes, and FK correctly; same gen_random_uuid() fallback issue as the catchup sweep exists here too, but since the migration runs exactly once this is less impactful than in the sweep.
apps/sim/app/api/copilot/chat/update-messages/route.ts	Correctly wires replaceCopilotChatMessages after the JSONB write; missing chatModel option means model column will be null for messages that don't carry their own model field.
apps/sim/instrumentation-node.ts	Fire-and-forget catch-up sweep correctly registered at server boot with void and a top-level catch; will run on every pod restart during rolling deploy.
apps/sim/app/api/mothership/chats/[chatId]/fork/route.ts	Dual-write for forked transcript is correctly placed after the new chat row is committed; parent.model is passed as chatModel.
packages/db/schema.ts	New copilotChatMessages table definition matches the migration SQL exactly; all three indexes (unique, created_at partial, stream_id partial) are present.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Route as API Route / Executor
    participant JSONB as copilot_chats (JSONB)
    participant DW as messages-dual-write.ts
    participant NewTable as copilot_chat_messages
    participant Sweep as messages-catchup.ts

    Note over Route,NewTable: R+0 Dual-Write Flow (reads still use JSONB)

    Client->>Route: POST /api/copilot/chat
    Route->>JSONB: "UPDATE messages = messages || [userMsg]"
    JSONB-->>Route: updated row (source of truth)
    Route->>DW: appendCopilotChatMessages(chatId, [userMsg])
    DW->>NewTable: INSERT ON CONFLICT DO UPDATE (best-effort)
    NewTable-->>DW: ok / error (swallowed)

    Route->>JSONB: finalizeAssistantTurn (tx + FOR UPDATE)
    JSONB-->>Route: committed
    Route->>DW: appendCopilotChatMessages(chatId, [assistantMsg])
    DW->>NewTable: INSERT ON CONFLICT DO UPDATE (best-effort)

    Note over Sweep,NewTable: Server Boot Catch-Up Sweep
    Sweep->>JSONB: SELECT chats updated in last 7 days
    Sweep->>NewTable: INSERT ON CONFLICT DO NOTHING (idempotent only for messages WITH id)

Loading

_{Reviews (1): Last reviewed commit: "feat(copilot): add copilot_chat_messages..." | Re-trigger Greptile}