feat: introduce DCP pipeline layer for unified context assembly (#329)

* refactor: introduce DCP pipeline layer for unified context assembly Introduce a Deterministic Context Pipeline (DCP) inspired by Cahciua, providing event-driven context assembly for LLM conversations. - Add `internal/pipeline/` package with Canonical Event types, Projection (reduce), Rendering (XML RC), Pipeline manager, and EventStore persistence - Change user message format from YAML front-matter to XML `<message>` tags with self-contained attributes (sender, channel, conversation, type) - Merge CLI/Web dual API into single `/local/` endpoint, remove CLI handler - Add `bot_session_events` table for event persistence and cold-start replay - Add `discuss` session type (reserved for future Cahciua-style mode) - Wire pipeline into HandleInbound: adapt → persist → push on every message - Lazy cold-start replay: load events from DB on first session access * feat: implement discuss mode with reactive driver and probe gate Add discuss session mode where the bot autonomously decides when to speak in group chats via tool-gated output (send tool only, no direct text reply). - Add discuss driver (per-session goroutine, RC watch, step loop via agent.Generate, TR persistence, late-binding prompt with mention hints) - Add system_discuss.md prompt template ("text = inner monologue, send = speak") - Add context composition (MergeContext, ComposeContext, TrimContext) for RC + assistant/tool message interleaving by timestamp - Add probe gate: when discuss_probe_model_id is set, cheap model pre-filters group messages; no tool calls = silence, tool calls = activate primary - Add /new [chat|discuss] command: explicit mode selection, defaults to discuss in groups, chat in DMs, chat-only for WebUI - Add ResolveRunConfig on flow.Resolver for discuss driver to reuse model/tools/system-prompt resolution without reimplementing - Fix send tool for discuss mode: same-conversation sends now go through SendDirect (channel adapter) instead of the local emitter shortcut - Add target attribute to XML message format (reply_target for routing) - Add discuss_probe_model_id to bots table settings - Remove pipeline compaction (SetCompactCursor) — reuse existing compaction.Service - Persist full SDK messages (including tool calls) in discuss mode * refactor: unify DCP event layer, fix persistence and local channel - Fix bot_session_events dedup index to include event_kind so that message + edit events for the same external_message_id coexist. - Change CreateSessionEvent from :one to :exec so ON CONFLICT DO NOTHING does not produce spurious errors on duplicate delivery. - Move ACL evaluation before event ingest; denied messages no longer enter bot_session_events or the in-memory pipeline. - Let chat mode consume RenderedContext from the DCP pipeline when available, sharing the same event-driven context assembly as discuss. - Collapse local WebSocket handler to route through HandleInbound instead of directly calling StreamChatWS, eliminating the dual business entry point. - Extract buildBaseRunConfig shared builder so resolve() and ResolveRunConfig() no longer duplicate model/credentials/skills setup. - Add StoreRound to RunConfigResolver interface so discuss driver persists assistant output with full metadata, usage, and memory extraction (same quality as chat mode). - Fix discuss driver context: use context.Background() instead of the short-lived HTTP request context that was getting cancelled. - Fix model ID passed to StoreRound: return database UUID from ResolveRunConfig instead of SDK model name. - Remove dead CLIAdapter/CLIType and update legacy web/cli references in tests and comments. * fix: stop idle discuss goroutines after 10min timeout Discuss session goroutines were never cleaned up when a session became inactive (e.g. after /new). Add a 10-minute idle timer that auto-exits the goroutine and removes it from the sessions map when no new RC arrives. * refactor: pipeline details — event types, structured reply, display content - Remove [User sent N attachments] placeholder text from buildInboundQuery; attachment info is now expressed via pipeline <attachment> tags. - Unify in-reply-to as structured ReplyRef (Sender/Preview fields) across Telegram, Discord, Feishu, and Matrix adapters instead of prepending [Reply to ...] text into the message body. Remove now-unused buildTelegramQuotedText, buildDiscordQuotedText, buildMatrixQuotedText. - Make AdaptInbound return CanonicalEvent interface and dispatch to adaptMessage/adaptEdit/adaptService based on metadata["event_type"]. - Add event_id column to bot_history_messages (migration 0059) so user messages can reference their canonical pipeline event. - PersistEvent now returns the event UUID; HandleInbound passes it through to both persistPassiveMessage and ChatRequest.EventID for storeRound. - Add FillDisplayContent to message service: extracts plain text from event_data for clean frontend display. - Frontend extractMessageText prefers display_content when available, falling back to legacy strip logic for old messages. - Fix: always generate headerifiedQuery for storage even when usePipeline is true, so user messages are persisted via storeRound in chat mode. * fix: use json.Marshal for pipeline context content serialization The manual string escaping in buildMessagesFromPipeline only handled double quotes but not newlines, backslashes, and other JSON special characters, producing invalid json.RawMessage values. The LLM then received empty/malformed context and complained about having no history. * fix: restore WebSocket handler to use StreamChatWS directly The previous refactoring replaced the WS handler with HandleInbound + RouteHub subscription, which broke streaming because RouteHub events use a different format (channel.StreamEvent) than what the frontend expects (flow.WSStreamEvent with text_delta, tool_call_start, etc.). Restore the original direct StreamChatWS call path so WebUI streaming works again. The WS handler now matches the pre-refactoring behavior while all other changes (pipeline, ACL, event types, etc.) are kept. * feat: store display_text directly in bot_history_messages Instead of computing display content at API response time by querying bot_session_events via event_id, store the raw user text in a dedicated display_text column at write time. This works for all paths including the WebSocket handler which does not go through the pipeline/event layer. - Migration 0060: add display_text TEXT column - PersistInput gains DisplayText; filled from trimmedText (passive) and req.Query (storeRound) - toMessageFields reads display_text into DisplayContent - Remove FillDisplayContent runtime query and ListSessionEventsByEventID - Frontend already prefers display_content when available (no change) * fix: display_text should contain raw user text, not XML-wrapped query req.Query gets overwritten to headerifiedQuery (with XML <message> tags) before storeRound runs. Add RawQuery field to ChatRequest to preserve the original user text, and use it for display_text in storeMessages. * fix(web): show discuss sessions * refactor: introduce DCP pipeline layer for unified context assembly Introduce a Deterministic Context Pipeline (DCP) inspired by Cahciua, providing event-driven context assembly for LLM conversations. - Add `internal/pipeline/` package with Canonical Event types, Projection (reduce), Rendering (XML RC), Pipeline manager, and EventStore persistence - Change user message format from YAML front-matter to XML `<message>` tags with self-contained attributes (sender, channel, conversation, type) - Merge CLI/Web dual API into single `/local/` endpoint, remove CLI handler - Add `bot_session_events` table for event persistence and cold-start replay - Add `discuss` session type (reserved for future Cahciua-style mode) - Wire pipeline into HandleInbound: adapt → persist → push on every message - Lazy cold-start replay: load events from DB on first session access * feat: implement discuss mode with reactive driver and probe gate Add discuss session mode where the bot autonomously decides when to speak in group chats via tool-gated output (send tool only, no direct text reply). - Add discuss driver (per-session goroutine, RC watch, step loop via agent.Generate, TR persistence, late-binding prompt with mention hints) - Add system_discuss.md prompt template ("text = inner monologue, send = speak") - Add context composition (MergeContext, ComposeContext, TrimContext) for RC + assistant/tool message interleaving by timestamp - Add probe gate: when discuss_probe_model_id is set, cheap model pre-filters group messages; no tool calls = silence, tool calls = activate primary - Add /new [chat|discuss] command: explicit mode selection, defaults to discuss in groups, chat in DMs, chat-only for WebUI - Add ResolveRunConfig on flow.Resolver for discuss driver to reuse model/tools/system-prompt resolution without reimplementing - Fix send tool for discuss mode: same-conversation sends now go through SendDirect (channel adapter) instead of the local emitter shortcut - Add target attribute to XML message format (reply_target for routing) - Add discuss_probe_model_id to bots table settings - Remove pipeline compaction (SetCompactCursor) — reuse existing compaction.Service - Persist full SDK messages (including tool calls) in discuss mode * refactor: unify DCP event layer, fix persistence and local channel - Fix bot_session_events dedup index to include event_kind so that message + edit events for the same external_message_id coexist. - Change CreateSessionEvent from :one to :exec so ON CONFLICT DO NOTHING does not produce spurious errors on duplicate delivery. - Move ACL evaluation before event ingest; denied messages no longer enter bot_session_events or the in-memory pipeline. - Let chat mode consume RenderedContext from the DCP pipeline when available, sharing the same event-driven context assembly as discuss. - Collapse local WebSocket handler to route through HandleInbound instead of directly calling StreamChatWS, eliminating the dual business entry point. - Extract buildBaseRunConfig shared builder so resolve() and ResolveRunConfig() no longer duplicate model/credentials/skills setup. - Add StoreRound to RunConfigResolver interface so discuss driver persists assistant output with full metadata, usage, and memory extraction (same quality as chat mode). - Fix discuss driver context: use context.Background() instead of the short-lived HTTP request context that was getting cancelled. - Fix model ID passed to StoreRound: return database UUID from ResolveRunConfig instead of SDK model name. - Remove dead CLIAdapter/CLIType and update legacy web/cli references in tests and comments. * fix: stop idle discuss goroutines after 10min timeout Discuss session goroutines were never cleaned up when a session became inactive (e.g. after /new). Add a 10-minute idle timer that auto-exits the goroutine and removes it from the sessions map when no new RC arrives. * refactor: pipeline details — event types, structured reply, display content - Remove [User sent N attachments] placeholder text from buildInboundQuery; attachment info is now expressed via pipeline <attachment> tags. - Unify in-reply-to as structured ReplyRef (Sender/Preview fields) across Telegram, Discord, Feishu, and Matrix adapters instead of prepending [Reply to ...] text into the message body. Remove now-unused buildTelegramQuotedText, buildDiscordQuotedText, buildMatrixQuotedText. - Make AdaptInbound return CanonicalEvent interface and dispatch to adaptMessage/adaptEdit/adaptService based on metadata["event_type"]. - Add event_id column to bot_history_messages (migration 0059) so user messages can reference their canonical pipeline event. - PersistEvent now returns the event UUID; HandleInbound passes it through to both persistPassiveMessage and ChatRequest.EventID for storeRound. - Add FillDisplayContent to message service: extracts plain text from event_data for clean frontend display. - Frontend extractMessageText prefers display_content when available, falling back to legacy strip logic for old messages. - Fix: always generate headerifiedQuery for storage even when usePipeline is true, so user messages are persisted via storeRound in chat mode. * fix: use json.Marshal for pipeline context content serialization The manual string escaping in buildMessagesFromPipeline only handled double quotes but not newlines, backslashes, and other JSON special characters, producing invalid json.RawMessage values. The LLM then received empty/malformed context and complained about having no history. * fix: restore WebSocket handler to use StreamChatWS directly The previous refactoring replaced the WS handler with HandleInbound + RouteHub subscription, which broke streaming because RouteHub events use a different format (channel.StreamEvent) than what the frontend expects (flow.WSStreamEvent with text_delta, tool_call_start, etc.). Restore the original direct StreamChatWS call path so WebUI streaming works again. The WS handler now matches the pre-refactoring behavior while all other changes (pipeline, ACL, event types, etc.) are kept. * feat: store display_text directly in bot_history_messages Instead of computing display content at API response time by querying bot_session_events via event_id, store the raw user text in a dedicated display_text column at write time. This works for all paths including the WebSocket handler which does not go through the pipeline/event layer. - Migration 0060: add display_text TEXT column - PersistInput gains DisplayText; filled from trimmedText (passive) and req.Query (storeRound) - toMessageFields reads display_text into DisplayContent - Remove FillDisplayContent runtime query and ListSessionEventsByEventID - Frontend already prefers display_content when available (no change) * fix: display_text should contain raw user text, not XML-wrapped query req.Query gets overwritten to headerifiedQuery (with XML <message> tags) before storeRound runs. Add RawQuery field to ChatRequest to preserve the original user text, and use it for display_text in storeMessages. * fix(web): show discuss sessions * chore(feishu): change discuss output to stream card * fix(channel): unify discuss/chat send path and card markdown delivery * feat(discuss): switch to stream execution with RouteHub broadcasting * refactor(pipeline): remove context trimming from ComposeContext The pipeline path should not trim context by token budget — the upstream IC/RC already bounds the event window. Remove TrimContext, FindWorkingWindowCursor, EstimateTokens, FormatLastProcessedMs (all unused or only used for trimming), the maxTokens parameter from ComposeContext, and MaxContextTokens from DiscussSessionConfig. --------- Co-authored-by: 晨苒 <16112591+chen-ran@users.noreply.github.com>
2026-04-25 07:00:48 +09:00 · 2026-04-06 21:56:25 +08:00
parent 830c521f11
commit 43c4153938
165 changed files with 5166 additions and 2097 deletions
@@ -178,6 +178,7 @@ CREATE TABLE IF NOT EXISTS bots (
  compaction_model_id UUID REFERENCES models(id) ON DELETE SET NULL,
  title_model_id UUID REFERENCES models(id) ON DELETE SET NULL,
  image_model_id UUID REFERENCES models(id) ON DELETE SET NULL,
+  discuss_probe_model_id UUID REFERENCES models(id) ON DELETE SET NULL,
  tts_model_id UUID REFERENCES tts_models(id) ON DELETE SET NULL,
  browser_context_id UUID REFERENCES browser_contexts(id) ON DELETE SET NULL,
  metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
@@ -347,7 +348,7 @@ CREATE TABLE IF NOT EXISTS bot_sessions (
  bot_id UUID NOT NULL REFERENCES bots(id) ON DELETE CASCADE,
  route_id UUID REFERENCES bot_channel_routes(id) ON DELETE SET NULL,
  channel_type TEXT,
-  type TEXT NOT NULL DEFAULT 'chat' CHECK (type IN ('chat', 'heartbeat', 'schedule', 'subagent')),
+  type TEXT NOT NULL DEFAULT 'chat' CHECK (type IN ('chat', 'heartbeat', 'schedule', 'subagent', 'discuss')),
  title TEXT NOT NULL DEFAULT '',
  metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
  parent_session_id UUID REFERENCES bot_sessions(id) ON DELETE SET NULL,
@@ -366,6 +367,25 @@ ALTER TABLE bot_channel_routes
  ADD CONSTRAINT fk_bot_channel_routes_active_session
  FOREIGN KEY (active_session_id) REFERENCES bot_sessions(id) ON DELETE SET NULL;

+-- bot_session_events: DCP pipeline event store for cold-start replay.
+CREATE TABLE IF NOT EXISTS bot_session_events (
+  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  bot_id UUID NOT NULL REFERENCES bots(id) ON DELETE CASCADE,
+  session_id UUID NOT NULL REFERENCES bot_sessions(id) ON DELETE CASCADE,
+  event_kind TEXT NOT NULL CHECK (event_kind IN ('message', 'edit', 'delete', 'service')),
+  event_data JSONB NOT NULL,
+  external_message_id TEXT,
+  sender_channel_identity_id UUID,
+  received_at_ms BIGINT NOT NULL,
+  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_session_events_session_received
+  ON bot_session_events (session_id, received_at_ms);
+CREATE UNIQUE INDEX IF NOT EXISTS idx_session_events_dedup
+  ON bot_session_events (session_id, event_kind, external_message_id)
+  WHERE external_message_id IS NOT NULL AND external_message_id != '';
+
 -- bot_history_messages: unified message history under bot scope.
 CREATE TABLE IF NOT EXISTS bot_history_messages (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
@@ -381,6 +401,8 @@ CREATE TABLE IF NOT EXISTS bot_history_messages (
  usage JSONB,
  model_id UUID REFERENCES models(id) ON DELETE SET NULL,
  compact_id UUID,
+  event_id UUID REFERENCES bot_session_events(id) ON DELETE SET NULL,
+  display_text TEXT,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
 );

@@ -0,0 +1,6 @@
+-- 0054_session_type_discuss (rollback)
+-- Revert to original CHECK constraint without 'discuss'.
+
+ALTER TABLE bot_sessions DROP CONSTRAINT IF EXISTS bot_sessions_type_check;
+ALTER TABLE bot_sessions ADD CONSTRAINT bot_sessions_type_check
+  CHECK (type IN ('chat', 'heartbeat', 'schedule', 'subagent'));
@@ -0,0 +1,6 @@
+-- 0054_session_type_discuss
+-- Add 'discuss' to the bot_sessions.type CHECK constraint for the new discuss session mode.
+
+ALTER TABLE bot_sessions DROP CONSTRAINT IF EXISTS bot_sessions_type_check;
+ALTER TABLE bot_sessions ADD CONSTRAINT bot_sessions_type_check
+  CHECK (type IN ('chat', 'heartbeat', 'schedule', 'subagent', 'discuss'));
@@ -0,0 +1,4 @@
+-- 0055_session_events (rollback)
+-- Drop the bot_session_events table.
+
+DROP TABLE IF EXISTS bot_session_events;
@@ -0,0 +1,22 @@
+-- 0055_session_events
+-- Create bot_session_events table for the DCP pipeline event store.
+-- Stores CanonicalEvent JSON for cold-start replay of the pipeline.
+
+CREATE TABLE IF NOT EXISTS bot_session_events (
+  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  bot_id UUID NOT NULL REFERENCES bots(id) ON DELETE CASCADE,
+  session_id UUID NOT NULL REFERENCES bot_sessions(id) ON DELETE CASCADE,
+  event_kind TEXT NOT NULL CHECK (event_kind IN ('message', 'edit', 'delete', 'service')),
+  event_data JSONB NOT NULL,
+  external_message_id TEXT,
+  sender_channel_identity_id UUID,
+  received_at_ms BIGINT NOT NULL,
+  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_session_events_session_received
+  ON bot_session_events (session_id, received_at_ms);
+
+CREATE UNIQUE INDEX IF NOT EXISTS idx_session_events_dedup
+  ON bot_session_events (session_id, event_kind, external_message_id)
+  WHERE external_message_id IS NOT NULL AND external_message_id != '';
@@ -0,0 +1,9 @@
+-- 0056_migrate_web_cli_to_local (rollback)
+-- Revert 'local' back to 'web'. Cannot restore 'cli' distinction since it was merged.
+
+UPDATE channel_identities SET channel_type = 'web' WHERE channel_type = 'local';
+UPDATE user_channel_bindings SET channel_type = 'web' WHERE channel_type = 'local';
+UPDATE bot_channel_configs SET channel_type = 'web' WHERE channel_type = 'local';
+UPDATE channel_identity_bind_codes SET channel_type = 'web' WHERE channel_type = 'local';
+UPDATE bot_channel_routes SET channel_type = 'web' WHERE channel_type = 'local';
+UPDATE bot_sessions SET channel_type = 'web' WHERE channel_type = 'local';
@@ -0,0 +1,9 @@
+-- 0056_migrate_web_cli_to_local
+-- Rename channel_type 'web' and 'cli' to 'local' across all tables.
+
+UPDATE channel_identities SET channel_type = 'local' WHERE channel_type IN ('web', 'cli');
+UPDATE user_channel_bindings SET channel_type = 'local' WHERE channel_type IN ('web', 'cli');
+UPDATE bot_channel_configs SET channel_type = 'local' WHERE channel_type IN ('web', 'cli');
+UPDATE channel_identity_bind_codes SET channel_type = 'local' WHERE channel_type IN ('web', 'cli');
+UPDATE bot_channel_routes SET channel_type = 'local' WHERE channel_type IN ('web', 'cli');
+UPDATE bot_sessions SET channel_type = 'local' WHERE channel_type IN ('web', 'cli');
@@ -0,0 +1,3 @@
+-- 0057_discuss_probe_model (rollback)
+
+ALTER TABLE bots DROP COLUMN IF EXISTS discuss_probe_model_id;
@@ -0,0 +1,4 @@
+-- 0057_discuss_probe_model
+-- Add discuss_probe_model_id column to bots table for probe gate configuration.
+
+ALTER TABLE bots ADD COLUMN IF NOT EXISTS discuss_probe_model_id UUID REFERENCES models(id) ON DELETE SET NULL;
@@ -0,0 +1,7 @@
+-- 0058_fix_session_events_dedup (rollback)
+-- Restore the original dedup index without event_kind.
+
+DROP INDEX IF EXISTS idx_session_events_dedup;
+CREATE UNIQUE INDEX IF NOT EXISTS idx_session_events_dedup
+  ON bot_session_events (session_id, external_message_id)
+  WHERE external_message_id IS NOT NULL AND external_message_id != '';
@@ -0,0 +1,8 @@
+-- 0058_fix_session_events_dedup
+-- Rebuild dedup index to include event_kind so that message + edit for the
+-- same external_message_id are stored as separate events.
+
+DROP INDEX IF EXISTS idx_session_events_dedup;
+CREATE UNIQUE INDEX IF NOT EXISTS idx_session_events_dedup
+  ON bot_session_events (session_id, event_kind, external_message_id)
+  WHERE external_message_id IS NOT NULL AND external_message_id != '';
@@ -0,0 +1,2 @@
+-- 0059_message_event_id (rollback)
+ALTER TABLE bot_history_messages DROP COLUMN IF EXISTS event_id;
@@ -0,0 +1,6 @@
+-- 0059_message_event_id
+-- Add event_id column to bot_history_messages so that user messages can
+-- reference their canonical event for clean frontend display.
+
+ALTER TABLE bot_history_messages
+  ADD COLUMN IF NOT EXISTS event_id UUID REFERENCES bot_session_events(id) ON DELETE SET NULL;
@@ -0,0 +1,2 @@
+-- 0060_message_display_text (rollback)
+ALTER TABLE bot_history_messages DROP COLUMN IF EXISTS display_text;
@@ -0,0 +1,5 @@
+-- 0060_message_display_text
+-- Add display_text column to store raw user message text for frontend display.
+
+ALTER TABLE bot_history_messages
+  ADD COLUMN IF NOT EXISTS display_text TEXT;