5 Commits

Author	SHA1	Message	Date
Acbox Liu	8d5c38f0e5	refactor: unify providers and models tables (#338) ... * refactor: unify providers and models tables - Rename `llm_providers` → `providers`, `llm_provider_oauth_tokens` → `provider_oauth_tokens` - Remove `tts_providers` and `tts_models` tables; speech models now live in the unified `models` table with `type = 'speech'` - Replace top-level `api_key`/`base_url` columns with a JSONB `config` field on `providers` - Rename `llm_provider_id` → `provider_id` across all references - Add `edge-speech` client type and `conf/providers/edge.yaml` default provider - Create new read-only speech endpoints (`/speech-providers`, `/speech-models`) backed by filtered views of the unified tables - Remove old TTS CRUD handlers; simplify speech page to read-only + test - Update registry loader to skip malformed YAML files instead of failing entirely - Fix YAML quoting for model names containing colons in openrouter.yaml - Regenerate sqlc, swagger, and TypeScript SDK * fix: exclude speech providers from providers list endpoint ListProviders now filters out client_type matching '%-speech' so Edge and future speech providers no longer appear on the Providers page. ListSpeechProviders uses the same pattern match instead of hard-coding 'edge-speech'. * fix: use explicit client_type list instead of LIKE pattern Replace '%-speech' pattern with explicit IN ('edge-speech') for both ListProviders (exclusion) and ListSpeechProviders (inclusion). New speech client types must be added to both queries. * fix: use EXECUTE for dynamic SQL in migrations referencing old schema PL/pgSQL pre-validates column/table references in static SQL statements inside DO blocks before evaluating IF/RETURN guards. This caused migrations 0010-0061 to fail on fresh databases where the canonical schema uses `providers`/`provider_id` instead of `llm_providers`/ `llm_provider_id`. Wrap all SQL that references potentially non-existent old schema objects (llm_providers, llm_provider_id, tts_providers, tts_models, etc.) in EXECUTE strings so they are only parsed at runtime when actually reached. * fix: revert canonical schema to use llm_providers for migration compatibility The CI migrations workflow (up → down → up) failed because 0061 down renames `providers` back to `llm_providers`, but 0001 down only dropped `providers` — leaving `llm_providers` as a remnant. On the second migrate up, 0010 found the stale `llm_providers` and tried to reference `models.llm_provider_id` which no longer existed. Revert 0001 canonical schema to use original names (llm_providers, tts_providers, tts_models) so incremental migrations work naturally and 0061 handles the final rename. Remove EXECUTE wrappers and unnecessary guards from migrations that now always operate on llm_providers. * fix: icons * fix: sync canonical schema with 0061 migration to fix sqlc column mismatch 0001_init.up.sql still used old names (llm_providers, llm_provider_id) and included dropped tts_providers/tts_models tables. sqlc could not parse the PL/pgSQL EXECUTE in migration 0061, so generated code retained stale columns (input_modalities, supports_reasoning) causing runtime "column does not exist" errors when adding models. - Update 0001_init.up.sql to current schema (providers, provider_id, no tts tables, add provider_oauth_tokens) - Use ALTER TABLE IF EXISTS in 0010/0041/0042 for backward compat - Regenerate sqlc * fix: guard all legacy migrations against fresh schema for CI compat On fresh databases, 0001_init.up.sql creates providers/provider_id (not llm_providers/llm_provider_id). Migrations 0013, 0041, 0046, 0047 referenced the old names without guards, causing CI migration failures. - 0013: check llm_provider_id column exists before adding old constraint - 0041: check llm_providers table exists before backfill/constraint DDL - 0046: wrap CREATE TABLE in DO block with llm_providers existence check - 0047: use ALTER TABLE IF EXISTS + DO block guard	2026-04-08 01:03:44 +08:00
Yiming Qi	64378d29ed	feat: openai codex support (#292) ... * feat(web): add provider oauth management ui * feat: add OAuth callback support on port 1455 * feat: enhance reasoning effort options and support for OpenAI Codex OAuth * feat: update twilight-ai dependency to v0.3.4 * refactor: promote openai-codex to first-class client_type, remove auth_type Replace the previous openai-responses + metadata auth_type=openai-codex-oauth combo with a dedicated openai-codex client_type. OAuth requirement is now determined solely by client_type, eliminating the auth_type concept from the LLM provider domain entirely. - Add openai-codex to DB CHECK constraint (migration 0047) with data migration - Add ClientTypeOpenAICodex constant and dedicated SDK/probe branches - Remove AuthType from SDKModelConfig, ModelCredentials, TriggerConfig, etc. - Simplify supportsOAuth to check client_type == openai-codex - Add conf/providers/codex.yaml preset with Codex catalog models - Frontend: replace auth_type selector with client_type-driven OAuth UI --------- Co-authored-by: Acbox <acbox0328@gmail.com>	2026-03-27 19:30:45 +08:00
Acbox Liu	b3a39ad93d	refactor: replace persistent subagents with ephemeral spawn tool (#280) ... * refactor: replace persistent subagents with ephemeral spawn tool (#subagent) - Drop subagents table, remove all persistent subagent infrastructure - Add 'subagent' session type with parent_session_id on bot_sessions - Rewrite subagent tool as single 'spawn' tool with parallel execution - Create system_subagent.md prompt, add _subagent.md include for chat - Limit subagent tools to file, exec, web_search, web_fetch only - Merge subagent token usage into parent chat session in reporting - Remove frontend subagent management page, update chat UI for spawn - Fix UTF-8 truncation in session title, fix query not passed to agent * refactor: remove history message page	2026-03-22 19:03:28 +08:00
Acbox Liu	b88ca96064	refactor: provider & models (#277) ... * refactor: move client_type to provider, replace model fields with config JSONB - Move `client_type` from `models` to `llm_providers` table - Add `icon` field to `llm_providers` - Replace `dimensions`, `input_modalities`, `supports_reasoning` on `models` with a single `config` JSONB column containing `dimensions`, `compatibilities` (vision, tool-call, image-output, reasoning), and `context_window` - Auto-imported models default to vision + tool-call + reasoning - Update all backend consumers (agent, flow resolver, handlers, memory) - Regenerate sqlc, swagger, and TypeScript SDK - Update frontend forms, display, and i18n for new schema * ui: show provider icon avatar in sidebar and detail header, remove icon input * feat: add built-in provider registry with YAML definitions and enable toggle - Add `enable` column to llm_providers (default true, backward-compatible) - Create internal/registry package to load YAML provider/model definitions on startup and upsert into database (new providers disabled by default) - Add conf/providers/ with OpenAI, Anthropic, Google YAML definitions - Add RegistryConfig to TOML config (providers_dir, default conf/providers) - Model listing APIs and conversation flow now filter by enabled providers - Frontend: enable switch in provider form, green status dot in sidebar, enabled providers sorted to top * fix: make 0041 migration idempotent for fresh databases Guard data migration steps with column-existence checks so the migration succeeds on databases created from the updated init schema.	2026-03-22 17:24:45 +08:00
Acbox Liu	1680316c7f	refactor(agent): remove agent gateway instead of twilight sdk (#264) ... * refactor(agent): replace TypeScript agent gateway with in-process Go agent using twilight-ai SDK - Remove apps/agent (Bun/Elysia gateway), packages/agent (@memoh/agent), internal/bun runtime manager, and all embedded agent/bun assets - Add internal/agent package powered by twilight-ai SDK for LLM calls, tool execution, streaming, sential logic, tag extraction, and prompts - Integrate ToolGatewayService in-process for both built-in and user MCP tools, eliminating HTTP round-trips to the old gateway - Update resolver to convert between sdk.Message and ModelMessage at the boundary (resolver_messages.go), keeping agent package free of persistence concerns - Prepend user message before storeRound since SDK only returns output messages (assistant + tool) - Clean up all Docker configs, TOML configs, nginx proxy, Dockerfile.agent, and Go config structs related to the removed agent gateway - Update cmd/agent and cmd/memoh entry points with setter-based ToolGateway injection to avoid FX dependency cycles * fix(web): move form declaration before computed properties that reference it The `form` reactive object was declared after computed properties like `selectedMemoryProvider` and `isSelectedMemoryProviderPersisted` that reference it, causing a TDZ ReferenceError during setup. * fix: prevent UTF-8 character corruption in streaming text output StreamTagExtractor.Push() used byte-level string slicing to hold back buffer tails for tag detection, which could split multi-byte UTF-8 characters. After json.Marshal replaced invalid bytes with U+FFFD, the corruption became permanent — causing garbled CJK characters (�) in agent responses. Add safeUTF8SplitIndex() to back up split points to valid character boundaries. Also fix byte-level truncation in command/formatter.go and command/fs.go to use rune-aware slicing. * fix: add agent error logging and fix Gemini tool schema validation - Log agent stream errors in both SSE and WebSocket paths with bot/model context - Fix send tool `attachments` parameter: empty `items` schema rejected by Google Gemini API (INVALID_ARGUMENT), now specifies `{"type": "string"}` - Upgrade twilight-ai to d898f0b (includes raw body in API error messages) * chore(ci): remove agent gateway from Docker build and release pipelines Agent gateway has been replaced by in-process Go agent; remove the obsolete Docker image matrix entry, Bun/UPX CI steps, and agent-binary build logic from the release script. * fix: preserve attachment filename, metadata, and container path through persistence - Add `name` column to `bot_history_message_assets` (migration 0034) to persist original filenames across page refreshes. - Add `metadata` JSONB column (migration 0035) to store source_path, source_url, and other context alongside each asset. - Update SQL queries, sqlc-generated code, and all Go types (MessageAsset, AssetRef, OutboundAssetRef, FileAttachment) to carry name and metadata through the full lifecycle. - Extract filenames from path/URL in AttachmentsResolver before clearing raw paths; enrich streaming event metadata with name, source_path, and source_url in both the WebSocket and channel inbound ingestion paths. - Implement `LinkAssets` on message service and `LinkOutboundAssets` on flow resolver so WebSocket-streamed bot attachments are persisted to the correct assistant message after streaming completes. - Frontend: update MessageAsset type with metadata field, pass metadata through to attachment items, and reorder attachment-block.vue template so container files (identified by metadata.source_path) open in the sidebar file manager instead of triggering a download. * refactor(agent): decouple built-in tools from MCP, load via ToolProvider interface Migrate all 13 built-in tool providers from internal/mcp/providers/ to internal/agent/tools/ using the twilight-ai sdk.Tool structure. The agent now loads tools through a ToolProvider interface instead of the MCP ToolGatewayService, which is simplified to only manage external federation sources. This enables selective tool loading and removes the coupling between business tools and the MCP protocol layer. * refactor(flow): split monolithic resolver.go into focused modules Break the 1959-line resolver.go into 12 files organized by concern: - resolver.go: core orchestration (Resolver struct, resolve, Chat, prepareRunConfig) - resolver_stream.go: streaming (StreamChat, StreamChatWS, tryStoreStream) - resolver_trigger.go: schedule/heartbeat triggers - resolver_attachments.go: attachment routing, inlining, encoding - resolver_history.go: message loading, deduplication, token trimming - resolver_store.go: persistence (storeRound, storeMessages, asset linking) - resolver_memory.go: memory provider integration - resolver_model_selection.go: model selection and candidate matching - resolver_identity.go: display name and channel identity resolution - resolver_settings.go: bot settings, loop detection, inbox - user_header.go: YAML front-matter formatting - resolver_util.go: shared utilities (sanitize, normalize, dedup, UUID) * fix(agent): enable Anthropic extended thinking by passing ReasoningConfig to provider Anthropic's thinking requires WithThinking() at provider creation time, unlike OpenAI which uses per-request ReasoningEffort. The config was never wired through, so Claude models could not trigger thinking. * refactor(agent): extract prompts into embedded markdown templates Move inline prompt strings from prompt.go into separate .md files under internal/agent/prompts/, using {{key}} placeholders and a simple render engine. Remove obsolete SystemPromptParams fields (Language, MaxContextLoadTime, Channels, CurrentChannel) and their call-site usage. * fix: lint	2026-03-19 13:31:54 +08:00