150 Commits

Author SHA1 Message Date
Acbox 70252124ba fix(filemanager): return raw file content in FSRead to avoid embedded line numbers
The FSRead handler was using client.ReadFile() which formats each line
with a line number prefix (for MCP AI tools). Switch to client.ReadRaw()
so the file viewer gets unmodified content — fixes duplicate line numbers
in the Monaco editor.
2026-03-11 17:24:39 +08:00
Yiming Qi a5c364911e feat(email/oauth): implement OAuth2 support for Gmail provider (#212) 2026-03-09 23:37:43 +08:00
Acbox Liu 23d49a1c7b feat: message abort and web socket support (#222)
* feat: message abort and web socket support

* fix(web): chat end

* fix: lint

* fix: lint
2026-03-09 23:27:50 +08:00
BBQ dae772f729 fix(containerd): backport network fallback fixes to v0.4 (#205)
* fix(containerd): prevent silent network failures from leaving containers unreachable (#202)

* fix(containerd): prevent silent network failures from leaving containers unreachable

Container network setup failures were silently swallowed at multiple
points in the call chain, leaving containers in a "running but
unreachable" ghost state. This patch closes every silent-failure path:

- setupCNINetwork: return error when CNI yields no usable IP
- Manager.Start: roll back container when IP is empty instead of
  returning success
- ensureContainerAndTask: extract setupNetworkOrFail with 1 retry,
  propagate error to callers
- ReconcileContainers: stop reporting "healthy" when network setup fails
- recoverContainerIP: retry up to 2 times with backoff for transient
  CNI failures (IPAM lock contention, etc.)
- gRPC Pool: evict connections stuck in Connecting state for >30s

* fix(containerd): clean stale cni0 bridge on startup to prevent MAC error

After a Docker container restart, the cni0 bridge interface can linger
with a zeroed MAC (00:00:00:00:00:00) and DOWN state. The CNI bridge
plugin then fails with "could not set bridge's mac: invalid argument",
making all MCP containers unreachable.

Two-layer fix:
- Entrypoint: delete cni0 and flush IPAM state before starting containerd
- Go: detect bridge MAC errors in setupCNINetwork and auto-delete cni0
  before retrying, as defense-in-depth for runtime recovery

* fix(containerd): use exec.CommandContext to satisfy noctx linter

* fix(mcp): propagate network errors from replaceContainerSnapshot

Network setup failure after snapshot replace (rollback/commit) was
silently swallowed — the container would start but remain unreachable
via gRPC. Return the error so callers (CreateSnapshot, RollbackVersion,
etc.) surface the failure instead of reporting success.
2026-03-07 18:13:06 +08:00
BBQ abbb14c59f fix(containerd): prevent silent network failures from leaving containers unreachable (#202)
* fix(containerd): prevent silent network failures from leaving containers unreachable

Container network setup failures were silently swallowed at multiple
points in the call chain, leaving containers in a "running but
unreachable" ghost state. This patch closes every silent-failure path:

- setupCNINetwork: return error when CNI yields no usable IP
- Manager.Start: roll back container when IP is empty instead of
  returning success
- ensureContainerAndTask: extract setupNetworkOrFail with 1 retry,
  propagate error to callers
- ReconcileContainers: stop reporting "healthy" when network setup fails
- recoverContainerIP: retry up to 2 times with backoff for transient
  CNI failures (IPAM lock contention, etc.)
- gRPC Pool: evict connections stuck in Connecting state for >30s

* fix(containerd): clean stale cni0 bridge on startup to prevent MAC error

After a Docker container restart, the cni0 bridge interface can linger
with a zeroed MAC (00:00:00:00:00:00) and DOWN state. The CNI bridge
plugin then fails with "could not set bridge's mac: invalid argument",
making all MCP containers unreachable.

Two-layer fix:
- Entrypoint: delete cni0 and flush IPAM state before starting containerd
- Go: detect bridge MAC errors in setupCNINetwork and auto-delete cni0
  before retrying, as defense-in-depth for runtime recovery

* fix(containerd): use exec.CommandContext to satisfy noctx linter
2026-03-07 17:50:01 +08:00
Acbox Liu bafd327b6b feat: agent browser (#200)
* feat: agent browser

* chore: complete docker and action config

* feat: more actions

* feat: browser tab switch

* fix: browser build

* fix: lint

* fix: migrations
2026-03-07 15:06:00 +08:00
BBQ 21999b49f4 feat(container): add explicit data workflows and snapshot rollback (#193)
* feat(container): add explicit data workflows and snapshot rollback

Make container upgrades and recreation data-safe by adding explicit preserve, export, import, restore, and rollback flows across the backend, SDK, and web UI.

* fix(container): resolve go lint issues

Fix formatting and lint violations introduced by the container data workflow changes so the Go CI lint job passes cleanly.
2026-03-06 17:57:48 +08:00
Acbox 4109a141f9 feat: move all tools from @memoh/agent into built-in mcp 2026-03-06 16:48:18 +08:00
BBQ 3feb03aca7 ci: add go lint and race test workflow (#187) 2026-03-05 11:25:33 +08:00
BBQ 9ceabf68c4 feat(mcp): replace bind-mount+exec with in-container gRPC service (#179)
Replace the host bind-mount + containerd exec approach with a per-bot
in-container gRPC server (ContainerService, port 9090). All file I/O,
exec, and MCP stdio sessions now go through gRPC instead of running
shell commands or reading host-mounted directories.

Architecture changes:
- cmd/mcp: rewritten as a gRPC server (ContainerService) with full
  file and exec API (ReadFile, WriteFile, ListDir, ReadRaw, WriteRaw,
  Exec, Stat, Mkdir, Rename, DeleteFile)
- internal/mcp/mcpcontainer: protobuf definitions and generated stubs
- internal/mcp/mcpclient: gRPC client wrapper with connection pool
  (Pool) and Provider interface for dependency injection
- mcp.Manager: add per-bot IP cache, gRPC connection pool, and
  SetContainerIP/MCPClient methods; remove DataDir/Exec helpers
- containerd.Service: remove ExecTask/ExecTaskStreaming; network setup
  now returns NetworkResult{IP} for pool routing
- internal/fs/service.go: deleted (replaced by mcpclient)
- handlers/fs.go: deleted; MCP stdio session logic moved to mcp_stdio.go
- container provider Executor: all tools (read/write/list/edit/exec)
  now call gRPC client instead of running shell via exec
- storefs, containerfs, media, skills, memory: all I/O ported to
  mcpclient.Provider

Database:
- migration 0022: drop host_path column from containers table

One-time data migration:
- migrateBindMountData: on first Start() after upgrade, copies old
  bind-mount data into the container via gRPC, then renames src dir
  to prevent re-migration; runs in background goroutine

Bug fixes:
- mcp_stdio: callRaw now returns full JSON-RPC envelope
  {"jsonrpc","id","result"|"error"} matching protocol spec;
  explicit "initialize" call now advances session init state to
  prevent duplicate handshake on next non-initialize call
- mcpclient Pool: properly evict stale gRPC connection after snapshot
  replace (container process recreated); use SetContainerIP instead
  of direct map write so IP changes always evict pool entry
- migrateBindMountData: walkErr on directories now counted as failure
  so partially-walked trees don't get incorrectly marked as migrated
- cmd/mcp/Dockerfile: removed dead file (docker/Dockerfile.mcp is the
  canonical production build)

Tests:
- provider_test.go: restored with bufconn in-process gRPC mock
  (fakeContainerService + staticProvider), 14 cases covering all 5
  tools plus edge cases
- mcp_session_test.go: new, covers JSON-RPC envelope, init state
  machine, pending cleanup on cancel/close, readLoop cancel
- storefs/service_test.go: restored (pure function roundtrip tests)
2026-03-04 21:50:08 +08:00
Acbox Liu 64609c2101 feat: MCP OAuth (#178)
* feat: MCP OAuth

* fix: redirect url and oauth
2026-03-04 00:41:05 +08:00
BBQ ee587b8ef5 fix(mcp): fix snapshot management and encapsulate locking (#169)
- Fix DeleteContainer FAILED_PRECONDITION by cleaning up stopped task
  entries before container deletion
- Fix CreateSnapshot leaving container in broken state: commit turns
  the active snapshot read-only, so the full cycle (stop → commit →
  prepare → delete → recreate → start) is now applied consistently
- Use context.WithoutCancel for atomic container replacement sequences
  to prevent cancelled HTTP requests from corrupting container state
- Use dctx for DB operations (recordSnapshotVersion/insertEvent) to
  avoid orphan snapshots in containerd without matching DB records
- Restart task + network after snapshot replacement, fixing Exec after
  CreateVersion where the container had no running task
- Extract replaceContainerSnapshot helper to deduplicate the prepare →
  delete → recreate → start pattern across three call sites
- Move snapshot list data fetching into Manager.ListBotSnapshotData to
  encapsulate per-container locking; remove exported LockBot method
- Use UnixNano for snapshot names to avoid second-precision collisions
2026-03-03 15:59:57 +08:00
Acbox Liu 5982bc6a42 feat: models import (#164) 2026-03-03 15:53:52 +08:00
Acbox Liu ea719f7ca7 refactor: memory provider (#140)
* refactor: memory provider

* fix: migrations

* feat: divide collection from different built-in memory

* feat: add `MEMORY.md` and `PROFILES.md`

* use .env for docker compose. fix #142 (#143)

* feat(web): add brand icons for search providers (#144)

Add custom FontAwesome icon definitions for all 9 search providers:
- Yandex: uses existing faYandex from FA free brands
- Tavily, Jina, Exa, Bocha, Serper: custom icons from brand SVGs
- DuckDuckGo, SearXNG, Sogou: custom icons from Simple Icons

Icons are registered with a custom 'fac' prefix and rendered as
monochrome (currentColor) via FontAwesome's standard rendering.

* fix: resolve multiple UI bugs (#147)

* feat: add email service with multi-adapter support (#146)

* feat: add email service with multi-adapter support

Implement a full-stack email service with global provider management,
per-bot bindings with granular read/write permissions, outbox audit
storage, and MCP tool integration for direct mailbox access.

Backend:
- Email providers: CRUD with dynamic config schema (generic SMTP/IMAP, Mailgun)
- Generic adapter: go-mail (SMTP) + go-imap/v2 (IMAP IDLE real-time push via
  UnilateralDataHandler + UID-based tracking + periodic check fallback)
- Mailgun adapter: mailgun-go/v5 with dual inbound mode (webhook + poll)
- Bot email bindings: per-bot provider binding with independent r/w permissions
- Outbox: outbound email audit log with status tracking
- Trigger: inbound emails push notification to bot_inbox (from/subject only,
  LLM reads full content on demand via MCP tools)
- MailboxReader interface: on-demand IMAP queries for listing/reading emails
- MCP tools: email_accounts, email_send, email_list (paginated mailbox),
  email_read (by UID) — all with multi-binding and provider_id selection
- Webhook: /email/mailgun/webhook/:config_id (JWT-skipped, signature-verified)
- DB migration: 0019_add_email (email_providers, bot_email_bindings, email_outbox)

Frontend:
- Email Providers page: /email-providers with MasterDetailSidebarLayout
- Dynamic config form rendered from ordered provider meta schema with i18n keys
- Bot detail: Email tab with bindings management + outbox audit table
- Sidebar navigation entry
- Full i18n support (en + zh)
- Auto-generated SDK from Swagger

Closes #17

* feat(email): trigger bot conversation immediately on inbound email

Instead of only storing an inbox item and waiting for the next chat,
the email trigger now proactively invokes the conversation resolver
so the bot processes new emails right away — aligned with the
schedule/heartbeat trigger pattern.

* fix: lint

---------

Co-authored-by: Acbox <acbox0328@gmail.com>

* chore: update AGENTS.md

* feat: files preview

* feat(web): improve MCP details page

* refactor(skills): import skill with pure markdown string

* merge main into refactor/memory

* fix: migration

* refactor: temp delete qdrant and bm25 index

* fix: clean merge code

* fix: update memory handler

---------

Co-authored-by: Leohearts <leohearts@leohearts.com>
Co-authored-by: Menci <mencici@msn.com>
Co-authored-by: Quincy <69751197+dqygit@users.noreply.github.com>
Co-authored-by: BBQ <35603386+HoneyBBQ@users.noreply.github.com>
Co-authored-by: Ran <16112591+chen-ran@users.noreply.github.com>
2026-03-03 15:33:50 +08:00
BBQ f9f968f13f feat(models): per-model probe testing with auto-detect UI (#133)
* feat(models): add per-model probe testing and auto-detect in UI

Move health probes from provider level to model level for precise
testing with real model_id and client_type. Provider test is now a
simple reachability check.

Backend:
- Add POST /models/:id/test endpoint that probes the model's provider
  using its actual model_id and client_type
- Add model healthcheck checker for bot health checks (chat/memory/embedding)
- Simplify provider test to reachability-only

Frontend:
- Auto-probe models on mount with status indicator (green/yellow/red dot + latency)
- Auto-probe provider reachability on load and on provider switch
- Fix missing faBolt icon registration
- Manual re-probe via refresh button

Closes #117

* fix(models): increase probe timeout to 15s for slow providers

Some providers (e.g. DashScope) exceed the 5s probe timeout, causing
false-negative "context deadline exceeded" errors. Increase per-probe
timeout to 15s and healthcheck overall timeout to 30s.

* fix(sdk): regenerate exports after merge conflict

Resolve duplicate SDK exports introduced by merge conflict resolution so the web build can compile again while preserving new model probe endpoints.
2026-03-02 14:59:15 +08:00
Acbox Liu 0cdf822603 feat: token usage state (#153)
* feat: token usage state

* fix: typo
2026-03-01 02:19:07 +08:00
Acbox 443ede30b4 refactor(skills): import skill with pure markdown string 2026-02-28 23:07:44 +08:00
BBQ cc5f00355f feat: add email service with multi-adapter support (#146)
* feat: add email service with multi-adapter support

Implement a full-stack email service with global provider management,
per-bot bindings with granular read/write permissions, outbox audit
storage, and MCP tool integration for direct mailbox access.

Backend:
- Email providers: CRUD with dynamic config schema (generic SMTP/IMAP, Mailgun)
- Generic adapter: go-mail (SMTP) + go-imap/v2 (IMAP IDLE real-time push via
  UnilateralDataHandler + UID-based tracking + periodic check fallback)
- Mailgun adapter: mailgun-go/v5 with dual inbound mode (webhook + poll)
- Bot email bindings: per-bot provider binding with independent r/w permissions
- Outbox: outbound email audit log with status tracking
- Trigger: inbound emails push notification to bot_inbox (from/subject only,
  LLM reads full content on demand via MCP tools)
- MailboxReader interface: on-demand IMAP queries for listing/reading emails
- MCP tools: email_accounts, email_send, email_list (paginated mailbox),
  email_read (by UID) — all with multi-binding and provider_id selection
- Webhook: /email/mailgun/webhook/:config_id (JWT-skipped, signature-verified)
- DB migration: 0019_add_email (email_providers, bot_email_bindings, email_outbox)

Frontend:
- Email Providers page: /email-providers with MasterDetailSidebarLayout
- Dynamic config form rendered from ordered provider meta schema with i18n keys
- Bot detail: Email tab with bindings management + outbox audit table
- Sidebar navigation entry
- Full i18n support (en + zh)
- Auto-generated SDK from Swagger

Closes #17

* feat(email): trigger bot conversation immediately on inbound email

Instead of only storing an inbox item and waiting for the next chat,
the email trigger now proactively invokes the conversation resolver
so the bot processes new emails right away — aligned with the
schedule/heartbeat trigger pattern.

* fix: lint

---------

Co-authored-by: Acbox <acbox0328@gmail.com>
2026-02-28 21:03:59 +08:00
Acbox Liu fe10abf3fc refactor: inbox (#137)
* refactor: inbox

* fix: migrations

* fix: migrations
2026-02-26 20:16:02 +08:00
BBQ d6aebf654f feat(devenv): add containerized development environment (#116)
* feat(devenv): add containerized development environment

Replace local-process dev workflow with a fully containerized stack
using docker compose. This enables consistent development across
machines without requiring local Go/Node toolchains or containerd.

- Add Dockerfile.server.dev with containerd + CNI networking support
- Add Dockerfile.web.dev for frontend dev server
- Add server-dev-entrypoint.sh for containerd lifecycle management
- Expand devenv/docker-compose.yml with server, agent, web, migrate
  and deps services with proper health checks and dependency ordering
- Update app.dev.toml to use container service names instead of localhost
- Refactor mise.toml dev tasks to drive docker compose workflow
- Support agent_gateway.server_addr in config package for inter-container
  communication

* feat(devenv): add hot-reload and registry mirror support

- Add air for Go server hot-reload in dev containers
- Fix agent_gateway host in dev config (0.0.0.0 -> agent)
- Add configurable registry mirror for China mainland users
- Unify MCP image refs via MCPConfig.ImageRef()

* feat(scripts): add China mainland mirror option to install script

Prompt users to opt-in to memoh.cn mirror during installation,
which applies docker-compose.cn.yml overlay and sets registry
in config.toml for MCP image pulls.
2026-02-26 17:32:19 +08:00
Acbox Liu 2f38662d4d feat: heartbeat (#108)
* feat: heartbeat

* feat: independent heartbeat model
2026-02-25 16:32:52 +08:00
Ran 30fe3edc39 fix(release): switch to agent-bin and gzip-only embedded web 2026-02-24 22:29:15 +08:00
Ran ffda558d24 feat(memoh): unify embedded runtime serving and release binaries 2026-02-24 19:39:01 +08:00
Ran 0c6a905317 feat(auth): implement JWT token refresh mechanism 2026-02-24 16:47:58 +08:00
Ran 5e12b5a53f fix: ensure unifying on hardcoded /data mount path 2026-02-24 03:35:27 +08:00
Ran 65c4d6f793 feat(container): support for apple container 2026-02-23 22:40:46 +08:00
Acbox 8590c53f3d feat(web): add sunagent, history, skills page 2026-02-23 15:49:28 +08:00
ringotypowriter 45e7362b44 Merge branch 'main' into fix/provider-scoped-model-id-resolution 2026-02-22 17:00:51 +08:00
Acbox Liu ee0aa319e2 feat: add restful apis of container file system (#92)
* feat: add restful apis of container file system

* feat: add fs tools in agent
2026-02-22 16:42:30 +08:00
Acbox 928b0c0ee5 refactor: subagent usage and related tools 2026-02-22 16:35:50 +08:00
ringotypowriter 8cd7c4aa86 Merge branch 'main' into fix/provider-scoped-model-id-resolution 2026-02-22 12:47:37 +08:00
Acbox Liu c591af14b0 feat: bot inbox (#77)
* feat: bot inbox

* feat: unified header

* fix: missing tool_call usage

* feat: add group name in header
2026-02-22 01:27:24 +08:00
ringotypowriter 50bdbd519c fix(models,settings,conversation): scope model_id uniqueness per
provider and harden model reference resolution
2026-02-21 22:31:32 +08:00
BBQ 4fc0ca5110 fix(container): propagate host timezone to all containers
Replace TZ env var with /etc/localtime bind-mount in docker-compose
and inject timezone spec opts into containerd bot containers.
2026-02-20 03:22:31 +08:00
BBQ bc374fe8cd refactor: content-addressed assets, cross-channel multimodal, infra simplification (#63)
* refactor(attachment): multimodal attachment refactor with snapshot schema and storage layer

- Add snapshot schema migration (0008) and update init/versions/snapshots
- Add internal/attachment and internal/channel normalize for unified attachment handling
- Move containerfs provider from internal/media to internal/storage
- Update agent types, channel adapters (Telegram/Feishu), inbound and handlers
- Add containerd snapshot lineage and local_channel tests
- Regenerate sqlc, swagger and SDK

* refactor(media): content-addressed asset system with unified naming

- Replace asset_id foreign key with content_hash as sole identifier
  for bot_history_message_assets (pure soft-link model)
- Remove mime, size_bytes, storage_key from DB; derive at read time
  via media.Resolve from actual storage
- Merge migrations 0008/0009 into single 0008; keep 0001 as canonical schema
- Add Docker initdb script for deterministic migration execution order
- Fix cross-channel real-time image display (Telegram → WebUI SSE)
- Fix message disappearing on refresh (null assets fallback)
- Fix file icon instead of image preview (mime derivation from storage)
- Unify AssetID → ContentHash naming across Go, Agent, and Frontend
- Change storage key prefix from 4-char to 2-char for directory sharding
- Add server-entrypoint.sh for Docker deployment migration handling

* refactor(infra): embedded migrations, Docker simplification, and config consolidation

- Embed SQL migrations into Go binary, removing shell-based migration scripts
- Consolidate config files into conf/ directory (app.example.toml, app.docker.toml, app.dev.toml)
- Simplify Docker setup: remove initdb.d scripts, streamline nginx config and entrypoint
- Remove legacy CLI, feishu-echo commands, and obsolete incremental migration files
- Update install script and docs to require sudo for one-click install
- Add mise tasks for dev environment orchestration

* chore: recover migrations

---------

Co-authored-by: Acbox <acbox0328@gmail.com>
2026-02-19 00:20:27 +08:00
Acbox 1294b4cf75 feat: connection test of provider 2026-02-18 18:52:26 +08:00
Acbox d6c47472b2 refactor: move client_type key from provider to model 2026-02-18 18:30:27 +08:00
BBQ df7876a30c feat: add media asset system, channel lifecycle refactor, and chat attachments (#54) 2026-02-17 19:06:46 +08:00
斬風千雪 0bdc31311c improvement(mcp): make CNI binary & data path configurable (#55) 2026-02-17 17:57:13 +08:00
Ringo.Typowriter 3aea635e44 fix: skill normalize (#57) 2026-02-17 17:55:07 +08:00
BBQ b8a6a85fbb fix(containerd): re-apply CNI network after server restart for running MCP tasks
Server container restart drops cni0 bridge, veth and iptables masquerade
in its network namespace while MCP tasks keep running in containerd.
Reconcile and ensureContainerAndTask now re-run SetupNetwork for already-
running tasks so outbound connectivity is restored.
2026-02-16 16:45:38 +08:00
Acbox 961ecbb96d feat: search provider 2026-02-16 01:22:15 +08:00
Ran 7817ec8147 fix(web): channel switch failure
Also add webui memory page
2026-02-14 07:30:21 +08:00
BBQ faaadf14c5 feat(channel): pass conversation type through to agent gateway and persist in route
Propagate conversation type (direct/group/thread) from channel adapters
all the way to the agent prompt. Store conversation_type on bot_channel_routes
so the bot knows whether a message originates from a p2p chat, group, or thread.

Schema changes are folded into the 0001 init migration (destructive update).
2026-02-13 06:22:24 +08:00
Ran 0406f42e86 feat: memory search/compact/rebuild api 2026-02-13 06:14:57 +08:00
BBQ 76dbae2844 feat(bots): per-key parallel health checks with MCP connection probing
- Refactor RuntimeChecker interface: CheckKeys() + RunCheck() for
  individual check dispatch instead of batch-all
- Add GET /bots/:id/checks/keys to list all available check keys
- Add GET /bots/:id/checks/run/:key to evaluate a single check
- MCP ConnectionChecker probes each active connection independently
  via tools/list with 8s timeout
- Keep container checks (init/record/task/data_path) as fast builtins
- Graceful network setup failure in containerd handler (log warning
  instead of killing task) for containerd-in-docker compatibility
2026-02-13 02:43:23 +08:00
BBQ fddacbb86d fix(containerd): keep task running when network setup fails
In containerd-in-docker mode, SetupNetwork fails because netns is
unavailable. Previously this killed the task, making stdio MCP tools
unusable. Now the task continues running with a warning log, since
stdio MCP communication does not require networking.
2026-02-13 01:50:59 +08:00
BBQ c7d66f89ab feat(bots): MCP add/import merge, batch select and batch delete/export 2026-02-13 01:11:40 +08:00
BBQ 3a1aa31611 fix(mcp): move import/export routes to /mcp-ops to avoid /:id conflict
Echo router treats /mcp/import and /mcp/export as /mcp/:id matches.
Move to /bots/:bot_id/mcp-ops/import and /mcp-ops/export.
2026-02-13 00:47:44 +08:00
BBQ 9dd7135820 refactor(mcp): standard mcpServers input format with type inference
- Accept standard mcpServers item format (command/args/env/url/headers)
- Auto-infer connection type: command -> stdio, url -> http/sse
- Add PUT /bots/:bot_id/mcp/import for batch import from mcpServers dict
- Add GET /bots/:bot_id/mcp/export for standard format export
- Add UpsertMCPConnectionByName SQL for import upsert by name
- Preserve is_active state on import upsert
2026-02-13 00:25:42 +08:00