Commit Graph

60 Commits

Author SHA1 Message Date
Acbox Liu 64609c2101 feat: MCP OAuth (#178)
* feat: MCP OAuth

* fix: redirect url and oauth
2026-03-04 00:41:05 +08:00
BBQ ee587b8ef5 fix(mcp): fix snapshot management and encapsulate locking (#169)
- Fix DeleteContainer FAILED_PRECONDITION by cleaning up stopped task
  entries before container deletion
- Fix CreateSnapshot leaving container in broken state: commit turns
  the active snapshot read-only, so the full cycle (stop → commit →
  prepare → delete → recreate → start) is now applied consistently
- Use context.WithoutCancel for atomic container replacement sequences
  to prevent cancelled HTTP requests from corrupting container state
- Use dctx for DB operations (recordSnapshotVersion/insertEvent) to
  avoid orphan snapshots in containerd without matching DB records
- Restart task + network after snapshot replacement, fixing Exec after
  CreateVersion where the container had no running task
- Extract replaceContainerSnapshot helper to deduplicate the prepare →
  delete → recreate → start pattern across three call sites
- Move snapshot list data fetching into Manager.ListBotSnapshotData to
  encapsulate per-container locking; remove exported LockBot method
- Use UnixNano for snapshot names to avoid second-precision collisions
2026-03-03 15:59:57 +08:00
Acbox Liu ea719f7ca7 refactor: memory provider (#140)
* refactor: memory provider

* fix: migrations

* feat: divide collection from different built-in memory

* feat: add `MEMORY.md` and `PROFILES.md`

* use .env for docker compose. fix #142 (#143)

* feat(web): add brand icons for search providers (#144)

Add custom FontAwesome icon definitions for all 9 search providers:
- Yandex: uses existing faYandex from FA free brands
- Tavily, Jina, Exa, Bocha, Serper: custom icons from brand SVGs
- DuckDuckGo, SearXNG, Sogou: custom icons from Simple Icons

Icons are registered with a custom 'fac' prefix and rendered as
monochrome (currentColor) via FontAwesome's standard rendering.

* fix: resolve multiple UI bugs (#147)

* feat: add email service with multi-adapter support (#146)

* feat: add email service with multi-adapter support

Implement a full-stack email service with global provider management,
per-bot bindings with granular read/write permissions, outbox audit
storage, and MCP tool integration for direct mailbox access.

Backend:
- Email providers: CRUD with dynamic config schema (generic SMTP/IMAP, Mailgun)
- Generic adapter: go-mail (SMTP) + go-imap/v2 (IMAP IDLE real-time push via
  UnilateralDataHandler + UID-based tracking + periodic check fallback)
- Mailgun adapter: mailgun-go/v5 with dual inbound mode (webhook + poll)
- Bot email bindings: per-bot provider binding with independent r/w permissions
- Outbox: outbound email audit log with status tracking
- Trigger: inbound emails push notification to bot_inbox (from/subject only,
  LLM reads full content on demand via MCP tools)
- MailboxReader interface: on-demand IMAP queries for listing/reading emails
- MCP tools: email_accounts, email_send, email_list (paginated mailbox),
  email_read (by UID) — all with multi-binding and provider_id selection
- Webhook: /email/mailgun/webhook/:config_id (JWT-skipped, signature-verified)
- DB migration: 0019_add_email (email_providers, bot_email_bindings, email_outbox)

Frontend:
- Email Providers page: /email-providers with MasterDetailSidebarLayout
- Dynamic config form rendered from ordered provider meta schema with i18n keys
- Bot detail: Email tab with bindings management + outbox audit table
- Sidebar navigation entry
- Full i18n support (en + zh)
- Auto-generated SDK from Swagger

Closes #17

* feat(email): trigger bot conversation immediately on inbound email

Instead of only storing an inbox item and waiting for the next chat,
the email trigger now proactively invokes the conversation resolver
so the bot processes new emails right away — aligned with the
schedule/heartbeat trigger pattern.

* fix: lint

---------

Co-authored-by: Acbox <acbox0328@gmail.com>

* chore: update AGENTS.md

* feat: files preview

* feat(web): improve MCP details page

* refactor(skills): import skill with pure markdown string

* merge main into refactor/memory

* fix: migration

* refactor: temp delete qdrant and bm25 index

* fix: clean merge code

* fix: update memory handler

---------

Co-authored-by: Leohearts <leohearts@leohearts.com>
Co-authored-by: Menci <mencici@msn.com>
Co-authored-by: Quincy <69751197+dqygit@users.noreply.github.com>
Co-authored-by: BBQ <35603386+HoneyBBQ@users.noreply.github.com>
Co-authored-by: Ran <16112591+chen-ran@users.noreply.github.com>
2026-03-03 15:33:50 +08:00
BBQ 04bce702b7 feat(devenv): MCP dev hot-reload with image-based approach (#145)
Add mcp-build.sh that compiles the MCP binary and packages it as an
OCI image layer on top of the base rootfs, imported directly into
containerd. Air triggers rebuild on code changes, cleaning stale
containers automatically.

Consolidate dev-only files (Dockerfiles, entrypoint, config, build
script) into devenv/ to separate dev tooling from production artifacts.
Skip image pull when already imported to speed up dev startup.
2026-03-02 14:59:48 +08:00
BBQ cc5f00355f feat: add email service with multi-adapter support (#146)
* feat: add email service with multi-adapter support

Implement a full-stack email service with global provider management,
per-bot bindings with granular read/write permissions, outbox audit
storage, and MCP tool integration for direct mailbox access.

Backend:
- Email providers: CRUD with dynamic config schema (generic SMTP/IMAP, Mailgun)
- Generic adapter: go-mail (SMTP) + go-imap/v2 (IMAP IDLE real-time push via
  UnilateralDataHandler + UID-based tracking + periodic check fallback)
- Mailgun adapter: mailgun-go/v5 with dual inbound mode (webhook + poll)
- Bot email bindings: per-bot provider binding with independent r/w permissions
- Outbox: outbound email audit log with status tracking
- Trigger: inbound emails push notification to bot_inbox (from/subject only,
  LLM reads full content on demand via MCP tools)
- MailboxReader interface: on-demand IMAP queries for listing/reading emails
- MCP tools: email_accounts, email_send, email_list (paginated mailbox),
  email_read (by UID) — all with multi-binding and provider_id selection
- Webhook: /email/mailgun/webhook/:config_id (JWT-skipped, signature-verified)
- DB migration: 0019_add_email (email_providers, bot_email_bindings, email_outbox)

Frontend:
- Email Providers page: /email-providers with MasterDetailSidebarLayout
- Dynamic config form rendered from ordered provider meta schema with i18n keys
- Bot detail: Email tab with bindings management + outbox audit table
- Sidebar navigation entry
- Full i18n support (en + zh)
- Auto-generated SDK from Swagger

Closes #17

* feat(email): trigger bot conversation immediately on inbound email

Instead of only storing an inbox item and waiting for the next chat,
the email trigger now proactively invokes the conversation resolver
so the bot processes new emails right away — aligned with the
schedule/heartbeat trigger pattern.

* fix: lint

---------

Co-authored-by: Acbox <acbox0328@gmail.com>
2026-02-28 21:03:59 +08:00
BBQ bf0eeb0e80 feat(search): add 8 new search providers (#135)
* feat(search): add Sogou search provider

* fix(search): use new endpoint and API version for sogou

* feat(search): add Serper, SearXNG, Jina, Exa, Bocha, DuckDuckGo search providers

Add six new search provider integrations:
- Serper: Google search via Serper API
- SearXNG: Self-hosted meta search engine
- Jina: Jina AI search API
- Exa: Exa neural search API
- Bocha: Bocha AI web search
- DuckDuckGo: DuckDuckGo HTML search (no API key required)

Each provider includes backend implementation, config schema,
i18n entries, and Vue settings component.

* feat(search): add Yandex search provider

Add Yandex search provider with XML response parsing and
configurable search type (RU/TR/COM).

---------

Co-authored-by: Menci <mencici@msn.com>
2026-02-27 00:00:44 +08:00
Acbox Liu fe10abf3fc refactor: inbox (#137)
* refactor: inbox

* fix: migrations

* fix: migrations
2026-02-26 20:16:02 +08:00
BBQ d6aebf654f feat(devenv): add containerized development environment (#116)
* feat(devenv): add containerized development environment

Replace local-process dev workflow with a fully containerized stack
using docker compose. This enables consistent development across
machines without requiring local Go/Node toolchains or containerd.

- Add Dockerfile.server.dev with containerd + CNI networking support
- Add Dockerfile.web.dev for frontend dev server
- Add server-dev-entrypoint.sh for containerd lifecycle management
- Expand devenv/docker-compose.yml with server, agent, web, migrate
  and deps services with proper health checks and dependency ordering
- Update app.dev.toml to use container service names instead of localhost
- Refactor mise.toml dev tasks to drive docker compose workflow
- Support agent_gateway.server_addr in config package for inter-container
  communication

* feat(devenv): add hot-reload and registry mirror support

- Add air for Go server hot-reload in dev containers
- Fix agent_gateway host in dev config (0.0.0.0 -> agent)
- Add configurable registry mirror for China mainland users
- Unify MCP image refs via MCPConfig.ImageRef()

* feat(scripts): add China mainland mirror option to install script

Prompt users to opt-in to memoh.cn mirror during installation,
which applies docker-compose.cn.yml overlay and sets registry
in config.toml for MCP image pulls.
2026-02-26 17:32:19 +08:00
Ringo.Typowriter 19ab2fef3a feat(tool): paginated file read with safety limits for container (#119)
* feat(tool): paginated file read with safety limits for container
provider

* fix(tool): harden container read pagination and binary safety
2026-02-26 16:17:12 +08:00
Menci ed90c63888 refactor(search): improve HTTP error reporting with status code and response detail 2026-02-26 15:54:14 +08:00
Menci 08f5130c66 feat(search): add Tavily search provider 2026-02-26 15:54:14 +08:00
Ran 5e12b5a53f fix: ensure unifying on hardcoded /data mount path 2026-02-24 03:35:27 +08:00
Ran 65c4d6f793 feat(container): support for apple container 2026-02-23 22:40:46 +08:00
Acbox a440bf122b feat(search): add bing and google support 2026-02-23 15:41:47 +08:00
Acbox 18535f97f2 feat: improve prompts 2026-02-23 14:18:41 +08:00
Acbox Liu c591af14b0 feat: bot inbox (#77)
* feat: bot inbox

* feat: unified header

* fix: missing tool_call usage

* feat: add group name in header
2026-02-22 01:27:24 +08:00
Ringo.Typowriter 9461f923df fix(flow): stabilize chunked SSE and unify prune limits for read/exec/gateway (#71)
* fix(agent): emit chunked SSE data

fix(flow): reassemble chunked SSE and prune tool payloads

fix: avoid whitespace prune bypass; optimize chunked SSE builder

* refactor: LLM provider pruning use shared textprune library

* chore: smaller range
2026-02-21 17:06:02 +08:00
Acbox e2652d0884 fix: keep ext name when file was missing mime type 2026-02-21 01:36:33 +08:00
Acbox 7b12fb0b0c fix: send file out of /data/media failed 2026-02-21 00:04:38 +08:00
Acbox 1a78ba3f53 feat: add platform metadata in contacts 2026-02-20 22:19:15 +08:00
Acbox 82cc9c357f feat: support attachment send to tool send 2026-02-20 22:04:00 +08:00
Acbox 4278675799 refactor(channel): add get_contacts tools and remove lookup_channel_user 2026-02-20 16:12:40 +08:00
BBQ 4fc0ca5110 fix(container): propagate host timezone to all containers
Replace TZ env var with /etc/localtime bind-mount in docker-compose
and inject timezone spec opts into containerd bot containers.
2026-02-20 03:22:31 +08:00
BBQ bc374fe8cd refactor: content-addressed assets, cross-channel multimodal, infra simplification (#63)
* refactor(attachment): multimodal attachment refactor with snapshot schema and storage layer

- Add snapshot schema migration (0008) and update init/versions/snapshots
- Add internal/attachment and internal/channel normalize for unified attachment handling
- Move containerfs provider from internal/media to internal/storage
- Update agent types, channel adapters (Telegram/Feishu), inbound and handlers
- Add containerd snapshot lineage and local_channel tests
- Regenerate sqlc, swagger and SDK

* refactor(media): content-addressed asset system with unified naming

- Replace asset_id foreign key with content_hash as sole identifier
  for bot_history_message_assets (pure soft-link model)
- Remove mime, size_bytes, storage_key from DB; derive at read time
  via media.Resolve from actual storage
- Merge migrations 0008/0009 into single 0008; keep 0001 as canonical schema
- Add Docker initdb script for deterministic migration execution order
- Fix cross-channel real-time image display (Telegram → WebUI SSE)
- Fix message disappearing on refresh (null assets fallback)
- Fix file icon instead of image preview (mime derivation from storage)
- Unify AssetID → ContentHash naming across Go, Agent, and Frontend
- Change storage key prefix from 4-char to 2-char for directory sharding
- Add server-entrypoint.sh for Docker deployment migration handling

* refactor(infra): embedded migrations, Docker simplification, and config consolidation

- Embed SQL migrations into Go binary, removing shell-based migration scripts
- Consolidate config files into conf/ directory (app.example.toml, app.docker.toml, app.dev.toml)
- Simplify Docker setup: remove initdb.d scripts, streamline nginx config and entrypoint
- Remove legacy CLI, feishu-echo commands, and obsolete incremental migration files
- Update install script and docs to require sudo for one-click install
- Add mise tasks for dev environment orchestration

* chore: recover migrations

---------

Co-authored-by: Acbox <acbox0328@gmail.com>
2026-02-19 00:20:27 +08:00
Kirikaze Chiyuki b69d355383 fix(mcp): replace dot with underscore in tool name to meet ai sdk's need 2026-02-18 21:24:08 +08:00
BBQ df7876a30c feat: add media asset system, channel lifecycle refactor, and chat attachments (#54) 2026-02-17 19:06:46 +08:00
斬風千雪 0bdc31311c improvement(mcp): make CNI binary & data path configurable (#55) 2026-02-17 17:57:13 +08:00
Acbox 961ecbb96d feat: search provider 2026-02-16 01:22:15 +08:00
Acbox 38753ef054 refactor: channel tools 2026-02-15 17:48:20 +08:00
Acbox c014fd3095 chore: change tool names of schedule 2026-02-15 15:09:03 +08:00
Ran 7817ec8147 fix(web): channel switch failure
Also add webui memory page
2026-02-14 07:30:21 +08:00
Ran 0406f42e86 feat: memory search/compact/rebuild api 2026-02-13 06:14:57 +08:00
BBQ 76dbae2844 feat(bots): per-key parallel health checks with MCP connection probing
- Refactor RuntimeChecker interface: CheckKeys() + RunCheck() for
  individual check dispatch instead of batch-all
- Add GET /bots/:id/checks/keys to list all available check keys
- Add GET /bots/:id/checks/run/:key to evaluate a single check
- MCP ConnectionChecker probes each active connection independently
  via tools/list with 8s timeout
- Keep container checks (init/record/task/data_path) as fast builtins
- Graceful network setup failure in containerd handler (log warning
  instead of killing task) for containerd-in-docker compatibility
2026-02-13 02:43:23 +08:00
BBQ 1d1586186c feat(bots): add MCP connection runtime health checks
- Add RuntimeChecker interface for extensible bot health checks
- Implement MCP ConnectionChecker: probes active connections via tools/list
- Show MCP server status (healthy/unreachable/no tools) in bot checks
- Register checker in main.go alongside container lifecycle
2026-02-13 02:13:52 +08:00
BBQ c7d66f89ab feat(bots): MCP add/import merge, batch select and batch delete/export 2026-02-13 01:11:40 +08:00
BBQ 9dd7135820 refactor(mcp): standard mcpServers input format with type inference
- Accept standard mcpServers item format (command/args/env/url/headers)
- Auto-infer connection type: command -> stdio, url -> http/sse
- Add PUT /bots/:bot_id/mcp/import for batch import from mcpServers dict
- Add GET /bots/:bot_id/mcp/export for standard format export
- Add UpsertMCPConnectionByName SQL for import upsert by name
- Preserve is_active state on import upsert
2026-02-13 00:25:42 +08:00
BBQ 85251a2905 refactor(core): codebase quality cleanup
- Remove user-level model settings (chat_model_id, memory_model_id,
  embedding_model_id, max_context_load_time, language) from users table
- Merge migration 0002 into 0001, remove compatibility migrations
- Delete dead conversation/resolver.go (1177 lines, only flow/resolver.go used)
- Remove type aliases (Chat=Conversation, types_alias.go)
- Fix SQL: remove AND false stub, fix UpdateChatTitle model_id,
  reset model IDs in DeleteSettings, add preauth expiry filter,
  add ListMessages limit, remove 10 dead queries
- Extract shared handler helpers (RequireChannelIdentityID, AuthorizeBotAccess)
- Rename internal/router to internal/channel/inbound
- Fix identity confusion: remove UserID->ChannelIdentityID fallbacks
- Fix all _ = var patterns with proper error logging
- Fix error propagation: storeMessages, rescheduleJob, botContainerID
- Fix naming: ModelId->ModelID, active->is_active, Duration semantic fix
- Remove dead code: mcpService, ReplyTarget, callMCPServer, sshShellQuote,
  buildSessionMetadata, ChatRequest.Language, TriggerPayload.ChatID
- Fix code quality: errors.Is(), remove goto, CreateHuman deprecated
- Remove Enable model endpoint and user-level settings CLI commands
- Regenerate sqlc, swagger, SDK
2026-02-12 23:50:48 +08:00
BBQ c53d35740e feat(deploy): self-contained containerd with embedded MCP image
- Add Dockerfile.containerd: multi-stage build that compiles MCP binary,
  assembles rootfs, creates Docker image tar, and bundles it with containerd
- Add containerd-entrypoint.sh: auto-imports MCP image on first start
- Fix MCP image reference: rename busybox_image to image in config,
  use fully-qualified docker.io/library/memoh-mcp:latest everywhere
- Make image ref configurable via config.toml instead of hardcoded
- Simplify deploy.sh: remove manual nerdctl/containerd-install steps
2026-02-12 23:50:48 +08:00
BBQ 30281742ef merge(github/main): integrate fx dependency injection framework
Merge upstream fx refactor and adapt all services to use go.uber.org/fx
for dependency injection. Resolve conflicts in main.go, server.go,
and service constructors while preserving our domain model changes.

- Fix telegram adapter panic on shutdown (double close channel)
- Fix feishu adapter processing messages after stop
- Increase directory lookup timeout from 2s to 5s
2026-02-12 16:42:44 +08:00
晨苒 5b05f13f1a Merge pull request #36 from ix64/refactor/fx
refactor: using fx
2026-02-12 15:35:01 +08:00
BBQ ca5c6a1866 refactor(core): restructure conversation, channel and message domains
- Rename chat module to conversation with flow-based architecture
- Move channelidentities into channel/identities subpackage
- Add channel/route for routing logic
- Add message service with event hub
- Add MCP providers: container, directory, schedule
- Refactor Feishu/Telegram adapters with directory and stream support
- Add platform management page and channel badges in web UI
- Update database schema for conversations, messages and channel routes
- Add @memoh/shared package for cross-package type definitions
2026-02-12 15:34:40 +08:00
BBQ d99ba38b7d merge(github): merge github/main, keep our code and docs/spec 2026-02-12 05:20:18 +08:00
BBQ a75fdb8040 refactor(mcp): standardize unified tool gateway on go-sdk
Split business executors from federation sources and migrate unified tool/federation transports to the official go-sdk for stricter MCP compliance and safer session lifecycle handling. Add targeted regression tests for accept compatibility, initialization retries, pending cleanup, and include updated swagger artifacts.
2026-02-11 17:37:16 +08:00
Acbox 1f30e666e0 feat(web): use generate sdk 2026-02-11 15:23:47 +08:00
BBQ 06e8619a37 refactor(core): migrate channel identity and binding across app
Align channel identity and bind flow across backend and app-facing layers, including generated swagger artifacts and package lock updates while excluding docs content changes.
2026-02-11 14:51:58 +08:00
MengYX 6548c31597 refactor: using fx 2026-02-11 10:25:40 +08:00
Acbox Liu 8ea779779e feat: mcp (#31)
* feat: add mcp connections table and related crud api

* feat: mcp-stdio api
2026-02-09 20:07:40 +08:00
Acbox 92838ef8da feat(mcp): change patch of edit tool to old_text-new_text 2026-02-09 18:29:33 +08:00
Ran 26dd8651b7 feat: go cni lifecycle manage 2026-02-08 21:39:34 +08:00
Ran 3f8cb3292c chore: optimize code structure 2026-02-08 01:45:53 +08:00