Commit Graph

30 Commits

Author SHA1 Message Date
Acbox fc54fcf573 chore(mise): add task install-workspace-toolkit 2026-03-18 17:34:56 +08:00
Menci d5b410d7e3 refactor(workspace): new workspace v3 container architecture (#244)
* feat(mcp): workspace container with bridge architecture

Migrate MCP containers to use UDS-based bridge communication instead of
TCP gRPC. Containers now mount runtime binaries and Unix domain sockets
from the host, eliminating the need for a dedicated MCP Docker image.

- Remove Dockerfile.mcp and entrypoint.sh in favor of standard base images
- Add toolkit Dockerfile for building MCP binary separately
- Containers use bind mounts for /opt/memoh (runtime) and /run/memoh (UDS)
- Update all config files with new runtime_path and socket_dir settings
- Support custom base images per bot (debian, alpine, ubuntu, etc.)
- Legacy container detection and TCP fallback for pre-bridge containers
- Frontend: add base image selector in container creation UI

* feat(container): SSE progress bar for container creation

Add real-time progress feedback during container image pull and creation
using Server-Sent Events, without breaking the existing synchronous JSON
API (content negotiation via Accept header).

Backend:
- Add PullProgress/LayerStatus types and OnProgress callback to
  PullImageOptions (containerd service layer)
- DefaultService.PullImage polls ContentStore.ListStatuses every 500ms
  when OnProgress is set; AppleService ignores it
- CreateContainer handler checks Accept: text/event-stream and switches
  to SSE branch: pulling → pull_progress → creating → complete/error

Frontend:
- handleCreateContainer/handleRecreateContainer use fetch + SSE instead
  of the SDK's synchronous postBotsByBotIdContainer
- Progress bar shows layer-level pull progress (offset/total) during
  pulling phase and indeterminate animation during creating phase
- i18n keys added for pullingImage and creatingContainer (en/zh)

* fix(container): clear stale legacy route and type create SSE

* fix(ci): resolve lint errors and arm64 musl node.js download

- Fix unused-receiver lint: rename `s` to `_` on stub methods in
  manager_legacy_test.go
- Fix sloglint: use slog.DiscardHandler instead of
  slog.NewTextHandler(io.Discard, nil)
- Handle missing arm64 musl Node.js builds: unofficial-builds.nodejs.org
  does not provide arm64 musl binaries, fall back to glibc build

* fix(lint): address errcheck, staticcheck, and gosec findings

- Discard os.Setenv/os.Remove return values explicitly with _
- Use omitted receiver name instead of _ (staticcheck ST1006)
- Tighten directory permissions from 0o755 to 0o750 (gosec G301)

* fix(lint): sanitize socket path to satisfy gosec G703

filepath.Clean the env-sourced socket path before os.Remove
to avoid path-traversal taint warning.

* fix(lint): use nolint directive for gosec G703 on socket path

filepath.Clean does not satisfy gosec's taint analysis. The socket
path comes from MCP_SOCKET_PATH env (operator-configured) or a
compiled-in default, not from end-user input.

* refactor: rename MCP container/bridge to workspace/bridge

Split internal/mcp/ to separate container lifecycle management from
Model Context Protocol connections, eliminating naming confusion:

- internal/mcp/ (container mgmt) → internal/workspace/
- internal/mcp/mcpclient/ → internal/workspace/bridge/
- internal/mcp/mcpcontainer/ → internal/workspace/bridgepb/
- cmd/mcp/ → cmd/bridge/
- config: MCPConfig → WorkspaceConfig, [mcp] → [workspace]
- container prefix: mcp-{id} → workspace-{id}
- labels: mcp.bot_id → memoh.bot_id, add memoh.workspace=v1
- socket: mcp.sock → bridge.sock, env BRIDGE_SOCKET_PATH
- runtime: /opt/memoh/runtime/mcp → /opt/memoh/runtime/bridge
- devenv: mcp-build.sh → bridge-build.sh

Legacy containers (mcp- prefix) detected by container name prefix
and handled via existing fallback path.

* fix(container): use memoh.workspace=v3 label value

* refactor(container): drop LegacyBotLabelKey, infer bot ID from container name

Legacy containers use mcp-{botID} naming, so bot ID can be derived
via TrimPrefix instead of looking up the mcp.bot_id label.

* fix(workspace): resolve containers via manager and drop gateway container ID

* docs: fix stale mcp references in AGENTS.md and DEPLOYMENT.md

* refactor(workspace): move container lifecycle ownership into manager

* dev: isolate local devenv from prod config

* toolkit: support musl node runtime

* containerd: fix fallback resolv.conf permissions

* web: preserve container create progress on completion

* web: add bot creation wait hint

* fix(workspace): preserve image selection across recreate

* feat(web): shorten default docker hub image refs

* fix(container): address code review findings

- Remove synchronous CreateContainer path (SSE-only now)
- Move flusher check before WriteHeader to avoid committed 200 on error
- Fix legacy container IP not cached via ensureContainerAndTask path
- Add atomic guard to prevent stale pull_progress after PullImage returns
- Defensive copy for tzEnv slice to avoid mutating shared backing array
- Restore network failure severity in restartContainer (return + Error)
- Extract duplicate progress bar into ContainerCreateProgress component
- Fix codesync comments to use repo-relative paths
- Add SaaS image validation note and kernel version comment on reaper

* refactor(devenv): extract toolkit install into shared script

Unify the Node.js + uv download logic into docker/toolkit/install.sh,
used by the production Dockerfile and runnable locally for dev.

Dev environment no longer bakes toolkit into the Docker image — it is
volume-mounted from .toolkit/ instead, so wrapper script changes take
effect immediately without rebuilding. The entrypoint checks for the
toolkit directory and prints a clear error if missing.

* fix(ci): address go ci failures

* chore(docker): remove unused containerd image

* refactor(config): rename workspace image key

* fix(workspace): fix legacy container data loss on migration and stop swallowing errors

Three root causes were identified and fixed:

1. Delete() used hardcoded "workspace-" prefix to look up legacy "mcp-"
   containers, causing GetContainer to return NotFound. CleanupBotContainer
   then silently skipped the error and deleted the DB record without ever
   calling PreserveData. Fix: resolve the actual container ID via
   ContainerID() (DB → label → scan) before operating.

2. Multiple restore error paths were silently swallowed (logged as Warn
   but not returned), so the user saw HTTP 200/204 with no data and no
   error. Fix: all errors in the preserve/restore chain now block the
   workflow and propagate to the caller.

3. tarGzDir used cached DirEntry.Info() for tar header size, which on
   overlayfs can differ from the actual file size, causing "archive/tar:
   write too long". Fix: open the file first, Fstat the fd for a
   race-free size, and use LimitReader as a safeguard.

Also adds a "restoring" SSE phase so the frontend shows a progress
indicator ("Restoring data, this may take a while...") during data
migration on container recreation.

* refactor(workspace): single-point container ID resolution

Replace the `containerID func(string) string` field with a single
`resolveContainerID(ctx, botID)` method that resolves the actual
container ID via DB → label → scan → fallback. All ~16 lookup
callsites across manager.go, dataio.go, versioning.go, and
manager_lifecycle.go now go through this single resolver, which
correctly handles both legacy "mcp-" and new "workspace-" containers.

Only `ensureBotWithImage` inlines `ContainerPrefix + botID` for
creating brand-new containers — every other path resolves dynamically.

* fix(web): show progress during data backup phase of container recreate

The recreate flow (delete with preserve_data + create with restore_data)
blocked on the DELETE call while backing up /data with no progress
indication. Add a 'preserving' phase to the progress component so
users see "正在备份数据..." instead of an unexplained hang.

* chore: remove [MYDEBUG] debug logging

Clean up all 112 temporary debug log statements added during the
legacy container migration investigation. Kept only meaningful
warn-level logs for non-fatal errors (network teardown, rename
failures).
2026-03-18 15:19:09 +08:00
Acbox be2bbaf74f docs: update to v0.5 2026-03-14 23:11:29 +08:00
Acbox Liu bafd327b6b feat: agent browser (#200)
* feat: agent browser

* chore: complete docker and action config

* feat: more actions

* feat: browser tab switch

* fix: browser build

* fix: lint

* fix: migrations
2026-03-07 15:06:00 +08:00
BBQ 04bce702b7 feat(devenv): MCP dev hot-reload with image-based approach (#145)
Add mcp-build.sh that compiles the MCP binary and packages it as an
OCI image layer on top of the base rootfs, imported directly into
containerd. Air triggers rebuild on code changes, cleaning stale
containers automatically.

Consolidate dev-only files (Dockerfiles, entrypoint, config, build
script) into devenv/ to separate dev tooling from production artifacts.
Skip image pull when already imported to speed up dev startup.
2026-03-02 14:59:48 +08:00
BBQ d6aebf654f feat(devenv): add containerized development environment (#116)
* feat(devenv): add containerized development environment

Replace local-process dev workflow with a fully containerized stack
using docker compose. This enables consistent development across
machines without requiring local Go/Node toolchains or containerd.

- Add Dockerfile.server.dev with containerd + CNI networking support
- Add Dockerfile.web.dev for frontend dev server
- Add server-dev-entrypoint.sh for containerd lifecycle management
- Expand devenv/docker-compose.yml with server, agent, web, migrate
  and deps services with proper health checks and dependency ordering
- Update app.dev.toml to use container service names instead of localhost
- Refactor mise.toml dev tasks to drive docker compose workflow
- Support agent_gateway.server_addr in config package for inter-container
  communication

* feat(devenv): add hot-reload and registry mirror support

- Add air for Go server hot-reload in dev containers
- Fix agent_gateway host in dev config (0.0.0.0 -> agent)
- Add configurable registry mirror for China mainland users
- Unify MCP image refs via MCPConfig.ImageRef()

* feat(scripts): add China mainland mirror option to install script

Prompt users to opt-in to memoh.cn mirror during installation,
which applies docker-compose.cn.yml overlay and sets registry
in config.toml for MCP image pulls.
2026-02-26 17:32:19 +08:00
Ran ffda558d24 feat(memoh): unify embedded runtime serving and release binaries 2026-02-24 19:39:01 +08:00
Ran 65c4d6f793 feat(container): support for apple container 2026-02-23 22:40:46 +08:00
Acbox ab84e29dde fix: change memory message role from system to user caused by the imcompatibility of anthropic messages api 2026-02-20 15:19:24 +08:00
BBQ bc374fe8cd refactor: content-addressed assets, cross-channel multimodal, infra simplification (#63)
* refactor(attachment): multimodal attachment refactor with snapshot schema and storage layer

- Add snapshot schema migration (0008) and update init/versions/snapshots
- Add internal/attachment and internal/channel normalize for unified attachment handling
- Move containerfs provider from internal/media to internal/storage
- Update agent types, channel adapters (Telegram/Feishu), inbound and handlers
- Add containerd snapshot lineage and local_channel tests
- Regenerate sqlc, swagger and SDK

* refactor(media): content-addressed asset system with unified naming

- Replace asset_id foreign key with content_hash as sole identifier
  for bot_history_message_assets (pure soft-link model)
- Remove mime, size_bytes, storage_key from DB; derive at read time
  via media.Resolve from actual storage
- Merge migrations 0008/0009 into single 0008; keep 0001 as canonical schema
- Add Docker initdb script for deterministic migration execution order
- Fix cross-channel real-time image display (Telegram → WebUI SSE)
- Fix message disappearing on refresh (null assets fallback)
- Fix file icon instead of image preview (mime derivation from storage)
- Unify AssetID → ContentHash naming across Go, Agent, and Frontend
- Change storage key prefix from 4-char to 2-char for directory sharding
- Add server-entrypoint.sh for Docker deployment migration handling

* refactor(infra): embedded migrations, Docker simplification, and config consolidation

- Embed SQL migrations into Go binary, removing shell-based migration scripts
- Consolidate config files into conf/ directory (app.example.toml, app.docker.toml, app.dev.toml)
- Simplify Docker setup: remove initdb.d scripts, streamline nginx config and entrypoint
- Remove legacy CLI, feishu-echo commands, and obsolete incremental migration files
- Update install script and docs to require sudo for one-click install
- Add mise tasks for dev environment orchestration

* chore: recover migrations

---------

Co-authored-by: Acbox <acbox0328@gmail.com>
2026-02-19 00:20:27 +08:00
Acbox b329475359 chore: add release scripts 2026-02-16 15:55:49 +08:00
BBQ c53d35740e feat(deploy): self-contained containerd with embedded MCP image
- Add Dockerfile.containerd: multi-stage build that compiles MCP binary,
  assembles rootfs, creates Docker image tar, and bundles it with containerd
- Add containerd-entrypoint.sh: auto-imports MCP image on first start
- Fix MCP image reference: rename busybox_image to image in config,
  use fully-qualified docker.io/library/memoh-mcp:latest everywhere
- Make image ref configurable via config.toml instead of hardcoded
- Simplify deploy.sh: remove manual nerdctl/containerd-install steps
2026-02-12 23:50:48 +08:00
Fodesu f8cd244d25 chore: add typos for spell check (#37) 2026-02-11 21:54:17 +08:00
Acbox 48681ce0bb feat: add hey-api sdk generate 2026-02-11 14:27:53 +08:00
Fodesu 4f422cc08a fix(typo): spell error (#33) 2026-02-10 19:38:22 +08:00
Fodesu bc7e8ccd29 chore(mise): add sqlc in tools (#32) 2026-02-10 15:19:12 +08:00
Ran 26dd8651b7 feat: go cni lifecycle manage 2026-02-08 21:39:34 +08:00
Acbox 60b4420235 Merge branch 'main' into refactor/channel-gateway 2026-02-07 17:29:14 +08:00
Acbox aa97083f65 docs: update trouble solution about lima and containerd in CONTRIBUTING.md 2026-02-07 17:19:19 +08:00
BBQ a246b79a4f refactor: restructure channel gateway and chat module architecture
- Refactor channel adapters (feishu, telegram, local) with enhanced descriptor and config
- Restructure channel manager, service, types, and outbound messaging
- Simplify chat module by removing normalize.go and chat.go, consolidating into resolver and types
- Update router channel handlers and tests
- Sync swagger documentation
2026-02-06 23:47:12 +08:00
Acbox 81155f2f78 chore: add cli build task 2026-02-06 00:29:29 +08:00
Ran cb36b68ee4 feat: Atomic update mcp image 2026-02-05 02:40:10 +08:00
BBQ 6aebbe9279 feat: refactor User/Bot architecture and implement multi-channel gateway
Major changes:
1. Core Architecture: Decoupled Bots from Users. Bots now have independent lifecycles, member management (bot_members), and dedicated configurations.
2. Channel Gateway:
   - Implemented a unified Channel Manager supporting Feishu, Telegram, and Local (Web/CLI) adapters.
   - Added message processing pipeline to normalize interactions across different platforms.
   - Introduced a Contact system for identity binding and guest access policies.
3. Database & Tooling:
   - Consolidated all migrations into 0001_init with updated schema for bots, channels, and contacts.
   - Optimized sqlc.yaml to automatically track the migrations directory.
4. Agent Enhancements:
   - Introduced ToolContext to provide Agents with platform-aware execution capabilities (e.g., messaging, contact lookups).
   - Added tool logging and fallback mechanisms for toolChoice execution.
5. UI & Docs: Updated frontend stores, UI components, and Swagger documentation to align with the new Bot-centric model.
2026-02-04 23:49:50 +08:00
Acbox 0326d03edb feat: add cli setup 2026-01-31 21:31:52 +08:00
Acbox ba34fb156d feat: schedule task 2026-01-30 23:32:23 +08:00
BBQ 8f3ee7582c Merge pull request #3 from HoneyBBQ/chore/fix-setup-help-message-20260130
chore(mise): update setup completion message
2026-01-30 07:18:20 +08:00
Acbox 715429dbb6 feat: add migration & drop scripts of database 2026-01-29 22:08:17 +08:00
Acbox 2f43e82c18 feat: add swagger & sqlc generate task to mise config 2026-01-29 16:09:56 +08:00
Acbox 54c9143a9c feat: add mise config for web 2026-01-29 15:49:51 +08:00
Acbox bfea20482b feat: add mise config 2026-01-29 15:45:41 +08:00