Compare commits

...

1696 Commits

Author SHA1 Message Date
Willem Jiang 2895600974 fixed the lint error 2026-05-04 08:59:25 +08:00
copilot-swe-agent[bot] 4e1c1618ed fix(subagents): update SubagentConfig.system_prompt to str | None and add astream regression test
Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/2ee03a26-e19b-4106-abc5-c76a2906383b

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
2026-05-03 15:50:03 +00:00
Willem Jiang afdaf9a2ae fix(subagents): consolidate system_prompt and skills into single SystemMessage
Some LLM APIs (vLLM, Xinference, Chinese LLM providers) reject multiple
  system messages with \”System message must be at the beginning.\” The
  subagent executor was sending separate SystemMessages for the configured
  system_prompt and each loaded skill, which caused failures when calling
  task tool with sub-agents.

  Merge system_prompt and all skill content into one SystemMessage in the
  initial state, and pass system_prompt=None to create_agent() so the
  factory doesn't prepend a second one.

Fixes #2693
2026-05-03 23:07:17 +08:00
wanxsb 44ab21fc44 feat(community): add Serper web search provider (#2630)
* feat(community): add Serper web search provider

Add a new community search provider backed by the Serper Google Search
API (https://serper.dev). Serper returns real-time Google results via a
simple JSON API and requires only an API key — no extra Python package.

Changes:
- backend/packages/harness/deerflow/community/serper/__init__.py
- backend/packages/harness/deerflow/community/serper/tools.py
  Implements web_search_tool using httpx (already a project dependency).
  API key is read from config.yaml `api_key` field or SERPER_API_KEY env var.
  Follows the same interface / output shape as the existing ddg_search provider.
  Exposes max_results parameter (default 5) with config override logic.
- backend/tests/test_serper_tools.py
  Unit tests covering API key resolution, config overrides, HTTP errors,
  empty results, and parameter passing.
- config.example.yaml: add commented-out Serper example alongside other providers
- .env.example: add SERPER_API_KEY placeholder

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix the lint error

* Fix the lint error

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-05-02 16:22:35 +08:00
Hinotobi e543bbf5d6 [security] fix(upload): reject symlinked upload destinations (#2623)
* fix: reject symlinked upload destinations

* test: harden upload destination checks

* fix: address PR feedback for #2623

* test: cover safe upload re-uploads

* fix: preserve upload limit checks after rebase

* fix(upload): stream safe HTTP upload writes
2026-05-02 15:19:28 +08:00
Xinmin Zeng ca3332f8bf fix(gateway): return ISO 8601 timestamps from threads endpoints (#2599)
* fix(gateway): return ISO 8601 timestamps from threads endpoints (#2594)

ThreadResponse documents created_at / updated_at as ISO timestamps,
matching the LangGraph Platform schema (langgraph_sdk.schema.Thread
exposes them as datetime, JSON-encoded as ISO 8601). The gateway
threads router was instead emitting str(time.time()) — unix-second
floats — breaking frontend new Date() parsing and producing a mixed
ISO/unix wire format that also corrupted the search sort order.

Centralize timestamp generation in deerflow.utils.time:
- now_iso()       — datetime.now(UTC).isoformat()
- coerce_iso(x)   — heals legacy unix-timestamp strings on read so the
                    store converges to ISO without a one-shot migration

threads.py: replace 6 time.time() call sites with now_iso(); wrap all
read paths and Phase-2 checkpoint metadata with coerce_iso(); _store_upsert
opportunistically heals legacy created_at on update; drop unused time import.

thread_runs.py: reuse now_iso() instead of a private duplicate _now_iso(),
preventing future drift between the two timestamp call sites.

Tests: 9 unit tests for the helper; 5 integration tests pinning the ISO
contract for create/get/patch/search and the legacy-healing path on the
internal store upsert. Full suite: 2144 passed, 15 skipped, 0 failed.

Closes #2594

* fix(gateway): coerce checkpoint metadata timestamps to ISO on read

After the merge with main, three additional read paths in ``threads.py``
were still emitting raw ``str(metadata.get("created_at", ""))`` —
``get_thread_state``, ``update_thread_state``, and ``get_thread_history``.

Same root cause as #2594: when the checkpoint metadata's ``created_at``
is a unix-second float (legacy data, or a checkpoint written by an older
Gateway version), ``str(float)`` produces ``"1777252410.411327"`` and the
frontend's ``new Date(...)`` returns ``Invalid Date``. The fix on the
``/threads/{id}`` GET path was already in place; these three sibling
endpoints needed the same treatment.

All four call sites now flow through ``coerce_iso``, so:
- legacy float metadata heals to ISO on the way out,
- ISO metadata passes through unchanged,
- ``datetime`` instances (which the new ``coerce_iso`` branch handles
  explicitly) emit with the ``T`` separator instead of falling through
  to the space-separated ``str(datetime)`` form.

Coverage added for the two endpoints not already pinned by the merge:
- ``test_get_thread_state_returns_iso_for_legacy_checkpoint_metadata``
- ``test_get_thread_history_returns_iso_for_legacy_checkpoint_metadata``

Both pre-seed a checkpoint whose metadata carries the literal float
from the issue body and assert the wire format is ISO.
2026-05-02 15:16:16 +08:00
Willem Jiang bb8b234d85 chroe(2585): keep polishing the code of codex token usage (#2689) 2026-05-02 15:04:11 +08:00
KiteEater 17447fccbe fix(runtime): make rollback restore checkpoint supersede newer checkpoints (#2582)
* Restore rollback checkpoints with fresh ids

* Tighten rollback checkpoint tests and imports

* Update test_run_worker_rollback.py

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-05-02 11:25:45 +08:00
KiteEater 866d1ca409 Populate Codex usage metadata for token accounting (#2585) 2026-05-02 11:16:03 +08:00
greatmengqi 8ba01dfd83 refactor: thread app_config through lead and subagent task path (#2666)
* refactor: thread app config through lead prompt

* fix: honor explicit app config across runtime paths

* style: format subagent executor tests

* fix: thread resolved app config and guard subagents-only fallback

Address two PR review findings:

1. _create_summarization_middleware passed the original (possibly None)
   app_config into create_chat_model, forcing the model factory back to
   ambient get_app_config() and risking config drift between the
   middleware's resolved view and the model's view. Pass the resolved
   AppConfig instance through end-to-end.

2. get_available_subagent_names accepted Any-typed config and forwarded
   it to is_host_bash_allowed, which reads ``.sandbox``. A
   SubagentsAppConfig (also accepted upstream as a sum-type input) has
   no ``.sandbox`` attribute and would be silently treated as "no
   sandbox configured", incorrectly disabling the bash subagent. Guard
   on hasattr and fall back to ambient lookup otherwise.

Adds regression tests for both paths.

* chore: simplify hasattr guard and tighten regression tests

- Collapse if/else into ternary in get_available_subagent_names; hasattr(None, ...) is False so the explicit None check was redundant.
- Drop comments that narrate the change rather than explain non-obvious WHY (test names already convey intent).
- Replace stringly-typed sentinel "no-arg" in regression test with direct args tuple comparison.

---------

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-05-02 06:37:49 +08:00
Willem Jiang 189b82405c fix(sandbox): pass no_change_timeout to exec_command to prevent 120s premature termination (#2685)
* fix(sandbox): pass no_change_timeout to exec_command to prevent 120s premature termination

  The agent_sandbox library's shell API defaults no_change_timeout to 120
  seconds. When AioSandbox.execute_command() called exec_command() without
  this parameter, commands producing no output for 120s would return with
  NO_CHANGE_TIMEOUT status even though the script was still running.

  Pass no_change_timeout=600 to all exec_command calls (matching the
  client-level HTTP timeout) so long-running commands are not cut short.

  Fixes #2668

* test(sandbox): add assertions for no_change_timeout in execute_command and list_dir

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/2f37bc72-0826-4443-a6ba-e5b78c22fb5a

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-05-01 22:27:02 +08:00
Nan Gao 487c1d939f fix(subagents): use model override for tools and middleware (#2641)
* fix(subagents): use model override for tools and middleware

* fix(config): resolve effective subagent model

* fix(subagents): defer app config loading

* fix(subagents): fully defer config.yaml load in executor __init__

The previous attempt only relocated the explicit get_app_config() call,
but left resolve_subagent_model_name(...) running eagerly in __init__.
That helper has its own internal get_app_config() fallback, which still
fired when both app_config and parent_model were None and
config.model == "inherit" — exactly the path unit tests hit, breaking
21 tests in CI with FileNotFoundError: config.yaml.

Skip the eager resolve in __init__ when it would require loading the
config file, and defer to _create_agent (which already has the
app_config or get_app_config() fallback).
2026-05-01 22:21:10 +08:00
Nan Gao c09c334544 fix(harness): resolve runtime paths from project root (#2642)
* fix(harness): resolve runtime paths from project root

* docs(config): update

* fix(config): address runtime path review feedback

* test(config): fix skills path e2e root

* test(config): cover legacy config fallback when project root lacks config files

Verifies that when DEER_FLOW_PROJECT_ROOT is unset and cwd has no
config.yaml/extensions_config.json, AppConfig and ExtensionsConfig fall back
to the legacy backend/repo-root candidates — the backward-compat path
requested in PR #2642 review.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-05-01 22:19:50 +08:00
KiteEater 8939ccaed2 fix(uploads): enforce streaming upload limits in gateway (#2589)
* fix: enforce gateway upload limits

* fix: acquire sandbox before upload writes

* Fix upload limit config wiring

* Sanitize upload size error filenames

* test: call upload routes unwrapped

* fix: guard upload limits endpoint

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-05-01 20:19:30 +08:00
JerryLee 83938cf35a fix(subagents): propagate user context across threaded execution (#2676) 2026-05-01 16:27:18 +08:00
yangzheli 78633c69ac fix(agents): propagate agent_name into ToolRuntime.context for setup_agent (#2679)
* fix(agents): propagate agent_name into ToolRuntime.context for setup_agent (#2677)

When creating a custom agent via the web UI, SOUL.md was always written
to the global base_dir/SOUL.md instead of agents/<name>/SOUL.md.

Root cause: the bootstrap flow sends agent_name via body.context, but
two layers were broken:

1. services.py only forwarded body.context keys into config["configurable"];
   config["context"] was never populated.
2. worker.py constructed the parent Runtime with a hard-coded
   {thread_id, run_id} context, ignoring config["context"] entirely.

After the langgraph >= 1.1.9 bump (#98a5b34f), ToolRuntime.context no
longer falls back to configurable, so setup_agent's
runtime.context.get("agent_name") returned None and the tool's silent
agent_name=None -> base_dir fallback kicked in, overwriting the global
SOUL.md.

Fix:
- services.py: extract merge_run_context_overrides() and write the
  whitelisted context keys into both configurable (legacy readers) and
  context (langgraph 1.1+ ToolRuntime consumers).
- worker.py: extract _build_runtime_context() and merge config["context"]
  into the Runtime's context (without letting callers override
  thread_id/run_id).

The base_dir fallback in setup_agent_tool.py is left in place because
the IM /bootstrap channel command depends on it. That code path can
be tightened in a follow-up.

Adds regression tests covering both helpers.

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-05-01 16:00:11 +08:00
greatmengqi 8b61c94e1d fix: keep lead agent graph factory signature compatible (#2678)
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-05-01 15:43:28 +08:00
Xun 1ad1420e31 refactor(skills): Unified skill storage capability (#2613) 2026-05-01 13:23:26 +08:00
He Wang eba3b9e18d fix(config): unify log_level from config.yaml across Gateway and debug entry points (#2601)
Centralize log level parsing in `logging_level_from_config()` and
application in `apply_logging_level()` within `deerflow.config.app_config`.

- Gateway lifespan applies configured log level on startup
- `debug.py` uses shared helpers instead of local duplicates
- `apply_logging_level()` targets only `deerflow`/`app` logger hierarchies
  so third-party library verbosity is not affected; root handler levels
  are only lowered (never raised) to allow configured loggers through
  without suppressing third-party output; root logger level is not modified
- Config field description updated to clarify scope
- Tests save/restore global logging state to avoid test pollution

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-30 22:27:14 +08:00
Willem Jiang c0da278269 fix(memory): replace short-lived asyncio.run() with persistent event loop (#2627)
* fix(memory): replace short-lived asyncio.run() with persistent event loop to prevent zombie httpx connections

  The memory updater used asyncio.run() inside daemon threads, creating
  and destroying short-lived event loops on every update. Langchain
  providers (e.g. langchain-anthropic) cache httpx AsyncClient instances
  globally via @lru_cache, so SSL connections created on a loop that is
  subsequently destroyed become zombie connections in the shared pool.
  When the main agent's lead run later reuses one of these connections,
  httpx/anyio triggers RuntimeError: Event loop is closed during
  connection cleanup.

  Replace the ThreadPoolExecutor + asyncio.run() pattern with a
  _MemoryLoopRunner that maintains a single persistent event loop in a
  daemon thread for the process lifetime. Since the loop never closes,
  connections bound to it never become invalid. The _run_async_update_sync
  function now submits coroutines to this persistent loop via
  run_coroutine_threadsafe instead of creating throwaway loops.

* update the code to address the review comments

* Fix the review comments of 2615

 P1 — user_id forwarded through sync path: Added user_id parameter to _prepare_update_prompt, _finalize_update, and _do_update_memory_sync, and forwarded it to get_memory_data(agent_name, user_id=user_id) and
  save(..., user_id=user_id). The update_memory() entry point now passes user_id through both the executor.submit path and the direct call path. Added TestUserIdForwarding with two regression tests (sync + async)
   verifying get_memory_data and save receive the correct user_id.

  P2 — aupdate_memory() delegates to sync: Replaced the model.ainvoke() call with asyncio.to_thread(self._do_update_memory_sync, ...). This eliminates the unsafe async provider client path entirely — all memory
  updater entry points now use the isolated sync model.invoke() path. Updated the test from asserting ainvoke is awaited to asserting invoke is called and ainvoke is not.

  Nit — duplicate comment removed: Removed the duplicated # Matches sentences... comment on line 230.

* Chore(test): update the code of test_memory_updater

---------

Co-authored-by: rayhpeng <rayhpeng@gmail.com>
2026-04-30 17:59:57 +08:00
KiteEater 7dea1666ce fix: avoid temporary event loops in async subagent execution (#2414)
* fix: avoid temporary event loops in async subagent execution

* Rename isolated subagent loop globals

* Harden isolated subagent loop shutdown and logging

* Sort subagent executor imports

* Format subagent executor

* Remove isolated loop pool from subagent executor

* Format subagent executor cleanup

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-30 15:29:17 +08:00
Chincherry93 88d47f677f fix(nginx): add catch-all /api/ location for auth routes (#2657)
The recent refactor (7bf618de) removed the langgraph upstream and added
individual /api/* prefix locations for models, memory, mcp, skills,
agents, threads, and sandboxes. However, /api/v1/auth/* routes (login,
register, setup-status, etc.) were not covered by any explicit location
block, causing them to fall through to the frontend catch-all.

In Docker production builds, Next.js rewrites are baked at build time
with http://127.0.0.1:8001 (the gateway is unreachable from the
frontend container's localhost), resulting in ECONNREFUSED errors for
all auth operations.

Adding a catch-all `location /api/` block after the more specific
prefix/regex locations ensures all gateway API routes are properly
proxied. nginx's location matching priority means the more specific
locations above still take precedence for /api/langgraph/, /api/models,
/api/memory, /api/mcp, /api/skills, /api/agents, /api/threads/*, and
/api/sandboxes.

Co-authored-by: Chincherry93 <Chincherry93@users.noreply.github.com>
2026-04-30 15:21:22 +08:00
greatmengqi 38714b6ceb refactor: thread app_config through middleware factories (#2652)
* refactor: thread app_config through middleware factories

Continues the incremental config-refactor sequence (#2611 root, #2612 lead
path) one layer deeper into the middleware factories. Two ambient lookups
inside _build_runtime_middlewares are eliminated and the LLMErrorHandling
band-aid removed:

- _build_runtime_middlewares / build_lead_runtime_middlewares /
  build_subagent_runtime_middlewares now require app_config: AppConfig.
- get_guardrails_config() inside the factory is replaced with
  app_config.guardrails (semantically identical — same default-factory
  GuardrailsConfig — verified by direct equality check).
- LLMErrorHandlingMiddleware.__init__ now requires app_config and reads
  circuit_breaker fields directly. The class-level
  circuit_failure_threshold / circuit_recovery_timeout_sec defaults are
  removed along with the try/except (FileNotFoundError, RuntimeError):
  pass band-aid — the let-it-crash invariant the rest of the refactor
  enforces.

Caller chain (already-resolved app_config sources):
- _build_middlewares in lead_agent/agent.py: reorder so
  resolved_app_config = app_config or get_app_config() is computed BEFORE
  build_lead_runtime_middlewares is called, then passed as kwarg.
- SubagentExecutor: optional app_config parameter (mirrors the lead-agent
  pattern); _create_agent does the same `or get_app_config()` fallback at
  agent-build time, so task_tool callers don't need to plumb app_config
  through yet (typed-context plumbing for tool runtimes is a separate
  refactor).

Tests:
- test_llm_error_handling_middleware: _make_app_config helper using
  AppConfig(sandbox=SandboxConfig(use="test")) — same minimal-config
  pattern conftest already uses. Three direct LLMErrorHandlingMiddleware()
  calls each followed by post-construction circuit_breaker mutation fold
  cleanly into _build_middleware(circuit_failure_threshold=...,
  circuit_recovery_timeout_sec=...).

Verification:
- tests/test_llm_error_handling_middleware.py — 14 passed
- tests/test_subagent_executor.py — 28 passed
- tests/test_tool_error_handling_middleware.py — 6 passed
- tests/test_task_tool_core_logic.py — 18 passed (verifies task_tool
  unchanged behavior)
- Full suite: 2697 passed, 3 skipped. The single intermittent failure in
  tests/test_client_e2e.py::test_tool_call_produces_events is pre-existing
  LLM flakiness (the test asserts the model decided to call a tool;
  reproduces 1/3 on unchanged main as well).

* fix: address middleware app config review comments

* fix: satisfy app config annotation lint

* test: cover explicit app config middleware wiring

---------

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-04-30 12:41:09 +08:00
Hinotobi 74081a85a6 [security] fix(sandbox): bind local Docker ports to loopback (#2633)
* fix(sandbox): bind local Docker ports to loopback

* fix(sandbox): preserve IPv6 loopback Docker binds

* fix(sandbox): log Docker bind host selection
2026-04-30 11:40:28 +08:00
Jsonz 24a5a00679 fix: avoid duplicate call to extractReasoningContentFromMessage (#2661)
In convertToSteps(), the extractReasoningContentFromMessage function was
called twice for the same message - once to check if reasoning exists and
again to assign it to the step object. Reuse the already-extracted value
from the local variable instead.
2026-04-30 11:33:49 +08:00
He Wang 08afdcb907 feat(channels): add DingTalk channel integration (#2628)
* feat(channels): add DingTalk channel integration

Add a new DingTalk messaging channel using the dingtalk-stream SDK
with Stream Push (WebSocket), requiring no public IP. Supports both
plain sampleMarkdown replies and optional AI Card streaming for a
typewriter effect when card_template_id is configured.

- Add DingTalkChannel implementation with token management, message
  routing, allowed_users filtering, and markdown adaptation
- Register dingtalk in channel service registry and capability map
- Propagate inbound metadata to outbound messages in ChannelManager
  for DingTalk sender context (sender_staff_id, conversation_type)
- Add dingtalk-stream dependency to pyproject.toml
- Add configuration examples in config.example.yaml and .env.example
- Update all README translations with setup instructions
- Add comprehensive test suite (test_dingtalk_channel.py) and
  metadata propagation test in test_channels.py
- Update backend CLAUDE.md to document DingTalk channel

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(channels): address PR review feedback for DingTalk integration

- Replace runtime mutation of CHANNEL_CAPABILITIES with a
  `supports_streaming` property on the Channel base class, overridden
  by DingTalkChannel, FeishuChannel, and WeComChannel
- Store stream client reference and attempt graceful disconnect in
  stop(); guard _on_chatbot_message with _running check to prevent
  post-stop message processing
- Use msg.chat_id as the primary routing key in send/send_file via
  a shared _resolve_routing helper, with metadata as fallback
- Fix process() return type annotation from tuple[str, str] to
  tuple[int, str] to match AckMessage.STATUS_OK
- Protect _incoming_messages with threading.Lock for cross-thread
  safety between the Stream Push thread and the asyncio loop
- Re-add Docker Compose URL guidance removed during DingTalk setup
  docs addition in README.md
- Fix incomplete sentence in README_zh.md (missing verb "启用")

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(docs): restore plain paragraph format for Docker Compose note

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(channels): fix isinstance TypeError and add file size guard in DingTalk channel

Use tuple syntax for isinstance() type check to avoid runtime TypeError
with PEP 604 union types. Add upload size limit (20MB) before reading
files into memory. Narrow exception handlers to specific types.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(channels): propagate markdown fallback errors and validate access token response

- Re-raise exceptions in _send_markdown_fallback to prevent partial
  deliveries (files sent without accompanying text)
- Validate _get_access_token response: reject non-dict bodies, empty
  tokens, and coerce invalid expireIn to a safe default
- Add tests for both fixes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(channels): validate upload response and broaden send_file exception handling

- Validate _upload_media JSON response: handle JSONDecodeError and
  non-dict payloads gracefully by returning None
- Broaden send_file exception tuple to include TypeError and
  AttributeError for unexpected JSON shapes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(channels): fix streaming race on channel registration and slim outbound metadata

- Register channel in service before calling start() to avoid race
  where background receiver publishes inbound before registration,
  causing manager to fall back to static CHANNEL_CAPABILITIES
- Strip known-large metadata keys (raw_message, ref_msg) from outbound
  messages to prevent memory bloat from propagated inbound payloads

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Update service.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update CLAUDE.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-30 11:25:33 +08:00
sunsine 0691c4dda3 fix(security): allow disabling API docs in production via GATEWAY_ENABLE_DOCS (#2651)
* fix(security): allow disabling API docs in production via GATEWAY_ENABLE_DOCS

Expose /docs, /redoc, and /openapi.json only when GATEWAY_ENABLE_DOCS=true
(default). Setting GATEWAY_ENABLE_DOCS=false disables all three endpoints,
preventing unauthorized API surface discovery in production deployments.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test(security): add unit tests and docs for GATEWAY_ENABLE_DOCS

Add 7 tests covering default behavior, env var parsing (case-insensitive,
fail-closed), endpoint visibility, and health endpoint independence.
Update CONFIGURATION.md and CLAUDE.md with the new toggle.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style(security): apply ruff formatting to gateway app.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-30 10:58:32 +08:00
yangzheli f7b10d42e4 fix(frontend): create thread on first submit in new-agent page (#2656)
The new-agent page pre-generates a thread UUID and passed it directly
to useThreadStream, which made the LangGraph SDK POST to
/threads/{uuid}/runs/stream against a thread the backend had never
created. After PR #2566 introduced multi-tenant owner checks on the
runs endpoints, that request now 404s with "Thread not found".

Pass threadId: undefined to useThreadStream so the SDK takes the
create-then-run path. The pre-generated UUID is still forwarded via
SubmitOptions.threadId in sendMessage, so the new thread is created
with that exact id and onCreated rebinds the hook to it.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-30 06:41:54 +08:00
Willem Jiang 4a9f1d547b Merge pull request #2566 from bytedance/release/2.0-rc
introduces a complete authentication/authorization system, SQL persistence layer, run event history, user data isolation, and extensive documentation in both English and Chinese. The core additions are:

Auth system: JWT-based auth with local email/password provider, CSRF protection, rate limiting
Persistence layer: SQLAlchemy 2.0 async ORM with SQLite backend (users, threads, runs, events, feedback)
User isolation: Per-user data scoping via contextvars sentinel pattern
Frontend: Login/setup pages, AuthProvider, CSRF-aware fetcher
Documentation: Comprehensive EN/ZH docs for harness, application, tutorials
2026-04-28 21:56:35 +08:00
Willem Jiang 11afd32459 Fix the log Injection error of skills.py 2026-04-28 21:42:38 +08:00
Willem Jiang 64f4dc1639 fixed the CI build errors 2026-04-28 19:01:36 +08:00
Willem Jiang 844ad8e528 Merge branch 'main' into release/2.0-rc 2026-04-28 15:44:02 +08:00
pyp0327 395c14357b chore(adpator):Adapt MindIE engine model and improve testing and fixes (#2523)
* feat(models): 适配 MindIE引擎的模型

* test: add unit tests for MindIEChatModel adapter and fix PR review comments

* chore: update uv.lock with pytest-asyncio

* build: add pytest-asyncio to test dependencies

* fix: address PR review comments (lazy import, cache clients, safe newline escape, strict xml regex)

* fix(mindie): preserve string args without JSON quotes in XML tool call serialization

* fix(mindie): preserve string args without JSON quotes in XML tool call serialization

* test_mindie_provider:format

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix(mindie): prevent nested tool_call params from leaking into outer args

* fixed by escaping XML entities in _fix_messages and unescaping during parse, with regression tests added.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-04-28 15:09:31 +08:00
greatmengqi e82940c03d refactor: thread release config through lead path (#2612)
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-04-28 14:53:18 +08:00
DanielWalnut 6bd88fe14c fix(sandbox): block host bash traversal escapes (#2560)
* fix(sandbox): block host bash traversal escapes

Fixes #2535

* fix(sandbox): harden local bash path guards

* fix(sandbox): avoid bash cd argument false positives

* Fix the lint error

Add function to resolve and validate user data path.

* Fix the lint error

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-28 12:18:41 +08:00
DanielWalnut 39c5da94f3 fix(sandbox): prevent local custom mount symlink escapes (#2558)
* fix(sandbox): prevent local custom mount symlink escapes

Fixes #2506

* fix(sandbox): harden custom mount symlink handling

* fix(sandbox): format internal symlink directory listings
2026-04-28 11:59:46 +08:00
DanielWalnut 707ed328dd fix(skills): scan skill archives before install (#2561)
* fix(skills): scan skill archives before install

Fixes #2536

* fix(skills): scan archive support files before install

* style(skills): format archive installer

* fix(skills): address archive install review comments
2026-04-28 11:56:11 +08:00
DanielWalnut f7dfb88a30 fix(aio-sandbox): redact env values in container logs (#2562)
* fix(aio-sandbox): redact env values in container logs

Fixes #2534

* fix(aio-sandbox): address env log review comments
2026-04-28 11:47:56 +08:00
Willem Jiang 69649d8aae Fix the issues when reviewing 2566 persistant part (#2604)
* Fix the code review command of journal & event store P0,P1 issues

* Fix the code review command of journal & event store P2 issues

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update backend/packages/harness/deerflow/runtime/journal.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Refactor logger debug message formatting

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-28 11:44:40 +08:00
Willem Jiang 4e4e4f92a0 fix(security): harden auth system and fix run journal logic bug (#2593)
* fix(security): harden auth system and fix run journal logic bug

  - Fix inverted condition in RunJournal.on_chat_model_start that prevented
    first human message capture (not messages → messages)
  - Pre-hash passwords with SHA-256 before bcrypt to avoid silent 72-byte
    truncation vulnerability
  - Move load_dotenv() from module scope into get_auth_config() to prevent
    import-time os.environ mutation breaking test isolation
  - Return generic ‘Invalid token’ instead of exposing specific error
    variants (expired, malformed, invalid_signature) to clients
  - Make @require_auth independently enforce 401 instead of silently
    passing through when AuthMiddleware is absent
  - Rate-limit /setup-status endpoint with per-IP cooldown to mitigate
    initialization-state information leak
  - Document in-process rate limiter limitation for multi-worker deployments

* fix(security): return 429+Retry-After on setup-status rate limit, bound cooldown dict

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/070d0be8-99a5-46c8-85bb-6b81b5284021

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

* fix(security): add versioned password hashes with auto-migration on login

  The SHA-256 pre-hash change silently broke verification for any existing
  bcrypt-only password hashes. Introduce a <N>$ prefix scheme so hashes
  are self-describing:

  - v2 (current): bcrypt(b64(sha256(password))) with $ prefix
  - v1 (legacy): plain bcrypt, prefixed $ or bare (no prefix)

  verify_password auto-detects the version and falls back to v1 for older
  hashes. LocalAuthProvider.authenticate() now rehashes legacy hashes to v2
  on successful login via needs_rehash(), so existing users upgrade
  transparently without a dedicated migration step.

* fix(auth): harden verify_password, best-effort rehash, update require_auth docstring, downgrade journal logging

- password.py: wrap bcrypt.checkpw in try/except → return False for malformed/corrupt hashes instead of crashing
- local_provider.py: wrap auto-rehash update_user() in try/except so transient DB errors don't fail valid logins
- authz.py: update require_auth docstring to reflect independent 401 enforcement
- journal.py: downgrade on_chat_model_start from INFO to DEBUG, log only metadata (batch_count, message_counts) instead of full serialized/messages content

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/48c5cf31-a4ab-418a-982a-6343c37bb299

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

* fix(auth): address code review - narrow ValueError catch, add rehash warning log, rename num_batches

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/48c5cf31-a4ab-418a-982a-6343c37bb299

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-04-28 11:34:07 +08:00
DanielWalnut af8c0cfb78 fix(harness): constrain view_image to thread data paths (#2557)
* fix(harness): constrain view_image to thread data paths

Fixes #2530

* fix(harness): address view_image review findings

* style(harness): format view_image changes

* fix(harness): address view_image review comments
2026-04-28 11:13:17 +08:00
greatmengqi b8bc4826d8 refactor: root release config in gateway runtime (#2611)
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-04-28 00:13:04 +08:00
yangzheli 748429ef0d fix(frontend): add missing mock routes for runs-list, models, and suggestions (#2578)
* fix(frontend): add missing mock routes for runs-list, models, and suggestions

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-04-26 23:29:59 +08:00
JeffJiang ed9ebfac4d fix: enforce 'request' parameter requirement in require_auth decorator 2026-04-26 22:49:27 +08:00
JeffJiang da174dfd4d feat: implement process-local internal authentication for Gateway and enhance CSRF handling 2026-04-26 22:20:57 +08:00
Willem Jiang 897dae5475 fix the lint error of backend 2026-04-26 22:16:04 +08:00
Willem Jiang eba6c0eab2 fix unit tests of test_upload_files and test_shutdown 2026-04-26 22:10:54 +08:00
Willem Jiang 60754f0c50 fix the unit tests error of agent provider 2026-04-26 22:10:54 +08:00
Willem Jiang ac18b9c424 Apply the code reviewer suggestion of abstractmethod 2026-04-26 22:10:54 +08:00
JeffJiang 35ef8b7c13 feat: add default database configuration for AppConfig and update example config 2026-04-26 20:46:45 +08:00
JeffJiang 7bf618de67 Refactor DeerFlow to use Gateway's LangGraph-compatible API
- Updated documentation and comments to reflect the transition from LangGraph Server to Gateway.
- Changed default URLs in ChannelManager and tests to point to Gateway.
- Removed references to LangGraph Server in deployment scripts and configurations.
- Updated Nginx configuration to route API traffic to Gateway.
- Adjusted frontend configurations to utilize Gateway's API.
- Removed LangGraph service from Docker Compose files, consolidating services under Gateway.
- Added regression tests to ensure Gateway integration works as expected.

Co-authored-by: Copilot <copilot@github.com>
2026-04-26 20:38:34 +08:00
Willem Jiang 653b7ae17a Apply the code reviewer suggestion of abstractmethod 2026-04-26 20:31:14 +08:00
Willem Jiang 16aedf459a Potential fix for pull request finding 'Unused import'
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-04-26 20:29:27 +08:00
yangzheli c5d57b4533 fix: resolve make dev and test-e2e errors (#2570) 2026-04-26 17:27:32 +08:00
Willem Jiang e4ff444a71 Fixed the warning message of uv 2026-04-26 17:02:04 +08:00
Willem Jiang 64a43bc448 fix the lint error by updating the .prettierignore 2026-04-26 16:19:36 +08:00
Willem Jiang 3f88045b98 try to fix the frontend e2e test errors 2026-04-26 15:48:57 +08:00
Willem Jiang 9eca429a29 fix the lint errors in the frontend 2026-04-26 15:37:16 +08:00
Willem Jiang 28381e1383 fix the lint errors in frontend 2026-04-26 15:11:22 +08:00
Willem Jiang 829e82a9af fix the lint error in backend 2026-04-26 15:09:25 +08:00
JeffJiang 3b71e2d377 feat: add request parameter to generate_suggestions endpoint for enhanced context
Co-authored-by: Copilot <copilot@github.com>
2026-04-26 12:36:05 +08:00
JeffJiang 98a5b34f76 fix: resolve merge conflict in pnpm-lock.yaml and clean up better-auth dependencies 2026-04-26 12:31:52 +08:00
JeffJiang db5ad86381 feat: enhance chat history loading with new hooks and UI components (#2338)
* Refactor API fetch calls to use a unified fetch function; enhance chat history loading with new hooks and UI components

- Replaced `fetchWithAuth` with a generic `fetch` function across various API modules for consistency.
- Updated `useThreadStream` and `useThreadHistory` hooks to manage chat history loading, including loading states and pagination.
- Introduced `LoadMoreHistoryIndicator` component for better user experience when loading more chat history.
- Enhanced message handling in `MessageList` to accommodate new loading states and history management.
- Added support for run messages in the thread context, improving the overall message handling logic.
- Updated translations for loading indicators in English and Chinese.

* Fix test assertions for run ordering in RunManager tests

- Updated assertions in `test_list_by_thread` to reflect correct ordering of runs.
- Modified `test_list_by_thread_is_stable_when_timestamps_tie` to ensure stable ordering when timestamps are tied.
2026-04-26 11:20:17 +08:00
rayhpeng 2e05f380c4 feat(persistence): per-user filesystem isolation, run-scoped APIs, and state/history simplification (#2153)
* feat(persistence): add unified persistence layer with event store, token tracking, and feedback (#1930)

* feat(persistence): add SQLAlchemy 2.0 async ORM scaffold

Introduce a unified database configuration (DatabaseConfig) that
controls both the LangGraph checkpointer and the DeerFlow application
persistence layer from a single `database:` config section.

New modules:
- deerflow.config.database_config — Pydantic config with memory/sqlite/postgres backends
- deerflow.persistence — async engine lifecycle, DeclarativeBase with to_dict mixin, Alembic skeleton
- deerflow.runtime.runs.store — RunStore ABC + MemoryRunStore implementation

Gateway integration initializes/tears down the persistence engine in
the existing langgraph_runtime() context manager. Legacy checkpointer
config is preserved for backward compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add RunEventStore ABC + MemoryRunEventStore

Phase 2-A prerequisite for event storage: adds the unified run event
stream interface (RunEventStore) with an in-memory implementation,
RunEventsConfig, gateway integration, and comprehensive tests (27 cases).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add ORM models, repositories, DB/JSONL event stores, RunJournal, and API endpoints

Phase 2-B: run persistence + event storage + token tracking.

- ORM models: RunRow (with token fields), ThreadMetaRow, RunEventRow
- RunRepository implements RunStore ABC via SQLAlchemy ORM
- ThreadMetaRepository with owner access control
- DbRunEventStore with trace content truncation and cursor pagination
- JsonlRunEventStore with per-run files and seq recovery from disk
- RunJournal (BaseCallbackHandler) captures LLM/tool/lifecycle events,
  accumulates token usage by caller type, buffers and flushes to store
- RunManager now accepts optional RunStore for persistent backing
- Worker creates RunJournal, writes human_message, injects callbacks
- Gateway deps use factory functions (RunRepository when DB available)
- New endpoints: messages, run messages, run events, token-usage
- ThreadCreateRequest gains assistant_id field
- 92 tests pass (33 new), zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add user feedback + follow-up run association

Phase 2-C: feedback and follow-up tracking.

- FeedbackRow ORM model (rating +1/-1, optional message_id, comment)
- FeedbackRepository with CRUD, list_by_run/thread, aggregate stats
- Feedback API endpoints: create, list, stats, delete
- follow_up_to_run_id in RunCreateRequest (explicit or auto-detected
  from latest successful run on the thread)
- Worker writes follow_up_to_run_id into human_message event metadata
- Gateway deps: feedback_repo factory + getter
- 17 new tests (14 FeedbackRepository + 3 follow-up association)
- 109 total tests pass, zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test+config: comprehensive Phase 2 test coverage + deprecate checkpointer config

- config.example.yaml: deprecate standalone checkpointer section, activate
  unified database:sqlite as default (drives both checkpointer + app data)
- New: test_thread_meta_repo.py (14 tests) — full ThreadMetaRepository coverage
  including check_access owner logic, list_by_owner pagination
- Extended test_run_repository.py (+4 tests) — completion preserves fields,
  list ordering desc, limit, owner_none returns all
- Extended test_run_journal.py (+8 tests) — on_chain_error, track_tokens=false,
  middleware no ai_message, unknown caller tokens, convenience fields,
  tool_error, non-summarization custom event
- Extended test_run_event_store.py (+7 tests) — DB batch seq continuity,
  make_run_event_store factory (memory/db/jsonl/fallback/unknown)
- Extended test_phase2b_integration.py (+4 tests) — create_or_reject persists,
  follow-up metadata, summarization in history, full DB-backed lifecycle
- Fixed DB integration test to use proper fake objects (not MagicMock)
  for JSON-serializable metadata
- 157 total Phase 2 tests pass, zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* config: move default sqlite_dir to .deer-flow/data

Keep SQLite databases alongside other DeerFlow-managed data
(threads, memory) under the .deer-flow/ directory instead of a
top-level ./data folder.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(persistence): remove UTFJSON, use engine-level json_serializer + datetime.now()

- Replace custom UTFJSON type with standard sqlalchemy.JSON in all ORM
  models. Add json_serializer=json.dumps(ensure_ascii=False) to all
  create_async_engine calls so non-ASCII text (Chinese etc.) is stored
  as-is in both SQLite and Postgres.
- Change ORM datetime defaults from datetime.now(UTC) to datetime.now(),
  remove UTC imports.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): simplify deps.py with getter factory + inline repos

- Replace 6 identical getter functions with _require() factory.
- Inline 3 _make_*_repo() factories into langgraph_runtime(), call
  get_session_factory() once instead of 3 times.
- Add thread_meta upsert in start_run (services.py).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(docker): add UV_EXTRAS build arg for optional dependencies

Support installing optional dependency groups (e.g. postgres) at
Docker build time via UV_EXTRAS build arg:
  UV_EXTRAS=postgres docker compose build

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(journal): fix flush, token tracking, and consolidate tests

RunJournal fixes:
- _flush_sync: retain events in buffer when no event loop instead of
  dropping them; worker's finally block flushes via async flush().
- on_llm_end: add tool_calls filter and caller=="lead_agent" guard for
  ai_message events; mark message IDs for dedup with record_llm_usage.
- worker.py: persist completion data (tokens, message count) to RunStore
  in finally block.

Model factory:
- Auto-inject stream_usage=True for BaseChatOpenAI subclasses with
  custom api_base, so usage_metadata is populated in streaming responses.

Test consolidation:
- Delete test_phase2b_integration.py (redundant with existing tests).
- Move DB-backed lifecycle test into test_run_journal.py.
- Add tests for stream_usage injection in test_model_factory.py.
- Clean up executor/task_tool dead journal references.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): widen content type to str|dict in all store backends

Allow event content to be a dict (for structured OpenAI-format messages)
in addition to plain strings. Dict values are JSON-serialized for the DB
backend and deserialized on read; memory and JSONL backends handle dicts
natively. Trace truncation now serializes dicts to JSON before measuring.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(events): use metadata flag instead of heuristic for dict content detection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(converters): add LangChain-to-OpenAI message format converters

Pure functions langchain_to_openai_message, langchain_to_openai_completion,
langchain_messages_to_openai, and _infer_finish_reason for converting
LangChain BaseMessage objects to OpenAI Chat Completions format, used by
RunJournal for event storage. 15 unit tests added.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(converters): handle empty list content as null, clean up test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): human_message content uses OpenAI user message format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): ai_message uses OpenAI format, add ai_tool_call message event

- ai_message content now uses {"role": "assistant", "content": "..."} format
- New ai_tool_call message event emitted when lead_agent LLM responds with tool_calls
- ai_tool_call uses langchain_to_openai_message converter for consistent format
- Both events include finish_reason in metadata ("stop" or "tool_calls")

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): add tool_result message event with OpenAI tool message format

Cache tool_call_id from on_tool_start keyed by run_id as fallback for on_tool_end,
then emit a tool_result message event (role=tool, tool_call_id, content) after each
successful tool completion.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): summary content uses OpenAI system message format

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): replace llm_start/llm_end with llm_request/llm_response in OpenAI format

Add on_chat_model_start to capture structured prompt messages as llm_request events.
Replace llm_end trace events with llm_response using OpenAI Chat Completions format.
Track llm_call_index to pair request/response events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): add record_middleware method for middleware trace events

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(events): add full run sequence integration test for OpenAI content format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): align message events with checkpoint format and add middleware tag injection

- Message events (ai_message, ai_tool_call, tool_result, human_message) now use
  BaseMessage.model_dump() format, matching LangGraph checkpoint values.messages
- on_tool_end extracts tool_call_id/name/status from ToolMessage objects
- on_tool_error now emits tool_result message events with error status
- record_middleware uses middleware:{tag} event_type and middleware category
- Summarization custom events use middleware:summarize category
- TitleMiddleware injects middleware:title tag via get_config() inheritance
- SummarizationMiddleware model bound with middleware:summarize tag
- Worker writes human_message using HumanMessage.model_dump()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(threads): switch search endpoint to threads_meta table and sync title

- POST /api/threads/search now queries threads_meta table directly,
  removing the two-phase Store + Checkpointer scan approach
- Add ThreadMetaRepository.search() with metadata/status filters
- Add ThreadMetaRepository.update_display_name() for title sync
- Worker syncs checkpoint title to threads_meta.display_name on run completion
- Map display_name to values.title in search response for API compatibility

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(threads): history endpoint reads messages from event store

- POST /api/threads/{thread_id}/history now combines two data sources:
  checkpointer for checkpoint_id, metadata, title, thread_data;
  event store for messages (complete history, not truncated by summarization)
- Strip internal LangGraph metadata keys from response
- Remove full channel_values serialization in favor of selective fields

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove duplicate optional-dependencies header in pyproject.toml

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(middleware): pass tagged config to TitleMiddleware ainvoke call

Without the config, the middleware:title tag was not injected,
causing the LLM response to be recorded as a lead_agent ai_message
in run_events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: resolve merge conflict in .env.example

Keep both DATABASE_URL (from persistence-scaffold) and WECOM
credentials (from main) after the merge.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address review feedback on PR #1851

- Fix naive datetime.now() → datetime.now(UTC) in all ORM models
- Fix seq race condition in DbRunEventStore.put() with FOR UPDATE
  and UNIQUE(thread_id, seq) constraint
- Encapsulate _store access in RunManager.update_run_completion()
- Deduplicate _store.put() logic in RunManager via _persist_to_store()
- Add update_run_completion to RunStore ABC + MemoryRunStore
- Wire follow_up_to_run_id through the full create path
- Add error recovery to RunJournal._flush_sync() lost-event scenario
- Add migration note for search_threads breaking change
- Fix test_checkpointer_none_fix mock to set database=None

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update uv.lock

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address 22 review comments from CodeQL, Copilot, and Code Quality

Bug fixes:
- Sanitize log params to prevent log injection (CodeQL)
- Reset threads_meta.status to idle/error when run completes
- Attach messages only to latest checkpoint in /history response
- Write threads_meta on POST /threads so new threads appear in search

Lint fixes:
- Remove unused imports (journal.py, migrations/env.py, test_converters.py)
- Convert lambda to named function (engine.py, Ruff E731)
- Remove unused logger definitions in repos (Ruff F841)
- Add logging to JSONL decode errors and empty except blocks
- Separate assert side-effects in tests (CodeQL)
- Remove unused local variables in tests (Ruff F841)
- Fix max_trace_content truncation to use byte length, not char length

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: apply ruff format to persistence and runtime files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Potential fix for pull request finding 'Statement has no effect'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

* refactor(runtime): introduce RunContext to reduce run_agent parameter bloat

Extract checkpointer, store, event_store, run_events_config, thread_meta_repo,
and follow_up_to_run_id into a frozen RunContext dataclass. Add get_run_context()
in deps.py to build the base context from app.state singletons. start_run() uses
dataclasses.replace() to enrich per-run fields before passing ctx to run_agent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): move sanitize_log_param to app/gateway/utils.py

Extract the log-injection sanitizer from routers/threads.py into a shared
utils module and rename to sanitize_log_param (public API). Eliminates the
reverse service → router import in services.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf: use SQL aggregation for feedback stats and thread token usage

Replace Python-side counting in FeedbackRepository.aggregate_by_run with
a single SELECT COUNT/SUM query. Add RunStore.aggregate_tokens_by_thread
abstract method with SQL GROUP BY implementation in RunRepository and
Python fallback in MemoryRunStore. Simplify the thread_token_usage
endpoint to delegate to the new method, eliminating the limit=10000
truncation risk.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: annotate DbRunEventStore.put() as low-frequency path

Add docstring clarifying that put() opens a per-call transaction with
FOR UPDATE and should only be used for infrequent writes (currently
just the initial human_message event). High-throughput callers should
use put_batch() instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(threads): fall back to Store search when ThreadMetaRepository is unavailable

When database.backend=memory (default) or no SQL session factory is
configured, search_threads now queries the LangGraph Store instead of
returning 503. Returns empty list if neither Store nor repo is available.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(persistence): introduce ThreadMetaStore ABC for backend-agnostic thread metadata

Add ThreadMetaStore abstract base class with create/get/search/update/delete
interface. ThreadMetaRepository (SQL) now inherits from it. New
MemoryThreadMetaStore wraps LangGraph BaseStore for memory-mode deployments.

deps.py now always provides a non-None thread_meta_repo, eliminating all
`if thread_meta_repo is not None` guards in services.py, worker.py, and
routers/threads.py. search_threads no longer needs a Store fallback branch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(history): read messages from checkpointer instead of RunEventStore

The /history endpoint now reads messages directly from the
checkpointer's channel_values (the authoritative source) instead of
querying RunEventStore.list_messages(). The RunEventStore API is
preserved for other consumers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address new Copilot review comments

- feedback.py: validate thread_id/run_id before deleting feedback
- jsonl.py: add path traversal protection with ID validation
- run_repo.py: parse `before` to datetime for PostgreSQL compat
- thread_meta_repo.py: fix pagination when metadata filter is active
- database_config.py: use resolve_path for sqlite_dir consistency

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Implement skill self-evolution and skill_manage flow (#1874)

* chore: ignore .worktrees directory

* Add skill_manage self-evolution flow

* Fix CI regressions for skill_manage

* Address PR review feedback for skill evolution

* fix(skill-evolution): preserve history on delete

* fix(skill-evolution): tighten scanner fallbacks

* docs: add skill_manage e2e evidence screenshot

* fix(skill-manage): avoid blocking fs ops in session runtime

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

* fix(config): resolve sqlite_dir relative to CWD, not Paths.base_dir

resolve_path() resolves relative to Paths.base_dir (.deer-flow),
which double-nested the path to .deer-flow/.deer-flow/data/app.db.
Use Path.resolve() (CWD-relative) instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Feature/feishu receive file (#1608)

* feat(feishu): add channel file materialization hook for inbound messages

- Introduce Channel.receive_file(msg, thread_id) as a base method for file materialization; default is no-op.
- Implement FeishuChannel.receive_file to download files/images from Feishu messages, save to sandbox, and inject virtual paths into msg.text.
- Update ChannelManager to call receive_file for any channel if msg.files is present, enabling downstream model access to user-uploaded files.
- No impact on Slack/Telegram or other channels (they inherit the default no-op).

* style(backend): format code with ruff for lint compliance

- Auto-formatted packages/harness/deerflow/agents/factory.py and tests/test_create_deerflow_agent.py using `ruff format`
- Ensured both files conform to project linting standards
- Fixes CI lint check failures caused by code style issues

* fix(feishu): handle file write operation asynchronously to prevent blocking

* fix(feishu): rename GetMessageResourceRequest to _GetMessageResourceRequest and remove redundant code

* test(feishu): add tests for receive_file method and placeholder replacement

* fix(manager): remove unnecessary type casting for channel retrieval

* fix(feishu): update logging messages to reflect resource handling instead of image

* fix(feishu): sanitize filename by replacing invalid characters in file uploads

* fix(feishu): improve filename sanitization and reorder image key handling in message processing

* fix(feishu): add thread lock to prevent filename conflicts during file downloads

* fix(test): correct bad merge in test_feishu_parser.py

* chore: run ruff and apply formatting cleanup
fix(feishu): preserve rich-text attachment order and improve fallback filename handling

* fix(docker): restore gateway env vars and fix langgraph empty arg issue (#1915)

Two production docker-compose.yaml bugs prevent `make up` from working:

1. Gateway missing DEER_FLOW_CONFIG_PATH and DEER_FLOW_EXTENSIONS_CONFIG_PATH
   environment overrides. Added in fb2d99f (#1836) but accidentally reverted
   by ca2fb95 (#1847). Without them, gateway reads host paths from .env via
   env_file, causing FileNotFoundError inside the container.

2. Langgraph command fails when LANGGRAPH_ALLOW_BLOCKING is unset (default).
   Empty $${allow_blocking} inserts a bare space between flags, causing
   ' --no-reload' to be parsed as unexpected extra argument. Fix by building
   args string first and conditionally appending --allow-blocking.

Co-authored-by: cooper <cooperfu@tencent.com>

* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities (#1904)

* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities

Fix `<button>` inside `<a>` invalid HTML in artifact components and add
missing `noopener,noreferrer` to `window.open` calls to prevent reverse
tabnabbing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(frontend): address Copilot review on tabnabbing and double-tab-open

Remove redundant parent onClick on web_fetch ChainOfThoughtStep to
prevent opening two tabs on link click, and explicitly null out
window.opener after window.open() for defensive tabnabbing hardening.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(persistence): organize entities into per-entity directories

Restructure the persistence layer from horizontal "models/ + repositories/"
split into vertical entity-aligned directories. Each entity (thread_meta,
run, feedback) now owns its ORM model, abstract interface (where applicable),
and concrete implementations under a single directory with an aggregating
__init__.py for one-line imports.

Layout:
  persistence/thread_meta/{base,model,sql,memory}.py
  persistence/run/{model,sql}.py
  persistence/feedback/{model,sql}.py

models/__init__.py is kept as a facade so Alembic autogenerate continues to
discover all ORM tables via Base.metadata. RunEventRow remains under
models/run_event.py because its storage implementation lives in
runtime/events/store/db.py and has no matching repository directory.

The repositories/ directory is removed entirely. All call sites in
gateway/deps.py and tests are updated to import from the new entity
packages, e.g.:

    from deerflow.persistence.thread_meta import ThreadMetaRepository
    from deerflow.persistence.run import RunRepository
    from deerflow.persistence.feedback import FeedbackRepository

Full test suite passes (1690 passed, 14 skipped).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(gateway): sync thread rename and delete through ThreadMetaStore

The POST /threads/{id}/state endpoint previously synced title changes
only to the LangGraph Store via _store_upsert. In sqlite mode the search
endpoint reads from the ThreadMetaRepository SQL table, so renames never
appeared in /threads/search until the next agent run completed (worker.py
syncs title from checkpoint to thread_meta in its finally block).

Likewise the DELETE /threads/{id} endpoint cleaned up the filesystem,
Store, and checkpointer but left the threads_meta row orphaned in sqlite,
so deleted threads kept appearing in /threads/search.

Fix both endpoints by routing through the ThreadMetaStore abstraction
which already has the correct sqlite/memory implementations wired up by
deps.py. The rename path now calls update_display_name() and the delete
path calls delete() — both work uniformly across backends.

Verified end-to-end with curl in gateway mode against sqlite backend.
Existing test suite (1690 passed) and focused router/repo tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): route all thread metadata access through ThreadMetaStore

Following the rename/delete bug fix in PR1, migrate the remaining direct
LangGraph Store reads/writes in the threads router and services to the
ThreadMetaStore abstraction so that the sqlite and memory backends behave
identically and the legacy dual-write paths can be removed.

Migrated endpoints (threads.py):
- create_thread: idempotency check + write now use thread_meta_repo.get/create
  instead of dual-writing the LangGraph Store and the SQL row.
- get_thread: reads from thread_meta_repo.get; the checkpoint-only fallback
  for legacy threads is preserved.
- patch_thread: replaced _store_get/_store_put with thread_meta_repo.update_metadata.
- delete_thread_data: dropped the legacy store.adelete; thread_meta_repo.delete
  already covers it.

Removed dead code (services.py):
- _upsert_thread_in_store — redundant with the immediately following
  thread_meta_repo.create() call.
- _sync_thread_title_after_run — worker.py's finally block already syncs
  the title via thread_meta_repo.update_display_name() after each run.

Removed dead code (threads.py):
- _store_get / _store_put / _store_upsert helpers (no remaining callers).
- THREADS_NS constant.
- get_store import (router no longer touches the LangGraph Store directly).

New abstract method:
- ThreadMetaStore.update_metadata(thread_id, metadata) merges metadata into
  the thread's metadata field. Implemented in both ThreadMetaRepository (SQL,
  read-modify-write inside one session) and MemoryThreadMetaStore. Three new
  unit tests cover merge / empty / nonexistent behaviour.

Net change: -134 lines. Full test suite: 1693 passed, 14 skipped.
Verified end-to-end with curl in gateway mode against sqlite backend
(create / patch / get / rename / search / delete).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: JilongSun <965640067@qq.com>
Co-authored-by: jie <49781832+stan-fu@users.noreply.github.com>
Co-authored-by: cooper <cooperfu@tencent.com>
Co-authored-by: yangzheli <43645580+yangzheli@users.noreply.github.com>

* feat(auth): release-validation pass for 2.0-rc — 12 blockers + simplify follow-ups (#2008)

* feat(auth): introduce backend auth module

Port RFC-001 authentication core from PR #1728:
- JWT token handling (create_access_token, decode_token, TokenPayload)
- Password hashing (bcrypt) with verify_password
- SQLite UserRepository with base interface
- Provider Factory pattern (LocalAuthProvider)
- CLI reset_admin tool
- Auth-specific errors (AuthErrorCode, TokenError, AuthErrorResponse)

Deps:
- bcrypt>=4.0.0
- pyjwt>=2.9.0
- email-validator>=2.0.0
- backend/uv.toml pins public PyPI index

Tests: 12 pure unit tests (test_auth_config.py, test_auth_errors.py).

Scope note: authz.py, test_auth.py, and test_auth_type_system.py are
deferred to commit 2 because they depend on middleware and deps wiring
that is not yet in place. Commit 1 stays "pure new files only" as the
spec mandates.

* feat(auth): wire auth end-to-end (middleware + frontend replacement)

Backend:
- Port auth_middleware, csrf_middleware, langgraph_auth, routers/auth
- Port authz decorator (owner_filter_key defaults to 'owner_id')
- Merge app.py: register AuthMiddleware + CSRFMiddleware + CORS, add
  _ensure_admin_user lifespan hook, _migrate_orphaned_threads helper,
  register auth router
- Merge deps.py: add get_local_provider, get_current_user_from_request,
  get_optional_user_from_request; keep get_current_user as thin str|None
  adapter for feedback router
- langgraph.json: add auth path pointing to langgraph_auth.py:auth
- Rename metadata['user_id'] -> metadata['owner_id'] in langgraph_auth
  (both metadata write and LangGraph filter dict) + test fixtures

Frontend:
- Delete better-auth library and api catch-all route
- Remove better-auth npm dependency and env vars (BETTER_AUTH_SECRET,
  BETTER_AUTH_GITHUB_*) from env.js
- Port frontend/src/core/auth/* (AuthProvider, gateway-config,
  proxy-policy, server-side getServerSideUser, types)
- Port frontend/src/core/api/fetcher.ts
- Port (auth)/layout, (auth)/login, (auth)/setup pages
- Rewrite workspace/layout.tsx as server component that calls
  getServerSideUser and wraps in AuthProvider
- Port workspace/workspace-content.tsx for the client-side sidebar logic

Tests:
- Port 5 auth test files (test_auth, test_auth_middleware,
  test_auth_type_system, test_ensure_admin, test_langgraph_auth)
- 176 auth tests PASS

After this commit: login/logout/registration flow works, but persistence
layer does not yet filter by owner_id. Commit 4 closes that gap.

* feat(auth): account settings page + i18n

- Port account-settings-page.tsx (change password, change email, logout)
- Wire into settings-dialog.tsx as new "account" section with UserIcon,
  rendered first in the section list
- Add i18n keys:
  - en-US/zh-CN: settings.sections.account ("Account" / "账号")
  - en-US/zh-CN: button.logout ("Log out" / "退出登录")
  - types.ts: matching type declarations

* feat(auth): enforce owner_id across 2.0-rc persistence layer

Add request-scoped contextvar-based owner filtering to threads_meta,
runs, run_events, and feedback repositories. Router code is unchanged
— isolation is enforced at the storage layer so that any caller that
forgets to pass owner_id still gets filtered results, and new routes
cannot accidentally leak data.

Core infrastructure
-------------------
- deerflow/runtime/user_context.py (new):
  - ContextVar[CurrentUser | None] with default None
  - runtime_checkable CurrentUser Protocol (structural subtype with .id)
  - set/reset/get/require helpers
  - AUTO sentinel + resolve_owner_id(value, method_name) for sentinel
    three-state resolution: AUTO reads contextvar, explicit str
    overrides, explicit None bypasses the filter (for migration/CLI)

Repository changes
------------------
- ThreadMetaRepository: create/get/search/update_*/delete gain
  owner_id=AUTO kwarg; read paths filter by owner, writes stamp it,
  mutations check ownership before applying
- RunRepository: put/get/list_by_thread/delete gain owner_id=AUTO kwarg
- FeedbackRepository: create/get/list_by_run/list_by_thread/delete
  gain owner_id=AUTO kwarg
- DbRunEventStore: list_messages/list_events/list_messages_by_run/
  count_messages/delete_by_thread/delete_by_run gain owner_id=AUTO
  kwarg. Write paths (put/put_batch) read contextvar softly: when a
  request-scoped user is available, owner_id is stamped; background
  worker writes without a user context pass None which is valid
  (orphan row to be bound by migration)

Schema
------
- persistence/models/run_event.py: RunEventRow.owner_id = Mapped[
  str | None] = mapped_column(String(64), nullable=True, index=True)
- No alembic migration needed: 2.0 ships fresh, Base.metadata.create_all
  picks up the new column automatically

Middleware
----------
- auth_middleware.py: after cookie check, call get_optional_user_from_
  request to load the real User, stamp it into request.state.user AND
  the contextvar via set_current_user, reset in a try/finally. Public
  paths and unauthenticated requests continue without contextvar, and
  @require_auth handles the strict 401 path

Test infrastructure
-------------------
- tests/conftest.py: @pytest.fixture(autouse=True) _auto_user_context
  sets a default SimpleNamespace(id="test-user-autouse") on every test
  unless marked @pytest.mark.no_auto_user. Keeps existing 20+
  persistence tests passing without modification
- pyproject.toml [tool.pytest.ini_options]: register no_auto_user
  marker so pytest does not emit warnings for opt-out tests
- tests/test_user_context.py: 6 tests covering three-state semantics,
  Protocol duck typing, and require/optional APIs
- tests/test_thread_meta_repo.py: one test updated to pass owner_id=
  None explicitly where it was previously relying on the old default

Test results
------------
- test_user_context.py: 6 passed
- test_auth*.py + test_langgraph_auth.py + test_ensure_admin.py: 127
- test_run_event_store / test_run_repository / test_thread_meta_repo
  / test_feedback: 92 passed
- Full backend suite: 1905 passed, 2 failed (both @requires_llm flaky
  integration tests unrelated to auth), 1 skipped

* feat(auth): extend orphan migration to 2.0-rc persistence tables

_ensure_admin_user now runs a three-step pipeline on every boot:

  Step 1 (fatal):     admin user exists / is created / password is reset
  Step 2 (non-fatal): LangGraph store orphan threads → admin
  Step 3 (non-fatal): SQL persistence tables → admin
    - threads_meta
    - runs
    - run_events
    - feedback

Each step is idempotent. The fatal/non-fatal split mirrors PR #1728's
original philosophy: admin creation failure blocks startup (the system
is unusable without an admin), whereas migration failures log a warning
and let the service proceed (a partial migration is recoverable; a
missing admin is not).

Key helpers
-----------
- _iter_store_items(store, namespace, *, page_size=500):
  async generator that cursor-paginates across LangGraph store pages.
  Fixes PR #1728's hardcoded limit=1000 bug that would silently lose
  orphans beyond the first page.

- _migrate_orphaned_threads(store, admin_user_id):
  Rewritten to use _iter_store_items. Returns the migrated count so the
  caller can log it; raises only on unhandled exceptions.

- _migrate_orphan_sql_tables(admin_user_id):
  Imports the 4 ORM models lazily, grabs the shared session factory,
  runs one UPDATE per table in a single transaction, commits once.
  No-op when no persistence backend is configured (in-memory dev).

Tests: test_ensure_admin.py (8 passed)

* test(auth): port AUTH test plan docs + lint/format pass

- Port backend/docs/AUTH_TEST_PLAN.md and AUTH_UPGRADE.md from PR #1728
- Rename metadata.user_id → metadata.owner_id in AUTH_TEST_PLAN.md
  (4 occurrences from the original PR doc)
- ruff auto-fix UP037 in sentinel type annotations: drop quotes around
  "str | None | _AutoSentinel" now that from __future__ import
  annotations makes them implicit string forms
- ruff format: 2 files (app/gateway/app.py, runtime/user_context.py)

Note on test coverage additions:
- conftest.py autouse fixture was already added in commit 4 (had to
  be co-located with the repository changes to keep pre-existing
  persistence tests passing)
- cross-user isolation E2E tests (test_owner_isolation.py) deferred
  — enforcement is already proven by the 98-test repository suite
  via the autouse fixture + explicit _AUTO sentinel exercises
- New test cases (TC-API-17..20, TC-ATK-13, TC-MIG-01..07) listed
  in AUTH_TEST_PLAN.md are deferred to a follow-up PR — they are
  manual-QA test cases rather than pytest code, and the spec-level
  coverage is already met by test_user_context.py + the 98-test
  repository suite.

Final test results:
- Auth suite (test_auth*, test_langgraph_auth, test_ensure_admin,
  test_user_context): 186 passed
- Persistence suite (test_run_event_store, test_run_repository,
  test_thread_meta_repo, test_feedback): 98 passed
- Lint: ruff check + ruff format both clean

* test(auth): add cross-user isolation test suite

10 tests exercising the storage-layer owner filter by manually
switching the user_context contextvar between two users. Verifies
the safety invariant:

  After a repository write with owner_id=A, a subsequent read with
  owner_id=B must not return the row, and vice versa.

Covers all 4 tables that own user-scoped data:

TC-API-17  threads_meta  — read, search, update, delete cross-user
TC-API-18  runs          — get, list_by_thread, delete cross-user
TC-API-19  run_events    — list_messages, list_events, count_messages,
                           delete_by_thread (CRITICAL: raw conversation
                           content leak vector)
TC-API-20  feedback      — get, list_by_run, delete cross-user

Plus two meta-tests verifying the sentinel pattern itself:
- AUTO + unset contextvar raises RuntimeError
- explicit owner_id=None bypasses the filter (migration escape hatch)

Architecture note
-----------------
These tests bypass the HTTP layer by design. The full chain
(cookie → middleware → contextvar → repository) is covered piecewise:

- test_auth_middleware.py: middleware sets contextvar from cookies
- test_owner_isolation.py: repositories enforce isolation when
  contextvar is set to different users

Together they prove the end-to-end safety property without the
ceremony of spinning up a full TestClient + in-memory DB for every
router endpoint.

Tests pass: 231 (full auth + persistence + isolation suite)
Lint: clean

* refactor(auth): migrate user repository to SQLAlchemy ORM

Move the users table into the shared persistence engine so auth
matches the pattern of threads_meta, runs, run_events, and feedback —
one engine, one session factory, one schema init codepath.

New files
---------
- persistence/user/__init__.py, persistence/user/model.py: UserRow
  ORM class with partial unique index on (oauth_provider, oauth_id)
- Registered in persistence/models/__init__.py so
  Base.metadata.create_all() picks it up

Modified
--------
- auth/repositories/sqlite.py: rewritten as async SQLAlchemy,
  identical constructor pattern to the other four repositories
  (def __init__(self, session_factory) + self._sf = session_factory)
- auth/config.py: drop users_db_path field — storage is configured
  through config.database like every other table
- deps.py/get_local_provider: construct SQLiteUserRepository with
  the shared session factory, fail fast if engine is not initialised
- tests/test_auth.py: rewrite test_sqlite_round_trip_new_fields to
  use the shared engine (init_engine + close_engine in a tempdir)
- tests/test_auth_type_system.py: add per-test autouse fixture that
  spins up a scratch engine and resets deps._cached_* singletons

* refactor(auth): remove SQL orphan migration (unused in supported scenarios)

The _migrate_orphan_sql_tables helper existed to bind NULL owner_id
rows in threads_meta, runs, run_events, and feedback to the admin on
first boot. But in every supported upgrade path, it's a no-op:

  1. Fresh install: create_all builds fresh tables, no legacy rows
  2. No-auth → with-auth (no existing persistence DB): persistence
     tables are created fresh by create_all, no legacy rows
  3. No-auth → with-auth (has existing persistence DB from #1930):
     NOT a supported upgrade path — "有 DB 到有 DB" schema evolution
     is out of scope; users wipe DB or run manual ALTER

So the SQL orphan migration never has anything to do in the
supported matrix. Delete the function, simplify _ensure_admin_user
from a 3-step pipeline to a 2-step one (admin creation + LangGraph
store orphan migration only).

LangGraph store orphan migration stays: it serves the real
"no-auth → with-auth" upgrade path where a user's existing LangGraph
thread metadata has no owner_id field and needs to be stamped with
the newly-created admin's id.

Tests: 284 passed (auth + persistence + isolation)
Lint: clean

* security(auth): write initial admin password to 0600 file instead of logs

CodeQL py/clear-text-logging-sensitive-data flagged 3 call sites that
logged the auto-generated admin password to stdout via logger.info().
Production log aggregators (ELK/Splunk/etc) would have captured those
cleartext secrets. Replace with a shared helper that writes to
.deer-flow/admin_initial_credentials.txt with mode 0600, and log only
the path.

New file
--------
- app/gateway/auth/credential_file.py: write_initial_credentials()
  helper. Takes email, password, and a "initial"/"reset" label.
  Creates .deer-flow/ if missing, writes a header comment plus the
  email+password, chmods 0o600, returns the absolute Path.

Modified
--------
- app/gateway/app.py: both _ensure_admin_user paths (fresh creation
  + needs_setup password reset) now write to file and log the path
- app/gateway/auth/reset_admin.py: rewritten to use the shared ORM
  repo (SQLiteUserRepository with session_factory) and the
  credential_file helper. The previous implementation was broken
  after the earlier ORM refactor — it still imported _get_users_conn
  and constructed SQLiteUserRepository() without a session factory.

No tests changed — the three password-log sites are all exercised
via existing test_ensure_admin.py which checks that startup
succeeds, not that a specific string appears in logs.

CodeQL alerts 272, 283, 284: all resolved.

* security(auth): strict JWT validation in middleware (fix junk cookie bypass)

AUTH_TEST_PLAN test 7.5.8 expects junk cookies to be rejected with
401. The previous middleware behaviour was "presence-only": check
that some access_token cookie exists, then pass through. In
combination with my Task-12 decision to skip @require_auth
decorators on routes, this created a gap where a request with any
cookie-shaped string (e.g. access_token=not-a-jwt) would bypass
authentication on routes that do not touch the repository
(/api/models, /api/mcp/config, /api/memory, /api/skills, …).

Fix: middleware now calls get_current_user_from_request() strictly
and catches the resulting HTTPException to render a 401 with the
proper fine-grained error code (token_invalid, token_expired,
user_not_found, …). On success it stamps request.state.user and
the contextvar so repository-layer owner filters work downstream.

The 4 old "_with_cookie_passes" tests in test_auth_middleware.py
were written for the presence-only behaviour; they asserted that
a junk cookie would make the handler return 200. They are renamed
to "_with_junk_cookie_rejected" and their assertions flipped to
401. The negative path (no cookie → 401 not_authenticated)
is unchanged.

Verified:
  no cookie       → 401 not_authenticated
  junk cookie     → 401 token_invalid     (the fixed bug)
  expired cookie  → 401 token_expired

Tests: 284 passed (auth + persistence + isolation)
Lint: clean

* security(auth): wire @require_permission(owner_check=True) on isolation routes

Apply the require_permission decorator to all 28 routes that take a
{thread_id} path parameter. Combined with the strict middleware
(previous commit), this gives the double-layer protection that
AUTH_TEST_PLAN test 7.5.9 documents:

  Layer 1 (AuthMiddleware): cookie + JWT validation, rejects junk
                            cookies and stamps request.state.user
  Layer 2 (@require_permission with owner_check=True): per-resource
                            ownership verification via
                            ThreadMetaStore.check_access — returns
                            404 if a different user owns the thread

The decorator's owner_check branch is rewritten to use the SQL
thread_meta_repo (the 2.0-rc persistence layer) instead of the
LangGraph store path that PR #1728 used (_store_get / get_store
in routers/threads.py). The inject_record convenience is dropped
— no caller in 2.0 needs the LangGraph blob, and the SQL repo has
a different shape.

Routes decorated (28 total):
- threads.py: delete, patch, get, get-state, post-state, post-history
- thread_runs.py: post-runs, post-runs-stream, post-runs-wait,
  list_runs, get_run, cancel_run, join_run, stream_existing_run,
  list_thread_messages, list_run_messages, list_run_events,
  thread_token_usage
- feedback.py: create, list, stats, delete
- uploads.py: upload (added Request param), list, delete
- artifacts.py: get_artifact
- suggestions.py: generate (renamed body parameter to avoid
  conflict with FastAPI Request)

Test fixes:
- test_suggestions_router.py: bypass the decorator via __wrapped__
  (the unit tests cover parsing logic, not auth — no point spinning
  up a thread_meta_repo just to test JSON unwrapping)
- test_auth_middleware.py 4 fake-cookie tests: already updated in
  the previous commit (745bf432)

Tests: 293 passed (auth + persistence + isolation + suggestions)
Lint: clean

* security(auth): defense-in-depth fixes from release validation pass

Eight findings caught while running the AUTH_TEST_PLAN end-to-end against
the deployed sg_dev stack. Each is a pre-condition for shipping
release/2.0-rc that the previous PRs missed.

Backend hardening
- routers/auth.py: rate limiter X-Real-IP now requires AUTH_TRUSTED_PROXIES
  whitelist (CIDR/IP allowlist). Without nginx in front, the previous code
  honored arbitrary X-Real-IP, letting an attacker rotate the header to
  fully bypass the per-IP login lockout.
- routers/auth.py: 36-entry common-password blocklist via Pydantic
  field_validator on RegisterRequest + ChangePasswordRequest. The shared
  _validate_strong_password helper keeps the constraint in one place.
- routers/threads.py: ThreadCreateRequest + ThreadPatchRequest strip
  server-reserved metadata keys (owner_id, user_id) via Pydantic
  field_validator so a forged value can never round-trip back to other
  clients reading the same thread. The actual ownership invariant stays
  on the threads_meta row; this closes the metadata-blob echo gap.
- authz.py + thread_meta/sql.py: require_permission gains a require_existing
  flag plumbed through check_access(require_existing=True). Destructive
  routes (DELETE/PATCH/state-update/runs/feedback) now treat a missing
  thread_meta row as 404 instead of "untracked legacy thread, allow",
  closing the cross-user delete-idempotence gap where any user could
  successfully DELETE another user's deleted thread.
- repositories/sqlite.py + base.py: update_user raises UserNotFoundError
  on a vanished row instead of silently returning the input. Concurrent
  delete during password reset can no longer look like a successful update.
- runtime/user_context.py: resolve_owner_id() coerces User.id (UUID) to
  str at the contextvar boundary so SQLAlchemy String(64) columns can
  bind it. The whole 2.0-rc isolation pipeline was previously broken
  end-to-end (POST /api/threads → 500 "type 'UUID' is not supported").
- persistence/engine.py: SQLAlchemy listener enables PRAGMA journal_mode=WAL,
  synchronous=NORMAL, foreign_keys=ON on every new SQLite connection.
  TC-UPG-06 in the test plan expects WAL; previous code shipped with the
  default 'delete' journal.
- auth_middleware.py: stamp request.state.auth = AuthContext(...) so
  @require_permission's short-circuit fires; previously every isolation
  request did a duplicate JWT decode + users SELECT. Also unifies the
  401 payload through AuthErrorResponse(...).model_dump().
- app.py: _ensure_admin_user restructure removes the noqa F821 scoping
  bug where 'password' was referenced outside the branch that defined it.
  New _announce_credentials helper absorbs the duplicate log block in
  the fresh-admin and reset-admin branches.

* fix(frontend+nginx): rollout CSRF on every state-changing client path

The frontend was 100% broken in gateway-pro mode for any user trying to
open a specific chat thread. Three cumulative bugs each silently
masked the next.

LangGraph SDK CSRF gap (api-client.ts)
- The Client constructor took only apiUrl, no defaultHeaders, no fetch
  interceptor. The SDK's internal fetch never sent X-CSRF-Token, so
  every state-changing /api/langgraph-compat/* call (runs/stream,
  threads/search, threads/{tid}/history, ...) hit CSRFMiddleware and
  got 403 before reaching the auth check. UI symptom: empty thread page
  with no error message; the SPA's hooks swallowed the rejection.
- Fix: pass an onRequest hook that injects X-CSRF-Token from the
  csrf_token cookie per request. Reading the cookie per call (not at
  construction time) handles login / logout / password-change cookie
  rotation transparently. The SDK's prepareFetchOptions calls
  onRequest for both regular requests AND streaming/SSE/reconnect, so
  the same hook covers runs.stream and runs.joinStream.

Raw fetch CSRF gap (7 files)
- Audit: 11 frontend fetch sites, only 2 included CSRF (login/setup +
  account-settings change-password). The other 7 routed through raw
  fetch() with no header — suggestions, memory, agents, mcp, skills,
  uploads, and the local thread cleanup hook all 403'd silently.
- Fix: enhance fetcher.ts:fetchWithAuth to auto-inject X-CSRF-Token on
  POST/PUT/DELETE/PATCH from a single shared readCsrfCookie() helper.
  Convert all 7 raw fetch() callers to fetchWithAuth so the contract
  is centrally enforced. api-client.ts and fetcher.ts share
  readCsrfCookie + STATE_CHANGING_METHODS to avoid drift.

nginx routing + buffering (nginx.local.conf)
- The auth feature shipped without updating the nginx config: per-API
  explicit location blocks but no /api/v1/auth/, /api/feedback, /api/runs.
  The frontend's client-side fetches to /api/v1/auth/login/local 404'd
  from the Next.js side because nginx routed /api/* to the frontend.
- Fix: add catch-all `location /api/` that proxies to the gateway.
  nginx longest-prefix matching keeps the explicit blocks (/api/models,
  /api/threads regex, /api/langgraph/, ...) winning for their paths.
- Fix: disable proxy_buffering + proxy_request_buffering for the
  frontend `location /` block. Without it, nginx tries to spool large
  Next.js chunks into /var/lib/nginx/proxy (root-owned) and fails with
  Permission denied → ERR_INCOMPLETE_CHUNKED_ENCODING → ChunkLoadError.

* test(auth): release-validation test infra and new coverage

Test fixtures and unit tests added during the validation pass.

Router test helpers (NEW: tests/_router_auth_helpers.py)
- make_authed_test_app(): builds a FastAPI test app with a stub
  middleware that stamps request.state.user + request.state.auth and a
  permissive thread_meta_repo mock. TestClient-based router tests
  (test_artifacts_router, test_threads_router) use it instead of bare
  FastAPI() so the new @require_permission(owner_check=True) decorators
  short-circuit cleanly.
- call_unwrapped(): walks the __wrapped__ chain to invoke the underlying
  handler without going through the authz wrappers. Direct-call tests
  (test_uploads_router) use it. Typed with ParamSpec so the wrapped
  signature flows through.

Backend test additions
- test_auth.py: 7 tests for the new _get_client_ip trust model (no
  proxy / trusted proxy / untrusted peer / XFF rejection / invalid
  CIDR / no client). 5 tests for the password blocklist (literal,
  case-insensitive, strong password accepted, change-password binding,
  short-password length-check still fires before blocklist).
  test_update_user_raises_when_row_concurrently_deleted: closes a
  shipped-without-coverage gap on the new UserNotFoundError contract.
- test_thread_meta_repo.py: 4 tests for check_access(require_existing=True)
  — strict missing-row denial, strict owner match, strict owner mismatch,
  strict null-owner still allowed (shared rows survive the tightening).
- test_ensure_admin.py: 3 tests for _migrate_orphaned_threads /
  _iter_store_items pagination, covering the TC-UPG-02 upgrade story
  end-to-end via mock store. Closes the gap where the cursor pagination
  was untested even though the previous PR rewrote it.
- test_threads_router.py: 5 tests for _strip_reserved_metadata
  (owner_id removal, user_id removal, safe-keys passthrough, empty
  input, both-stripped).
- test_auth_type_system.py: replace "password123" fixtures with
  Tr0ub4dor3a / AnotherStr0ngPwd! so the new password blocklist
  doesn't reject the test data.

* docs(auth): refresh TC-DOCKER-05 + document Docker validation gap

- AUTH_TEST_PLAN.md TC-DOCKER-05: the previous expectation
  ("admin password visible in docker logs") was stale after the simplify
  pass that moved credentials to a 0600 file. The grep "Password:" check
  would have silently failed and given a false sense of coverage. New
  expectation matches the actual file-based path: 0600 file in
  DEER_FLOW_HOME, log shows the path (not the secret), reverse-grep
  asserts no leaked password in container logs.
- NEW: docs/AUTH_TEST_DOCKER_GAP.md documents the only un-executed
  block in the test plan (TC-DOCKER-01..06). Reason: sg_dev validation
  host has no Docker daemon installed. The doc maps each Docker case
  to an already-validated bare-metal equivalent (TC-1.1, TC-REENT-01,
  TC-API-02 etc.) so the gap is auditable, and includes pre-flight
  reproduction steps for whoever has Docker available.

---------

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>

* refactor(persistence): unify SQLite to single deerflow.db and move checkpointer to runtime

Merge checkpoints.db and app.db into a single deerflow.db file (WAL mode
handles concurrent access safely). Move checkpointer module from
agents/checkpointer to runtime/checkpointer to better reflect its role
as a runtime infrastructure concern.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(persistence): rename owner_id to user_id and thread_meta_repo to thread_store

Rename owner_id to user_id across all persistence models, repositories,
stores, routers, and tests for clearer semantics. Rename thread_meta_repo
to thread_store for consistency with run_store/run_event_store naming.
Add ThreadMetaStore return type annotation to get_thread_store().

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): unify ThreadMetaStore interface with user isolation and factory

Add user_id parameter to all ThreadMetaStore abstract methods. Implement
owner isolation in MemoryThreadMetaStore with _get_owned_record helper.
Add check_access to base class and memory implementation. Add
make_thread_store factory to simplify deps.py initialization. Add
memory-backend isolation tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(feedback): add UNIQUE(thread_id, run_id, user_id) constraint

Add UNIQUE constraint to FeedbackRow to enforce one feedback per user per run,
enabling upsert behavior in Task 2. Update tests to use distinct user_ids for
multiple feedback records per run, and pass user_id=None to list_by_run for
admin-style queries that bypass user isolation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(feedback): add upsert() method with UNIQUE enforcement

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(feedback): add delete_by_run() and list_by_thread_grouped()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(feedback): add PUT upsert and DELETE-by-run endpoints

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(feedback): enrich messages endpoint with per-run feedback data

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(feedback): add frontend feedback API client

Adds upsertFeedback and deleteFeedback API functions backed by
fetchWithAuth, targeting the /api/threads/{id}/runs/{id}/feedback
endpoint.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(feedback): wire feedback data into message rendering for history echo

Adds useThreadFeedback hook that fetches run-level feedback from the
messages API and builds a runId->FeedbackData map. MessageList now calls
this hook and passes feedback and runId to each MessageListItem so
previously-submitted thumbs are pre-filled when revisiting a thread.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(feedback): correct run_id mapping for feedback echo

The feedbackMap was keyed by run_id but looked up by LangGraph message ID.
Fixed by tracking AI message ordinal index to correlate event store
run_ids with LangGraph SDK messages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(feedback): use real threadId and refresh after stream

- Pass threadId prop to MessageListItem instead of reading "new" from URL params
- Invalidate thread-feedback query on stream finish so buttons appear immediately
- Show feedback buttons always visible, copy button on hover only

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style(feedback): group copy and feedback buttons together on the left

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style(feedback): always show toolbar buttons without hover

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): stream hang when run_events.backend=db

DbRunEventStore._user_id_from_context() returned user.id without
coercing it to str. User.id is a Pydantic UUID, and aiosqlite cannot
bind a raw UUID object to a VARCHAR column, so the INSERT for the
initial human_message event silently rolled back and raised out of
the worker task. Because that put() sat outside the worker's try
block, the finally-clause that publishes end-of-stream never ran
and the SSE stream hung forever.

jsonl mode was unaffected because json.dumps(default=str) coerces
UUID objects transparently.

Fixes:
- db.py: coerce user.id to str at the context-read boundary (matches
  what resolve_user_id already does for the other repositories)
- worker.py: move RunJournal init + human_message put inside the try
  block so any failure flows through the finally/publish_end path
  instead of hanging the subscriber

Defense-in-depth:
- engine.py: add PRAGMA busy_timeout=5000 so checkpointer and event
  store wait for each other on the shared deerflow.db file instead
  of failing immediately under write-lock contention
- journal.py: skip fire-and-forget _flush_sync when a previous flush
  task is still in flight, to avoid piling up concurrent put_batch
  writes on the same SQLAlchemy engine during streaming; flush() now
  waits for pending tasks before draining the buffer
- database_config.py: doc-only update clarifying WAL + busy_timeout
  keep the unified deerflow.db safe for both workloads

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore(persistence): drop redundant busy_timeout PRAGMA

Python's sqlite3 driver defaults to a 5-second busy timeout via the
``timeout`` kwarg of ``sqlite3.connect``, and aiosqlite + SQLAlchemy's
aiosqlite dialect inherit that default. Setting ``PRAGMA busy_timeout=5000``
explicitly was a no-op — verified by reading back the PRAGMA on a fresh
connection (it already reports 5000ms without our PRAGMA).

Concurrent stress test (50 checkpoint writes + 20 event batches + 50
thread_meta updates on the same deerflow.db) still completes with zero
errors and 200/200 rows after removing the explicit PRAGMA.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(journal): unwrap Command tool results in on_tool_end

Tools that update graph state (e.g. ``present_files``) return
``Command(update={'messages': [ToolMessage(...)], 'artifacts': [...]})``.
LangGraph later unwraps the inner ``ToolMessage`` into checkpoint state,
but ``RunJournal.on_tool_end`` was receiving the ``Command`` object
directly via the LangChain callback chain and storing
``str(Command(update={...}))`` as the tool_result content.

This produced a visible divergence between the event-store and the
checkpoint for any thread that used a Command-returning tool, blocking
the event-store-backed history fix in the follow-up commit. Concrete
example from thread ``6d30913e-dcd4-41c8-8941-f66c716cf359`` (seq=48):
checkpoint had ``'Successfully presented files'`` while event_store
stored the full Command repr.

The fix detects ``Command`` in ``on_tool_end``, extracts the first
``ToolMessage`` from ``update['messages']``, and lets the existing
ToolMessage branch handle the ``model_dump()`` path. Legacy rows still
containing the Command repr are separately cleaned up by the history
helper in the follow-up commit.

Tests:
- ``test_tool_end_unwraps_command_with_inner_tool_message`` — unit test
  of the unwrap branch with a constructed Command
- ``test_tool_invoke_end_to_end_unwraps_command`` — end-to-end via
  ``CallbackManager`` + ``tool.invoke`` to exercise the real LangChain
  dispatch path that production uses, matching the repro shape from
  ``present_files``
- Counter-proof: temporarily reverted the patch, both tests failed with
  the exact ``Command(update={...})`` repr that was stored in the
  production SQLite row at seq=48, confirming LangChain does pass the
  ``Command`` through callbacks (the unwrap is load-bearing)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(threads): load history messages from event store, immune to summarize

``get_thread_history`` and ``get_thread_state`` in Gateway mode read
messages from ``checkpoint.channel_values["messages"]``. After
SummarizationMiddleware runs mid-run, that list is rewritten in-place:
pre-summarize messages are dropped and a synthetic summary-as-human
message takes position 0. The frontend then renders a chat history that
starts with ``"Here is a summary of the conversation to date:..."``
instead of the user's original query, and all earlier turns are gone.

The event store (``RunEventStore``) is append-only and never rewritten,
so it retains the full transcript. This commit adds a helper
``_get_event_store_messages`` that loads the event store's message
stream and overrides ``values["messages"]`` in both endpoints; the
checkpoint fallback kicks in only when the event store is unavailable.

Behavior contract of the helper:

- **Full pagination.** ``list_messages`` returns the newest ``limit``
  records when no cursor is given, so a fixed limit silently drops
  older messages on long threads. The helper sizes the read from
  ``count_messages()`` and pages forward with ``after_seq`` cursors.
- **Copy-on-read.** Each content dict is copied before ``id`` is
  patched so the live store object (``MemoryRunEventStore`` returns
  references) is never mutated.
- **Stable ids.** Messages with ``id=None`` (human + tool_result,
  which don't receive an id until checkpoint persistence) get a
  deterministic ``uuid5(NAMESPACE_URL, f"{thread_id}:{seq}")`` so
  React keys stay stable across requests. AI messages keep their
  LLM-assigned ``lc_run--*`` ids.
- **Legacy ``Command`` repr sanitization.** Rows captured before the
  ``journal.py`` ``on_tool_end`` fix (previous commit) stored
  ``str(Command(update={'messages': [ToolMessage(content='X', ...)]}))``
  as the tool_result content. ``_sanitize_legacy_command_repr``
  regex-extracts the inner text so old threads render cleanly.
- **Inline feedback.** When loading the stream, the helper also pulls
  ``feedback_repo.list_by_thread_grouped`` and attaches ``run_id`` to
  every message plus ``feedback`` to the final ``ai_message`` of each
  run. This removes the frontend's need to fetch a second endpoint
  and positional-index-map its way back to the right run. When the
  feedback subsystem is unavailable, the ``feedback`` field is left
  absent entirely so the frontend hides the button rather than
  rendering it over a broken write path.
- **User context.** ``DbRunEventStore`` is user-scoped by default via
  ``resolve_user_id(AUTO)``. The helper relies on the ``@require_permission``
  decorator having populated the user contextvar on both callers; the
  docstring documents this dependency explicitly so nobody wires it
  into a CLI or migration script without passing ``user_id=None``.

Real data verification against thread
``6d30913e-dcd4-41c8-8941-f66c716cf359``: checkpoint showed 12 messages
(summarize-corrupted), event store had 16. The original human message
``"最新伊美局势"`` was preserved as seq=1 in the event store and
correctly restored to position 0 in the helper output. Helper output
for AI messages was byte-identical to checkpoint for every overlapping
message; only tool_result ids differed (patched to uuid5) and the
legacy Command repr at seq=48 was sanitized.

Tests:
- ``test_thread_state_event_store.py`` — 18 tests covering
  ``_sanitize_legacy_command_repr`` (passthrough, single/double-quote
  extraction, unparseable fallback), helper happy path (all message
  types, stable uuid5, store non-mutation), multi-page pagination,
  summarize regression (recovers pre-summarize messages), feedback
  attachment (per-run, multi-run threads, repo failure graceful),
  and dependency failure fallback to ``None``.

Docs:
- ``docs/superpowers/plans/2026-04-10-event-store-history.md`` — the
  implementation plan this commit realizes, with Task 1 revised after
  the evaluation findings (pagination, copy-on-read, Command wrap
  already landed in journal.py, frontend feedback pagination in the
  follow-up commit, Standard-mode follow-up noted).
- ``docs/superpowers/specs/2026-04-11-runjournal-history-evaluation.md``
  — the Claude + second-opinion evaluation document that drove the
  plan revisions (pagination bug, dict-mutation bug, feedback hidden
  bug, Command bug).
- ``docs/superpowers/specs/2026-04-11-summarize-marker-design.md`` —
  design for a follow-up PR that visually marks summarize events in
  history, based on a verified ``adispatch_custom_event`` experiment
  (``trace=False`` middleware nodes can still forward the Pregel task
  config via explicit signature injection).

Scope: Gateway mode only (``make dev-pro``). Standard mode
(``make dev``) hits LangGraph Server directly and bypasses these
endpoints; the summarize symptom is still present there and is
tracked as a separate follow-up in the plan.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(feedback): inline feedback on history and drop positional mapping

The old ``useThreadFeedback`` hook loaded ``GET /api/threads/{id}/messages?limit=200``
and built two parallel lookup tables: ``runIdByAiIndex`` (an ordinal array of
run_ids for every ``ai_message``-typed event) and ``feedbackByRunId``. The render
loop in ``message-list.tsx`` walked the AI messages in order, incrementing
``aiMessageIndex`` on each non-human message, and used that ordinal to look up
the run_id and feedback.

This shape had three latent bugs we could observe on real threads:

1. **Fetch was capped at 200 messages.** Long or tool-heavy threads silently
   dropped earlier entries from the map, so feedback buttons could be missing
   on messages they should own.
2. **Ordinal mismatch.** The render loop counted every non-human message
   (including each intermediate ``ai_tool_call``), but ``runIdByAiIndex`` only
   pushed entries for ``event_type == "ai_message"``. A run with 3 tool_calls
   + 1 final AI message would push 1 entry while the render consumed 4
   positions, so buttons mapped to the wrong positions across multi-run
   threads.
3. **Two parallel data paths.** The ``/history`` render path and the
   ``/messages`` feedback-lookup path could drift in-between an
   ``invalidateQueries`` call and the next refetch, producing transient
   mismaps.

The previous commit moved the authoritative message source for history to
the event store and added ``run_id`` + ``feedback`` inline on each message
dict returned by ``_get_event_store_messages``. This commit aligns the
frontend with that contract:

- **Delete** ``useThreadFeedback``, ``ThreadFeedbackData``,
  ``runIdByAiIndex``, ``feedbackByRunId``, and ``fetchAllThreadMessages``.
- **Introduce** ``useThreadMessageEnrichment`` that fetches
  ``POST /history?limit=1`` once, indexes the returned messages by
  ``message.id`` into a ``Map<id, {run_id, feedback?}>``, and invalidates
  on stream completion (``onFinish`` in ``useThreadStream``). Keying by
  ``message.id`` is stable across runs, tool_call chains, and summarize.
- **Simplify** ``message-list.tsx`` to drop the ``aiMessageIndex``
  counter and read ``enrichment?.get(msg.id)`` at each render step.
- **Rewire** ``message-list-item.tsx`` so the feedback button renders
  when ``feedback !== undefined`` rather than when the message happens
  to be non-human. ``feedback`` is ``undefined`` for non-eligible
  messages (humans, non-final AI, tools), ``null`` for the final
  ai_message of an unrated run, and a ``FeedbackData`` object once
  rated — cleanly distinguishing "not eligible" from "eligible but
  unrated".

``/api/threads/{id}/messages`` is kept as a debug/export surface; no
frontend code calls it anymore but the backend router is untouched.

Validation:
- ``pnpm check`` clean (0 errors, 1 pre-existing unrelated warning)
- Live test on thread ``3d5dea4a`` after gateway restart confirmed the
  original user query is restored to position 0 and the feedback
  button behaves correctly on the final AI message.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(rebase): remove duplicate definitions and update stale module paths

Rebase left duplicate function blocks in worker.py (triple human_message
write causing 3x user messages in /history), deps.py, and prompt.py.
Also update checkpointer imports from the old deerflow.agents.checkpointer
path to deerflow.runtime.checkpointer, and clean up orphaned feedback
props in the frontend message components.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(rebase): restore FeedbackButtons component and enrichment lost during rebase

The FeedbackButtons component (defined inline in message-list-item.tsx)
was introduced in commit 95df8d13 but lost during rebase. The previous
rebase cleanup commit incorrectly removed the feedback/runId props and
enrichment hook as "orphaned code" instead of restoring the missing
component. This commit restores:

- FeedbackButtons component with thumbs up/down toggle and optimistic state
- FeedbackData/upsertFeedback/deleteFeedback imports
- feedback and runId props on MessageListItem
- useThreadMessageEnrichment hook and entry lookup in message-list.tsx

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(user-context): add DEFAULT_USER_ID and get_effective_user_id helper

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(paths): add user-aware path methods with optional user_id parameter

Add _validate_user_id(), user_dir(), user_memory_file(), user_agent_memory_file()
and optional keyword-only user_id parameter to all thread-related path methods.
When user_id is provided, paths resolve under users/{user_id}/threads/{thread_id}/;
when omitted, legacy layout is preserved for backward compatibility.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(memory): add user_id to MemoryStorage interface for per-user isolation

Thread user_id through MemoryStorage.load/reload/save abstract methods and
FileMemoryStorage, re-keying the in-memory cache from bare agent_name to a
(user_id, agent_name) tuple to prevent cross-user cache collisions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(memory): thread user_id through memory updater layer

Add `user_id` keyword-only parameter to all public updater functions
(_save_memory_to_file, get_memory_data, reload_memory_data, import_memory_data,
clear_memory_data, create/delete/update_memory_fact) and regular keyword param
to MemoryUpdater.update_memory + update_memory_from_conversation, propagating
it to every storage load/save/reload call.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(memory): capture user_id at enqueue time for async-safe thread isolation

Add user_id field to ConversationContext and MemoryUpdateQueue.add() so the
user identity is stored explicitly at request time, before threading.Timer
fires on a different thread where ContextVar values do not propagate.
MemoryMiddleware.after_agent() now calls get_effective_user_id() at enqueue
time and passes the value through to updater.update_memory().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(isolation): wire user_id through all Paths and memory callsites

Pass user_id=get_effective_user_id() at every callsite that invokes
Paths methods or memory functions, enabling per-user filesystem isolation
throughout the harness and app layers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(migration): add idempotent script for per-user data migration

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update CLAUDE.md and config docs for per-user isolation

* feat(events): add pagination to list_messages_by_run on all store backends

Replicates the existing before_seq/after_seq/limit cursor-pagination pattern
from list_messages onto list_messages_by_run across the abstract interface,
MemoryRunEventStore, JsonlRunEventStore, and DbRunEventStore.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(api): add GET /api/runs/{run_id}/messages with cursor pagination

New endpoint resolves thread_id from the run record and delegates to
RunEventStore.list_messages_by_run for cursor-based pagination.
Ownership is enforced implicitly via RunStore.get() user filtering.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(api): add GET /api/runs/{run_id}/feedback

Delegates to FeedbackRepository.list_by_run via the existing _resolve_run
helper; includes tests for success, 404, empty list, and 503 (no DB).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(api): retrofit cursor pagination onto GET /threads/{tid}/runs/{rid}/messages

Replace bare list[dict] response with {data: [...], has_more: bool} envelope,
forwarding limit/before_seq/after_seq query params to the event store.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add run-level API endpoints to CLAUDE.md routers table

* refactor(threads): remove event-store message loader and feedback from state/history endpoints

State and history endpoints now return messages purely from the
checkpointer's channel_values. The _get_event_store_messages helper
(which loaded the full event-store transcript with feedback attached)
is removed along with its tests. Frontend will use the dedicated
GET /api/runs/{run_id}/messages and /feedback endpoints instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add unified persistence layer with event store, token tracking, and feedback (#1930)

* feat(persistence): add SQLAlchemy 2.0 async ORM scaffold

Introduce a unified database configuration (DatabaseConfig) that
controls both the LangGraph checkpointer and the DeerFlow application
persistence layer from a single `database:` config section.

New modules:
- deerflow.config.database_config — Pydantic config with memory/sqlite/postgres backends
- deerflow.persistence — async engine lifecycle, DeclarativeBase with to_dict mixin, Alembic skeleton
- deerflow.runtime.runs.store — RunStore ABC + MemoryRunStore implementation

Gateway integration initializes/tears down the persistence engine in
the existing langgraph_runtime() context manager. Legacy checkpointer
config is preserved for backward compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add RunEventStore ABC + MemoryRunEventStore

Phase 2-A prerequisite for event storage: adds the unified run event
stream interface (RunEventStore) with an in-memory implementation,
RunEventsConfig, gateway integration, and comprehensive tests (27 cases).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add ORM models, repositories, DB/JSONL event stores, RunJournal, and API endpoints

Phase 2-B: run persistence + event storage + token tracking.

- ORM models: RunRow (with token fields), ThreadMetaRow, RunEventRow
- RunRepository implements RunStore ABC via SQLAlchemy ORM
- ThreadMetaRepository with owner access control
- DbRunEventStore with trace content truncation and cursor pagination
- JsonlRunEventStore with per-run files and seq recovery from disk
- RunJournal (BaseCallbackHandler) captures LLM/tool/lifecycle events,
  accumulates token usage by caller type, buffers and flushes to store
- RunManager now accepts optional RunStore for persistent backing
- Worker creates RunJournal, writes human_message, injects callbacks
- Gateway deps use factory functions (RunRepository when DB available)
- New endpoints: messages, run messages, run events, token-usage
- ThreadCreateRequest gains assistant_id field
- 92 tests pass (33 new), zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add user feedback + follow-up run association

Phase 2-C: feedback and follow-up tracking.

- FeedbackRow ORM model (rating +1/-1, optional message_id, comment)
- FeedbackRepository with CRUD, list_by_run/thread, aggregate stats
- Feedback API endpoints: create, list, stats, delete
- follow_up_to_run_id in RunCreateRequest (explicit or auto-detected
  from latest successful run on the thread)
- Worker writes follow_up_to_run_id into human_message event metadata
- Gateway deps: feedback_repo factory + getter
- 17 new tests (14 FeedbackRepository + 3 follow-up association)
- 109 total tests pass, zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test+config: comprehensive Phase 2 test coverage + deprecate checkpointer config

- config.example.yaml: deprecate standalone checkpointer section, activate
  unified database:sqlite as default (drives both checkpointer + app data)
- New: test_thread_meta_repo.py (14 tests) — full ThreadMetaRepository coverage
  including check_access owner logic, list_by_owner pagination
- Extended test_run_repository.py (+4 tests) — completion preserves fields,
  list ordering desc, limit, owner_none returns all
- Extended test_run_journal.py (+8 tests) — on_chain_error, track_tokens=false,
  middleware no ai_message, unknown caller tokens, convenience fields,
  tool_error, non-summarization custom event
- Extended test_run_event_store.py (+7 tests) — DB batch seq continuity,
  make_run_event_store factory (memory/db/jsonl/fallback/unknown)
- Extended test_phase2b_integration.py (+4 tests) — create_or_reject persists,
  follow-up metadata, summarization in history, full DB-backed lifecycle
- Fixed DB integration test to use proper fake objects (not MagicMock)
  for JSON-serializable metadata
- 157 total Phase 2 tests pass, zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* config: move default sqlite_dir to .deer-flow/data

Keep SQLite databases alongside other DeerFlow-managed data
(threads, memory) under the .deer-flow/ directory instead of a
top-level ./data folder.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(persistence): remove UTFJSON, use engine-level json_serializer + datetime.now()

- Replace custom UTFJSON type with standard sqlalchemy.JSON in all ORM
  models. Add json_serializer=json.dumps(ensure_ascii=False) to all
  create_async_engine calls so non-ASCII text (Chinese etc.) is stored
  as-is in both SQLite and Postgres.
- Change ORM datetime defaults from datetime.now(UTC) to datetime.now(),
  remove UTC imports.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): simplify deps.py with getter factory + inline repos

- Replace 6 identical getter functions with _require() factory.
- Inline 3 _make_*_repo() factories into langgraph_runtime(), call
  get_session_factory() once instead of 3 times.
- Add thread_meta upsert in start_run (services.py).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(docker): add UV_EXTRAS build arg for optional dependencies

Support installing optional dependency groups (e.g. postgres) at
Docker build time via UV_EXTRAS build arg:
  UV_EXTRAS=postgres docker compose build

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(journal): fix flush, token tracking, and consolidate tests

RunJournal fixes:
- _flush_sync: retain events in buffer when no event loop instead of
  dropping them; worker's finally block flushes via async flush().
- on_llm_end: add tool_calls filter and caller=="lead_agent" guard for
  ai_message events; mark message IDs for dedup with record_llm_usage.
- worker.py: persist completion data (tokens, message count) to RunStore
  in finally block.

Model factory:
- Auto-inject stream_usage=True for BaseChatOpenAI subclasses with
  custom api_base, so usage_metadata is populated in streaming responses.

Test consolidation:
- Delete test_phase2b_integration.py (redundant with existing tests).
- Move DB-backed lifecycle test into test_run_journal.py.
- Add tests for stream_usage injection in test_model_factory.py.
- Clean up executor/task_tool dead journal references.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): widen content type to str|dict in all store backends

Allow event content to be a dict (for structured OpenAI-format messages)
in addition to plain strings. Dict values are JSON-serialized for the DB
backend and deserialized on read; memory and JSONL backends handle dicts
natively. Trace truncation now serializes dicts to JSON before measuring.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(events): use metadata flag instead of heuristic for dict content detection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(converters): add LangChain-to-OpenAI message format converters

Pure functions langchain_to_openai_message, langchain_to_openai_completion,
langchain_messages_to_openai, and _infer_finish_reason for converting
LangChain BaseMessage objects to OpenAI Chat Completions format, used by
RunJournal for event storage. 15 unit tests added.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(converters): handle empty list content as null, clean up test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): human_message content uses OpenAI user message format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): ai_message uses OpenAI format, add ai_tool_call message event

- ai_message content now uses {"role": "assistant", "content": "..."} format
- New ai_tool_call message event emitted when lead_agent LLM responds with tool_calls
- ai_tool_call uses langchain_to_openai_message converter for consistent format
- Both events include finish_reason in metadata ("stop" or "tool_calls")

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): add tool_result message event with OpenAI tool message format

Cache tool_call_id from on_tool_start keyed by run_id as fallback for on_tool_end,
then emit a tool_result message event (role=tool, tool_call_id, content) after each
successful tool completion.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): summary content uses OpenAI system message format

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): replace llm_start/llm_end with llm_request/llm_response in OpenAI format

Add on_chat_model_start to capture structured prompt messages as llm_request events.
Replace llm_end trace events with llm_response using OpenAI Chat Completions format.
Track llm_call_index to pair request/response events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): add record_middleware method for middleware trace events

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(events): add full run sequence integration test for OpenAI content format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): align message events with checkpoint format and add middleware tag injection

- Message events (ai_message, ai_tool_call, tool_result, human_message) now use
  BaseMessage.model_dump() format, matching LangGraph checkpoint values.messages
- on_tool_end extracts tool_call_id/name/status from ToolMessage objects
- on_tool_error now emits tool_result message events with error status
- record_middleware uses middleware:{tag} event_type and middleware category
- Summarization custom events use middleware:summarize category
- TitleMiddleware injects middleware:title tag via get_config() inheritance
- SummarizationMiddleware model bound with middleware:summarize tag
- Worker writes human_message using HumanMessage.model_dump()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(threads): switch search endpoint to threads_meta table and sync title

- POST /api/threads/search now queries threads_meta table directly,
  removing the two-phase Store + Checkpointer scan approach
- Add ThreadMetaRepository.search() with metadata/status filters
- Add ThreadMetaRepository.update_display_name() for title sync
- Worker syncs checkpoint title to threads_meta.display_name on run completion
- Map display_name to values.title in search response for API compatibility

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(threads): history endpoint reads messages from event store

- POST /api/threads/{thread_id}/history now combines two data sources:
  checkpointer for checkpoint_id, metadata, title, thread_data;
  event store for messages (complete history, not truncated by summarization)
- Strip internal LangGraph metadata keys from response
- Remove full channel_values serialization in favor of selective fields

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove duplicate optional-dependencies header in pyproject.toml

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(middleware): pass tagged config to TitleMiddleware ainvoke call

Without the config, the middleware:title tag was not injected,
causing the LLM response to be recorded as a lead_agent ai_message
in run_events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: resolve merge conflict in .env.example

Keep both DATABASE_URL (from persistence-scaffold) and WECOM
credentials (from main) after the merge.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address review feedback on PR #1851

- Fix naive datetime.now() → datetime.now(UTC) in all ORM models
- Fix seq race condition in DbRunEventStore.put() with FOR UPDATE
  and UNIQUE(thread_id, seq) constraint
- Encapsulate _store access in RunManager.update_run_completion()
- Deduplicate _store.put() logic in RunManager via _persist_to_store()
- Add update_run_completion to RunStore ABC + MemoryRunStore
- Wire follow_up_to_run_id through the full create path
- Add error recovery to RunJournal._flush_sync() lost-event scenario
- Add migration note for search_threads breaking change
- Fix test_checkpointer_none_fix mock to set database=None

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update uv.lock

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address 22 review comments from CodeQL, Copilot, and Code Quality

Bug fixes:
- Sanitize log params to prevent log injection (CodeQL)
- Reset threads_meta.status to idle/error when run completes
- Attach messages only to latest checkpoint in /history response
- Write threads_meta on POST /threads so new threads appear in search

Lint fixes:
- Remove unused imports (journal.py, migrations/env.py, test_converters.py)
- Convert lambda to named function (engine.py, Ruff E731)
- Remove unused logger definitions in repos (Ruff F841)
- Add logging to JSONL decode errors and empty except blocks
- Separate assert side-effects in tests (CodeQL)
- Remove unused local variables in tests (Ruff F841)
- Fix max_trace_content truncation to use byte length, not char length

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: apply ruff format to persistence and runtime files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Potential fix for pull request finding 'Statement has no effect'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

* refactor(runtime): introduce RunContext to reduce run_agent parameter bloat

Extract checkpointer, store, event_store, run_events_config, thread_meta_repo,
and follow_up_to_run_id into a frozen RunContext dataclass. Add get_run_context()
in deps.py to build the base context from app.state singletons. start_run() uses
dataclasses.replace() to enrich per-run fields before passing ctx to run_agent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): move sanitize_log_param to app/gateway/utils.py

Extract the log-injection sanitizer from routers/threads.py into a shared
utils module and rename to sanitize_log_param (public API). Eliminates the
reverse service → router import in services.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf: use SQL aggregation for feedback stats and thread token usage

Replace Python-side counting in FeedbackRepository.aggregate_by_run with
a single SELECT COUNT/SUM query. Add RunStore.aggregate_tokens_by_thread
abstract method with SQL GROUP BY implementation in RunRepository and
Python fallback in MemoryRunStore. Simplify the thread_token_usage
endpoint to delegate to the new method, eliminating the limit=10000
truncation risk.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: annotate DbRunEventStore.put() as low-frequency path

Add docstring clarifying that put() opens a per-call transaction with
FOR UPDATE and should only be used for infrequent writes (currently
just the initial human_message event). High-throughput callers should
use put_batch() instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(threads): fall back to Store search when ThreadMetaRepository is unavailable

When database.backend=memory (default) or no SQL session factory is
configured, search_threads now queries the LangGraph Store instead of
returning 503. Returns empty list if neither Store nor repo is available.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(persistence): introduce ThreadMetaStore ABC for backend-agnostic thread metadata

Add ThreadMetaStore abstract base class with create/get/search/update/delete
interface. ThreadMetaRepository (SQL) now inherits from it. New
MemoryThreadMetaStore wraps LangGraph BaseStore for memory-mode deployments.

deps.py now always provides a non-None thread_meta_repo, eliminating all
`if thread_meta_repo is not None` guards in services.py, worker.py, and
routers/threads.py. search_threads no longer needs a Store fallback branch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(history): read messages from checkpointer instead of RunEventStore

The /history endpoint now reads messages directly from the
checkpointer's channel_values (the authoritative source) instead of
querying RunEventStore.list_messages(). The RunEventStore API is
preserved for other consumers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address new Copilot review comments

- feedback.py: validate thread_id/run_id before deleting feedback
- jsonl.py: add path traversal protection with ID validation
- run_repo.py: parse `before` to datetime for PostgreSQL compat
- thread_meta_repo.py: fix pagination when metadata filter is active
- database_config.py: use resolve_path for sqlite_dir consistency

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Implement skill self-evolution and skill_manage flow (#1874)

* chore: ignore .worktrees directory

* Add skill_manage self-evolution flow

* Fix CI regressions for skill_manage

* Address PR review feedback for skill evolution

* fix(skill-evolution): preserve history on delete

* fix(skill-evolution): tighten scanner fallbacks

* docs: add skill_manage e2e evidence screenshot

* fix(skill-manage): avoid blocking fs ops in session runtime

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

* fix(config): resolve sqlite_dir relative to CWD, not Paths.base_dir

resolve_path() resolves relative to Paths.base_dir (.deer-flow),
which double-nested the path to .deer-flow/.deer-flow/data/app.db.
Use Path.resolve() (CWD-relative) instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Feature/feishu receive file (#1608)

* feat(feishu): add channel file materialization hook for inbound messages

- Introduce Channel.receive_file(msg, thread_id) as a base method for file materialization; default is no-op.
- Implement FeishuChannel.receive_file to download files/images from Feishu messages, save to sandbox, and inject virtual paths into msg.text.
- Update ChannelManager to call receive_file for any channel if msg.files is present, enabling downstream model access to user-uploaded files.
- No impact on Slack/Telegram or other channels (they inherit the default no-op).

* style(backend): format code with ruff for lint compliance

- Auto-formatted packages/harness/deerflow/agents/factory.py and tests/test_create_deerflow_agent.py using `ruff format`
- Ensured both files conform to project linting standards
- Fixes CI lint check failures caused by code style issues

* fix(feishu): handle file write operation asynchronously to prevent blocking

* fix(feishu): rename GetMessageResourceRequest to _GetMessageResourceRequest and remove redundant code

* test(feishu): add tests for receive_file method and placeholder replacement

* fix(manager): remove unnecessary type casting for channel retrieval

* fix(feishu): update logging messages to reflect resource handling instead of image

* fix(feishu): sanitize filename by replacing invalid characters in file uploads

* fix(feishu): improve filename sanitization and reorder image key handling in message processing

* fix(feishu): add thread lock to prevent filename conflicts during file downloads

* fix(test): correct bad merge in test_feishu_parser.py

* chore: run ruff and apply formatting cleanup
fix(feishu): preserve rich-text attachment order and improve fallback filename handling

* fix(docker): restore gateway env vars and fix langgraph empty arg issue (#1915)

Two production docker-compose.yaml bugs prevent `make up` from working:

1. Gateway missing DEER_FLOW_CONFIG_PATH and DEER_FLOW_EXTENSIONS_CONFIG_PATH
   environment overrides. Added in fb2d99f (#1836) but accidentally reverted
   by ca2fb95 (#1847). Without them, gateway reads host paths from .env via
   env_file, causing FileNotFoundError inside the container.

2. Langgraph command fails when LANGGRAPH_ALLOW_BLOCKING is unset (default).
   Empty $${allow_blocking} inserts a bare space between flags, causing
   ' --no-reload' to be parsed as unexpected extra argument. Fix by building
   args string first and conditionally appending --allow-blocking.

Co-authored-by: cooper <cooperfu@tencent.com>

* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities (#1904)

* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities

Fix `<button>` inside `<a>` invalid HTML in artifact components and add
missing `noopener,noreferrer` to `window.open` calls to prevent reverse
tabnabbing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(frontend): address Copilot review on tabnabbing and double-tab-open

Remove redundant parent onClick on web_fetch ChainOfThoughtStep to
prevent opening two tabs on link click, and explicitly null out
window.opener after window.open() for defensive tabnabbing hardening.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(persistence): organize entities into per-entity directories

Restructure the persistence layer from horizontal "models/ + repositories/"
split into vertical entity-aligned directories. Each entity (thread_meta,
run, feedback) now owns its ORM model, abstract interface (where applicable),
and concrete implementations under a single directory with an aggregating
__init__.py for one-line imports.

Layout:
  persistence/thread_meta/{base,model,sql,memory}.py
  persistence/run/{model,sql}.py
  persistence/feedback/{model,sql}.py

models/__init__.py is kept as a facade so Alembic autogenerate continues to
discover all ORM tables via Base.metadata. RunEventRow remains under
models/run_event.py because its storage implementation lives in
runtime/events/store/db.py and has no matching repository directory.

The repositories/ directory is removed entirely. All call sites in
gateway/deps.py and tests are updated to import from the new entity
packages, e.g.:

    from deerflow.persistence.thread_meta import ThreadMetaRepository
    from deerflow.persistence.run import RunRepository
    from deerflow.persistence.feedback import FeedbackRepository

Full test suite passes (1690 passed, 14 skipped).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(gateway): sync thread rename and delete through ThreadMetaStore

The POST /threads/{id}/state endpoint previously synced title changes
only to the LangGraph Store via _store_upsert. In sqlite mode the search
endpoint reads from the ThreadMetaRepository SQL table, so renames never
appeared in /threads/search until the next agent run completed (worker.py
syncs title from checkpoint to thread_meta in its finally block).

Likewise the DELETE /threads/{id} endpoint cleaned up the filesystem,
Store, and checkpointer but left the threads_meta row orphaned in sqlite,
so deleted threads kept appearing in /threads/search.

Fix both endpoints by routing through the ThreadMetaStore abstraction
which already has the correct sqlite/memory implementations wired up by
deps.py. The rename path now calls update_display_name() and the delete
path calls delete() — both work uniformly across backends.

Verified end-to-end with curl in gateway mode against sqlite backend.
Existing test suite (1690 passed) and focused router/repo tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): route all thread metadata access through ThreadMetaStore

Following the rename/delete bug fix in PR1, migrate the remaining direct
LangGraph Store reads/writes in the threads router and services to the
ThreadMetaStore abstraction so that the sqlite and memory backends behave
identically and the legacy dual-write paths can be removed.

Migrated endpoints (threads.py):
- create_thread: idempotency check + write now use thread_meta_repo.get/create
  instead of dual-writing the LangGraph Store and the SQL row.
- get_thread: reads from thread_meta_repo.get; the checkpoint-only fallback
  for legacy threads is preserved.
- patch_thread: replaced _store_get/_store_put with thread_meta_repo.update_metadata.
- delete_thread_data: dropped the legacy store.adelete; thread_meta_repo.delete
  already covers it.

Removed dead code (services.py):
- _upsert_thread_in_store — redundant with the immediately following
  thread_meta_repo.create() call.
- _sync_thread_title_after_run — worker.py's finally block already syncs
  the title via thread_meta_repo.update_display_name() after each run.

Removed dead code (threads.py):
- _store_get / _store_put / _store_upsert helpers (no remaining callers).
- THREADS_NS constant.
- get_store import (router no longer touches the LangGraph Store directly).

New abstract method:
- ThreadMetaStore.update_metadata(thread_id, metadata) merges metadata into
  the thread's metadata field. Implemented in both ThreadMetaRepository (SQL,
  read-modify-write inside one session) and MemoryThreadMetaStore. Three new
  unit tests cover merge / empty / nonexistent behaviour.

Net change: -134 lines. Full test suite: 1693 passed, 14 skipped.
Verified end-to-end with curl in gateway mode against sqlite backend
(create / patch / get / rename / search / delete).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: JilongSun <965640067@qq.com>
Co-authored-by: jie <49781832+stan-fu@users.noreply.github.com>
Co-authored-by: cooper <cooperfu@tencent.com>
Co-authored-by: yangzheli <43645580+yangzheli@users.noreply.github.com>

* feat(auth): release-validation pass for 2.0-rc — 12 blockers + simplify follow-ups (#2008)

* feat(auth): introduce backend auth module

Port RFC-001 authentication core from PR #1728:
- JWT token handling (create_access_token, decode_token, TokenPayload)
- Password hashing (bcrypt) with verify_password
- SQLite UserRepository with base interface
- Provider Factory pattern (LocalAuthProvider)
- CLI reset_admin tool
- Auth-specific errors (AuthErrorCode, TokenError, AuthErrorResponse)

Deps:
- bcrypt>=4.0.0
- pyjwt>=2.9.0
- email-validator>=2.0.0
- backend/uv.toml pins public PyPI index

Tests: 12 pure unit tests (test_auth_config.py, test_auth_errors.py).

Scope note: authz.py, test_auth.py, and test_auth_type_system.py are
deferred to commit 2 because they depend on middleware and deps wiring
that is not yet in place. Commit 1 stays "pure new files only" as the
spec mandates.

* feat(auth): wire auth end-to-end (middleware + frontend replacement)

Backend:
- Port auth_middleware, csrf_middleware, langgraph_auth, routers/auth
- Port authz decorator (owner_filter_key defaults to 'owner_id')
- Merge app.py: register AuthMiddleware + CSRFMiddleware + CORS, add
  _ensure_admin_user lifespan hook, _migrate_orphaned_threads helper,
  register auth router
- Merge deps.py: add get_local_provider, get_current_user_from_request,
  get_optional_user_from_request; keep get_current_user as thin str|None
  adapter for feedback router
- langgraph.json: add auth path pointing to langgraph_auth.py:auth
- Rename metadata['user_id'] -> metadata['owner_id'] in langgraph_auth
  (both metadata write and LangGraph filter dict) + test fixtures

Frontend:
- Delete better-auth library and api catch-all route
- Remove better-auth npm dependency and env vars (BETTER_AUTH_SECRET,
  BETTER_AUTH_GITHUB_*) from env.js
- Port frontend/src/core/auth/* (AuthProvider, gateway-config,
  proxy-policy, server-side getServerSideUser, types)
- Port frontend/src/core/api/fetcher.ts
- Port (auth)/layout, (auth)/login, (auth)/setup pages
- Rewrite workspace/layout.tsx as server component that calls
  getServerSideUser and wraps in AuthProvider
- Port workspace/workspace-content.tsx for the client-side sidebar logic

Tests:
- Port 5 auth test files (test_auth, test_auth_middleware,
  test_auth_type_system, test_ensure_admin, test_langgraph_auth)
- 176 auth tests PASS

After this commit: login/logout/registration flow works, but persistence
layer does not yet filter by owner_id. Commit 4 closes that gap.

* feat(auth): account settings page + i18n

- Port account-settings-page.tsx (change password, change email, logout)
- Wire into settings-dialog.tsx as new "account" section with UserIcon,
  rendered first in the section list
- Add i18n keys:
  - en-US/zh-CN: settings.sections.account ("Account" / "账号")
  - en-US/zh-CN: button.logout ("Log out" / "退出登录")
  - types.ts: matching type declarations

* feat(auth): enforce owner_id across 2.0-rc persistence layer

Add request-scoped contextvar-based owner filtering to threads_meta,
runs, run_events, and feedback repositories. Router code is unchanged
— isolation is enforced at the storage layer so that any caller that
forgets to pass owner_id still gets filtered results, and new routes
cannot accidentally leak data.

Core infrastructure
-------------------
- deerflow/runtime/user_context.py (new):
  - ContextVar[CurrentUser | None] with default None
  - runtime_checkable CurrentUser Protocol (structural subtype with .id)
  - set/reset/get/require helpers
  - AUTO sentinel + resolve_owner_id(value, method_name) for sentinel
    three-state resolution: AUTO reads contextvar, explicit str
    overrides, explicit None bypasses the filter (for migration/CLI)

Repository changes
------------------
- ThreadMetaRepository: create/get/search/update_*/delete gain
  owner_id=AUTO kwarg; read paths filter by owner, writes stamp it,
  mutations check ownership before applying
- RunRepository: put/get/list_by_thread/delete gain owner_id=AUTO kwarg
- FeedbackRepository: create/get/list_by_run/list_by_thread/delete
  gain owner_id=AUTO kwarg
- DbRunEventStore: list_messages/list_events/list_messages_by_run/
  count_messages/delete_by_thread/delete_by_run gain owner_id=AUTO
  kwarg. Write paths (put/put_batch) read contextvar softly: when a
  request-scoped user is available, owner_id is stamped; background
  worker writes without a user context pass None which is valid
  (orphan row to be bound by migration)

Schema
------
- persistence/models/run_event.py: RunEventRow.owner_id = Mapped[
  str | None] = mapped_column(String(64), nullable=True, index=True)
- No alembic migration needed: 2.0 ships fresh, Base.metadata.create_all
  picks up the new column automatically

Middleware
----------
- auth_middleware.py: after cookie check, call get_optional_user_from_
  request to load the real User, stamp it into request.state.user AND
  the contextvar via set_current_user, reset in a try/finally. Public
  paths and unauthenticated requests continue without contextvar, and
  @require_auth handles the strict 401 path

Test infrastructure
-------------------
- tests/conftest.py: @pytest.fixture(autouse=True) _auto_user_context
  sets a default SimpleNamespace(id="test-user-autouse") on every test
  unless marked @pytest.mark.no_auto_user. Keeps existing 20+
  persistence tests passing without modification
- pyproject.toml [tool.pytest.ini_options]: register no_auto_user
  marker so pytest does not emit warnings for opt-out tests
- tests/test_user_context.py: 6 tests covering three-state semantics,
  Protocol duck typing, and require/optional APIs
- tests/test_thread_meta_repo.py: one test updated to pass owner_id=
  None explicitly where it was previously relying on the old default

Test results
------------
- test_user_context.py: 6 passed
- test_auth*.py + test_langgraph_auth.py + test_ensure_admin.py: 127
- test_run_event_store / test_run_repository / test_thread_meta_repo
  / test_feedback: 92 passed
- Full backend suite: 1905 passed, 2 failed (both @requires_llm flaky
  integration tests unrelated to auth), 1 skipped

* feat(auth): extend orphan migration to 2.0-rc persistence tables

_ensure_admin_user now runs a three-step pipeline on every boot:

  Step 1 (fatal):     admin user exists / is created / password is reset
  Step 2 (non-fatal): LangGraph store orphan threads → admin
  Step 3 (non-fatal): SQL persistence tables → admin
    - threads_meta
    - runs
    - run_events
    - feedback

Each step is idempotent. The fatal/non-fatal split mirrors PR #1728's
original philosophy: admin creation failure blocks startup (the system
is unusable without an admin), whereas migration failures log a warning
and let the service proceed (a partial migration is recoverable; a
missing admin is not).

Key helpers
-----------
- _iter_store_items(store, namespace, *, page_size=500):
  async generator that cursor-paginates across LangGraph store pages.
  Fixes PR #1728's hardcoded limit=1000 bug that would silently lose
  orphans beyond the first page.

- _migrate_orphaned_threads(store, admin_user_id):
  Rewritten to use _iter_store_items. Returns the migrated count so the
  caller can log it; raises only on unhandled exceptions.

- _migrate_orphan_sql_tables(admin_user_id):
  Imports the 4 ORM models lazily, grabs the shared session factory,
  runs one UPDATE per table in a single transaction, commits once.
  No-op when no persistence backend is configured (in-memory dev).

Tests: test_ensure_admin.py (8 passed)

* test(auth): port AUTH test plan docs + lint/format pass

- Port backend/docs/AUTH_TEST_PLAN.md and AUTH_UPGRADE.md from PR #1728
- Rename metadata.user_id → metadata.owner_id in AUTH_TEST_PLAN.md
  (4 occurrences from the original PR doc)
- ruff auto-fix UP037 in sentinel type annotations: drop quotes around
  "str | None | _AutoSentinel" now that from __future__ import
  annotations makes them implicit string forms
- ruff format: 2 files (app/gateway/app.py, runtime/user_context.py)

Note on test coverage additions:
- conftest.py autouse fixture was already added in commit 4 (had to
  be co-located with the repository changes to keep pre-existing
  persistence tests passing)
- cross-user isolation E2E tests (test_owner_isolation.py) deferred
  — enforcement is already proven by the 98-test repository suite
  via the autouse fixture + explicit _AUTO sentinel exercises
- New test cases (TC-API-17..20, TC-ATK-13, TC-MIG-01..07) listed
  in AUTH_TEST_PLAN.md are deferred to a follow-up PR — they are
  manual-QA test cases rather than pytest code, and the spec-level
  coverage is already met by test_user_context.py + the 98-test
  repository suite.

Final test results:
- Auth suite (test_auth*, test_langgraph_auth, test_ensure_admin,
  test_user_context): 186 passed
- Persistence suite (test_run_event_store, test_run_repository,
  test_thread_meta_repo, test_feedback): 98 passed
- Lint: ruff check + ruff format both clean

* test(auth): add cross-user isolation test suite

10 tests exercising the storage-layer owner filter by manually
switching the user_context contextvar between two users. Verifies
the safety invariant:

  After a repository write with owner_id=A, a subsequent read with
  owner_id=B must not return the row, and vice versa.

Covers all 4 tables that own user-scoped data:

TC-API-17  threads_meta  — read, search, update, delete cross-user
TC-API-18  runs          — get, list_by_thread, delete cross-user
TC-API-19  run_events    — list_messages, list_events, count_messages,
                           delete_by_thread (CRITICAL: raw conversation
                           content leak vector)
TC-API-20  feedback      — get, list_by_run, delete cross-user

Plus two meta-tests verifying the sentinel pattern itself:
- AUTO + unset contextvar raises RuntimeError
- explicit owner_id=None bypasses the filter (migration escape hatch)

Architecture note
-----------------
These tests bypass the HTTP layer by design. The full chain
(cookie → middleware → contextvar → repository) is covered piecewise:

- test_auth_middleware.py: middleware sets contextvar from cookies
- test_owner_isolation.py: repositories enforce isolation when
  contextvar is set to different users

Together they prove the end-to-end safety property without the
ceremony of spinning up a full TestClient + in-memory DB for every
router endpoint.

Tests pass: 231 (full auth + persistence + isolation suite)
Lint: clean

* refactor(auth): migrate user repository to SQLAlchemy ORM

Move the users table into the shared persistence engine so auth
matches the pattern of threads_meta, runs, run_events, and feedback —
one engine, one session factory, one schema init codepath.

New files
---------
- persistence/user/__init__.py, persistence/user/model.py: UserRow
  ORM class with partial unique index on (oauth_provider, oauth_id)
- Registered in persistence/models/__init__.py so
  Base.metadata.create_all() picks it up

Modified
--------
- auth/repositories/sqlite.py: rewritten as async SQLAlchemy,
  identical constructor pattern to the other four repositories
  (def __init__(self, session_factory) + self._sf = session_factory)
- auth/config.py: drop users_db_path field — storage is configured
  through config.database like every other table
- deps.py/get_local_provider: construct SQLiteUserRepository with
  the shared session factory, fail fast if engine is not initialised
- tests/test_auth.py: rewrite test_sqlite_round_trip_new_fields to
  use the shared engine (init_engine + close_engine in a tempdir)
- tests/test_auth_type_system.py: add per-test autouse fixture that
  spins up a scratch engine and resets deps._cached_* singletons

* refactor(auth): remove SQL orphan migration (unused in supported scenarios)

The _migrate_orphan_sql_tables helper existed to bind NULL owner_id
rows in threads_meta, runs, run_events, and feedback to the admin on
first boot. But in every supported upgrade path, it's a no-op:

  1. Fresh install: create_all builds fresh tables, no legacy rows
  2. No-auth → with-auth (no existing persistence DB): persistence
     tables are created fresh by create_all, no legacy rows
  3. No-auth → with-auth (has existing persistence DB from #1930):
     NOT a supported upgrade path — "有 DB 到有 DB" schema evolution
     is out of scope; users wipe DB or run manual ALTER

So the SQL orphan migration never has anything to do in the
supported matrix. Delete the function, simplify _ensure_admin_user
from a 3-step pipeline to a 2-step one (admin creation + LangGraph
store orphan migration only).

LangGraph store orphan migration stays: it serves the real
"no-auth → with-auth" upgrade path where a user's existing LangGraph
thread metadata has no owner_id field and needs to be stamped with
the newly-created admin's id.

Tests: 284 passed (auth + persistence + isolation)
Lint: clean

* security(auth): write initial admin password to 0600 file instead of logs

CodeQL py/clear-text-logging-sensitive-data flagged 3 call sites that
logged the auto-generated admin password to stdout via logger.info().
Production log aggregators (ELK/Splunk/etc) would have captured those
cleartext secrets. Replace with a shared helper that writes to
.deer-flow/admin_initial_credentials.txt with mode 0600, and log only
the path.

New file
--------
- app/gateway/auth/credential_file.py: write_initial_credentials()
  helper. Takes email, password, and a "initial"/"reset" label.
  Creates .deer-flow/ if missing, writes a header comment plus the
  email+password, chmods 0o600, returns the absolute Path.

Modified
--------
- app/gateway/app.py: both _ensure_admin_user paths (fresh creation
  + needs_setup password reset) now write to file and log the path
- app/gateway/auth/reset_admin.py: rewritten to use the shared ORM
  repo (SQLiteUserRepository with session_factory) and the
  credential_file helper. The previous implementation was broken
  after the earlier ORM refactor — it still imported _get_users_conn
  and constructed SQLiteUserRepository() without a session factory.

No tests changed — the three password-log sites are all exercised
via existing test_ensure_admin.py which checks that startup
succeeds, not that a specific string appears in logs.

CodeQL alerts 272, 283, 284: all resolved.

* security(auth): strict JWT validation in middleware (fix junk cookie bypass)

AUTH_TEST_PLAN test 7.5.8 expects junk cookies to be rejected with
401. The previous middleware behaviour was "presence-only": check
that some access_token cookie exists, then pass through. In
combination with my Task-12 decision to skip @require_auth
decorators on routes, this created a gap where a request with any
cookie-shaped string (e.g. access_token=not-a-jwt) would bypass
authentication on routes that do not touch the repository
(/api/models, /api/mcp/config, /api/memory, /api/skills, …).

Fix: middleware now calls get_current_user_from_request() strictly
and catches the resulting HTTPException to render a 401 with the
proper fine-grained error code (token_invalid, token_expired,
user_not_found, …). On success it stamps request.state.user and
the contextvar so repository-layer owner filters work downstream.

The 4 old "_with_cookie_passes" tests in test_auth_middleware.py
were written for the presence-only behaviour; they asserted that
a junk cookie would make the handler return 200. They are renamed
to "_with_junk_cookie_rejected" and their assertions flipped to
401. The negative path (no cookie → 401 not_authenticated)
is unchanged.

Verified:
  no cookie       → 401 not_authenticated
  junk cookie     → 401 token_invalid     (the fixed bug)
  expired cookie  → 401 token_expired

Tests: 284 passed (auth + persistence + isolation)
Lint: clean

* security(auth): wire @require_permission(owner_check=True) on isolation routes

Apply the require_permission decorator to all 28 routes that take a
{thread_id} path parameter. Combined with the strict middleware
(previous commit), this gives the double-layer protection that
AUTH_TEST_PLAN test 7.5.9 documents:

  Layer 1 (AuthMiddleware): cookie + JWT validation, rejects junk
                            cookies and stamps request.state.user
  Layer 2 (@require_permission with owner_check=True): per-resource
                            ownership verification via
                            ThreadMetaStore.check_access — returns
                            404 if a different user owns the thread

The decorator's owner_check branch is rewritten to use the SQL
thread_meta_repo (the 2.0-rc persistence layer) instead of the
LangGraph store path that PR #1728 used (_store_get / get_store
in routers/threads.py). The inject_record convenience is dropped
— no caller in 2.0 needs the LangGraph blob, and the SQL repo has
a different shape.

Routes decorated (28 total):
- threads.py: delete, patch, get, get-state, post-state, post-history
- thread_runs.py: post-runs, post-runs-stream, post-runs-wait,
  list_runs, get_run, cancel_run, join_run, stream_existing_run,
  list_thread_messages, list_run_messages, list_run_events,
  thread_token_usage
- feedback.py: create, list, stats, delete
- uploads.py: upload (added Request param), list, delete
- artifacts.py: get_artifact
- suggestions.py: generate (renamed body parameter to avoid
  conflict with FastAPI Request)

Test fixes:
- test_suggestions_router.py: bypass the decorator via __wrapped__
  (the unit tests cover parsing logic, not auth — no point spinning
  up a thread_meta_repo just to test JSON unwrapping)
- test_auth_middleware.py 4 fake-cookie tests: already updated in
  the previous commit (745bf432)

Tests: 293 passed (auth + persistence + isolation + suggestions)
Lint: clean

* security(auth): defense-in-depth fixes from release validation pass

Eight findings caught while running the AUTH_TEST_PLAN end-to-end against
the deployed sg_dev stack. Each is a pre-condition for shipping
release/2.0-rc that the previous PRs missed.

Backend hardening
- routers/auth.py: rate limiter X-Real-IP now requires AUTH_TRUSTED_PROXIES
  whitelist (CIDR/IP allowlist). Without nginx in front, the previous code
  honored arbitrary X-Real-IP, letting an attacker rotate the header to
  fully bypass the per-IP login lockout.
- routers/auth.py: 36-entry common-password blocklist via Pydantic
  field_validator on RegisterRequest + ChangePasswordRequest. The shared
  _validate_strong_password helper keeps the constraint in one place.
- routers/threads.py: ThreadCreateRequest + ThreadPatchRequest strip
  server-reserved metadata keys (owner_id, user_id) via Pydantic
  field_validator so a forged value can never round-trip back to other
  clients reading the same thread. The actual ownership invariant stays
  on the threads_meta row; this closes the metadata-blob echo gap.
- authz.py + thread_meta/sql.py: require_permission gains a require_existing
  flag plumbed through check_access(require_existing=True). Destructive
  routes (DELETE/PATCH/state-update/runs/feedback) now treat a missing
  thread_meta row as 404 instead of "untracked legacy thread, allow",
  closing the cross-user delete-idempotence gap where any user could
  successfully DELETE another user's deleted thread.
- repositories/sqlite.py + base.py: update_user raises UserNotFoundError
  on a vanished row instead of silently returning the input. Concurrent
  delete during password reset can no longer look like a successful update.
- runtime/user_context.py: resolve_owner_id() coerces User.id (UUID) to
  str at the contextvar boundary so SQLAlchemy String(64) columns can
  bind it. The whole 2.0-rc isolation pipeline was previously broken
  end-to-end (POST /api/threads → 500 "type 'UUID' is not supported").
- persistence/engine.py: SQLAlchemy listener enables PRAGMA journal_mode=WAL,
  synchronous=NORMAL, foreign_keys=ON on every new SQLite connection.
  TC-UPG-06 in the test plan expects WAL; previous code shipped with the
  default 'delete' journal.
- auth_middleware.py: stamp request.state.auth = AuthContext(...) so
  @require_permission's short-circuit fires; previously every isolation
  request did a duplicate JWT decode + users SELECT. Also unifies the
  401 payload through AuthErrorResponse(...).model_dump().
- app.py: _ensure_admin_user restructure removes the noqa F821 scoping
  bug where 'password' was referenced outside the branch that defined it.
  New _announce_credentials helper absorbs the duplicate log block in
  the fresh-admin and reset-admin branches.

* fix(frontend+nginx): rollout CSRF on every state-changing client path

The frontend was 100% broken in gateway-pro mode for any user trying to
open a specific chat thread. Three cumulative bugs each silently
masked the next.

LangGraph SDK CSRF gap (api-client.ts)
- The Client constructor took only apiUrl, no defaultHeaders, no fetch
  interceptor. The SDK's internal fetch never sent X-CSRF-Token, so
  every state-changing /api/langgraph-compat/* call (runs/stream,
  threads/search, threads/{tid}/history, ...) hit CSRFMiddleware and
  got 403 before reaching the auth check. UI symptom: empty thread page
  with no error message; the SPA's hooks swallowed the rejection.
- Fix: pass an onRequest hook that injects X-CSRF-Token from the
  csrf_token cookie per request. Reading the cookie per call (not at
  construction time) handles login / logout / password-change cookie
  rotation transparently. The SDK's prepareFetchOptions calls
  onRequest for both regular requests AND streaming/SSE/reconnect, so
  the same hook covers runs.stream and runs.joinStream.

Raw fetch CSRF gap (7 files)
- Audit: 11 frontend fetch sites, only 2 included CSRF (login/setup +
  account-settings change-password). The other 7 routed through raw
  fetch() with no header — suggestions, memory, agents, mcp, skills,
  uploads, and the local thread cleanup hook all 403'd silently.
- Fix: enhance fetcher.ts:fetchWithAuth to auto-inject X-CSRF-Token on
  POST/PUT/DELETE/PATCH from a single shared readCsrfCookie() helper.
  Convert all 7 raw fetch() callers to fetchWithAuth so the contract
  is centrally enforced. api-client.ts and fetcher.ts share
  readCsrfCookie + STATE_CHANGING_METHODS to avoid drift.

nginx routing + buffering (nginx.local.conf)
- The auth feature shipped without updating the nginx config: per-API
  explicit location blocks but no /api/v1/auth/, /api/feedback, /api/runs.
  The frontend's client-side fetches to /api/v1/auth/login/local 404'd
  from the Next.js side because nginx routed /api/* to the frontend.
- Fix: add catch-all `location /api/` that proxies to the gateway.
  nginx longest-prefix matching keeps the explicit blocks (/api/models,
  /api/threads regex, /api/langgraph/, ...) winning for their paths.
- Fix: disable proxy_buffering + proxy_request_buffering for the
  frontend `location /` block. Without it, nginx tries to spool large
  Next.js chunks into /var/lib/nginx/proxy (root-owned) and fails with
  Permission denied → ERR_INCOMPLETE_CHUNKED_ENCODING → ChunkLoadError.

* test(auth): release-validation test infra and new coverage

Test fixtures and unit tests added during the validation pass.

Router test helpers (NEW: tests/_router_auth_helpers.py)
- make_authed_test_app(): builds a FastAPI test app with a stub
  middleware that stamps request.state.user + request.state.auth and a
  permissive thread_meta_repo mock. TestClient-based router tests
  (test_artifacts_router, test_threads_router) use it instead of bare
  FastAPI() so the new @require_permission(owner_check=True) decorators
  short-circuit cleanly.
- call_unwrapped(): walks the __wrapped__ chain to invoke the underlying
  handler without going through the authz wrappers. Direct-call tests
  (test_uploads_router) use it. Typed with ParamSpec so the wrapped
  signature flows through.

Backend test additions
- test_auth.py: 7 tests for the new _get_client_ip trust model (no
  proxy / trusted proxy / untrusted peer / XFF rejection / invalid
  CIDR / no client). 5 tests for the password blocklist (literal,
  case-insensitive, strong password accepted, change-password binding,
  short-password length-check still fires before blocklist).
  test_update_user_raises_when_row_concurrently_deleted: closes a
  shipped-without-coverage gap on the new UserNotFoundError contract.
- test_thread_meta_repo.py: 4 tests for check_access(require_existing=True)
  — strict missing-row denial, strict owner match, strict owner mismatch,
  strict null-owner still allowed (shared rows survive the tightening).
- test_ensure_admin.py: 3 tests for _migrate_orphaned_threads /
  _iter_store_items pagination, covering the TC-UPG-02 upgrade story
  end-to-end via mock store. Closes the gap where the cursor pagination
  was untested even though the previous PR rewrote it.
- test_threads_router.py: 5 tests for _strip_reserved_metadata
  (owner_id removal, user_id removal, safe-keys passthrough, empty
  input, both-stripped).
- test_auth_type_system.py: replace "password123" fixtures with
  Tr0ub4dor3a / AnotherStr0ngPwd! so the new password blocklist
  doesn't reject the test data.

* docs(auth): refresh TC-DOCKER-05 + document Docker validation gap

- AUTH_TEST_PLAN.md TC-DOCKER-05: the previous expectation
  ("admin password visible in docker logs") was stale after the simplify
  pass that moved credentials to a 0600 file. The grep "Password:" check
  would have silently failed and given a false sense of coverage. New
  expectation matches the actual file-based path: 0600 file in
  DEER_FLOW_HOME, log shows the path (not the secret), reverse-grep
  asserts no leaked password in container logs.
- NEW: docs/AUTH_TEST_DOCKER_GAP.md documents the only un-executed
  block in the test plan (TC-DOCKER-01..06). Reason: sg_dev validation
  host has no Docker daemon installed. The doc maps each Docker case
  to an already-validated bare-metal equivalent (TC-1.1, TC-REENT-01,
  TC-API-02 etc.) so the gap is auditable, and includes pre-flight
  reproduction steps for whoever has Docker available.

---------

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>

* fix(persistence): stream hang when run_events.backend=db

DbRunEventStore._user_id_from_context() returned user.id without
coercing it to str. User.id is a Pydantic UUID, and aiosqlite cannot
bind a raw UUID object to a VARCHAR column, so the INSERT for the
initial human_message event silently rolled back and raised out of
the worker task. Because that put() sat outside the worker's try
block, the finally-clause that publishes end-of-stream never ran
and the SSE stream hung forever.

jsonl mode was unaffected because json.dumps(default=str) coerces
UUID objects transparently.

Fixes:
- db.py: coerce user.id to str at the context-read boundary (matches
  what resolve_user_id already does for the other repositories)
- worker.py: move RunJournal init + human_message put inside the try
  block so any failure flows through the finally/publish_end path
  instead of hanging the subscriber

Defense-in-depth:
- engine.py: add PRAGMA busy_timeout=5000 so checkpointer and event
  store wait for each other on the shared deerflow.db file instead
  of failing immediately under write-lock contention
- journal.py: skip fire-and-forget _flush_sync when a previous flush
  task is still in flight, to avoid piling up concurrent put_batch
  writes on the same SQLAlchemy engine during streaming; flush() now
  waits for pending tasks before draining the buffer
- database_config.py: doc-only update clarifying WAL + busy_timeout
  keep the unified deerflow.db safe for both workloads

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore(persistence): drop redundant busy_timeout PRAGMA

Python's sqlite3 driver defaults to a 5-second busy timeout via the
``timeout`` kwarg of ``sqlite3.connect``, and aiosqlite + SQLAlchemy's
aiosqlite dialect inherit that default. Setting ``PRAGMA busy_timeout=5000``
explicitly was a no-op — verified by reading back the PRAGMA on a fresh
connection (it already reports 5000ms without our PRAGMA).

Concurrent stress test (50 checkpoint writes + 20 event batches + 50
thread_meta updates on the same deerflow.db) still completes with zero
errors and 200/200 rows after removing the explicit PRAGMA.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(rebase): remove duplicate definitions and update stale module paths

Rebase left duplicate function blocks in worker.py (triple human_message
write causing 3x user messages in /history), deps.py, and prompt.py.
Also update checkpointer imports from the old deerflow.agents.checkpointer
path to deerflow.runtime.checkpointer, and clean up orphaned feedback
props in the frontend message components.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(user-context): add DEFAULT_USER_ID and get_effective_user_id helper

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(paths): add user-aware path methods with optional user_id parameter

Add _validate_user_id(), user_dir(), user_memory_file(), user_agent_memory_file()
and optional keyword-only user_id parameter to all thread-related path methods.
When user_id is provided, paths resolve under users/{user_id}/threads/{thread_id}/;
when omitted, legacy layout is preserved for backward compatibility.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(memory): add user_id to MemoryStorage interface for per-user isolation

Thread user_id through MemoryStorage.load/reload/save abstract methods and
FileMemoryStorage, re-keying the in-memory cache from bare agent_name to a
(user_id, agent_name) tuple to prevent cross-user cache collisions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(memory): thread user_id through memory updater layer

Add `user_id` keyword-only parameter to all public updater functions
(_save_memory_to_file, get_memory_data, reload_memory_data, import_memory_data,
clear_memory_data, create/delete/update_memory_fact) and regular keyword param
to MemoryUpdater.update_memory + update_memory_from_conversation, propagating
it to every storage load/save/reload call.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(memory): capture user_id at enqueue time for async-safe thread isolation

Add user_id field to ConversationContext and MemoryUpdateQueue.add() so the
user identity is stored explicitly at request time, before threading.Timer
fires on a different thread where ContextVar values do not propagate.
MemoryMiddleware.after_agent() now calls get_effective_user_id() at enqueue
time and passes the value through to updater.update_memory().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(isolation): wire user_id through all Paths and memory callsites

Pass user_id=get_effective_user_id() at every callsite that invokes
Paths methods or memory functions, enabling per-user filesystem isolation
throughout the harness and app layers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(migration): add idempotent script for per-user data migration

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update CLAUDE.md and config docs for per-user isolation

* feat(events): add pagination to list_messages_by_run on all store backends

Replicates the existing before_seq/after_seq/limit cursor-pagination pattern
from list_messages onto list_messages_by_run across the abstract interface,
MemoryRunEventStore, JsonlRunEventStore, and DbRunEventStore.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(api): add GET /api/runs/{run_id}/messages with cursor pagination

New endpoint resolves thread_id from the run record and delegates to
RunEventStore.list_messages_by_run for cursor-based pagination.
Ownership is enforced implicitly via RunStore.get() user filtering.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(api): add GET /api/runs/{run_id}/feedback

Delegates to FeedbackRepository.list_by_run via the existing _resolve_run
helper; includes tests for success, 404, empty list, and 503 (no DB).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(api): retrofit cursor pagination onto GET /threads/{tid}/runs/{rid}/messages

Replace bare list[dict] response with {data: [...], has_more: bool} envelope,
forwarding limit/before_seq/after_seq query params to the event store.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add run-level API endpoints to CLAUDE.md routers table

* refactor(threads): remove event-store message loader and feedback from state/history endpoints

State and history endpoints now return messages purely from the
checkpointer's channel_values. The _get_event_store_messages helper
(which loaded the full event-store transcript with feedback attached)
is removed along with its tests. Frontend will use the dedicated
GET /api/runs/{run_id}/messages and /feedback endpoints instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: JilongSun <965640067@qq.com>
Co-authored-by: jie <49781832+stan-fu@users.noreply.github.com>
Co-authored-by: cooper <cooperfu@tencent.com>
Co-authored-by: yangzheli <43645580+yangzheli@users.noreply.github.com>
Co-authored-by: greatmengqi <chenmengqi.0376@gmail.com>
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-04-26 11:13:01 +08:00
foreleven 00a90bbd3d refactor: Remove init_token handling from admin initialization logic and related tests 2026-04-26 11:09:56 +08:00
JeffJiang 44d9953e2e feat: Add metadata and descriptions to various documentation pages in Chinese
- Added titles and descriptions to workspace usage, configuration, customization, design principles, installation, integration guide, lead agent, MCP integration, memory system, middleware, quick start, sandbox, skills, subagents, and tools documentation.
- Removed outdated API/Gateway reference and concepts glossary pages.
- Updated configuration reference to reflect current structure and removed unnecessary sections.
- Introduced new model provider documentation for Ark and updated the index page for model providers.
- Enhanced tutorials with titles and descriptions for better clarity and navigation.
2026-04-26 11:09:55 +08:00
rayhpeng 56d5fa3337 feat(persistence):Unified persistence layer with event store, feedback, and rebase cleanup (#2134)
* feat(persistence): add unified persistence layer with event store, token tracking, and feedback (#1930)

* feat(persistence): add SQLAlchemy 2.0 async ORM scaffold

Introduce a unified database configuration (DatabaseConfig) that
controls both the LangGraph checkpointer and the DeerFlow application
persistence layer from a single `database:` config section.

New modules:
- deerflow.config.database_config — Pydantic config with memory/sqlite/postgres backends
- deerflow.persistence — async engine lifecycle, DeclarativeBase with to_dict mixin, Alembic skeleton
- deerflow.runtime.runs.store — RunStore ABC + MemoryRunStore implementation

Gateway integration initializes/tears down the persistence engine in
the existing langgraph_runtime() context manager. Legacy checkpointer
config is preserved for backward compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add RunEventStore ABC + MemoryRunEventStore

Phase 2-A prerequisite for event storage: adds the unified run event
stream interface (RunEventStore) with an in-memory implementation,
RunEventsConfig, gateway integration, and comprehensive tests (27 cases).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add ORM models, repositories, DB/JSONL event stores, RunJournal, and API endpoints

Phase 2-B: run persistence + event storage + token tracking.

- ORM models: RunRow (with token fields), ThreadMetaRow, RunEventRow
- RunRepository implements RunStore ABC via SQLAlchemy ORM
- ThreadMetaRepository with owner access control
- DbRunEventStore with trace content truncation and cursor pagination
- JsonlRunEventStore with per-run files and seq recovery from disk
- RunJournal (BaseCallbackHandler) captures LLM/tool/lifecycle events,
  accumulates token usage by caller type, buffers and flushes to store
- RunManager now accepts optional RunStore for persistent backing
- Worker creates RunJournal, writes human_message, injects callbacks
- Gateway deps use factory functions (RunRepository when DB available)
- New endpoints: messages, run messages, run events, token-usage
- ThreadCreateRequest gains assistant_id field
- 92 tests pass (33 new), zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add user feedback + follow-up run association

Phase 2-C: feedback and follow-up tracking.

- FeedbackRow ORM model (rating +1/-1, optional message_id, comment)
- FeedbackRepository with CRUD, list_by_run/thread, aggregate stats
- Feedback API endpoints: create, list, stats, delete
- follow_up_to_run_id in RunCreateRequest (explicit or auto-detected
  from latest successful run on the thread)
- Worker writes follow_up_to_run_id into human_message event metadata
- Gateway deps: feedback_repo factory + getter
- 17 new tests (14 FeedbackRepository + 3 follow-up association)
- 109 total tests pass, zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test+config: comprehensive Phase 2 test coverage + deprecate checkpointer config

- config.example.yaml: deprecate standalone checkpointer section, activate
  unified database:sqlite as default (drives both checkpointer + app data)
- New: test_thread_meta_repo.py (14 tests) — full ThreadMetaRepository coverage
  including check_access owner logic, list_by_owner pagination
- Extended test_run_repository.py (+4 tests) — completion preserves fields,
  list ordering desc, limit, owner_none returns all
- Extended test_run_journal.py (+8 tests) — on_chain_error, track_tokens=false,
  middleware no ai_message, unknown caller tokens, convenience fields,
  tool_error, non-summarization custom event
- Extended test_run_event_store.py (+7 tests) — DB batch seq continuity,
  make_run_event_store factory (memory/db/jsonl/fallback/unknown)
- Extended test_phase2b_integration.py (+4 tests) — create_or_reject persists,
  follow-up metadata, summarization in history, full DB-backed lifecycle
- Fixed DB integration test to use proper fake objects (not MagicMock)
  for JSON-serializable metadata
- 157 total Phase 2 tests pass, zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* config: move default sqlite_dir to .deer-flow/data

Keep SQLite databases alongside other DeerFlow-managed data
(threads, memory) under the .deer-flow/ directory instead of a
top-level ./data folder.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(persistence): remove UTFJSON, use engine-level json_serializer + datetime.now()

- Replace custom UTFJSON type with standard sqlalchemy.JSON in all ORM
  models. Add json_serializer=json.dumps(ensure_ascii=False) to all
  create_async_engine calls so non-ASCII text (Chinese etc.) is stored
  as-is in both SQLite and Postgres.
- Change ORM datetime defaults from datetime.now(UTC) to datetime.now(),
  remove UTC imports.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): simplify deps.py with getter factory + inline repos

- Replace 6 identical getter functions with _require() factory.
- Inline 3 _make_*_repo() factories into langgraph_runtime(), call
  get_session_factory() once instead of 3 times.
- Add thread_meta upsert in start_run (services.py).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(docker): add UV_EXTRAS build arg for optional dependencies

Support installing optional dependency groups (e.g. postgres) at
Docker build time via UV_EXTRAS build arg:
  UV_EXTRAS=postgres docker compose build

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(journal): fix flush, token tracking, and consolidate tests

RunJournal fixes:
- _flush_sync: retain events in buffer when no event loop instead of
  dropping them; worker's finally block flushes via async flush().
- on_llm_end: add tool_calls filter and caller=="lead_agent" guard for
  ai_message events; mark message IDs for dedup with record_llm_usage.
- worker.py: persist completion data (tokens, message count) to RunStore
  in finally block.

Model factory:
- Auto-inject stream_usage=True for BaseChatOpenAI subclasses with
  custom api_base, so usage_metadata is populated in streaming responses.

Test consolidation:
- Delete test_phase2b_integration.py (redundant with existing tests).
- Move DB-backed lifecycle test into test_run_journal.py.
- Add tests for stream_usage injection in test_model_factory.py.
- Clean up executor/task_tool dead journal references.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): widen content type to str|dict in all store backends

Allow event content to be a dict (for structured OpenAI-format messages)
in addition to plain strings. Dict values are JSON-serialized for the DB
backend and deserialized on read; memory and JSONL backends handle dicts
natively. Trace truncation now serializes dicts to JSON before measuring.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(events): use metadata flag instead of heuristic for dict content detection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(converters): add LangChain-to-OpenAI message format converters

Pure functions langchain_to_openai_message, langchain_to_openai_completion,
langchain_messages_to_openai, and _infer_finish_reason for converting
LangChain BaseMessage objects to OpenAI Chat Completions format, used by
RunJournal for event storage. 15 unit tests added.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(converters): handle empty list content as null, clean up test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): human_message content uses OpenAI user message format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): ai_message uses OpenAI format, add ai_tool_call message event

- ai_message content now uses {"role": "assistant", "content": "..."} format
- New ai_tool_call message event emitted when lead_agent LLM responds with tool_calls
- ai_tool_call uses langchain_to_openai_message converter for consistent format
- Both events include finish_reason in metadata ("stop" or "tool_calls")

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): add tool_result message event with OpenAI tool message format

Cache tool_call_id from on_tool_start keyed by run_id as fallback for on_tool_end,
then emit a tool_result message event (role=tool, tool_call_id, content) after each
successful tool completion.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): summary content uses OpenAI system message format

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): replace llm_start/llm_end with llm_request/llm_response in OpenAI format

Add on_chat_model_start to capture structured prompt messages as llm_request events.
Replace llm_end trace events with llm_response using OpenAI Chat Completions format.
Track llm_call_index to pair request/response events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): add record_middleware method for middleware trace events

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(events): add full run sequence integration test for OpenAI content format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): align message events with checkpoint format and add middleware tag injection

- Message events (ai_message, ai_tool_call, tool_result, human_message) now use
  BaseMessage.model_dump() format, matching LangGraph checkpoint values.messages
- on_tool_end extracts tool_call_id/name/status from ToolMessage objects
- on_tool_error now emits tool_result message events with error status
- record_middleware uses middleware:{tag} event_type and middleware category
- Summarization custom events use middleware:summarize category
- TitleMiddleware injects middleware:title tag via get_config() inheritance
- SummarizationMiddleware model bound with middleware:summarize tag
- Worker writes human_message using HumanMessage.model_dump()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(threads): switch search endpoint to threads_meta table and sync title

- POST /api/threads/search now queries threads_meta table directly,
  removing the two-phase Store + Checkpointer scan approach
- Add ThreadMetaRepository.search() with metadata/status filters
- Add ThreadMetaRepository.update_display_name() for title sync
- Worker syncs checkpoint title to threads_meta.display_name on run completion
- Map display_name to values.title in search response for API compatibility

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(threads): history endpoint reads messages from event store

- POST /api/threads/{thread_id}/history now combines two data sources:
  checkpointer for checkpoint_id, metadata, title, thread_data;
  event store for messages (complete history, not truncated by summarization)
- Strip internal LangGraph metadata keys from response
- Remove full channel_values serialization in favor of selective fields

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove duplicate optional-dependencies header in pyproject.toml

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(middleware): pass tagged config to TitleMiddleware ainvoke call

Without the config, the middleware:title tag was not injected,
causing the LLM response to be recorded as a lead_agent ai_message
in run_events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: resolve merge conflict in .env.example

Keep both DATABASE_URL (from persistence-scaffold) and WECOM
credentials (from main) after the merge.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address review feedback on PR #1851

- Fix naive datetime.now() → datetime.now(UTC) in all ORM models
- Fix seq race condition in DbRunEventStore.put() with FOR UPDATE
  and UNIQUE(thread_id, seq) constraint
- Encapsulate _store access in RunManager.update_run_completion()
- Deduplicate _store.put() logic in RunManager via _persist_to_store()
- Add update_run_completion to RunStore ABC + MemoryRunStore
- Wire follow_up_to_run_id through the full create path
- Add error recovery to RunJournal._flush_sync() lost-event scenario
- Add migration note for search_threads breaking change
- Fix test_checkpointer_none_fix mock to set database=None

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update uv.lock

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address 22 review comments from CodeQL, Copilot, and Code Quality

Bug fixes:
- Sanitize log params to prevent log injection (CodeQL)
- Reset threads_meta.status to idle/error when run completes
- Attach messages only to latest checkpoint in /history response
- Write threads_meta on POST /threads so new threads appear in search

Lint fixes:
- Remove unused imports (journal.py, migrations/env.py, test_converters.py)
- Convert lambda to named function (engine.py, Ruff E731)
- Remove unused logger definitions in repos (Ruff F841)
- Add logging to JSONL decode errors and empty except blocks
- Separate assert side-effects in tests (CodeQL)
- Remove unused local variables in tests (Ruff F841)
- Fix max_trace_content truncation to use byte length, not char length

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: apply ruff format to persistence and runtime files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Potential fix for pull request finding 'Statement has no effect'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

* refactor(runtime): introduce RunContext to reduce run_agent parameter bloat

Extract checkpointer, store, event_store, run_events_config, thread_meta_repo,
and follow_up_to_run_id into a frozen RunContext dataclass. Add get_run_context()
in deps.py to build the base context from app.state singletons. start_run() uses
dataclasses.replace() to enrich per-run fields before passing ctx to run_agent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): move sanitize_log_param to app/gateway/utils.py

Extract the log-injection sanitizer from routers/threads.py into a shared
utils module and rename to sanitize_log_param (public API). Eliminates the
reverse service → router import in services.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf: use SQL aggregation for feedback stats and thread token usage

Replace Python-side counting in FeedbackRepository.aggregate_by_run with
a single SELECT COUNT/SUM query. Add RunStore.aggregate_tokens_by_thread
abstract method with SQL GROUP BY implementation in RunRepository and
Python fallback in MemoryRunStore. Simplify the thread_token_usage
endpoint to delegate to the new method, eliminating the limit=10000
truncation risk.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: annotate DbRunEventStore.put() as low-frequency path

Add docstring clarifying that put() opens a per-call transaction with
FOR UPDATE and should only be used for infrequent writes (currently
just the initial human_message event). High-throughput callers should
use put_batch() instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(threads): fall back to Store search when ThreadMetaRepository is unavailable

When database.backend=memory (default) or no SQL session factory is
configured, search_threads now queries the LangGraph Store instead of
returning 503. Returns empty list if neither Store nor repo is available.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(persistence): introduce ThreadMetaStore ABC for backend-agnostic thread metadata

Add ThreadMetaStore abstract base class with create/get/search/update/delete
interface. ThreadMetaRepository (SQL) now inherits from it. New
MemoryThreadMetaStore wraps LangGraph BaseStore for memory-mode deployments.

deps.py now always provides a non-None thread_meta_repo, eliminating all
`if thread_meta_repo is not None` guards in services.py, worker.py, and
routers/threads.py. search_threads no longer needs a Store fallback branch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(history): read messages from checkpointer instead of RunEventStore

The /history endpoint now reads messages directly from the
checkpointer's channel_values (the authoritative source) instead of
querying RunEventStore.list_messages(). The RunEventStore API is
preserved for other consumers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address new Copilot review comments

- feedback.py: validate thread_id/run_id before deleting feedback
- jsonl.py: add path traversal protection with ID validation
- run_repo.py: parse `before` to datetime for PostgreSQL compat
- thread_meta_repo.py: fix pagination when metadata filter is active
- database_config.py: use resolve_path for sqlite_dir consistency

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Implement skill self-evolution and skill_manage flow (#1874)

* chore: ignore .worktrees directory

* Add skill_manage self-evolution flow

* Fix CI regressions for skill_manage

* Address PR review feedback for skill evolution

* fix(skill-evolution): preserve history on delete

* fix(skill-evolution): tighten scanner fallbacks

* docs: add skill_manage e2e evidence screenshot

* fix(skill-manage): avoid blocking fs ops in session runtime

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

* fix(config): resolve sqlite_dir relative to CWD, not Paths.base_dir

resolve_path() resolves relative to Paths.base_dir (.deer-flow),
which double-nested the path to .deer-flow/.deer-flow/data/app.db.
Use Path.resolve() (CWD-relative) instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Feature/feishu receive file (#1608)

* feat(feishu): add channel file materialization hook for inbound messages

- Introduce Channel.receive_file(msg, thread_id) as a base method for file materialization; default is no-op.
- Implement FeishuChannel.receive_file to download files/images from Feishu messages, save to sandbox, and inject virtual paths into msg.text.
- Update ChannelManager to call receive_file for any channel if msg.files is present, enabling downstream model access to user-uploaded files.
- No impact on Slack/Telegram or other channels (they inherit the default no-op).

* style(backend): format code with ruff for lint compliance

- Auto-formatted packages/harness/deerflow/agents/factory.py and tests/test_create_deerflow_agent.py using `ruff format`
- Ensured both files conform to project linting standards
- Fixes CI lint check failures caused by code style issues

* fix(feishu): handle file write operation asynchronously to prevent blocking

* fix(feishu): rename GetMessageResourceRequest to _GetMessageResourceRequest and remove redundant code

* test(feishu): add tests for receive_file method and placeholder replacement

* fix(manager): remove unnecessary type casting for channel retrieval

* fix(feishu): update logging messages to reflect resource handling instead of image

* fix(feishu): sanitize filename by replacing invalid characters in file uploads

* fix(feishu): improve filename sanitization and reorder image key handling in message processing

* fix(feishu): add thread lock to prevent filename conflicts during file downloads

* fix(test): correct bad merge in test_feishu_parser.py

* chore: run ruff and apply formatting cleanup
fix(feishu): preserve rich-text attachment order and improve fallback filename handling

* fix(docker): restore gateway env vars and fix langgraph empty arg issue (#1915)

Two production docker-compose.yaml bugs prevent `make up` from working:

1. Gateway missing DEER_FLOW_CONFIG_PATH and DEER_FLOW_EXTENSIONS_CONFIG_PATH
   environment overrides. Added in fb2d99f (#1836) but accidentally reverted
   by ca2fb95 (#1847). Without them, gateway reads host paths from .env via
   env_file, causing FileNotFoundError inside the container.

2. Langgraph command fails when LANGGRAPH_ALLOW_BLOCKING is unset (default).
   Empty $${allow_blocking} inserts a bare space between flags, causing
   ' --no-reload' to be parsed as unexpected extra argument. Fix by building
   args string first and conditionally appending --allow-blocking.

Co-authored-by: cooper <cooperfu@tencent.com>

* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities (#1904)

* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities

Fix `<button>` inside `<a>` invalid HTML in artifact components and add
missing `noopener,noreferrer` to `window.open` calls to prevent reverse
tabnabbing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(frontend): address Copilot review on tabnabbing and double-tab-open

Remove redundant parent onClick on web_fetch ChainOfThoughtStep to
prevent opening two tabs on link click, and explicitly null out
window.opener after window.open() for defensive tabnabbing hardening.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(persistence): organize entities into per-entity directories

Restructure the persistence layer from horizontal "models/ + repositories/"
split into vertical entity-aligned directories. Each entity (thread_meta,
run, feedback) now owns its ORM model, abstract interface (where applicable),
and concrete implementations under a single directory with an aggregating
__init__.py for one-line imports.

Layout:
  persistence/thread_meta/{base,model,sql,memory}.py
  persistence/run/{model,sql}.py
  persistence/feedback/{model,sql}.py

models/__init__.py is kept as a facade so Alembic autogenerate continues to
discover all ORM tables via Base.metadata. RunEventRow remains under
models/run_event.py because its storage implementation lives in
runtime/events/store/db.py and has no matching repository directory.

The repositories/ directory is removed entirely. All call sites in
gateway/deps.py and tests are updated to import from the new entity
packages, e.g.:

    from deerflow.persistence.thread_meta import ThreadMetaRepository
    from deerflow.persistence.run import RunRepository
    from deerflow.persistence.feedback import FeedbackRepository

Full test suite passes (1690 passed, 14 skipped).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(gateway): sync thread rename and delete through ThreadMetaStore

The POST /threads/{id}/state endpoint previously synced title changes
only to the LangGraph Store via _store_upsert. In sqlite mode the search
endpoint reads from the ThreadMetaRepository SQL table, so renames never
appeared in /threads/search until the next agent run completed (worker.py
syncs title from checkpoint to thread_meta in its finally block).

Likewise the DELETE /threads/{id} endpoint cleaned up the filesystem,
Store, and checkpointer but left the threads_meta row orphaned in sqlite,
so deleted threads kept appearing in /threads/search.

Fix both endpoints by routing through the ThreadMetaStore abstraction
which already has the correct sqlite/memory implementations wired up by
deps.py. The rename path now calls update_display_name() and the delete
path calls delete() — both work uniformly across backends.

Verified end-to-end with curl in gateway mode against sqlite backend.
Existing test suite (1690 passed) and focused router/repo tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): route all thread metadata access through ThreadMetaStore

Following the rename/delete bug fix in PR1, migrate the remaining direct
LangGraph Store reads/writes in the threads router and services to the
ThreadMetaStore abstraction so that the sqlite and memory backends behave
identically and the legacy dual-write paths can be removed.

Migrated endpoints (threads.py):
- create_thread: idempotency check + write now use thread_meta_repo.get/create
  instead of dual-writing the LangGraph Store and the SQL row.
- get_thread: reads from thread_meta_repo.get; the checkpoint-only fallback
  for legacy threads is preserved.
- patch_thread: replaced _store_get/_store_put with thread_meta_repo.update_metadata.
- delete_thread_data: dropped the legacy store.adelete; thread_meta_repo.delete
  already covers it.

Removed dead code (services.py):
- _upsert_thread_in_store — redundant with the immediately following
  thread_meta_repo.create() call.
- _sync_thread_title_after_run — worker.py's finally block already syncs
  the title via thread_meta_repo.update_display_name() after each run.

Removed dead code (threads.py):
- _store_get / _store_put / _store_upsert helpers (no remaining callers).
- THREADS_NS constant.
- get_store import (router no longer touches the LangGraph Store directly).

New abstract method:
- ThreadMetaStore.update_metadata(thread_id, metadata) merges metadata into
  the thread's metadata field. Implemented in both ThreadMetaRepository (SQL,
  read-modify-write inside one session) and MemoryThreadMetaStore. Three new
  unit tests cover merge / empty / nonexistent behaviour.

Net change: -134 lines. Full test suite: 1693 passed, 14 skipped.
Verified end-to-end with curl in gateway mode against sqlite backend
(create / patch / get / rename / search / delete).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: JilongSun <965640067@qq.com>
Co-authored-by: jie <49781832+stan-fu@users.noreply.github.com>
Co-authored-by: cooper <cooperfu@tencent.com>
Co-authored-by: yangzheli <43645580+yangzheli@users.noreply.github.com>

* feat(auth): release-validation pass for 2.0-rc — 12 blockers + simplify follow-ups (#2008)

* feat(auth): introduce backend auth module

Port RFC-001 authentication core from PR #1728:
- JWT token handling (create_access_token, decode_token, TokenPayload)
- Password hashing (bcrypt) with verify_password
- SQLite UserRepository with base interface
- Provider Factory pattern (LocalAuthProvider)
- CLI reset_admin tool
- Auth-specific errors (AuthErrorCode, TokenError, AuthErrorResponse)

Deps:
- bcrypt>=4.0.0
- pyjwt>=2.9.0
- email-validator>=2.0.0
- backend/uv.toml pins public PyPI index

Tests: 12 pure unit tests (test_auth_config.py, test_auth_errors.py).

Scope note: authz.py, test_auth.py, and test_auth_type_system.py are
deferred to commit 2 because they depend on middleware and deps wiring
that is not yet in place. Commit 1 stays "pure new files only" as the
spec mandates.

* feat(auth): wire auth end-to-end (middleware + frontend replacement)

Backend:
- Port auth_middleware, csrf_middleware, langgraph_auth, routers/auth
- Port authz decorator (owner_filter_key defaults to 'owner_id')
- Merge app.py: register AuthMiddleware + CSRFMiddleware + CORS, add
  _ensure_admin_user lifespan hook, _migrate_orphaned_threads helper,
  register auth router
- Merge deps.py: add get_local_provider, get_current_user_from_request,
  get_optional_user_from_request; keep get_current_user as thin str|None
  adapter for feedback router
- langgraph.json: add auth path pointing to langgraph_auth.py:auth
- Rename metadata['user_id'] -> metadata['owner_id'] in langgraph_auth
  (both metadata write and LangGraph filter dict) + test fixtures

Frontend:
- Delete better-auth library and api catch-all route
- Remove better-auth npm dependency and env vars (BETTER_AUTH_SECRET,
  BETTER_AUTH_GITHUB_*) from env.js
- Port frontend/src/core/auth/* (AuthProvider, gateway-config,
  proxy-policy, server-side getServerSideUser, types)
- Port frontend/src/core/api/fetcher.ts
- Port (auth)/layout, (auth)/login, (auth)/setup pages
- Rewrite workspace/layout.tsx as server component that calls
  getServerSideUser and wraps in AuthProvider
- Port workspace/workspace-content.tsx for the client-side sidebar logic

Tests:
- Port 5 auth test files (test_auth, test_auth_middleware,
  test_auth_type_system, test_ensure_admin, test_langgraph_auth)
- 176 auth tests PASS

After this commit: login/logout/registration flow works, but persistence
layer does not yet filter by owner_id. Commit 4 closes that gap.

* feat(auth): account settings page + i18n

- Port account-settings-page.tsx (change password, change email, logout)
- Wire into settings-dialog.tsx as new "account" section with UserIcon,
  rendered first in the section list
- Add i18n keys:
  - en-US/zh-CN: settings.sections.account ("Account" / "账号")
  - en-US/zh-CN: button.logout ("Log out" / "退出登录")
  - types.ts: matching type declarations

* feat(auth): enforce owner_id across 2.0-rc persistence layer

Add request-scoped contextvar-based owner filtering to threads_meta,
runs, run_events, and feedback repositories. Router code is unchanged
— isolation is enforced at the storage layer so that any caller that
forgets to pass owner_id still gets filtered results, and new routes
cannot accidentally leak data.

Core infrastructure
-------------------
- deerflow/runtime/user_context.py (new):
  - ContextVar[CurrentUser | None] with default None
  - runtime_checkable CurrentUser Protocol (structural subtype with .id)
  - set/reset/get/require helpers
  - AUTO sentinel + resolve_owner_id(value, method_name) for sentinel
    three-state resolution: AUTO reads contextvar, explicit str
    overrides, explicit None bypasses the filter (for migration/CLI)

Repository changes
------------------
- ThreadMetaRepository: create/get/search/update_*/delete gain
  owner_id=AUTO kwarg; read paths filter by owner, writes stamp it,
  mutations check ownership before applying
- RunRepository: put/get/list_by_thread/delete gain owner_id=AUTO kwarg
- FeedbackRepository: create/get/list_by_run/list_by_thread/delete
  gain owner_id=AUTO kwarg
- DbRunEventStore: list_messages/list_events/list_messages_by_run/
  count_messages/delete_by_thread/delete_by_run gain owner_id=AUTO
  kwarg. Write paths (put/put_batch) read contextvar softly: when a
  request-scoped user is available, owner_id is stamped; background
  worker writes without a user context pass None which is valid
  (orphan row to be bound by migration)

Schema
------
- persistence/models/run_event.py: RunEventRow.owner_id = Mapped[
  str | None] = mapped_column(String(64), nullable=True, index=True)
- No alembic migration needed: 2.0 ships fresh, Base.metadata.create_all
  picks up the new column automatically

Middleware
----------
- auth_middleware.py: after cookie check, call get_optional_user_from_
  request to load the real User, stamp it into request.state.user AND
  the contextvar via set_current_user, reset in a try/finally. Public
  paths and unauthenticated requests continue without contextvar, and
  @require_auth handles the strict 401 path

Test infrastructure
-------------------
- tests/conftest.py: @pytest.fixture(autouse=True) _auto_user_context
  sets a default SimpleNamespace(id="test-user-autouse") on every test
  unless marked @pytest.mark.no_auto_user. Keeps existing 20+
  persistence tests passing without modification
- pyproject.toml [tool.pytest.ini_options]: register no_auto_user
  marker so pytest does not emit warnings for opt-out tests
- tests/test_user_context.py: 6 tests covering three-state semantics,
  Protocol duck typing, and require/optional APIs
- tests/test_thread_meta_repo.py: one test updated to pass owner_id=
  None explicitly where it was previously relying on the old default

Test results
------------
- test_user_context.py: 6 passed
- test_auth*.py + test_langgraph_auth.py + test_ensure_admin.py: 127
- test_run_event_store / test_run_repository / test_thread_meta_repo
  / test_feedback: 92 passed
- Full backend suite: 1905 passed, 2 failed (both @requires_llm flaky
  integration tests unrelated to auth), 1 skipped

* feat(auth): extend orphan migration to 2.0-rc persistence tables

_ensure_admin_user now runs a three-step pipeline on every boot:

  Step 1 (fatal):     admin user exists / is created / password is reset
  Step 2 (non-fatal): LangGraph store orphan threads → admin
  Step 3 (non-fatal): SQL persistence tables → admin
    - threads_meta
    - runs
    - run_events
    - feedback

Each step is idempotent. The fatal/non-fatal split mirrors PR #1728's
original philosophy: admin creation failure blocks startup (the system
is unusable without an admin), whereas migration failures log a warning
and let the service proceed (a partial migration is recoverable; a
missing admin is not).

Key helpers
-----------
- _iter_store_items(store, namespace, *, page_size=500):
  async generator that cursor-paginates across LangGraph store pages.
  Fixes PR #1728's hardcoded limit=1000 bug that would silently lose
  orphans beyond the first page.

- _migrate_orphaned_threads(store, admin_user_id):
  Rewritten to use _iter_store_items. Returns the migrated count so the
  caller can log it; raises only on unhandled exceptions.

- _migrate_orphan_sql_tables(admin_user_id):
  Imports the 4 ORM models lazily, grabs the shared session factory,
  runs one UPDATE per table in a single transaction, commits once.
  No-op when no persistence backend is configured (in-memory dev).

Tests: test_ensure_admin.py (8 passed)

* test(auth): port AUTH test plan docs + lint/format pass

- Port backend/docs/AUTH_TEST_PLAN.md and AUTH_UPGRADE.md from PR #1728
- Rename metadata.user_id → metadata.owner_id in AUTH_TEST_PLAN.md
  (4 occurrences from the original PR doc)
- ruff auto-fix UP037 in sentinel type annotations: drop quotes around
  "str | None | _AutoSentinel" now that from __future__ import
  annotations makes them implicit string forms
- ruff format: 2 files (app/gateway/app.py, runtime/user_context.py)

Note on test coverage additions:
- conftest.py autouse fixture was already added in commit 4 (had to
  be co-located with the repository changes to keep pre-existing
  persistence tests passing)
- cross-user isolation E2E tests (test_owner_isolation.py) deferred
  — enforcement is already proven by the 98-test repository suite
  via the autouse fixture + explicit _AUTO sentinel exercises
- New test cases (TC-API-17..20, TC-ATK-13, TC-MIG-01..07) listed
  in AUTH_TEST_PLAN.md are deferred to a follow-up PR — they are
  manual-QA test cases rather than pytest code, and the spec-level
  coverage is already met by test_user_context.py + the 98-test
  repository suite.

Final test results:
- Auth suite (test_auth*, test_langgraph_auth, test_ensure_admin,
  test_user_context): 186 passed
- Persistence suite (test_run_event_store, test_run_repository,
  test_thread_meta_repo, test_feedback): 98 passed
- Lint: ruff check + ruff format both clean

* test(auth): add cross-user isolation test suite

10 tests exercising the storage-layer owner filter by manually
switching the user_context contextvar between two users. Verifies
the safety invariant:

  After a repository write with owner_id=A, a subsequent read with
  owner_id=B must not return the row, and vice versa.

Covers all 4 tables that own user-scoped data:

TC-API-17  threads_meta  — read, search, update, delete cross-user
TC-API-18  runs          — get, list_by_thread, delete cross-user
TC-API-19  run_events    — list_messages, list_events, count_messages,
                           delete_by_thread (CRITICAL: raw conversation
                           content leak vector)
TC-API-20  feedback      — get, list_by_run, delete cross-user

Plus two meta-tests verifying the sentinel pattern itself:
- AUTO + unset contextvar raises RuntimeError
- explicit owner_id=None bypasses the filter (migration escape hatch)

Architecture note
-----------------
These tests bypass the HTTP layer by design. The full chain
(cookie → middleware → contextvar → repository) is covered piecewise:

- test_auth_middleware.py: middleware sets contextvar from cookies
- test_owner_isolation.py: repositories enforce isolation when
  contextvar is set to different users

Together they prove the end-to-end safety property without the
ceremony of spinning up a full TestClient + in-memory DB for every
router endpoint.

Tests pass: 231 (full auth + persistence + isolation suite)
Lint: clean

* refactor(auth): migrate user repository to SQLAlchemy ORM

Move the users table into the shared persistence engine so auth
matches the pattern of threads_meta, runs, run_events, and feedback —
one engine, one session factory, one schema init codepath.

New files
---------
- persistence/user/__init__.py, persistence/user/model.py: UserRow
  ORM class with partial unique index on (oauth_provider, oauth_id)
- Registered in persistence/models/__init__.py so
  Base.metadata.create_all() picks it up

Modified
--------
- auth/repositories/sqlite.py: rewritten as async SQLAlchemy,
  identical constructor pattern to the other four repositories
  (def __init__(self, session_factory) + self._sf = session_factory)
- auth/config.py: drop users_db_path field — storage is configured
  through config.database like every other table
- deps.py/get_local_provider: construct SQLiteUserRepository with
  the shared session factory, fail fast if engine is not initialised
- tests/test_auth.py: rewrite test_sqlite_round_trip_new_fields to
  use the shared engine (init_engine + close_engine in a tempdir)
- tests/test_auth_type_system.py: add per-test autouse fixture that
  spins up a scratch engine and resets deps._cached_* singletons

* refactor(auth): remove SQL orphan migration (unused in supported scenarios)

The _migrate_orphan_sql_tables helper existed to bind NULL owner_id
rows in threads_meta, runs, run_events, and feedback to the admin on
first boot. But in every supported upgrade path, it's a no-op:

  1. Fresh install: create_all builds fresh tables, no legacy rows
  2. No-auth → with-auth (no existing persistence DB): persistence
     tables are created fresh by create_all, no legacy rows
  3. No-auth → with-auth (has existing persistence DB from #1930):
     NOT a supported upgrade path — "有 DB 到有 DB" schema evolution
     is out of scope; users wipe DB or run manual ALTER

So the SQL orphan migration never has anything to do in the
supported matrix. Delete the function, simplify _ensure_admin_user
from a 3-step pipeline to a 2-step one (admin creation + LangGraph
store orphan migration only).

LangGraph store orphan migration stays: it serves the real
"no-auth → with-auth" upgrade path where a user's existing LangGraph
thread metadata has no owner_id field and needs to be stamped with
the newly-created admin's id.

Tests: 284 passed (auth + persistence + isolation)
Lint: clean

* security(auth): write initial admin password to 0600 file instead of logs

CodeQL py/clear-text-logging-sensitive-data flagged 3 call sites that
logged the auto-generated admin password to stdout via logger.info().
Production log aggregators (ELK/Splunk/etc) would have captured those
cleartext secrets. Replace with a shared helper that writes to
.deer-flow/admin_initial_credentials.txt with mode 0600, and log only
the path.

New file
--------
- app/gateway/auth/credential_file.py: write_initial_credentials()
  helper. Takes email, password, and a "initial"/"reset" label.
  Creates .deer-flow/ if missing, writes a header comment plus the
  email+password, chmods 0o600, returns the absolute Path.

Modified
--------
- app/gateway/app.py: both _ensure_admin_user paths (fresh creation
  + needs_setup password reset) now write to file and log the path
- app/gateway/auth/reset_admin.py: rewritten to use the shared ORM
  repo (SQLiteUserRepository with session_factory) and the
  credential_file helper. The previous implementation was broken
  after the earlier ORM refactor — it still imported _get_users_conn
  and constructed SQLiteUserRepository() without a session factory.

No tests changed — the three password-log sites are all exercised
via existing test_ensure_admin.py which checks that startup
succeeds, not that a specific string appears in logs.

CodeQL alerts 272, 283, 284: all resolved.

* security(auth): strict JWT validation in middleware (fix junk cookie bypass)

AUTH_TEST_PLAN test 7.5.8 expects junk cookies to be rejected with
401. The previous middleware behaviour was "presence-only": check
that some access_token cookie exists, then pass through. In
combination with my Task-12 decision to skip @require_auth
decorators on routes, this created a gap where a request with any
cookie-shaped string (e.g. access_token=not-a-jwt) would bypass
authentication on routes that do not touch the repository
(/api/models, /api/mcp/config, /api/memory, /api/skills, …).

Fix: middleware now calls get_current_user_from_request() strictly
and catches the resulting HTTPException to render a 401 with the
proper fine-grained error code (token_invalid, token_expired,
user_not_found, …). On success it stamps request.state.user and
the contextvar so repository-layer owner filters work downstream.

The 4 old "_with_cookie_passes" tests in test_auth_middleware.py
were written for the presence-only behaviour; they asserted that
a junk cookie would make the handler return 200. They are renamed
to "_with_junk_cookie_rejected" and their assertions flipped to
401. The negative path (no cookie → 401 not_authenticated)
is unchanged.

Verified:
  no cookie       → 401 not_authenticated
  junk cookie     → 401 token_invalid     (the fixed bug)
  expired cookie  → 401 token_expired

Tests: 284 passed (auth + persistence + isolation)
Lint: clean

* security(auth): wire @require_permission(owner_check=True) on isolation routes

Apply the require_permission decorator to all 28 routes that take a
{thread_id} path parameter. Combined with the strict middleware
(previous commit), this gives the double-layer protection that
AUTH_TEST_PLAN test 7.5.9 documents:

  Layer 1 (AuthMiddleware): cookie + JWT validation, rejects junk
                            cookies and stamps request.state.user
  Layer 2 (@require_permission with owner_check=True): per-resource
                            ownership verification via
                            ThreadMetaStore.check_access — returns
                            404 if a different user owns the thread

The decorator's owner_check branch is rewritten to use the SQL
thread_meta_repo (the 2.0-rc persistence layer) instead of the
LangGraph store path that PR #1728 used (_store_get / get_store
in routers/threads.py). The inject_record convenience is dropped
— no caller in 2.0 needs the LangGraph blob, and the SQL repo has
a different shape.

Routes decorated (28 total):
- threads.py: delete, patch, get, get-state, post-state, post-history
- thread_runs.py: post-runs, post-runs-stream, post-runs-wait,
  list_runs, get_run, cancel_run, join_run, stream_existing_run,
  list_thread_messages, list_run_messages, list_run_events,
  thread_token_usage
- feedback.py: create, list, stats, delete
- uploads.py: upload (added Request param), list, delete
- artifacts.py: get_artifact
- suggestions.py: generate (renamed body parameter to avoid
  conflict with FastAPI Request)

Test fixes:
- test_suggestions_router.py: bypass the decorator via __wrapped__
  (the unit tests cover parsing logic, not auth — no point spinning
  up a thread_meta_repo just to test JSON unwrapping)
- test_auth_middleware.py 4 fake-cookie tests: already updated in
  the previous commit (745bf432)

Tests: 293 passed (auth + persistence + isolation + suggestions)
Lint: clean

* security(auth): defense-in-depth fixes from release validation pass

Eight findings caught while running the AUTH_TEST_PLAN end-to-end against
the deployed sg_dev stack. Each is a pre-condition for shipping
release/2.0-rc that the previous PRs missed.

Backend hardening
- routers/auth.py: rate limiter X-Real-IP now requires AUTH_TRUSTED_PROXIES
  whitelist (CIDR/IP allowlist). Without nginx in front, the previous code
  honored arbitrary X-Real-IP, letting an attacker rotate the header to
  fully bypass the per-IP login lockout.
- routers/auth.py: 36-entry common-password blocklist via Pydantic
  field_validator on RegisterRequest + ChangePasswordRequest. The shared
  _validate_strong_password helper keeps the constraint in one place.
- routers/threads.py: ThreadCreateRequest + ThreadPatchRequest strip
  server-reserved metadata keys (owner_id, user_id) via Pydantic
  field_validator so a forged value can never round-trip back to other
  clients reading the same thread. The actual ownership invariant stays
  on the threads_meta row; this closes the metadata-blob echo gap.
- authz.py + thread_meta/sql.py: require_permission gains a require_existing
  flag plumbed through check_access(require_existing=True). Destructive
  routes (DELETE/PATCH/state-update/runs/feedback) now treat a missing
  thread_meta row as 404 instead of "untracked legacy thread, allow",
  closing the cross-user delete-idempotence gap where any user could
  successfully DELETE another user's deleted thread.
- repositories/sqlite.py + base.py: update_user raises UserNotFoundError
  on a vanished row instead of silently returning the input. Concurrent
  delete during password reset can no longer look like a successful update.
- runtime/user_context.py: resolve_owner_id() coerces User.id (UUID) to
  str at the contextvar boundary so SQLAlchemy String(64) columns can
  bind it. The whole 2.0-rc isolation pipeline was previously broken
  end-to-end (POST /api/threads → 500 "type 'UUID' is not supported").
- persistence/engine.py: SQLAlchemy listener enables PRAGMA journal_mode=WAL,
  synchronous=NORMAL, foreign_keys=ON on every new SQLite connection.
  TC-UPG-06 in the test plan expects WAL; previous code shipped with the
  default 'delete' journal.
- auth_middleware.py: stamp request.state.auth = AuthContext(...) so
  @require_permission's short-circuit fires; previously every isolation
  request did a duplicate JWT decode + users SELECT. Also unifies the
  401 payload through AuthErrorResponse(...).model_dump().
- app.py: _ensure_admin_user restructure removes the noqa F821 scoping
  bug where 'password' was referenced outside the branch that defined it.
  New _announce_credentials helper absorbs the duplicate log block in
  the fresh-admin and reset-admin branches.

* fix(frontend+nginx): rollout CSRF on every state-changing client path

The frontend was 100% broken in gateway-pro mode for any user trying to
open a specific chat thread. Three cumulative bugs each silently
masked the next.

LangGraph SDK CSRF gap (api-client.ts)
- The Client constructor took only apiUrl, no defaultHeaders, no fetch
  interceptor. The SDK's internal fetch never sent X-CSRF-Token, so
  every state-changing /api/langgraph-compat/* call (runs/stream,
  threads/search, threads/{tid}/history, ...) hit CSRFMiddleware and
  got 403 before reaching the auth check. UI symptom: empty thread page
  with no error message; the SPA's hooks swallowed the rejection.
- Fix: pass an onRequest hook that injects X-CSRF-Token from the
  csrf_token cookie per request. Reading the cookie per call (not at
  construction time) handles login / logout / password-change cookie
  rotation transparently. The SDK's prepareFetchOptions calls
  onRequest for both regular requests AND streaming/SSE/reconnect, so
  the same hook covers runs.stream and runs.joinStream.

Raw fetch CSRF gap (7 files)
- Audit: 11 frontend fetch sites, only 2 included CSRF (login/setup +
  account-settings change-password). The other 7 routed through raw
  fetch() with no header — suggestions, memory, agents, mcp, skills,
  uploads, and the local thread cleanup hook all 403'd silently.
- Fix: enhance fetcher.ts:fetchWithAuth to auto-inject X-CSRF-Token on
  POST/PUT/DELETE/PATCH from a single shared readCsrfCookie() helper.
  Convert all 7 raw fetch() callers to fetchWithAuth so the contract
  is centrally enforced. api-client.ts and fetcher.ts share
  readCsrfCookie + STATE_CHANGING_METHODS to avoid drift.

nginx routing + buffering (nginx.local.conf)
- The auth feature shipped without updating the nginx config: per-API
  explicit location blocks but no /api/v1/auth/, /api/feedback, /api/runs.
  The frontend's client-side fetches to /api/v1/auth/login/local 404'd
  from the Next.js side because nginx routed /api/* to the frontend.
- Fix: add catch-all `location /api/` that proxies to the gateway.
  nginx longest-prefix matching keeps the explicit blocks (/api/models,
  /api/threads regex, /api/langgraph/, ...) winning for their paths.
- Fix: disable proxy_buffering + proxy_request_buffering for the
  frontend `location /` block. Without it, nginx tries to spool large
  Next.js chunks into /var/lib/nginx/proxy (root-owned) and fails with
  Permission denied → ERR_INCOMPLETE_CHUNKED_ENCODING → ChunkLoadError.

* test(auth): release-validation test infra and new coverage

Test fixtures and unit tests added during the validation pass.

Router test helpers (NEW: tests/_router_auth_helpers.py)
- make_authed_test_app(): builds a FastAPI test app with a stub
  middleware that stamps request.state.user + request.state.auth and a
  permissive thread_meta_repo mock. TestClient-based router tests
  (test_artifacts_router, test_threads_router) use it instead of bare
  FastAPI() so the new @require_permission(owner_check=True) decorators
  short-circuit cleanly.
- call_unwrapped(): walks the __wrapped__ chain to invoke the underlying
  handler without going through the authz wrappers. Direct-call tests
  (test_uploads_router) use it. Typed with ParamSpec so the wrapped
  signature flows through.

Backend test additions
- test_auth.py: 7 tests for the new _get_client_ip trust model (no
  proxy / trusted proxy / untrusted peer / XFF rejection / invalid
  CIDR / no client). 5 tests for the password blocklist (literal,
  case-insensitive, strong password accepted, change-password binding,
  short-password length-check still fires before blocklist).
  test_update_user_raises_when_row_concurrently_deleted: closes a
  shipped-without-coverage gap on the new UserNotFoundError contract.
- test_thread_meta_repo.py: 4 tests for check_access(require_existing=True)
  — strict missing-row denial, strict owner match, strict owner mismatch,
  strict null-owner still allowed (shared rows survive the tightening).
- test_ensure_admin.py: 3 tests for _migrate_orphaned_threads /
  _iter_store_items pagination, covering the TC-UPG-02 upgrade story
  end-to-end via mock store. Closes the gap where the cursor pagination
  was untested even though the previous PR rewrote it.
- test_threads_router.py: 5 tests for _strip_reserved_metadata
  (owner_id removal, user_id removal, safe-keys passthrough, empty
  input, both-stripped).
- test_auth_type_system.py: replace "password123" fixtures with
  Tr0ub4dor3a / AnotherStr0ngPwd! so the new password blocklist
  doesn't reject the test data.

* docs(auth): refresh TC-DOCKER-05 + document Docker validation gap

- AUTH_TEST_PLAN.md TC-DOCKER-05: the previous expectation
  ("admin password visible in docker logs") was stale after the simplify
  pass that moved credentials to a 0600 file. The grep "Password:" check
  would have silently failed and given a false sense of coverage. New
  expectation matches the actual file-based path: 0600 file in
  DEER_FLOW_HOME, log shows the path (not the secret), reverse-grep
  asserts no leaked password in container logs.
- NEW: docs/AUTH_TEST_DOCKER_GAP.md documents the only un-executed
  block in the test plan (TC-DOCKER-01..06). Reason: sg_dev validation
  host has no Docker daemon installed. The doc maps each Docker case
  to an already-validated bare-metal equivalent (TC-1.1, TC-REENT-01,
  TC-API-02 etc.) so the gap is auditable, and includes pre-flight
  reproduction steps for whoever has Docker available.

---------

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>

* refactor(persistence): unify SQLite to single deerflow.db and move checkpointer to runtime

Merge checkpoints.db and app.db into a single deerflow.db file (WAL mode
handles concurrent access safely). Move checkpointer module from
agents/checkpointer to runtime/checkpointer to better reflect its role
as a runtime infrastructure concern.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(persistence): rename owner_id to user_id and thread_meta_repo to thread_store

Rename owner_id to user_id across all persistence models, repositories,
stores, routers, and tests for clearer semantics. Rename thread_meta_repo
to thread_store for consistency with run_store/run_event_store naming.
Add ThreadMetaStore return type annotation to get_thread_store().

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): unify ThreadMetaStore interface with user isolation and factory

Add user_id parameter to all ThreadMetaStore abstract methods. Implement
owner isolation in MemoryThreadMetaStore with _get_owned_record helper.
Add check_access to base class and memory implementation. Add
make_thread_store factory to simplify deps.py initialization. Add
memory-backend isolation tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(feedback): add UNIQUE(thread_id, run_id, user_id) constraint

Add UNIQUE constraint to FeedbackRow to enforce one feedback per user per run,
enabling upsert behavior in Task 2. Update tests to use distinct user_ids for
multiple feedback records per run, and pass user_id=None to list_by_run for
admin-style queries that bypass user isolation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(feedback): add upsert() method with UNIQUE enforcement

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(feedback): add delete_by_run() and list_by_thread_grouped()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(feedback): add PUT upsert and DELETE-by-run endpoints

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(feedback): enrich messages endpoint with per-run feedback data

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(feedback): add frontend feedback API client

Adds upsertFeedback and deleteFeedback API functions backed by
fetchWithAuth, targeting the /api/threads/{id}/runs/{id}/feedback
endpoint.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(feedback): wire feedback data into message rendering for history echo

Adds useThreadFeedback hook that fetches run-level feedback from the
messages API and builds a runId->FeedbackData map. MessageList now calls
this hook and passes feedback and runId to each MessageListItem so
previously-submitted thumbs are pre-filled when revisiting a thread.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(feedback): correct run_id mapping for feedback echo

The feedbackMap was keyed by run_id but looked up by LangGraph message ID.
Fixed by tracking AI message ordinal index to correlate event store
run_ids with LangGraph SDK messages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(feedback): use real threadId and refresh after stream

- Pass threadId prop to MessageListItem instead of reading "new" from URL params
- Invalidate thread-feedback query on stream finish so buttons appear immediately
- Show feedback buttons always visible, copy button on hover only

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style(feedback): group copy and feedback buttons together on the left

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style(feedback): always show toolbar buttons without hover

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): stream hang when run_events.backend=db

DbRunEventStore._user_id_from_context() returned user.id without
coercing it to str. User.id is a Pydantic UUID, and aiosqlite cannot
bind a raw UUID object to a VARCHAR column, so the INSERT for the
initial human_message event silently rolled back and raised out of
the worker task. Because that put() sat outside the worker's try
block, the finally-clause that publishes end-of-stream never ran
and the SSE stream hung forever.

jsonl mode was unaffected because json.dumps(default=str) coerces
UUID objects transparently.

Fixes:
- db.py: coerce user.id to str at the context-read boundary (matches
  what resolve_user_id already does for the other repositories)
- worker.py: move RunJournal init + human_message put inside the try
  block so any failure flows through the finally/publish_end path
  instead of hanging the subscriber

Defense-in-depth:
- engine.py: add PRAGMA busy_timeout=5000 so checkpointer and event
  store wait for each other on the shared deerflow.db file instead
  of failing immediately under write-lock contention
- journal.py: skip fire-and-forget _flush_sync when a previous flush
  task is still in flight, to avoid piling up concurrent put_batch
  writes on the same SQLAlchemy engine during streaming; flush() now
  waits for pending tasks before draining the buffer
- database_config.py: doc-only update clarifying WAL + busy_timeout
  keep the unified deerflow.db safe for both workloads

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore(persistence): drop redundant busy_timeout PRAGMA

Python's sqlite3 driver defaults to a 5-second busy timeout via the
``timeout`` kwarg of ``sqlite3.connect``, and aiosqlite + SQLAlchemy's
aiosqlite dialect inherit that default. Setting ``PRAGMA busy_timeout=5000``
explicitly was a no-op — verified by reading back the PRAGMA on a fresh
connection (it already reports 5000ms without our PRAGMA).

Concurrent stress test (50 checkpoint writes + 20 event batches + 50
thread_meta updates on the same deerflow.db) still completes with zero
errors and 200/200 rows after removing the explicit PRAGMA.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(journal): unwrap Command tool results in on_tool_end

Tools that update graph state (e.g. ``present_files``) return
``Command(update={'messages': [ToolMessage(...)], 'artifacts': [...]})``.
LangGraph later unwraps the inner ``ToolMessage`` into checkpoint state,
but ``RunJournal.on_tool_end`` was receiving the ``Command`` object
directly via the LangChain callback chain and storing
``str(Command(update={...}))`` as the tool_result content.

This produced a visible divergence between the event-store and the
checkpoint for any thread that used a Command-returning tool, blocking
the event-store-backed history fix in the follow-up commit. Concrete
example from thread ``6d30913e-dcd4-41c8-8941-f66c716cf359`` (seq=48):
checkpoint had ``'Successfully presented files'`` while event_store
stored the full Command repr.

The fix detects ``Command`` in ``on_tool_end``, extracts the first
``ToolMessage`` from ``update['messages']``, and lets the existing
ToolMessage branch handle the ``model_dump()`` path. Legacy rows still
containing the Command repr are separately cleaned up by the history
helper in the follow-up commit.

Tests:
- ``test_tool_end_unwraps_command_with_inner_tool_message`` — unit test
  of the unwrap branch with a constructed Command
- ``test_tool_invoke_end_to_end_unwraps_command`` — end-to-end via
  ``CallbackManager`` + ``tool.invoke`` to exercise the real LangChain
  dispatch path that production uses, matching the repro shape from
  ``present_files``
- Counter-proof: temporarily reverted the patch, both tests failed with
  the exact ``Command(update={...})`` repr that was stored in the
  production SQLite row at seq=48, confirming LangChain does pass the
  ``Command`` through callbacks (the unwrap is load-bearing)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(threads): load history messages from event store, immune to summarize

``get_thread_history`` and ``get_thread_state`` in Gateway mode read
messages from ``checkpoint.channel_values["messages"]``. After
SummarizationMiddleware runs mid-run, that list is rewritten in-place:
pre-summarize messages are dropped and a synthetic summary-as-human
message takes position 0. The frontend then renders a chat history that
starts with ``"Here is a summary of the conversation to date:..."``
instead of the user's original query, and all earlier turns are gone.

The event store (``RunEventStore``) is append-only and never rewritten,
so it retains the full transcript. This commit adds a helper
``_get_event_store_messages`` that loads the event store's message
stream and overrides ``values["messages"]`` in both endpoints; the
checkpoint fallback kicks in only when the event store is unavailable.

Behavior contract of the helper:

- **Full pagination.** ``list_messages`` returns the newest ``limit``
  records when no cursor is given, so a fixed limit silently drops
  older messages on long threads. The helper sizes the read from
  ``count_messages()`` and pages forward with ``after_seq`` cursors.
- **Copy-on-read.** Each content dict is copied before ``id`` is
  patched so the live store object (``MemoryRunEventStore`` returns
  references) is never mutated.
- **Stable ids.** Messages with ``id=None`` (human + tool_result,
  which don't receive an id until checkpoint persistence) get a
  deterministic ``uuid5(NAMESPACE_URL, f"{thread_id}:{seq}")`` so
  React keys stay stable across requests. AI messages keep their
  LLM-assigned ``lc_run--*`` ids.
- **Legacy ``Command`` repr sanitization.** Rows captured before the
  ``journal.py`` ``on_tool_end`` fix (previous commit) stored
  ``str(Command(update={'messages': [ToolMessage(content='X', ...)]}))``
  as the tool_result content. ``_sanitize_legacy_command_repr``
  regex-extracts the inner text so old threads render cleanly.
- **Inline feedback.** When loading the stream, the helper also pulls
  ``feedback_repo.list_by_thread_grouped`` and attaches ``run_id`` to
  every message plus ``feedback`` to the final ``ai_message`` of each
  run. This removes the frontend's need to fetch a second endpoint
  and positional-index-map its way back to the right run. When the
  feedback subsystem is unavailable, the ``feedback`` field is left
  absent entirely so the frontend hides the button rather than
  rendering it over a broken write path.
- **User context.** ``DbRunEventStore`` is user-scoped by default via
  ``resolve_user_id(AUTO)``. The helper relies on the ``@require_permission``
  decorator having populated the user contextvar on both callers; the
  docstring documents this dependency explicitly so nobody wires it
  into a CLI or migration script without passing ``user_id=None``.

Real data verification against thread
``6d30913e-dcd4-41c8-8941-f66c716cf359``: checkpoint showed 12 messages
(summarize-corrupted), event store had 16. The original human message
``"最新伊美局势"`` was preserved as seq=1 in the event store and
correctly restored to position 0 in the helper output. Helper output
for AI messages was byte-identical to checkpoint for every overlapping
message; only tool_result ids differed (patched to uuid5) and the
legacy Command repr at seq=48 was sanitized.

Tests:
- ``test_thread_state_event_store.py`` — 18 tests covering
  ``_sanitize_legacy_command_repr`` (passthrough, single/double-quote
  extraction, unparseable fallback), helper happy path (all message
  types, stable uuid5, store non-mutation), multi-page pagination,
  summarize regression (recovers pre-summarize messages), feedback
  attachment (per-run, multi-run threads, repo failure graceful),
  and dependency failure fallback to ``None``.

Docs:
- ``docs/superpowers/plans/2026-04-10-event-store-history.md`` — the
  implementation plan this commit realizes, with Task 1 revised after
  the evaluation findings (pagination, copy-on-read, Command wrap
  already landed in journal.py, frontend feedback pagination in the
  follow-up commit, Standard-mode follow-up noted).
- ``docs/superpowers/specs/2026-04-11-runjournal-history-evaluation.md``
  — the Claude + second-opinion evaluation document that drove the
  plan revisions (pagination bug, dict-mutation bug, feedback hidden
  bug, Command bug).
- ``docs/superpowers/specs/2026-04-11-summarize-marker-design.md`` —
  design for a follow-up PR that visually marks summarize events in
  history, based on a verified ``adispatch_custom_event`` experiment
  (``trace=False`` middleware nodes can still forward the Pregel task
  config via explicit signature injection).

Scope: Gateway mode only (``make dev-pro``). Standard mode
(``make dev``) hits LangGraph Server directly and bypasses these
endpoints; the summarize symptom is still present there and is
tracked as a separate follow-up in the plan.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(feedback): inline feedback on history and drop positional mapping

The old ``useThreadFeedback`` hook loaded ``GET /api/threads/{id}/messages?limit=200``
and built two parallel lookup tables: ``runIdByAiIndex`` (an ordinal array of
run_ids for every ``ai_message``-typed event) and ``feedbackByRunId``. The render
loop in ``message-list.tsx`` walked the AI messages in order, incrementing
``aiMessageIndex`` on each non-human message, and used that ordinal to look up
the run_id and feedback.

This shape had three latent bugs we could observe on real threads:

1. **Fetch was capped at 200 messages.** Long or tool-heavy threads silently
   dropped earlier entries from the map, so feedback buttons could be missing
   on messages they should own.
2. **Ordinal mismatch.** The render loop counted every non-human message
   (including each intermediate ``ai_tool_call``), but ``runIdByAiIndex`` only
   pushed entries for ``event_type == "ai_message"``. A run with 3 tool_calls
   + 1 final AI message would push 1 entry while the render consumed 4
   positions, so buttons mapped to the wrong positions across multi-run
   threads.
3. **Two parallel data paths.** The ``/history`` render path and the
   ``/messages`` feedback-lookup path could drift in-between an
   ``invalidateQueries`` call and the next refetch, producing transient
   mismaps.

The previous commit moved the authoritative message source for history to
the event store and added ``run_id`` + ``feedback`` inline on each message
dict returned by ``_get_event_store_messages``. This commit aligns the
frontend with that contract:

- **Delete** ``useThreadFeedback``, ``ThreadFeedbackData``,
  ``runIdByAiIndex``, ``feedbackByRunId``, and ``fetchAllThreadMessages``.
- **Introduce** ``useThreadMessageEnrichment`` that fetches
  ``POST /history?limit=1`` once, indexes the returned messages by
  ``message.id`` into a ``Map<id, {run_id, feedback?}>``, and invalidates
  on stream completion (``onFinish`` in ``useThreadStream``). Keying by
  ``message.id`` is stable across runs, tool_call chains, and summarize.
- **Simplify** ``message-list.tsx`` to drop the ``aiMessageIndex``
  counter and read ``enrichment?.get(msg.id)`` at each render step.
- **Rewire** ``message-list-item.tsx`` so the feedback button renders
  when ``feedback !== undefined`` rather than when the message happens
  to be non-human. ``feedback`` is ``undefined`` for non-eligible
  messages (humans, non-final AI, tools), ``null`` for the final
  ai_message of an unrated run, and a ``FeedbackData`` object once
  rated — cleanly distinguishing "not eligible" from "eligible but
  unrated".

``/api/threads/{id}/messages`` is kept as a debug/export surface; no
frontend code calls it anymore but the backend router is untouched.

Validation:
- ``pnpm check`` clean (0 errors, 1 pre-existing unrelated warning)
- Live test on thread ``3d5dea4a`` after gateway restart confirmed the
  original user query is restored to position 0 and the feedback
  button behaves correctly on the final AI message.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(rebase): remove duplicate definitions and update stale module paths

Rebase left duplicate function blocks in worker.py (triple human_message
write causing 3x user messages in /history), deps.py, and prompt.py.
Also update checkpointer imports from the old deerflow.agents.checkpointer
path to deerflow.runtime.checkpointer, and clean up orphaned feedback
props in the frontend message components.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(rebase): restore FeedbackButtons component and enrichment lost during rebase

The FeedbackButtons component (defined inline in message-list-item.tsx)
was introduced in commit 95df8d13 but lost during rebase. The previous
rebase cleanup commit incorrectly removed the feedback/runId props and
enrichment hook as "orphaned code" instead of restoring the missing
component. This commit restores:

- FeedbackButtons component with thumbs up/down toggle and optimistic state
- FeedbackData/upsertFeedback/deleteFeedback imports
- feedback and runId props on MessageListItem
- useThreadMessageEnrichment hook and entry lookup in message-list.tsx

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: JilongSun <965640067@qq.com>
Co-authored-by: jie <49781832+stan-fu@users.noreply.github.com>
Co-authored-by: cooper <cooperfu@tencent.com>
Co-authored-by: yangzheli <43645580+yangzheli@users.noreply.github.com>
Co-authored-by: greatmengqi <chenmengqi.0376@gmail.com>
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-04-26 11:09:55 +08:00
copilot-swe-agent[bot] 88f822a8b3 docs: fill all TBD documentation pages and add new harness module pages
Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/ff389ed8-31c9-430c-85ff-cc1b52b8239c

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>
2026-04-26 11:08:49 +08:00
copilot-swe-agent[bot] 814a488bcb docs: complete all English and Chinese documentation pages
Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/a5f192e7-8034-4e46-af22-60b90ee27d40

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>
2026-04-26 11:08:49 +08:00
copilot-swe-agent[bot] 716cae20c6 docs: fix review feedback - source-map paths, memory API routes, supports_thinking, checkpointer callout
Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/fb75dc8c-18a4-4a23-9229-25b3c5e545cf

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>
2026-04-26 11:08:49 +08:00
jiangfeng.11 7ff9077074 feat(dependencies): add langchain-ollama and ollama packages with optional dependencies 2026-04-26 11:08:49 +08:00
Copilot 848ace98cb feat: replace auto-admin creation with secure interactive first-boot setup (#2063)
* feat(persistence): add unified persistence layer with event store, token tracking, and feedback (#1930)

* feat(persistence): add SQLAlchemy 2.0 async ORM scaffold

Introduce a unified database configuration (DatabaseConfig) that
controls both the LangGraph checkpointer and the DeerFlow application
persistence layer from a single `database:` config section.

New modules:
- deerflow.config.database_config — Pydantic config with memory/sqlite/postgres backends
- deerflow.persistence — async engine lifecycle, DeclarativeBase with to_dict mixin, Alembic skeleton
- deerflow.runtime.runs.store — RunStore ABC + MemoryRunStore implementation

Gateway integration initializes/tears down the persistence engine in
the existing langgraph_runtime() context manager. Legacy checkpointer
config is preserved for backward compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add RunEventStore ABC + MemoryRunEventStore

Phase 2-A prerequisite for event storage: adds the unified run event
stream interface (RunEventStore) with an in-memory implementation,
RunEventsConfig, gateway integration, and comprehensive tests (27 cases).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add ORM models, repositories, DB/JSONL event stores, RunJournal, and API endpoints

Phase 2-B: run persistence + event storage + token tracking.

- ORM models: RunRow (with token fields), ThreadMetaRow, RunEventRow
- RunRepository implements RunStore ABC via SQLAlchemy ORM
- ThreadMetaRepository with owner access control
- DbRunEventStore with trace content truncation and cursor pagination
- JsonlRunEventStore with per-run files and seq recovery from disk
- RunJournal (BaseCallbackHandler) captures LLM/tool/lifecycle events,
  accumulates token usage by caller type, buffers and flushes to store
- RunManager now accepts optional RunStore for persistent backing
- Worker creates RunJournal, writes human_message, injects callbacks
- Gateway deps use factory functions (RunRepository when DB available)
- New endpoints: messages, run messages, run events, token-usage
- ThreadCreateRequest gains assistant_id field
- 92 tests pass (33 new), zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add user feedback + follow-up run association

Phase 2-C: feedback and follow-up tracking.

- FeedbackRow ORM model (rating +1/-1, optional message_id, comment)
- FeedbackRepository with CRUD, list_by_run/thread, aggregate stats
- Feedback API endpoints: create, list, stats, delete
- follow_up_to_run_id in RunCreateRequest (explicit or auto-detected
  from latest successful run on the thread)
- Worker writes follow_up_to_run_id into human_message event metadata
- Gateway deps: feedback_repo factory + getter
- 17 new tests (14 FeedbackRepository + 3 follow-up association)
- 109 total tests pass, zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test+config: comprehensive Phase 2 test coverage + deprecate checkpointer config

- config.example.yaml: deprecate standalone checkpointer section, activate
  unified database:sqlite as default (drives both checkpointer + app data)
- New: test_thread_meta_repo.py (14 tests) — full ThreadMetaRepository coverage
  including check_access owner logic, list_by_owner pagination
- Extended test_run_repository.py (+4 tests) — completion preserves fields,
  list ordering desc, limit, owner_none returns all
- Extended test_run_journal.py (+8 tests) — on_chain_error, track_tokens=false,
  middleware no ai_message, unknown caller tokens, convenience fields,
  tool_error, non-summarization custom event
- Extended test_run_event_store.py (+7 tests) — DB batch seq continuity,
  make_run_event_store factory (memory/db/jsonl/fallback/unknown)
- Extended test_phase2b_integration.py (+4 tests) — create_or_reject persists,
  follow-up metadata, summarization in history, full DB-backed lifecycle
- Fixed DB integration test to use proper fake objects (not MagicMock)
  for JSON-serializable metadata
- 157 total Phase 2 tests pass, zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* config: move default sqlite_dir to .deer-flow/data

Keep SQLite databases alongside other DeerFlow-managed data
(threads, memory) under the .deer-flow/ directory instead of a
top-level ./data folder.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(persistence): remove UTFJSON, use engine-level json_serializer + datetime.now()

- Replace custom UTFJSON type with standard sqlalchemy.JSON in all ORM
  models. Add json_serializer=json.dumps(ensure_ascii=False) to all
  create_async_engine calls so non-ASCII text (Chinese etc.) is stored
  as-is in both SQLite and Postgres.
- Change ORM datetime defaults from datetime.now(UTC) to datetime.now(),
  remove UTC imports.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): simplify deps.py with getter factory + inline repos

- Replace 6 identical getter functions with _require() factory.
- Inline 3 _make_*_repo() factories into langgraph_runtime(), call
  get_session_factory() once instead of 3 times.
- Add thread_meta upsert in start_run (services.py).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(docker): add UV_EXTRAS build arg for optional dependencies

Support installing optional dependency groups (e.g. postgres) at
Docker build time via UV_EXTRAS build arg:
  UV_EXTRAS=postgres docker compose build

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(journal): fix flush, token tracking, and consolidate tests

RunJournal fixes:
- _flush_sync: retain events in buffer when no event loop instead of
  dropping them; worker's finally block flushes via async flush().
- on_llm_end: add tool_calls filter and caller=="lead_agent" guard for
  ai_message events; mark message IDs for dedup with record_llm_usage.
- worker.py: persist completion data (tokens, message count) to RunStore
  in finally block.

Model factory:
- Auto-inject stream_usage=True for BaseChatOpenAI subclasses with
  custom api_base, so usage_metadata is populated in streaming responses.

Test consolidation:
- Delete test_phase2b_integration.py (redundant with existing tests).
- Move DB-backed lifecycle test into test_run_journal.py.
- Add tests for stream_usage injection in test_model_factory.py.
- Clean up executor/task_tool dead journal references.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): widen content type to str|dict in all store backends

Allow event content to be a dict (for structured OpenAI-format messages)
in addition to plain strings. Dict values are JSON-serialized for the DB
backend and deserialized on read; memory and JSONL backends handle dicts
natively. Trace truncation now serializes dicts to JSON before measuring.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(events): use metadata flag instead of heuristic for dict content detection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(converters): add LangChain-to-OpenAI message format converters

Pure functions langchain_to_openai_message, langchain_to_openai_completion,
langchain_messages_to_openai, and _infer_finish_reason for converting
LangChain BaseMessage objects to OpenAI Chat Completions format, used by
RunJournal for event storage. 15 unit tests added.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(converters): handle empty list content as null, clean up test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): human_message content uses OpenAI user message format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): ai_message uses OpenAI format, add ai_tool_call message event

- ai_message content now uses {"role": "assistant", "content": "..."} format
- New ai_tool_call message event emitted when lead_agent LLM responds with tool_calls
- ai_tool_call uses langchain_to_openai_message converter for consistent format
- Both events include finish_reason in metadata ("stop" or "tool_calls")

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): add tool_result message event with OpenAI tool message format

Cache tool_call_id from on_tool_start keyed by run_id as fallback for on_tool_end,
then emit a tool_result message event (role=tool, tool_call_id, content) after each
successful tool completion.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): summary content uses OpenAI system message format

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): replace llm_start/llm_end with llm_request/llm_response in OpenAI format

Add on_chat_model_start to capture structured prompt messages as llm_request events.
Replace llm_end trace events with llm_response using OpenAI Chat Completions format.
Track llm_call_index to pair request/response events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): add record_middleware method for middleware trace events

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(events): add full run sequence integration test for OpenAI content format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): align message events with checkpoint format and add middleware tag injection

- Message events (ai_message, ai_tool_call, tool_result, human_message) now use
  BaseMessage.model_dump() format, matching LangGraph checkpoint values.messages
- on_tool_end extracts tool_call_id/name/status from ToolMessage objects
- on_tool_error now emits tool_result message events with error status
- record_middleware uses middleware:{tag} event_type and middleware category
- Summarization custom events use middleware:summarize category
- TitleMiddleware injects middleware:title tag via get_config() inheritance
- SummarizationMiddleware model bound with middleware:summarize tag
- Worker writes human_message using HumanMessage.model_dump()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(threads): switch search endpoint to threads_meta table and sync title

- POST /api/threads/search now queries threads_meta table directly,
  removing the two-phase Store + Checkpointer scan approach
- Add ThreadMetaRepository.search() with metadata/status filters
- Add ThreadMetaRepository.update_display_name() for title sync
- Worker syncs checkpoint title to threads_meta.display_name on run completion
- Map display_name to values.title in search response for API compatibility

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(threads): history endpoint reads messages from event store

- POST /api/threads/{thread_id}/history now combines two data sources:
  checkpointer for checkpoint_id, metadata, title, thread_data;
  event store for messages (complete history, not truncated by summarization)
- Strip internal LangGraph metadata keys from response
- Remove full channel_values serialization in favor of selective fields

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove duplicate optional-dependencies header in pyproject.toml

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(middleware): pass tagged config to TitleMiddleware ainvoke call

Without the config, the middleware:title tag was not injected,
causing the LLM response to be recorded as a lead_agent ai_message
in run_events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: resolve merge conflict in .env.example

Keep both DATABASE_URL (from persistence-scaffold) and WECOM
credentials (from main) after the merge.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address review feedback on PR #1851

- Fix naive datetime.now() → datetime.now(UTC) in all ORM models
- Fix seq race condition in DbRunEventStore.put() with FOR UPDATE
  and UNIQUE(thread_id, seq) constraint
- Encapsulate _store access in RunManager.update_run_completion()
- Deduplicate _store.put() logic in RunManager via _persist_to_store()
- Add update_run_completion to RunStore ABC + MemoryRunStore
- Wire follow_up_to_run_id through the full create path
- Add error recovery to RunJournal._flush_sync() lost-event scenario
- Add migration note for search_threads breaking change
- Fix test_checkpointer_none_fix mock to set database=None

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update uv.lock

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address 22 review comments from CodeQL, Copilot, and Code Quality

Bug fixes:
- Sanitize log params to prevent log injection (CodeQL)
- Reset threads_meta.status to idle/error when run completes
- Attach messages only to latest checkpoint in /history response
- Write threads_meta on POST /threads so new threads appear in search

Lint fixes:
- Remove unused imports (journal.py, migrations/env.py, test_converters.py)
- Convert lambda to named function (engine.py, Ruff E731)
- Remove unused logger definitions in repos (Ruff F841)
- Add logging to JSONL decode errors and empty except blocks
- Separate assert side-effects in tests (CodeQL)
- Remove unused local variables in tests (Ruff F841)
- Fix max_trace_content truncation to use byte length, not char length

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: apply ruff format to persistence and runtime files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Potential fix for pull request finding 'Statement has no effect'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

* refactor(runtime): introduce RunContext to reduce run_agent parameter bloat

Extract checkpointer, store, event_store, run_events_config, thread_meta_repo,
and follow_up_to_run_id into a frozen RunContext dataclass. Add get_run_context()
in deps.py to build the base context from app.state singletons. start_run() uses
dataclasses.replace() to enrich per-run fields before passing ctx to run_agent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): move sanitize_log_param to app/gateway/utils.py

Extract the log-injection sanitizer from routers/threads.py into a shared
utils module and rename to sanitize_log_param (public API). Eliminates the
reverse service → router import in services.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf: use SQL aggregation for feedback stats and thread token usage

Replace Python-side counting in FeedbackRepository.aggregate_by_run with
a single SELECT COUNT/SUM query. Add RunStore.aggregate_tokens_by_thread
abstract method with SQL GROUP BY implementation in RunRepository and
Python fallback in MemoryRunStore. Simplify the thread_token_usage
endpoint to delegate to the new method, eliminating the limit=10000
truncation risk.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: annotate DbRunEventStore.put() as low-frequency path

Add docstring clarifying that put() opens a per-call transaction with
FOR UPDATE and should only be used for infrequent writes (currently
just the initial human_message event). High-throughput callers should
use put_batch() instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(threads): fall back to Store search when ThreadMetaRepository is unavailable

When database.backend=memory (default) or no SQL session factory is
configured, search_threads now queries the LangGraph Store instead of
returning 503. Returns empty list if neither Store nor repo is available.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(persistence): introduce ThreadMetaStore ABC for backend-agnostic thread metadata

Add ThreadMetaStore abstract base class with create/get/search/update/delete
interface. ThreadMetaRepository (SQL) now inherits from it. New
MemoryThreadMetaStore wraps LangGraph BaseStore for memory-mode deployments.

deps.py now always provides a non-None thread_meta_repo, eliminating all
`if thread_meta_repo is not None` guards in services.py, worker.py, and
routers/threads.py. search_threads no longer needs a Store fallback branch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(history): read messages from checkpointer instead of RunEventStore

The /history endpoint now reads messages directly from the
checkpointer's channel_values (the authoritative source) instead of
querying RunEventStore.list_messages(). The RunEventStore API is
preserved for other consumers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address new Copilot review comments

- feedback.py: validate thread_id/run_id before deleting feedback
- jsonl.py: add path traversal protection with ID validation
- run_repo.py: parse `before` to datetime for PostgreSQL compat
- thread_meta_repo.py: fix pagination when metadata filter is active
- database_config.py: use resolve_path for sqlite_dir consistency

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Implement skill self-evolution and skill_manage flow (#1874)

* chore: ignore .worktrees directory

* Add skill_manage self-evolution flow

* Fix CI regressions for skill_manage

* Address PR review feedback for skill evolution

* fix(skill-evolution): preserve history on delete

* fix(skill-evolution): tighten scanner fallbacks

* docs: add skill_manage e2e evidence screenshot

* fix(skill-manage): avoid blocking fs ops in session runtime

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

* fix(config): resolve sqlite_dir relative to CWD, not Paths.base_dir

resolve_path() resolves relative to Paths.base_dir (.deer-flow),
which double-nested the path to .deer-flow/.deer-flow/data/app.db.
Use Path.resolve() (CWD-relative) instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Feature/feishu receive file (#1608)

* feat(feishu): add channel file materialization hook for inbound messages

- Introduce Channel.receive_file(msg, thread_id) as a base method for file materialization; default is no-op.
- Implement FeishuChannel.receive_file to download files/images from Feishu messages, save to sandbox, and inject virtual paths into msg.text.
- Update ChannelManager to call receive_file for any channel if msg.files is present, enabling downstream model access to user-uploaded files.
- No impact on Slack/Telegram or other channels (they inherit the default no-op).

* style(backend): format code with ruff for lint compliance

- Auto-formatted packages/harness/deerflow/agents/factory.py and tests/test_create_deerflow_agent.py using `ruff format`
- Ensured both files conform to project linting standards
- Fixes CI lint check failures caused by code style issues

* fix(feishu): handle file write operation asynchronously to prevent blocking

* fix(feishu): rename GetMessageResourceRequest to _GetMessageResourceRequest and remove redundant code

* test(feishu): add tests for receive_file method and placeholder replacement

* fix(manager): remove unnecessary type casting for channel retrieval

* fix(feishu): update logging messages to reflect resource handling instead of image

* fix(feishu): sanitize filename by replacing invalid characters in file uploads

* fix(feishu): improve filename sanitization and reorder image key handling in message processing

* fix(feishu): add thread lock to prevent filename conflicts during file downloads

* fix(test): correct bad merge in test_feishu_parser.py

* chore: run ruff and apply formatting cleanup
fix(feishu): preserve rich-text attachment order and improve fallback filename handling

* fix(docker): restore gateway env vars and fix langgraph empty arg issue (#1915)

Two production docker-compose.yaml bugs prevent `make up` from working:

1. Gateway missing DEER_FLOW_CONFIG_PATH and DEER_FLOW_EXTENSIONS_CONFIG_PATH
   environment overrides. Added in fb2d99f (#1836) but accidentally reverted
   by ca2fb95 (#1847). Without them, gateway reads host paths from .env via
   env_file, causing FileNotFoundError inside the container.

2. Langgraph command fails when LANGGRAPH_ALLOW_BLOCKING is unset (default).
   Empty $${allow_blocking} inserts a bare space between flags, causing
   ' --no-reload' to be parsed as unexpected extra argument. Fix by building
   args string first and conditionally appending --allow-blocking.

Co-authored-by: cooper <cooperfu@tencent.com>

* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities (#1904)

* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities

Fix `<button>` inside `<a>` invalid HTML in artifact components and add
missing `noopener,noreferrer` to `window.open` calls to prevent reverse
tabnabbing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(frontend): address Copilot review on tabnabbing and double-tab-open

Remove redundant parent onClick on web_fetch ChainOfThoughtStep to
prevent opening two tabs on link click, and explicitly null out
window.opener after window.open() for defensive tabnabbing hardening.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(persistence): organize entities into per-entity directories

Restructure the persistence layer from horizontal "models/ + repositories/"
split into vertical entity-aligned directories. Each entity (thread_meta,
run, feedback) now owns its ORM model, abstract interface (where applicable),
and concrete implementations under a single directory with an aggregating
__init__.py for one-line imports.

Layout:
  persistence/thread_meta/{base,model,sql,memory}.py
  persistence/run/{model,sql}.py
  persistence/feedback/{model,sql}.py

models/__init__.py is kept as a facade so Alembic autogenerate continues to
discover all ORM tables via Base.metadata. RunEventRow remains under
models/run_event.py because its storage implementation lives in
runtime/events/store/db.py and has no matching repository directory.

The repositories/ directory is removed entirely. All call sites in
gateway/deps.py and tests are updated to import from the new entity
packages, e.g.:

    from deerflow.persistence.thread_meta import ThreadMetaRepository
    from deerflow.persistence.run import RunRepository
    from deerflow.persistence.feedback import FeedbackRepository

Full test suite passes (1690 passed, 14 skipped).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(gateway): sync thread rename and delete through ThreadMetaStore

The POST /threads/{id}/state endpoint previously synced title changes
only to the LangGraph Store via _store_upsert. In sqlite mode the search
endpoint reads from the ThreadMetaRepository SQL table, so renames never
appeared in /threads/search until the next agent run completed (worker.py
syncs title from checkpoint to thread_meta in its finally block).

Likewise the DELETE /threads/{id} endpoint cleaned up the filesystem,
Store, and checkpointer but left the threads_meta row orphaned in sqlite,
so deleted threads kept appearing in /threads/search.

Fix both endpoints by routing through the ThreadMetaStore abstraction
which already has the correct sqlite/memory implementations wired up by
deps.py. The rename path now calls update_display_name() and the delete
path calls delete() — both work uniformly across backends.

Verified end-to-end with curl in gateway mode against sqlite backend.
Existing test suite (1690 passed) and focused router/repo tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): route all thread metadata access through ThreadMetaStore

Following the rename/delete bug fix in PR1, migrate the remaining direct
LangGraph Store reads/writes in the threads router and services to the
ThreadMetaStore abstraction so that the sqlite and memory backends behave
identically and the legacy dual-write paths can be removed.

Migrated endpoints (threads.py):
- create_thread: idempotency check + write now use thread_meta_repo.get/create
  instead of dual-writing the LangGraph Store and the SQL row.
- get_thread: reads from thread_meta_repo.get; the checkpoint-only fallback
  for legacy threads is preserved.
- patch_thread: replaced _store_get/_store_put with thread_meta_repo.update_metadata.
- delete_thread_data: dropped the legacy store.adelete; thread_meta_repo.delete
  already covers it.

Removed dead code (services.py):
- _upsert_thread_in_store — redundant with the immediately following
  thread_meta_repo.create() call.
- _sync_thread_title_after_run — worker.py's finally block already syncs
  the title via thread_meta_repo.update_display_name() after each run.

Removed dead code (threads.py):
- _store_get / _store_put / _store_upsert helpers (no remaining callers).
- THREADS_NS constant.
- get_store import (router no longer touches the LangGraph Store directly).

New abstract method:
- ThreadMetaStore.update_metadata(thread_id, metadata) merges metadata into
  the thread's metadata field. Implemented in both ThreadMetaRepository (SQL,
  read-modify-write inside one session) and MemoryThreadMetaStore. Three new
  unit tests cover merge / empty / nonexistent behaviour.

Net change: -134 lines. Full test suite: 1693 passed, 14 skipped.
Verified end-to-end with curl in gateway mode against sqlite backend
(create / patch / get / rename / search / delete).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: JilongSun <965640067@qq.com>
Co-authored-by: jie <49781832+stan-fu@users.noreply.github.com>
Co-authored-by: cooper <cooperfu@tencent.com>
Co-authored-by: yangzheli <43645580+yangzheli@users.noreply.github.com>

* feat(auth): release-validation pass for 2.0-rc — 12 blockers + simplify follow-ups (#2008)

* feat(auth): introduce backend auth module

Port RFC-001 authentication core from PR #1728:
- JWT token handling (create_access_token, decode_token, TokenPayload)
- Password hashing (bcrypt) with verify_password
- SQLite UserRepository with base interface
- Provider Factory pattern (LocalAuthProvider)
- CLI reset_admin tool
- Auth-specific errors (AuthErrorCode, TokenError, AuthErrorResponse)

Deps:
- bcrypt>=4.0.0
- pyjwt>=2.9.0
- email-validator>=2.0.0
- backend/uv.toml pins public PyPI index

Tests: 12 pure unit tests (test_auth_config.py, test_auth_errors.py).

Scope note: authz.py, test_auth.py, and test_auth_type_system.py are
deferred to commit 2 because they depend on middleware and deps wiring
that is not yet in place. Commit 1 stays "pure new files only" as the
spec mandates.

* feat(auth): wire auth end-to-end (middleware + frontend replacement)

Backend:
- Port auth_middleware, csrf_middleware, langgraph_auth, routers/auth
- Port authz decorator (owner_filter_key defaults to 'owner_id')
- Merge app.py: register AuthMiddleware + CSRFMiddleware + CORS, add
  _ensure_admin_user lifespan hook, _migrate_orphaned_threads helper,
  register auth router
- Merge deps.py: add get_local_provider, get_current_user_from_request,
  get_optional_user_from_request; keep get_current_user as thin str|None
  adapter for feedback router
- langgraph.json: add auth path pointing to langgraph_auth.py:auth
- Rename metadata['user_id'] -> metadata['owner_id'] in langgraph_auth
  (both metadata write and LangGraph filter dict) + test fixtures

Frontend:
- Delete better-auth library and api catch-all route
- Remove better-auth npm dependency and env vars (BETTER_AUTH_SECRET,
  BETTER_AUTH_GITHUB_*) from env.js
- Port frontend/src/core/auth/* (AuthProvider, gateway-config,
  proxy-policy, server-side getServerSideUser, types)
- Port frontend/src/core/api/fetcher.ts
- Port (auth)/layout, (auth)/login, (auth)/setup pages
- Rewrite workspace/layout.tsx as server component that calls
  getServerSideUser and wraps in AuthProvider
- Port workspace/workspace-content.tsx for the client-side sidebar logic

Tests:
- Port 5 auth test files (test_auth, test_auth_middleware,
  test_auth_type_system, test_ensure_admin, test_langgraph_auth)
- 176 auth tests PASS

After this commit: login/logout/registration flow works, but persistence
layer does not yet filter by owner_id. Commit 4 closes that gap.

* feat(auth): account settings page + i18n

- Port account-settings-page.tsx (change password, change email, logout)
- Wire into settings-dialog.tsx as new "account" section with UserIcon,
  rendered first in the section list
- Add i18n keys:
  - en-US/zh-CN: settings.sections.account ("Account" / "账号")
  - en-US/zh-CN: button.logout ("Log out" / "退出登录")
  - types.ts: matching type declarations

* feat(auth): enforce owner_id across 2.0-rc persistence layer

Add request-scoped contextvar-based owner filtering to threads_meta,
runs, run_events, and feedback repositories. Router code is unchanged
— isolation is enforced at the storage layer so that any caller that
forgets to pass owner_id still gets filtered results, and new routes
cannot accidentally leak data.

Core infrastructure
-------------------
- deerflow/runtime/user_context.py (new):
  - ContextVar[CurrentUser | None] with default None
  - runtime_checkable CurrentUser Protocol (structural subtype with .id)
  - set/reset/get/require helpers
  - AUTO sentinel + resolve_owner_id(value, method_name) for sentinel
    three-state resolution: AUTO reads contextvar, explicit str
    overrides, explicit None bypasses the filter (for migration/CLI)

Repository changes
------------------
- ThreadMetaRepository: create/get/search/update_*/delete gain
  owner_id=AUTO kwarg; read paths filter by owner, writes stamp it,
  mutations check ownership before applying
- RunRepository: put/get/list_by_thread/delete gain owner_id=AUTO kwarg
- FeedbackRepository: create/get/list_by_run/list_by_thread/delete
  gain owner_id=AUTO kwarg
- DbRunEventStore: list_messages/list_events/list_messages_by_run/
  count_messages/delete_by_thread/delete_by_run gain owner_id=AUTO
  kwarg. Write paths (put/put_batch) read contextvar softly: when a
  request-scoped user is available, owner_id is stamped; background
  worker writes without a user context pass None which is valid
  (orphan row to be bound by migration)

Schema
------
- persistence/models/run_event.py: RunEventRow.owner_id = Mapped[
  str | None] = mapped_column(String(64), nullable=True, index=True)
- No alembic migration needed: 2.0 ships fresh, Base.metadata.create_all
  picks up the new column automatically

Middleware
----------
- auth_middleware.py: after cookie check, call get_optional_user_from_
  request to load the real User, stamp it into request.state.user AND
  the contextvar via set_current_user, reset in a try/finally. Public
  paths and unauthenticated requests continue without contextvar, and
  @require_auth handles the strict 401 path

Test infrastructure
-------------------
- tests/conftest.py: @pytest.fixture(autouse=True) _auto_user_context
  sets a default SimpleNamespace(id="test-user-autouse") on every test
  unless marked @pytest.mark.no_auto_user. Keeps existing 20+
  persistence tests passing without modification
- pyproject.toml [tool.pytest.ini_options]: register no_auto_user
  marker so pytest does not emit warnings for opt-out tests
- tests/test_user_context.py: 6 tests covering three-state semantics,
  Protocol duck typing, and require/optional APIs
- tests/test_thread_meta_repo.py: one test updated to pass owner_id=
  None explicitly where it was previously relying on the old default

Test results
------------
- test_user_context.py: 6 passed
- test_auth*.py + test_langgraph_auth.py + test_ensure_admin.py: 127
- test_run_event_store / test_run_repository / test_thread_meta_repo
  / test_feedback: 92 passed
- Full backend suite: 1905 passed, 2 failed (both @requires_llm flaky
  integration tests unrelated to auth), 1 skipped

* feat(auth): extend orphan migration to 2.0-rc persistence tables

_ensure_admin_user now runs a three-step pipeline on every boot:

  Step 1 (fatal):     admin user exists / is created / password is reset
  Step 2 (non-fatal): LangGraph store orphan threads → admin
  Step 3 (non-fatal): SQL persistence tables → admin
    - threads_meta
    - runs
    - run_events
    - feedback

Each step is idempotent. The fatal/non-fatal split mirrors PR #1728's
original philosophy: admin creation failure blocks startup (the system
is unusable without an admin), whereas migration failures log a warning
and let the service proceed (a partial migration is recoverable; a
missing admin is not).

Key helpers
-----------
- _iter_store_items(store, namespace, *, page_size=500):
  async generator that cursor-paginates across LangGraph store pages.
  Fixes PR #1728's hardcoded limit=1000 bug that would silently lose
  orphans beyond the first page.

- _migrate_orphaned_threads(store, admin_user_id):
  Rewritten to use _iter_store_items. Returns the migrated count so the
  caller can log it; raises only on unhandled exceptions.

- _migrate_orphan_sql_tables(admin_user_id):
  Imports the 4 ORM models lazily, grabs the shared session factory,
  runs one UPDATE per table in a single transaction, commits once.
  No-op when no persistence backend is configured (in-memory dev).

Tests: test_ensure_admin.py (8 passed)

* test(auth): port AUTH test plan docs + lint/format pass

- Port backend/docs/AUTH_TEST_PLAN.md and AUTH_UPGRADE.md from PR #1728
- Rename metadata.user_id → metadata.owner_id in AUTH_TEST_PLAN.md
  (4 occurrences from the original PR doc)
- ruff auto-fix UP037 in sentinel type annotations: drop quotes around
  "str | None | _AutoSentinel" now that from __future__ import
  annotations makes them implicit string forms
- ruff format: 2 files (app/gateway/app.py, runtime/user_context.py)

Note on test coverage additions:
- conftest.py autouse fixture was already added in commit 4 (had to
  be co-located with the repository changes to keep pre-existing
  persistence tests passing)
- cross-user isolation E2E tests (test_owner_isolation.py) deferred
  — enforcement is already proven by the 98-test repository suite
  via the autouse fixture + explicit _AUTO sentinel exercises
- New test cases (TC-API-17..20, TC-ATK-13, TC-MIG-01..07) listed
  in AUTH_TEST_PLAN.md are deferred to a follow-up PR — they are
  manual-QA test cases rather than pytest code, and the spec-level
  coverage is already met by test_user_context.py + the 98-test
  repository suite.

Final test results:
- Auth suite (test_auth*, test_langgraph_auth, test_ensure_admin,
  test_user_context): 186 passed
- Persistence suite (test_run_event_store, test_run_repository,
  test_thread_meta_repo, test_feedback): 98 passed
- Lint: ruff check + ruff format both clean

* test(auth): add cross-user isolation test suite

10 tests exercising the storage-layer owner filter by manually
switching the user_context contextvar between two users. Verifies
the safety invariant:

  After a repository write with owner_id=A, a subsequent read with
  owner_id=B must not return the row, and vice versa.

Covers all 4 tables that own user-scoped data:

TC-API-17  threads_meta  — read, search, update, delete cross-user
TC-API-18  runs          — get, list_by_thread, delete cross-user
TC-API-19  run_events    — list_messages, list_events, count_messages,
                           delete_by_thread (CRITICAL: raw conversation
                           content leak vector)
TC-API-20  feedback      — get, list_by_run, delete cross-user

Plus two meta-tests verifying the sentinel pattern itself:
- AUTO + unset contextvar raises RuntimeError
- explicit owner_id=None bypasses the filter (migration escape hatch)

Architecture note
-----------------
These tests bypass the HTTP layer by design. The full chain
(cookie → middleware → contextvar → repository) is covered piecewise:

- test_auth_middleware.py: middleware sets contextvar from cookies
- test_owner_isolation.py: repositories enforce isolation when
  contextvar is set to different users

Together they prove the end-to-end safety property without the
ceremony of spinning up a full TestClient + in-memory DB for every
router endpoint.

Tests pass: 231 (full auth + persistence + isolation suite)
Lint: clean

* refactor(auth): migrate user repository to SQLAlchemy ORM

Move the users table into the shared persistence engine so auth
matches the pattern of threads_meta, runs, run_events, and feedback —
one engine, one session factory, one schema init codepath.

New files
---------
- persistence/user/__init__.py, persistence/user/model.py: UserRow
  ORM class with partial unique index on (oauth_provider, oauth_id)
- Registered in persistence/models/__init__.py so
  Base.metadata.create_all() picks it up

Modified
--------
- auth/repositories/sqlite.py: rewritten as async SQLAlchemy,
  identical constructor pattern to the other four repositories
  (def __init__(self, session_factory) + self._sf = session_factory)
- auth/config.py: drop users_db_path field — storage is configured
  through config.database like every other table
- deps.py/get_local_provider: construct SQLiteUserRepository with
  the shared session factory, fail fast if engine is not initialised
- tests/test_auth.py: rewrite test_sqlite_round_trip_new_fields to
  use the shared engine (init_engine + close_engine in a tempdir)
- tests/test_auth_type_system.py: add per-test autouse fixture that
  spins up a scratch engine and resets deps._cached_* singletons

* refactor(auth): remove SQL orphan migration (unused in supported scenarios)

The _migrate_orphan_sql_tables helper existed to bind NULL owner_id
rows in threads_meta, runs, run_events, and feedback to the admin on
first boot. But in every supported upgrade path, it's a no-op:

  1. Fresh install: create_all builds fresh tables, no legacy rows
  2. No-auth → with-auth (no existing persistence DB): persistence
     tables are created fresh by create_all, no legacy rows
  3. No-auth → with-auth (has existing persistence DB from #1930):
     NOT a supported upgrade path — "有 DB 到有 DB" schema evolution
     is out of scope; users wipe DB or run manual ALTER

So the SQL orphan migration never has anything to do in the
supported matrix. Delete the function, simplify _ensure_admin_user
from a 3-step pipeline to a 2-step one (admin creation + LangGraph
store orphan migration only).

LangGraph store orphan migration stays: it serves the real
"no-auth → with-auth" upgrade path where a user's existing LangGraph
thread metadata has no owner_id field and needs to be stamped with
the newly-created admin's id.

Tests: 284 passed (auth + persistence + isolation)
Lint: clean

* security(auth): write initial admin password to 0600 file instead of logs

CodeQL py/clear-text-logging-sensitive-data flagged 3 call sites that
logged the auto-generated admin password to stdout via logger.info().
Production log aggregators (ELK/Splunk/etc) would have captured those
cleartext secrets. Replace with a shared helper that writes to
.deer-flow/admin_initial_credentials.txt with mode 0600, and log only
the path.

New file
--------
- app/gateway/auth/credential_file.py: write_initial_credentials()
  helper. Takes email, password, and a "initial"/"reset" label.
  Creates .deer-flow/ if missing, writes a header comment plus the
  email+password, chmods 0o600, returns the absolute Path.

Modified
--------
- app/gateway/app.py: both _ensure_admin_user paths (fresh creation
  + needs_setup password reset) now write to file and log the path
- app/gateway/auth/reset_admin.py: rewritten to use the shared ORM
  repo (SQLiteUserRepository with session_factory) and the
  credential_file helper. The previous implementation was broken
  after the earlier ORM refactor — it still imported _get_users_conn
  and constructed SQLiteUserRepository() without a session factory.

No tests changed — the three password-log sites are all exercised
via existing test_ensure_admin.py which checks that startup
succeeds, not that a specific string appears in logs.

CodeQL alerts 272, 283, 284: all resolved.

* security(auth): strict JWT validation in middleware (fix junk cookie bypass)

AUTH_TEST_PLAN test 7.5.8 expects junk cookies to be rejected with
401. The previous middleware behaviour was "presence-only": check
that some access_token cookie exists, then pass through. In
combination with my Task-12 decision to skip @require_auth
decorators on routes, this created a gap where a request with any
cookie-shaped string (e.g. access_token=not-a-jwt) would bypass
authentication on routes that do not touch the repository
(/api/models, /api/mcp/config, /api/memory, /api/skills, …).

Fix: middleware now calls get_current_user_from_request() strictly
and catches the resulting HTTPException to render a 401 with the
proper fine-grained error code (token_invalid, token_expired,
user_not_found, …). On success it stamps request.state.user and
the contextvar so repository-layer owner filters work downstream.

The 4 old "_with_cookie_passes" tests in test_auth_middleware.py
were written for the presence-only behaviour; they asserted that
a junk cookie would make the handler return 200. They are renamed
to "_with_junk_cookie_rejected" and their assertions flipped to
401. The negative path (no cookie → 401 not_authenticated)
is unchanged.

Verified:
  no cookie       → 401 not_authenticated
  junk cookie     → 401 token_invalid     (the fixed bug)
  expired cookie  → 401 token_expired

Tests: 284 passed (auth + persistence + isolation)
Lint: clean

* security(auth): wire @require_permission(owner_check=True) on isolation routes

Apply the require_permission decorator to all 28 routes that take a
{thread_id} path parameter. Combined with the strict middleware
(previous commit), this gives the double-layer protection that
AUTH_TEST_PLAN test 7.5.9 documents:

  Layer 1 (AuthMiddleware): cookie + JWT validation, rejects junk
                            cookies and stamps request.state.user
  Layer 2 (@require_permission with owner_check=True): per-resource
                            ownership verification via
                            ThreadMetaStore.check_access — returns
                            404 if a different user owns the thread

The decorator's owner_check branch is rewritten to use the SQL
thread_meta_repo (the 2.0-rc persistence layer) instead of the
LangGraph store path that PR #1728 used (_store_get / get_store
in routers/threads.py). The inject_record convenience is dropped
— no caller in 2.0 needs the LangGraph blob, and the SQL repo has
a different shape.

Routes decorated (28 total):
- threads.py: delete, patch, get, get-state, post-state, post-history
- thread_runs.py: post-runs, post-runs-stream, post-runs-wait,
  list_runs, get_run, cancel_run, join_run, stream_existing_run,
  list_thread_messages, list_run_messages, list_run_events,
  thread_token_usage
- feedback.py: create, list, stats, delete
- uploads.py: upload (added Request param), list, delete
- artifacts.py: get_artifact
- suggestions.py: generate (renamed body parameter to avoid
  conflict with FastAPI Request)

Test fixes:
- test_suggestions_router.py: bypass the decorator via __wrapped__
  (the unit tests cover parsing logic, not auth — no point spinning
  up a thread_meta_repo just to test JSON unwrapping)
- test_auth_middleware.py 4 fake-cookie tests: already updated in
  the previous commit (745bf432)

Tests: 293 passed (auth + persistence + isolation + suggestions)
Lint: clean

* security(auth): defense-in-depth fixes from release validation pass

Eight findings caught while running the AUTH_TEST_PLAN end-to-end against
the deployed sg_dev stack. Each is a pre-condition for shipping
release/2.0-rc that the previous PRs missed.

Backend hardening
- routers/auth.py: rate limiter X-Real-IP now requires AUTH_TRUSTED_PROXIES
  whitelist (CIDR/IP allowlist). Without nginx in front, the previous code
  honored arbitrary X-Real-IP, letting an attacker rotate the header to
  fully bypass the per-IP login lockout.
- routers/auth.py: 36-entry common-password blocklist via Pydantic
  field_validator on RegisterRequest + ChangePasswordRequest. The shared
  _validate_strong_password helper keeps the constraint in one place.
- routers/threads.py: ThreadCreateRequest + ThreadPatchRequest strip
  server-reserved metadata keys (owner_id, user_id) via Pydantic
  field_validator so a forged value can never round-trip back to other
  clients reading the same thread. The actual ownership invariant stays
  on the threads_meta row; this closes the metadata-blob echo gap.
- authz.py + thread_meta/sql.py: require_permission gains a require_existing
  flag plumbed through check_access(require_existing=True). Destructive
  routes (DELETE/PATCH/state-update/runs/feedback) now treat a missing
  thread_meta row as 404 instead of "untracked legacy thread, allow",
  closing the cross-user delete-idempotence gap where any user could
  successfully DELETE another user's deleted thread.
- repositories/sqlite.py + base.py: update_user raises UserNotFoundError
  on a vanished row instead of silently returning the input. Concurrent
  delete during password reset can no longer look like a successful update.
- runtime/user_context.py: resolve_owner_id() coerces User.id (UUID) to
  str at the contextvar boundary so SQLAlchemy String(64) columns can
  bind it. The whole 2.0-rc isolation pipeline was previously broken
  end-to-end (POST /api/threads → 500 "type 'UUID' is not supported").
- persistence/engine.py: SQLAlchemy listener enables PRAGMA journal_mode=WAL,
  synchronous=NORMAL, foreign_keys=ON on every new SQLite connection.
  TC-UPG-06 in the test plan expects WAL; previous code shipped with the
  default 'delete' journal.
- auth_middleware.py: stamp request.state.auth = AuthContext(...) so
  @require_permission's short-circuit fires; previously every isolation
  request did a duplicate JWT decode + users SELECT. Also unifies the
  401 payload through AuthErrorResponse(...).model_dump().
- app.py: _ensure_admin_user restructure removes the noqa F821 scoping
  bug where 'password' was referenced outside the branch that defined it.
  New _announce_credentials helper absorbs the duplicate log block in
  the fresh-admin and reset-admin branches.

* fix(frontend+nginx): rollout CSRF on every state-changing client path

The frontend was 100% broken in gateway-pro mode for any user trying to
open a specific chat thread. Three cumulative bugs each silently
masked the next.

LangGraph SDK CSRF gap (api-client.ts)
- The Client constructor took only apiUrl, no defaultHeaders, no fetch
  interceptor. The SDK's internal fetch never sent X-CSRF-Token, so
  every state-changing /api/langgraph-compat/* call (runs/stream,
  threads/search, threads/{tid}/history, ...) hit CSRFMiddleware and
  got 403 before reaching the auth check. UI symptom: empty thread page
  with no error message; the SPA's hooks swallowed the rejection.
- Fix: pass an onRequest hook that injects X-CSRF-Token from the
  csrf_token cookie per request. Reading the cookie per call (not at
  construction time) handles login / logout / password-change cookie
  rotation transparently. The SDK's prepareFetchOptions calls
  onRequest for both regular requests AND streaming/SSE/reconnect, so
  the same hook covers runs.stream and runs.joinStream.

Raw fetch CSRF gap (7 files)
- Audit: 11 frontend fetch sites, only 2 included CSRF (login/setup +
  account-settings change-password). The other 7 routed through raw
  fetch() with no header — suggestions, memory, agents, mcp, skills,
  uploads, and the local thread cleanup hook all 403'd silently.
- Fix: enhance fetcher.ts:fetchWithAuth to auto-inject X-CSRF-Token on
  POST/PUT/DELETE/PATCH from a single shared readCsrfCookie() helper.
  Convert all 7 raw fetch() callers to fetchWithAuth so the contract
  is centrally enforced. api-client.ts and fetcher.ts share
  readCsrfCookie + STATE_CHANGING_METHODS to avoid drift.

nginx routing + buffering (nginx.local.conf)
- The auth feature shipped without updating the nginx config: per-API
  explicit location blocks but no /api/v1/auth/, /api/feedback, /api/runs.
  The frontend's client-side fetches to /api/v1/auth/login/local 404'd
  from the Next.js side because nginx routed /api/* to the frontend.
- Fix: add catch-all `location /api/` that proxies to the gateway.
  nginx longest-prefix matching keeps the explicit blocks (/api/models,
  /api/threads regex, /api/langgraph/, ...) winning for their paths.
- Fix: disable proxy_buffering + proxy_request_buffering for the
  frontend `location /` block. Without it, nginx tries to spool large
  Next.js chunks into /var/lib/nginx/proxy (root-owned) and fails with
  Permission denied → ERR_INCOMPLETE_CHUNKED_ENCODING → ChunkLoadError.

* test(auth): release-validation test infra and new coverage

Test fixtures and unit tests added during the validation pass.

Router test helpers (NEW: tests/_router_auth_helpers.py)
- make_authed_test_app(): builds a FastAPI test app with a stub
  middleware that stamps request.state.user + request.state.auth and a
  permissive thread_meta_repo mock. TestClient-based router tests
  (test_artifacts_router, test_threads_router) use it instead of bare
  FastAPI() so the new @require_permission(owner_check=True) decorators
  short-circuit cleanly.
- call_unwrapped(): walks the __wrapped__ chain to invoke the underlying
  handler without going through the authz wrappers. Direct-call tests
  (test_uploads_router) use it. Typed with ParamSpec so the wrapped
  signature flows through.

Backend test additions
- test_auth.py: 7 tests for the new _get_client_ip trust model (no
  proxy / trusted proxy / untrusted peer / XFF rejection / invalid
  CIDR / no client). 5 tests for the password blocklist (literal,
  case-insensitive, strong password accepted, change-password binding,
  short-password length-check still fires before blocklist).
  test_update_user_raises_when_row_concurrently_deleted: closes a
  shipped-without-coverage gap on the new UserNotFoundError contract.
- test_thread_meta_repo.py: 4 tests for check_access(require_existing=True)
  — strict missing-row denial, strict owner match, strict owner mismatch,
  strict null-owner still allowed (shared rows survive the tightening).
- test_ensure_admin.py: 3 tests for _migrate_orphaned_threads /
  _iter_store_items pagination, covering the TC-UPG-02 upgrade story
  end-to-end via mock store. Closes the gap where the cursor pagination
  was untested even though the previous PR rewrote it.
- test_threads_router.py: 5 tests for _strip_reserved_metadata
  (owner_id removal, user_id removal, safe-keys passthrough, empty
  input, both-stripped).
- test_auth_type_system.py: replace "password123" fixtures with
  Tr0ub4dor3a / AnotherStr0ngPwd! so the new password blocklist
  doesn't reject the test data.

* docs(auth): refresh TC-DOCKER-05 + document Docker validation gap

- AUTH_TEST_PLAN.md TC-DOCKER-05: the previous expectation
  ("admin password visible in docker logs") was stale after the simplify
  pass that moved credentials to a 0600 file. The grep "Password:" check
  would have silently failed and given a false sense of coverage. New
  expectation matches the actual file-based path: 0600 file in
  DEER_FLOW_HOME, log shows the path (not the secret), reverse-grep
  asserts no leaked password in container logs.
- NEW: docs/AUTH_TEST_DOCKER_GAP.md documents the only un-executed
  block in the test plan (TC-DOCKER-01..06). Reason: sg_dev validation
  host has no Docker daemon installed. The doc maps each Docker case
  to an already-validated bare-metal equivalent (TC-1.1, TC-REENT-01,
  TC-API-02 etc.) so the gap is auditable, and includes pre-flight
  reproduction steps for whoever has Docker available.

---------

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>

* feat: replace auto-admin creation with interactive setup flow

On first boot, instead of auto-creating admin@deerflow.dev with a
random password written to a credential file, DeerFlow now redirects
to /setup where the user creates the admin account interactively.

Backend:
- Remove auto admin creation from _ensure_admin_user (now only runs
  orphan thread migration when an admin already exists)
- Add POST /api/v1/auth/initialize endpoint (public, only callable
  when 0 users exist; auto-logs in after creation)
- Add /api/v1/auth/initialize to public paths in auth_middleware.py
  and CSRF exempt paths in csrf_middleware.py
- Update test_ensure_admin.py to match new behavior
- Add test_initialize_admin.py with 8 tests for the new endpoint

Frontend:
- Add system_setup_required to AuthResult type
- getServerSideUser() checks setup-status when unauthenticated
- Auth layout allows system_setup_required (renders children)
- Workspace layout redirects system_setup_required → /setup
- Login page redirects to /setup when system not initialized
- Setup page detects mode via isAuthenticated: unauth = create-admin
  form (calls /initialize), auth = change-password form (existing)

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/9c2471c5-d6e9-4ada-9192-61b56007b8d7

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

* fix: add cleanup flags to useEffect async fetches in setup/login pages

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/9c2471c5-d6e9-4ada-9192-61b56007b8d7

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

* fix: address reviewer feedback on /initialize endpoint security and robustness

1. Concurrency/register-blocking: switch setup-status and /initialize to
   check admin_count (via new count_admin_users()) instead of total
   user_count — /register can no longer block admin initialization

2. Dedicated error code: add SYSTEM_ALREADY_INITIALIZED to AuthErrorCode
   and use it in /initialize 409 responses; add to frontend types

3. Init token security: generate a one-time token at startup (logged to
   stdout) and require it in the /initialize request body — prevents
   an attacker from claiming admin on an exposed first-boot instance

4. Setup-status fetch timeout: apply SSR_AUTH_TIMEOUT_MS abort-controller
   pattern to the setup-status fetch in server.ts (same as /auth/me)

Backend repo/provider: add count_admin_users() to base, SQLite, and
LocalAuthProvider. Tests updated + new token-validation/register-blocking
test cases added.

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/b9f531fc-8ed3-41db-b416-237f243b45fd

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

* fix: address code review nits — move secrets import, add INVALID_INIT_TOKEN error code, fix test assertions

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/b9f531fc-8ed3-41db-b416-237f243b45fd

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

* refactor: remove init_token generation and validation from admin setup flow

* fix: re-apply init_token security for /initialize endpoint

Re-adds the one-time init_token requirement to the /initialize endpoint,
building on the human's UI improvements in 5eeeb09. This addresses the
two remaining unresolved review threads:

1. Dedicated error code (SYSTEM_ALREADY_INITIALIZED + INVALID_INIT_TOKEN)
2. Init token security gate — requires the token logged at startup

Changes:
- errors.py: re-add INVALID_INIT_TOKEN error code
- routers/auth.py: re-add `import secrets`, `init_token` field,
  token validation with secrets.compare_digest, and token consumption
- app.py: re-add token generation/logging and app.state.init_token = None
- setup/page.tsx: re-add initToken state + input field (human's UI kept)
- types.ts: re-add invalid_init_token error code
- test_initialize_admin.py: restore full token test coverage
- test_ensure_admin.py: restore init_token assertions

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/646fb5c0-ec09-41aa-9fe9-e6f7c32364e8

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

* fix: make init_token optional (403 not 422 on missing), don't consume token on error paths

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/646fb5c0-ec09-41aa-9fe9-e6f7c32364e8

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

* refactor: remove redundant skill-related functions and documentation

---------

Co-authored-by: rayhpeng <rayhpeng@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: JilongSun <965640067@qq.com>
Co-authored-by: jie <49781832+stan-fu@users.noreply.github.com>
Co-authored-by: cooper <cooperfu@tencent.com>
Co-authored-by: yangzheli <43645580+yangzheli@users.noreply.github.com>
Co-authored-by: greatmengqi <chenmengqi.0376@gmail.com>
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>
Co-authored-by: jiangfeng.11 <jiangfeng.11@bytedance.com>
2026-04-26 11:08:41 +08:00
greatmengqi 94eee95fe0 feat(auth): release-validation pass for 2.0-rc — 12 blockers + simplify follow-ups (#2008)
* feat(auth): introduce backend auth module

Port RFC-001 authentication core from PR #1728:
- JWT token handling (create_access_token, decode_token, TokenPayload)
- Password hashing (bcrypt) with verify_password
- SQLite UserRepository with base interface
- Provider Factory pattern (LocalAuthProvider)
- CLI reset_admin tool
- Auth-specific errors (AuthErrorCode, TokenError, AuthErrorResponse)

Deps:
- bcrypt>=4.0.0
- pyjwt>=2.9.0
- email-validator>=2.0.0
- backend/uv.toml pins public PyPI index

Tests: 12 pure unit tests (test_auth_config.py, test_auth_errors.py).

Scope note: authz.py, test_auth.py, and test_auth_type_system.py are
deferred to commit 2 because they depend on middleware and deps wiring
that is not yet in place. Commit 1 stays "pure new files only" as the
spec mandates.

* feat(auth): wire auth end-to-end (middleware + frontend replacement)

Backend:
- Port auth_middleware, csrf_middleware, langgraph_auth, routers/auth
- Port authz decorator (owner_filter_key defaults to 'owner_id')
- Merge app.py: register AuthMiddleware + CSRFMiddleware + CORS, add
  _ensure_admin_user lifespan hook, _migrate_orphaned_threads helper,
  register auth router
- Merge deps.py: add get_local_provider, get_current_user_from_request,
  get_optional_user_from_request; keep get_current_user as thin str|None
  adapter for feedback router
- langgraph.json: add auth path pointing to langgraph_auth.py:auth
- Rename metadata['user_id'] -> metadata['owner_id'] in langgraph_auth
  (both metadata write and LangGraph filter dict) + test fixtures

Frontend:
- Delete better-auth library and api catch-all route
- Remove better-auth npm dependency and env vars (BETTER_AUTH_SECRET,
  BETTER_AUTH_GITHUB_*) from env.js
- Port frontend/src/core/auth/* (AuthProvider, gateway-config,
  proxy-policy, server-side getServerSideUser, types)
- Port frontend/src/core/api/fetcher.ts
- Port (auth)/layout, (auth)/login, (auth)/setup pages
- Rewrite workspace/layout.tsx as server component that calls
  getServerSideUser and wraps in AuthProvider
- Port workspace/workspace-content.tsx for the client-side sidebar logic

Tests:
- Port 5 auth test files (test_auth, test_auth_middleware,
  test_auth_type_system, test_ensure_admin, test_langgraph_auth)
- 176 auth tests PASS

After this commit: login/logout/registration flow works, but persistence
layer does not yet filter by owner_id. Commit 4 closes that gap.

* feat(auth): account settings page + i18n

- Port account-settings-page.tsx (change password, change email, logout)
- Wire into settings-dialog.tsx as new "account" section with UserIcon,
  rendered first in the section list
- Add i18n keys:
  - en-US/zh-CN: settings.sections.account ("Account" / "账号")
  - en-US/zh-CN: button.logout ("Log out" / "退出登录")
  - types.ts: matching type declarations

* feat(auth): enforce owner_id across 2.0-rc persistence layer

Add request-scoped contextvar-based owner filtering to threads_meta,
runs, run_events, and feedback repositories. Router code is unchanged
— isolation is enforced at the storage layer so that any caller that
forgets to pass owner_id still gets filtered results, and new routes
cannot accidentally leak data.

Core infrastructure
-------------------
- deerflow/runtime/user_context.py (new):
  - ContextVar[CurrentUser | None] with default None
  - runtime_checkable CurrentUser Protocol (structural subtype with .id)
  - set/reset/get/require helpers
  - AUTO sentinel + resolve_owner_id(value, method_name) for sentinel
    three-state resolution: AUTO reads contextvar, explicit str
    overrides, explicit None bypasses the filter (for migration/CLI)

Repository changes
------------------
- ThreadMetaRepository: create/get/search/update_*/delete gain
  owner_id=AUTO kwarg; read paths filter by owner, writes stamp it,
  mutations check ownership before applying
- RunRepository: put/get/list_by_thread/delete gain owner_id=AUTO kwarg
- FeedbackRepository: create/get/list_by_run/list_by_thread/delete
  gain owner_id=AUTO kwarg
- DbRunEventStore: list_messages/list_events/list_messages_by_run/
  count_messages/delete_by_thread/delete_by_run gain owner_id=AUTO
  kwarg. Write paths (put/put_batch) read contextvar softly: when a
  request-scoped user is available, owner_id is stamped; background
  worker writes without a user context pass None which is valid
  (orphan row to be bound by migration)

Schema
------
- persistence/models/run_event.py: RunEventRow.owner_id = Mapped[
  str | None] = mapped_column(String(64), nullable=True, index=True)
- No alembic migration needed: 2.0 ships fresh, Base.metadata.create_all
  picks up the new column automatically

Middleware
----------
- auth_middleware.py: after cookie check, call get_optional_user_from_
  request to load the real User, stamp it into request.state.user AND
  the contextvar via set_current_user, reset in a try/finally. Public
  paths and unauthenticated requests continue without contextvar, and
  @require_auth handles the strict 401 path

Test infrastructure
-------------------
- tests/conftest.py: @pytest.fixture(autouse=True) _auto_user_context
  sets a default SimpleNamespace(id="test-user-autouse") on every test
  unless marked @pytest.mark.no_auto_user. Keeps existing 20+
  persistence tests passing without modification
- pyproject.toml [tool.pytest.ini_options]: register no_auto_user
  marker so pytest does not emit warnings for opt-out tests
- tests/test_user_context.py: 6 tests covering three-state semantics,
  Protocol duck typing, and require/optional APIs
- tests/test_thread_meta_repo.py: one test updated to pass owner_id=
  None explicitly where it was previously relying on the old default

Test results
------------
- test_user_context.py: 6 passed
- test_auth*.py + test_langgraph_auth.py + test_ensure_admin.py: 127
- test_run_event_store / test_run_repository / test_thread_meta_repo
  / test_feedback: 92 passed
- Full backend suite: 1905 passed, 2 failed (both @requires_llm flaky
  integration tests unrelated to auth), 1 skipped

* feat(auth): extend orphan migration to 2.0-rc persistence tables

_ensure_admin_user now runs a three-step pipeline on every boot:

  Step 1 (fatal):     admin user exists / is created / password is reset
  Step 2 (non-fatal): LangGraph store orphan threads → admin
  Step 3 (non-fatal): SQL persistence tables → admin
    - threads_meta
    - runs
    - run_events
    - feedback

Each step is idempotent. The fatal/non-fatal split mirrors PR #1728's
original philosophy: admin creation failure blocks startup (the system
is unusable without an admin), whereas migration failures log a warning
and let the service proceed (a partial migration is recoverable; a
missing admin is not).

Key helpers
-----------
- _iter_store_items(store, namespace, *, page_size=500):
  async generator that cursor-paginates across LangGraph store pages.
  Fixes PR #1728's hardcoded limit=1000 bug that would silently lose
  orphans beyond the first page.

- _migrate_orphaned_threads(store, admin_user_id):
  Rewritten to use _iter_store_items. Returns the migrated count so the
  caller can log it; raises only on unhandled exceptions.

- _migrate_orphan_sql_tables(admin_user_id):
  Imports the 4 ORM models lazily, grabs the shared session factory,
  runs one UPDATE per table in a single transaction, commits once.
  No-op when no persistence backend is configured (in-memory dev).

Tests: test_ensure_admin.py (8 passed)

* test(auth): port AUTH test plan docs + lint/format pass

- Port backend/docs/AUTH_TEST_PLAN.md and AUTH_UPGRADE.md from PR #1728
- Rename metadata.user_id → metadata.owner_id in AUTH_TEST_PLAN.md
  (4 occurrences from the original PR doc)
- ruff auto-fix UP037 in sentinel type annotations: drop quotes around
  "str | None | _AutoSentinel" now that from __future__ import
  annotations makes them implicit string forms
- ruff format: 2 files (app/gateway/app.py, runtime/user_context.py)

Note on test coverage additions:
- conftest.py autouse fixture was already added in commit 4 (had to
  be co-located with the repository changes to keep pre-existing
  persistence tests passing)
- cross-user isolation E2E tests (test_owner_isolation.py) deferred
  — enforcement is already proven by the 98-test repository suite
  via the autouse fixture + explicit _AUTO sentinel exercises
- New test cases (TC-API-17..20, TC-ATK-13, TC-MIG-01..07) listed
  in AUTH_TEST_PLAN.md are deferred to a follow-up PR — they are
  manual-QA test cases rather than pytest code, and the spec-level
  coverage is already met by test_user_context.py + the 98-test
  repository suite.

Final test results:
- Auth suite (test_auth*, test_langgraph_auth, test_ensure_admin,
  test_user_context): 186 passed
- Persistence suite (test_run_event_store, test_run_repository,
  test_thread_meta_repo, test_feedback): 98 passed
- Lint: ruff check + ruff format both clean

* test(auth): add cross-user isolation test suite

10 tests exercising the storage-layer owner filter by manually
switching the user_context contextvar between two users. Verifies
the safety invariant:

  After a repository write with owner_id=A, a subsequent read with
  owner_id=B must not return the row, and vice versa.

Covers all 4 tables that own user-scoped data:

TC-API-17  threads_meta  — read, search, update, delete cross-user
TC-API-18  runs          — get, list_by_thread, delete cross-user
TC-API-19  run_events    — list_messages, list_events, count_messages,
                           delete_by_thread (CRITICAL: raw conversation
                           content leak vector)
TC-API-20  feedback      — get, list_by_run, delete cross-user

Plus two meta-tests verifying the sentinel pattern itself:
- AUTO + unset contextvar raises RuntimeError
- explicit owner_id=None bypasses the filter (migration escape hatch)

Architecture note
-----------------
These tests bypass the HTTP layer by design. The full chain
(cookie → middleware → contextvar → repository) is covered piecewise:

- test_auth_middleware.py: middleware sets contextvar from cookies
- test_owner_isolation.py: repositories enforce isolation when
  contextvar is set to different users

Together they prove the end-to-end safety property without the
ceremony of spinning up a full TestClient + in-memory DB for every
router endpoint.

Tests pass: 231 (full auth + persistence + isolation suite)
Lint: clean

* refactor(auth): migrate user repository to SQLAlchemy ORM

Move the users table into the shared persistence engine so auth
matches the pattern of threads_meta, runs, run_events, and feedback —
one engine, one session factory, one schema init codepath.

New files
---------
- persistence/user/__init__.py, persistence/user/model.py: UserRow
  ORM class with partial unique index on (oauth_provider, oauth_id)
- Registered in persistence/models/__init__.py so
  Base.metadata.create_all() picks it up

Modified
--------
- auth/repositories/sqlite.py: rewritten as async SQLAlchemy,
  identical constructor pattern to the other four repositories
  (def __init__(self, session_factory) + self._sf = session_factory)
- auth/config.py: drop users_db_path field — storage is configured
  through config.database like every other table
- deps.py/get_local_provider: construct SQLiteUserRepository with
  the shared session factory, fail fast if engine is not initialised
- tests/test_auth.py: rewrite test_sqlite_round_trip_new_fields to
  use the shared engine (init_engine + close_engine in a tempdir)
- tests/test_auth_type_system.py: add per-test autouse fixture that
  spins up a scratch engine and resets deps._cached_* singletons

* refactor(auth): remove SQL orphan migration (unused in supported scenarios)

The _migrate_orphan_sql_tables helper existed to bind NULL owner_id
rows in threads_meta, runs, run_events, and feedback to the admin on
first boot. But in every supported upgrade path, it's a no-op:

  1. Fresh install: create_all builds fresh tables, no legacy rows
  2. No-auth → with-auth (no existing persistence DB): persistence
     tables are created fresh by create_all, no legacy rows
  3. No-auth → with-auth (has existing persistence DB from #1930):
     NOT a supported upgrade path — "有 DB 到有 DB" schema evolution
     is out of scope; users wipe DB or run manual ALTER

So the SQL orphan migration never has anything to do in the
supported matrix. Delete the function, simplify _ensure_admin_user
from a 3-step pipeline to a 2-step one (admin creation + LangGraph
store orphan migration only).

LangGraph store orphan migration stays: it serves the real
"no-auth → with-auth" upgrade path where a user's existing LangGraph
thread metadata has no owner_id field and needs to be stamped with
the newly-created admin's id.

Tests: 284 passed (auth + persistence + isolation)
Lint: clean

* security(auth): write initial admin password to 0600 file instead of logs

CodeQL py/clear-text-logging-sensitive-data flagged 3 call sites that
logged the auto-generated admin password to stdout via logger.info().
Production log aggregators (ELK/Splunk/etc) would have captured those
cleartext secrets. Replace with a shared helper that writes to
.deer-flow/admin_initial_credentials.txt with mode 0600, and log only
the path.

New file
--------
- app/gateway/auth/credential_file.py: write_initial_credentials()
  helper. Takes email, password, and a "initial"/"reset" label.
  Creates .deer-flow/ if missing, writes a header comment plus the
  email+password, chmods 0o600, returns the absolute Path.

Modified
--------
- app/gateway/app.py: both _ensure_admin_user paths (fresh creation
  + needs_setup password reset) now write to file and log the path
- app/gateway/auth/reset_admin.py: rewritten to use the shared ORM
  repo (SQLiteUserRepository with session_factory) and the
  credential_file helper. The previous implementation was broken
  after the earlier ORM refactor — it still imported _get_users_conn
  and constructed SQLiteUserRepository() without a session factory.

No tests changed — the three password-log sites are all exercised
via existing test_ensure_admin.py which checks that startup
succeeds, not that a specific string appears in logs.

CodeQL alerts 272, 283, 284: all resolved.

* security(auth): strict JWT validation in middleware (fix junk cookie bypass)

AUTH_TEST_PLAN test 7.5.8 expects junk cookies to be rejected with
401. The previous middleware behaviour was "presence-only": check
that some access_token cookie exists, then pass through. In
combination with my Task-12 decision to skip @require_auth
decorators on routes, this created a gap where a request with any
cookie-shaped string (e.g. access_token=not-a-jwt) would bypass
authentication on routes that do not touch the repository
(/api/models, /api/mcp/config, /api/memory, /api/skills, …).

Fix: middleware now calls get_current_user_from_request() strictly
and catches the resulting HTTPException to render a 401 with the
proper fine-grained error code (token_invalid, token_expired,
user_not_found, …). On success it stamps request.state.user and
the contextvar so repository-layer owner filters work downstream.

The 4 old "_with_cookie_passes" tests in test_auth_middleware.py
were written for the presence-only behaviour; they asserted that
a junk cookie would make the handler return 200. They are renamed
to "_with_junk_cookie_rejected" and their assertions flipped to
401. The negative path (no cookie → 401 not_authenticated)
is unchanged.

Verified:
  no cookie       → 401 not_authenticated
  junk cookie     → 401 token_invalid     (the fixed bug)
  expired cookie  → 401 token_expired

Tests: 284 passed (auth + persistence + isolation)
Lint: clean

* security(auth): wire @require_permission(owner_check=True) on isolation routes

Apply the require_permission decorator to all 28 routes that take a
{thread_id} path parameter. Combined with the strict middleware
(previous commit), this gives the double-layer protection that
AUTH_TEST_PLAN test 7.5.9 documents:

  Layer 1 (AuthMiddleware): cookie + JWT validation, rejects junk
                            cookies and stamps request.state.user
  Layer 2 (@require_permission with owner_check=True): per-resource
                            ownership verification via
                            ThreadMetaStore.check_access — returns
                            404 if a different user owns the thread

The decorator's owner_check branch is rewritten to use the SQL
thread_meta_repo (the 2.0-rc persistence layer) instead of the
LangGraph store path that PR #1728 used (_store_get / get_store
in routers/threads.py). The inject_record convenience is dropped
— no caller in 2.0 needs the LangGraph blob, and the SQL repo has
a different shape.

Routes decorated (28 total):
- threads.py: delete, patch, get, get-state, post-state, post-history
- thread_runs.py: post-runs, post-runs-stream, post-runs-wait,
  list_runs, get_run, cancel_run, join_run, stream_existing_run,
  list_thread_messages, list_run_messages, list_run_events,
  thread_token_usage
- feedback.py: create, list, stats, delete
- uploads.py: upload (added Request param), list, delete
- artifacts.py: get_artifact
- suggestions.py: generate (renamed body parameter to avoid
  conflict with FastAPI Request)

Test fixes:
- test_suggestions_router.py: bypass the decorator via __wrapped__
  (the unit tests cover parsing logic, not auth — no point spinning
  up a thread_meta_repo just to test JSON unwrapping)
- test_auth_middleware.py 4 fake-cookie tests: already updated in
  the previous commit (745bf432)

Tests: 293 passed (auth + persistence + isolation + suggestions)
Lint: clean

* security(auth): defense-in-depth fixes from release validation pass

Eight findings caught while running the AUTH_TEST_PLAN end-to-end against
the deployed sg_dev stack. Each is a pre-condition for shipping
release/2.0-rc that the previous PRs missed.

Backend hardening
- routers/auth.py: rate limiter X-Real-IP now requires AUTH_TRUSTED_PROXIES
  whitelist (CIDR/IP allowlist). Without nginx in front, the previous code
  honored arbitrary X-Real-IP, letting an attacker rotate the header to
  fully bypass the per-IP login lockout.
- routers/auth.py: 36-entry common-password blocklist via Pydantic
  field_validator on RegisterRequest + ChangePasswordRequest. The shared
  _validate_strong_password helper keeps the constraint in one place.
- routers/threads.py: ThreadCreateRequest + ThreadPatchRequest strip
  server-reserved metadata keys (owner_id, user_id) via Pydantic
  field_validator so a forged value can never round-trip back to other
  clients reading the same thread. The actual ownership invariant stays
  on the threads_meta row; this closes the metadata-blob echo gap.
- authz.py + thread_meta/sql.py: require_permission gains a require_existing
  flag plumbed through check_access(require_existing=True). Destructive
  routes (DELETE/PATCH/state-update/runs/feedback) now treat a missing
  thread_meta row as 404 instead of "untracked legacy thread, allow",
  closing the cross-user delete-idempotence gap where any user could
  successfully DELETE another user's deleted thread.
- repositories/sqlite.py + base.py: update_user raises UserNotFoundError
  on a vanished row instead of silently returning the input. Concurrent
  delete during password reset can no longer look like a successful update.
- runtime/user_context.py: resolve_owner_id() coerces User.id (UUID) to
  str at the contextvar boundary so SQLAlchemy String(64) columns can
  bind it. The whole 2.0-rc isolation pipeline was previously broken
  end-to-end (POST /api/threads → 500 "type 'UUID' is not supported").
- persistence/engine.py: SQLAlchemy listener enables PRAGMA journal_mode=WAL,
  synchronous=NORMAL, foreign_keys=ON on every new SQLite connection.
  TC-UPG-06 in the test plan expects WAL; previous code shipped with the
  default 'delete' journal.
- auth_middleware.py: stamp request.state.auth = AuthContext(...) so
  @require_permission's short-circuit fires; previously every isolation
  request did a duplicate JWT decode + users SELECT. Also unifies the
  401 payload through AuthErrorResponse(...).model_dump().
- app.py: _ensure_admin_user restructure removes the noqa F821 scoping
  bug where 'password' was referenced outside the branch that defined it.
  New _announce_credentials helper absorbs the duplicate log block in
  the fresh-admin and reset-admin branches.

* fix(frontend+nginx): rollout CSRF on every state-changing client path

The frontend was 100% broken in gateway-pro mode for any user trying to
open a specific chat thread. Three cumulative bugs each silently
masked the next.

LangGraph SDK CSRF gap (api-client.ts)
- The Client constructor took only apiUrl, no defaultHeaders, no fetch
  interceptor. The SDK's internal fetch never sent X-CSRF-Token, so
  every state-changing /api/langgraph-compat/* call (runs/stream,
  threads/search, threads/{tid}/history, ...) hit CSRFMiddleware and
  got 403 before reaching the auth check. UI symptom: empty thread page
  with no error message; the SPA's hooks swallowed the rejection.
- Fix: pass an onRequest hook that injects X-CSRF-Token from the
  csrf_token cookie per request. Reading the cookie per call (not at
  construction time) handles login / logout / password-change cookie
  rotation transparently. The SDK's prepareFetchOptions calls
  onRequest for both regular requests AND streaming/SSE/reconnect, so
  the same hook covers runs.stream and runs.joinStream.

Raw fetch CSRF gap (7 files)
- Audit: 11 frontend fetch sites, only 2 included CSRF (login/setup +
  account-settings change-password). The other 7 routed through raw
  fetch() with no header — suggestions, memory, agents, mcp, skills,
  uploads, and the local thread cleanup hook all 403'd silently.
- Fix: enhance fetcher.ts:fetchWithAuth to auto-inject X-CSRF-Token on
  POST/PUT/DELETE/PATCH from a single shared readCsrfCookie() helper.
  Convert all 7 raw fetch() callers to fetchWithAuth so the contract
  is centrally enforced. api-client.ts and fetcher.ts share
  readCsrfCookie + STATE_CHANGING_METHODS to avoid drift.

nginx routing + buffering (nginx.local.conf)
- The auth feature shipped without updating the nginx config: per-API
  explicit location blocks but no /api/v1/auth/, /api/feedback, /api/runs.
  The frontend's client-side fetches to /api/v1/auth/login/local 404'd
  from the Next.js side because nginx routed /api/* to the frontend.
- Fix: add catch-all `location /api/` that proxies to the gateway.
  nginx longest-prefix matching keeps the explicit blocks (/api/models,
  /api/threads regex, /api/langgraph/, ...) winning for their paths.
- Fix: disable proxy_buffering + proxy_request_buffering for the
  frontend `location /` block. Without it, nginx tries to spool large
  Next.js chunks into /var/lib/nginx/proxy (root-owned) and fails with
  Permission denied → ERR_INCOMPLETE_CHUNKED_ENCODING → ChunkLoadError.

* test(auth): release-validation test infra and new coverage

Test fixtures and unit tests added during the validation pass.

Router test helpers (NEW: tests/_router_auth_helpers.py)
- make_authed_test_app(): builds a FastAPI test app with a stub
  middleware that stamps request.state.user + request.state.auth and a
  permissive thread_meta_repo mock. TestClient-based router tests
  (test_artifacts_router, test_threads_router) use it instead of bare
  FastAPI() so the new @require_permission(owner_check=True) decorators
  short-circuit cleanly.
- call_unwrapped(): walks the __wrapped__ chain to invoke the underlying
  handler without going through the authz wrappers. Direct-call tests
  (test_uploads_router) use it. Typed with ParamSpec so the wrapped
  signature flows through.

Backend test additions
- test_auth.py: 7 tests for the new _get_client_ip trust model (no
  proxy / trusted proxy / untrusted peer / XFF rejection / invalid
  CIDR / no client). 5 tests for the password blocklist (literal,
  case-insensitive, strong password accepted, change-password binding,
  short-password length-check still fires before blocklist).
  test_update_user_raises_when_row_concurrently_deleted: closes a
  shipped-without-coverage gap on the new UserNotFoundError contract.
- test_thread_meta_repo.py: 4 tests for check_access(require_existing=True)
  — strict missing-row denial, strict owner match, strict owner mismatch,
  strict null-owner still allowed (shared rows survive the tightening).
- test_ensure_admin.py: 3 tests for _migrate_orphaned_threads /
  _iter_store_items pagination, covering the TC-UPG-02 upgrade story
  end-to-end via mock store. Closes the gap where the cursor pagination
  was untested even though the previous PR rewrote it.
- test_threads_router.py: 5 tests for _strip_reserved_metadata
  (owner_id removal, user_id removal, safe-keys passthrough, empty
  input, both-stripped).
- test_auth_type_system.py: replace "password123" fixtures with
  Tr0ub4dor3a / AnotherStr0ngPwd! so the new password blocklist
  doesn't reject the test data.

* docs(auth): refresh TC-DOCKER-05 + document Docker validation gap

- AUTH_TEST_PLAN.md TC-DOCKER-05: the previous expectation
  ("admin password visible in docker logs") was stale after the simplify
  pass that moved credentials to a 0600 file. The grep "Password:" check
  would have silently failed and given a false sense of coverage. New
  expectation matches the actual file-based path: 0600 file in
  DEER_FLOW_HOME, log shows the path (not the secret), reverse-grep
  asserts no leaked password in container logs.
- NEW: docs/AUTH_TEST_DOCKER_GAP.md documents the only un-executed
  block in the test plan (TC-DOCKER-01..06). Reason: sg_dev validation
  host has no Docker daemon installed. The doc maps each Docker case
  to an already-validated bare-metal equivalent (TC-1.1, TC-REENT-01,
  TC-API-02 etc.) so the gap is auditable, and includes pre-flight
  reproduction steps for whoever has Docker available.

---------

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-04-26 11:08:11 +08:00
rayhpeng d8ecaf46c9 feat(persistence): add unified persistence layer with event store, token tracking, and feedback (#1930)
* feat(persistence): add SQLAlchemy 2.0 async ORM scaffold

Introduce a unified database configuration (DatabaseConfig) that
controls both the LangGraph checkpointer and the DeerFlow application
persistence layer from a single `database:` config section.

New modules:
- deerflow.config.database_config — Pydantic config with memory/sqlite/postgres backends
- deerflow.persistence — async engine lifecycle, DeclarativeBase with to_dict mixin, Alembic skeleton
- deerflow.runtime.runs.store — RunStore ABC + MemoryRunStore implementation

Gateway integration initializes/tears down the persistence engine in
the existing langgraph_runtime() context manager. Legacy checkpointer
config is preserved for backward compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add RunEventStore ABC + MemoryRunEventStore

Phase 2-A prerequisite for event storage: adds the unified run event
stream interface (RunEventStore) with an in-memory implementation,
RunEventsConfig, gateway integration, and comprehensive tests (27 cases).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add ORM models, repositories, DB/JSONL event stores, RunJournal, and API endpoints

Phase 2-B: run persistence + event storage + token tracking.

- ORM models: RunRow (with token fields), ThreadMetaRow, RunEventRow
- RunRepository implements RunStore ABC via SQLAlchemy ORM
- ThreadMetaRepository with owner access control
- DbRunEventStore with trace content truncation and cursor pagination
- JsonlRunEventStore with per-run files and seq recovery from disk
- RunJournal (BaseCallbackHandler) captures LLM/tool/lifecycle events,
  accumulates token usage by caller type, buffers and flushes to store
- RunManager now accepts optional RunStore for persistent backing
- Worker creates RunJournal, writes human_message, injects callbacks
- Gateway deps use factory functions (RunRepository when DB available)
- New endpoints: messages, run messages, run events, token-usage
- ThreadCreateRequest gains assistant_id field
- 92 tests pass (33 new), zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(persistence): add user feedback + follow-up run association

Phase 2-C: feedback and follow-up tracking.

- FeedbackRow ORM model (rating +1/-1, optional message_id, comment)
- FeedbackRepository with CRUD, list_by_run/thread, aggregate stats
- Feedback API endpoints: create, list, stats, delete
- follow_up_to_run_id in RunCreateRequest (explicit or auto-detected
  from latest successful run on the thread)
- Worker writes follow_up_to_run_id into human_message event metadata
- Gateway deps: feedback_repo factory + getter
- 17 new tests (14 FeedbackRepository + 3 follow-up association)
- 109 total tests pass, zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test+config: comprehensive Phase 2 test coverage + deprecate checkpointer config

- config.example.yaml: deprecate standalone checkpointer section, activate
  unified database:sqlite as default (drives both checkpointer + app data)
- New: test_thread_meta_repo.py (14 tests) — full ThreadMetaRepository coverage
  including check_access owner logic, list_by_owner pagination
- Extended test_run_repository.py (+4 tests) — completion preserves fields,
  list ordering desc, limit, owner_none returns all
- Extended test_run_journal.py (+8 tests) — on_chain_error, track_tokens=false,
  middleware no ai_message, unknown caller tokens, convenience fields,
  tool_error, non-summarization custom event
- Extended test_run_event_store.py (+7 tests) — DB batch seq continuity,
  make_run_event_store factory (memory/db/jsonl/fallback/unknown)
- Extended test_phase2b_integration.py (+4 tests) — create_or_reject persists,
  follow-up metadata, summarization in history, full DB-backed lifecycle
- Fixed DB integration test to use proper fake objects (not MagicMock)
  for JSON-serializable metadata
- 157 total Phase 2 tests pass, zero regressions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* config: move default sqlite_dir to .deer-flow/data

Keep SQLite databases alongside other DeerFlow-managed data
(threads, memory) under the .deer-flow/ directory instead of a
top-level ./data folder.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(persistence): remove UTFJSON, use engine-level json_serializer + datetime.now()

- Replace custom UTFJSON type with standard sqlalchemy.JSON in all ORM
  models. Add json_serializer=json.dumps(ensure_ascii=False) to all
  create_async_engine calls so non-ASCII text (Chinese etc.) is stored
  as-is in both SQLite and Postgres.
- Change ORM datetime defaults from datetime.now(UTC) to datetime.now(),
  remove UTC imports.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): simplify deps.py with getter factory + inline repos

- Replace 6 identical getter functions with _require() factory.
- Inline 3 _make_*_repo() factories into langgraph_runtime(), call
  get_session_factory() once instead of 3 times.
- Add thread_meta upsert in start_run (services.py).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(docker): add UV_EXTRAS build arg for optional dependencies

Support installing optional dependency groups (e.g. postgres) at
Docker build time via UV_EXTRAS build arg:
  UV_EXTRAS=postgres docker compose build

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(journal): fix flush, token tracking, and consolidate tests

RunJournal fixes:
- _flush_sync: retain events in buffer when no event loop instead of
  dropping them; worker's finally block flushes via async flush().
- on_llm_end: add tool_calls filter and caller=="lead_agent" guard for
  ai_message events; mark message IDs for dedup with record_llm_usage.
- worker.py: persist completion data (tokens, message count) to RunStore
  in finally block.

Model factory:
- Auto-inject stream_usage=True for BaseChatOpenAI subclasses with
  custom api_base, so usage_metadata is populated in streaming responses.

Test consolidation:
- Delete test_phase2b_integration.py (redundant with existing tests).
- Move DB-backed lifecycle test into test_run_journal.py.
- Add tests for stream_usage injection in test_model_factory.py.
- Clean up executor/task_tool dead journal references.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): widen content type to str|dict in all store backends

Allow event content to be a dict (for structured OpenAI-format messages)
in addition to plain strings. Dict values are JSON-serialized for the DB
backend and deserialized on read; memory and JSONL backends handle dicts
natively. Trace truncation now serializes dicts to JSON before measuring.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(events): use metadata flag instead of heuristic for dict content detection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(converters): add LangChain-to-OpenAI message format converters

Pure functions langchain_to_openai_message, langchain_to_openai_completion,
langchain_messages_to_openai, and _infer_finish_reason for converting
LangChain BaseMessage objects to OpenAI Chat Completions format, used by
RunJournal for event storage. 15 unit tests added.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(converters): handle empty list content as null, clean up test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): human_message content uses OpenAI user message format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): ai_message uses OpenAI format, add ai_tool_call message event

- ai_message content now uses {"role": "assistant", "content": "..."} format
- New ai_tool_call message event emitted when lead_agent LLM responds with tool_calls
- ai_tool_call uses langchain_to_openai_message converter for consistent format
- Both events include finish_reason in metadata ("stop" or "tool_calls")

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): add tool_result message event with OpenAI tool message format

Cache tool_call_id from on_tool_start keyed by run_id as fallback for on_tool_end,
then emit a tool_result message event (role=tool, tool_call_id, content) after each
successful tool completion.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): summary content uses OpenAI system message format

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): replace llm_start/llm_end with llm_request/llm_response in OpenAI format

Add on_chat_model_start to capture structured prompt messages as llm_request events.
Replace llm_end trace events with llm_response using OpenAI Chat Completions format.
Track llm_call_index to pair request/response events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(events): add record_middleware method for middleware trace events

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(events): add full run sequence integration test for OpenAI content format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(events): align message events with checkpoint format and add middleware tag injection

- Message events (ai_message, ai_tool_call, tool_result, human_message) now use
  BaseMessage.model_dump() format, matching LangGraph checkpoint values.messages
- on_tool_end extracts tool_call_id/name/status from ToolMessage objects
- on_tool_error now emits tool_result message events with error status
- record_middleware uses middleware:{tag} event_type and middleware category
- Summarization custom events use middleware:summarize category
- TitleMiddleware injects middleware:title tag via get_config() inheritance
- SummarizationMiddleware model bound with middleware:summarize tag
- Worker writes human_message using HumanMessage.model_dump()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(threads): switch search endpoint to threads_meta table and sync title

- POST /api/threads/search now queries threads_meta table directly,
  removing the two-phase Store + Checkpointer scan approach
- Add ThreadMetaRepository.search() with metadata/status filters
- Add ThreadMetaRepository.update_display_name() for title sync
- Worker syncs checkpoint title to threads_meta.display_name on run completion
- Map display_name to values.title in search response for API compatibility

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(threads): history endpoint reads messages from event store

- POST /api/threads/{thread_id}/history now combines two data sources:
  checkpointer for checkpoint_id, metadata, title, thread_data;
  event store for messages (complete history, not truncated by summarization)
- Strip internal LangGraph metadata keys from response
- Remove full channel_values serialization in favor of selective fields

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove duplicate optional-dependencies header in pyproject.toml

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(middleware): pass tagged config to TitleMiddleware ainvoke call

Without the config, the middleware:title tag was not injected,
causing the LLM response to be recorded as a lead_agent ai_message
in run_events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: resolve merge conflict in .env.example

Keep both DATABASE_URL (from persistence-scaffold) and WECOM
credentials (from main) after the merge.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address review feedback on PR #1851

- Fix naive datetime.now() → datetime.now(UTC) in all ORM models
- Fix seq race condition in DbRunEventStore.put() with FOR UPDATE
  and UNIQUE(thread_id, seq) constraint
- Encapsulate _store access in RunManager.update_run_completion()
- Deduplicate _store.put() logic in RunManager via _persist_to_store()
- Add update_run_completion to RunStore ABC + MemoryRunStore
- Wire follow_up_to_run_id through the full create path
- Add error recovery to RunJournal._flush_sync() lost-event scenario
- Add migration note for search_threads breaking change
- Fix test_checkpointer_none_fix mock to set database=None

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update uv.lock

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address 22 review comments from CodeQL, Copilot, and Code Quality

Bug fixes:
- Sanitize log params to prevent log injection (CodeQL)
- Reset threads_meta.status to idle/error when run completes
- Attach messages only to latest checkpoint in /history response
- Write threads_meta on POST /threads so new threads appear in search

Lint fixes:
- Remove unused imports (journal.py, migrations/env.py, test_converters.py)
- Convert lambda to named function (engine.py, Ruff E731)
- Remove unused logger definitions in repos (Ruff F841)
- Add logging to JSONL decode errors and empty except blocks
- Separate assert side-effects in tests (CodeQL)
- Remove unused local variables in tests (Ruff F841)
- Fix max_trace_content truncation to use byte length, not char length

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: apply ruff format to persistence and runtime files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Potential fix for pull request finding 'Statement has no effect'

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

* refactor(runtime): introduce RunContext to reduce run_agent parameter bloat

Extract checkpointer, store, event_store, run_events_config, thread_meta_repo,
and follow_up_to_run_id into a frozen RunContext dataclass. Add get_run_context()
in deps.py to build the base context from app.state singletons. start_run() uses
dataclasses.replace() to enrich per-run fields before passing ctx to run_agent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): move sanitize_log_param to app/gateway/utils.py

Extract the log-injection sanitizer from routers/threads.py into a shared
utils module and rename to sanitize_log_param (public API). Eliminates the
reverse service → router import in services.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf: use SQL aggregation for feedback stats and thread token usage

Replace Python-side counting in FeedbackRepository.aggregate_by_run with
a single SELECT COUNT/SUM query. Add RunStore.aggregate_tokens_by_thread
abstract method with SQL GROUP BY implementation in RunRepository and
Python fallback in MemoryRunStore. Simplify the thread_token_usage
endpoint to delegate to the new method, eliminating the limit=10000
truncation risk.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: annotate DbRunEventStore.put() as low-frequency path

Add docstring clarifying that put() opens a per-call transaction with
FOR UPDATE and should only be used for infrequent writes (currently
just the initial human_message event). High-throughput callers should
use put_batch() instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(threads): fall back to Store search when ThreadMetaRepository is unavailable

When database.backend=memory (default) or no SQL session factory is
configured, search_threads now queries the LangGraph Store instead of
returning 503. Returns empty list if neither Store nor repo is available.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(persistence): introduce ThreadMetaStore ABC for backend-agnostic thread metadata

Add ThreadMetaStore abstract base class with create/get/search/update/delete
interface. ThreadMetaRepository (SQL) now inherits from it. New
MemoryThreadMetaStore wraps LangGraph BaseStore for memory-mode deployments.

deps.py now always provides a non-None thread_meta_repo, eliminating all
`if thread_meta_repo is not None` guards in services.py, worker.py, and
routers/threads.py. search_threads no longer needs a Store fallback branch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(history): read messages from checkpointer instead of RunEventStore

The /history endpoint now reads messages directly from the
checkpointer's channel_values (the authoritative source) instead of
querying RunEventStore.list_messages(). The RunEventStore API is
preserved for other consumers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(persistence): address new Copilot review comments

- feedback.py: validate thread_id/run_id before deleting feedback
- jsonl.py: add path traversal protection with ID validation
- run_repo.py: parse `before` to datetime for PostgreSQL compat
- thread_meta_repo.py: fix pagination when metadata filter is active
- database_config.py: use resolve_path for sqlite_dir consistency

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Implement skill self-evolution and skill_manage flow (#1874)

* chore: ignore .worktrees directory

* Add skill_manage self-evolution flow

* Fix CI regressions for skill_manage

* Address PR review feedback for skill evolution

* fix(skill-evolution): preserve history on delete

* fix(skill-evolution): tighten scanner fallbacks

* docs: add skill_manage e2e evidence screenshot

* fix(skill-manage): avoid blocking fs ops in session runtime

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

* fix(config): resolve sqlite_dir relative to CWD, not Paths.base_dir

resolve_path() resolves relative to Paths.base_dir (.deer-flow),
which double-nested the path to .deer-flow/.deer-flow/data/app.db.
Use Path.resolve() (CWD-relative) instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Feature/feishu receive file (#1608)

* feat(feishu): add channel file materialization hook for inbound messages

- Introduce Channel.receive_file(msg, thread_id) as a base method for file materialization; default is no-op.
- Implement FeishuChannel.receive_file to download files/images from Feishu messages, save to sandbox, and inject virtual paths into msg.text.
- Update ChannelManager to call receive_file for any channel if msg.files is present, enabling downstream model access to user-uploaded files.
- No impact on Slack/Telegram or other channels (they inherit the default no-op).

* style(backend): format code with ruff for lint compliance

- Auto-formatted packages/harness/deerflow/agents/factory.py and tests/test_create_deerflow_agent.py using `ruff format`
- Ensured both files conform to project linting standards
- Fixes CI lint check failures caused by code style issues

* fix(feishu): handle file write operation asynchronously to prevent blocking

* fix(feishu): rename GetMessageResourceRequest to _GetMessageResourceRequest and remove redundant code

* test(feishu): add tests for receive_file method and placeholder replacement

* fix(manager): remove unnecessary type casting for channel retrieval

* fix(feishu): update logging messages to reflect resource handling instead of image

* fix(feishu): sanitize filename by replacing invalid characters in file uploads

* fix(feishu): improve filename sanitization and reorder image key handling in message processing

* fix(feishu): add thread lock to prevent filename conflicts during file downloads

* fix(test): correct bad merge in test_feishu_parser.py

* chore: run ruff and apply formatting cleanup
fix(feishu): preserve rich-text attachment order and improve fallback filename handling

* fix(docker): restore gateway env vars and fix langgraph empty arg issue (#1915)

Two production docker-compose.yaml bugs prevent `make up` from working:

1. Gateway missing DEER_FLOW_CONFIG_PATH and DEER_FLOW_EXTENSIONS_CONFIG_PATH
   environment overrides. Added in fb2d99f (#1836) but accidentally reverted
   by ca2fb95 (#1847). Without them, gateway reads host paths from .env via
   env_file, causing FileNotFoundError inside the container.

2. Langgraph command fails when LANGGRAPH_ALLOW_BLOCKING is unset (default).
   Empty $${allow_blocking} inserts a bare space between flags, causing
   ' --no-reload' to be parsed as unexpected extra argument. Fix by building
   args string first and conditionally appending --allow-blocking.

Co-authored-by: cooper <cooperfu@tencent.com>

* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities (#1904)

* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities

Fix `<button>` inside `<a>` invalid HTML in artifact components and add
missing `noopener,noreferrer` to `window.open` calls to prevent reverse
tabnabbing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(frontend): address Copilot review on tabnabbing and double-tab-open

Remove redundant parent onClick on web_fetch ChainOfThoughtStep to
prevent opening two tabs on link click, and explicitly null out
window.opener after window.open() for defensive tabnabbing hardening.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(persistence): organize entities into per-entity directories

Restructure the persistence layer from horizontal "models/ + repositories/"
split into vertical entity-aligned directories. Each entity (thread_meta,
run, feedback) now owns its ORM model, abstract interface (where applicable),
and concrete implementations under a single directory with an aggregating
__init__.py for one-line imports.

Layout:
  persistence/thread_meta/{base,model,sql,memory}.py
  persistence/run/{model,sql}.py
  persistence/feedback/{model,sql}.py

models/__init__.py is kept as a facade so Alembic autogenerate continues to
discover all ORM tables via Base.metadata. RunEventRow remains under
models/run_event.py because its storage implementation lives in
runtime/events/store/db.py and has no matching repository directory.

The repositories/ directory is removed entirely. All call sites in
gateway/deps.py and tests are updated to import from the new entity
packages, e.g.:

    from deerflow.persistence.thread_meta import ThreadMetaRepository
    from deerflow.persistence.run import RunRepository
    from deerflow.persistence.feedback import FeedbackRepository

Full test suite passes (1690 passed, 14 skipped).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(gateway): sync thread rename and delete through ThreadMetaStore

The POST /threads/{id}/state endpoint previously synced title changes
only to the LangGraph Store via _store_upsert. In sqlite mode the search
endpoint reads from the ThreadMetaRepository SQL table, so renames never
appeared in /threads/search until the next agent run completed (worker.py
syncs title from checkpoint to thread_meta in its finally block).

Likewise the DELETE /threads/{id} endpoint cleaned up the filesystem,
Store, and checkpointer but left the threads_meta row orphaned in sqlite,
so deleted threads kept appearing in /threads/search.

Fix both endpoints by routing through the ThreadMetaStore abstraction
which already has the correct sqlite/memory implementations wired up by
deps.py. The rename path now calls update_display_name() and the delete
path calls delete() — both work uniformly across backends.

Verified end-to-end with curl in gateway mode against sqlite backend.
Existing test suite (1690 passed) and focused router/repo tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(gateway): route all thread metadata access through ThreadMetaStore

Following the rename/delete bug fix in PR1, migrate the remaining direct
LangGraph Store reads/writes in the threads router and services to the
ThreadMetaStore abstraction so that the sqlite and memory backends behave
identically and the legacy dual-write paths can be removed.

Migrated endpoints (threads.py):
- create_thread: idempotency check + write now use thread_meta_repo.get/create
  instead of dual-writing the LangGraph Store and the SQL row.
- get_thread: reads from thread_meta_repo.get; the checkpoint-only fallback
  for legacy threads is preserved.
- patch_thread: replaced _store_get/_store_put with thread_meta_repo.update_metadata.
- delete_thread_data: dropped the legacy store.adelete; thread_meta_repo.delete
  already covers it.

Removed dead code (services.py):
- _upsert_thread_in_store — redundant with the immediately following
  thread_meta_repo.create() call.
- _sync_thread_title_after_run — worker.py's finally block already syncs
  the title via thread_meta_repo.update_display_name() after each run.

Removed dead code (threads.py):
- _store_get / _store_put / _store_upsert helpers (no remaining callers).
- THREADS_NS constant.
- get_store import (router no longer touches the LangGraph Store directly).

New abstract method:
- ThreadMetaStore.update_metadata(thread_id, metadata) merges metadata into
  the thread's metadata field. Implemented in both ThreadMetaRepository (SQL,
  read-modify-write inside one session) and MemoryThreadMetaStore. Three new
  unit tests cover merge / empty / nonexistent behaviour.

Net change: -134 lines. Full test suite: 1693 passed, 14 skipped.
Verified end-to-end with curl in gateway mode against sqlite backend
(create / patch / get / rename / search / delete).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: JilongSun <965640067@qq.com>
Co-authored-by: jie <49781832+stan-fu@users.noreply.github.com>
Co-authored-by: cooper <cooperfu@tencent.com>
Co-authored-by: yangzheli <43645580+yangzheli@users.noreply.github.com>
2026-04-26 11:05:47 +08:00
Willem Jiang 9dc25987e0 fix(channles):update the logger for the channel config (#2524)
* fix(channles):update the logger for the channel config

* fix(channels): normalize credential values and add tests for disabled-but-configured warning

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/dfc0a566-aa59-49f9-a74d-610292fb0a63

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

* fix the backend lint error

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-04-26 10:09:55 +08:00
Willem Jiang 8a044142cb feat(dev): add pre-commit hooks for ruff, eslint, and prettier (#2525)
* feat(dev): add pre-commit hooks for ruff, eslint, and prettier

* fix: use local uv-based ruff hooks and uv run for pre-commit install

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/a1e34cc5-0d4b-4400-9e6a-e687d964ff1e

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-04-26 09:40:17 +08:00
ming1523 410f0c48b5 fix(channels): accept single slack allowed user (#2481)
* fix(channels): accept single slack allowed user

* docs: address Slack allowed_users review notes

* ci: rerun backend unit tests

* docs: clarify Slack allowed_users config

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-25 19:40:52 +08:00
Octopus 1f59e945af fix: cap prompt caching breakpoints at 4 to prevent API 400 errors (#2449)
* fix: cap prompt caching breakpoints at 4 to prevent API 400 errors (fixes #2448)

The previous _apply_prompt_caching() attached cache_control to every text
block in the system prompt, every content block in the last N messages, and
the last tool definition. In multi-turn conversations with structured content
blocks this easily exceeded the 4-breakpoint hard limit enforced by both the
Anthropic API and AWS Bedrock, producing a 400 Bad Request (or a silent
"No generations found in stream" when streaming).

Fix: collect all candidate blocks in document order, then apply cache_control
only to the last MAX_CACHE_BREAKPOINTS (4) of them. Later breakpoints cover a
larger prefix and therefore yield better cache hit rates, making this the
optimal placement strategy as well as the safe one.

Adds 13 unit tests covering the budget cap, edge cases, and correct
last-candidate placement.

* docs: clarify _apply_prompt_caching docstring includes tool definitions

Per Copilot review: the implementation also caches the last tool definition
(see the candidates list at lines 202-205), so the docstring summary should
explicitly mention tools alongside system and recent messages.

* Fix the lint error

* style: fix ruff format check for test_claude_provider_prompt_caching.py

Add the missing blank line before the 'Edge cases' section comment so
that ruff format --check passes in CI.

---------

Co-authored-by: octo-patch <octo-patch@github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-25 19:40:06 +08:00
IECspace f394c0d8c8 feat(mcp): support custom tool interceptors via extensions_config.json (#2451)
* feat(mcp): support custom tool interceptors via extensions_config.json

Add a generic extension point for registering custom MCP tool
interceptors through `extensions_config.json`. This allows downstream
projects to inject per-request header manipulation, auth context
propagation, or other cross-cutting concerns without modifying
DeerFlow source code.

Interceptors are declared as Python callable paths in a new
`mcpInterceptors` array field and loaded via the existing
`resolve_variable` reflection mechanism:

```json
{
  "mcpInterceptors": [
    "my_package.mcp.auth:build_auth_interceptor"
  ]
}
```

Each entry must resolve to a no-arg builder function that returns an
async interceptor compatible with `MultiServerMCPClient`'s
`tool_interceptors` interface.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(mcp): add unit tests for custom tool interceptors

Cover all branches of the mcpInterceptors loading logic:

- valid interceptor loaded and appended to tool_interceptors
- multiple interceptors loaded in declaration order
- builder returning None is skipped
- resolve_variable ImportError logged and skipped
- builder raising exception logged and skipped
- absent mcpInterceptors field is safe (no-op)
- custom interceptors coexist with OAuth interceptor

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix(mcp): validate mcpInterceptors type and fix lint warnings

Address review feedback:

1. Validate mcpInterceptors config value before iterating:
   - Accept a single string and normalize to [string]
   - Ignore None silently
   - Log warning and skip for non-list/non-string types

2. Fix ruff F841 lint errors in tests:
   - Rename _make_mock_env to _make_patches, embed mock_client
   - Remove unused `as mock_cls` bindings where not needed
   - Extract _get_interceptors() helper to reduce repetition

3. Add two new test cases for type validation:
   - test_mcp_interceptors_single_string_is_normalized
   - test_mcp_interceptors_invalid_type_logs_warning

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(mcp): validate interceptor return type and fix import mock path

Address review feedback:

1. Validate builder return type with callable() check:
   - callable interceptor → append to tool_interceptors
   - None → silently skip (builder opted out)
   - non-callable → log warning with type name and skip

2. Fix test mock path: resolve_variable is a top-level import in
   tools.py, so mock deerflow.mcp.tools.resolve_variable instead of
   deerflow.reflection.resolve_variable to correctly intercept calls.

3. Add test_custom_interceptor_non_callable_return_logs_warning to
   cover the new non-callable validation branch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs(mcp): add mcpInterceptors example and documentation

- Add mcpInterceptors field to extensions_config.example.json
- Add "Custom Tool Interceptors" section to MCP_SERVER.md with
  configuration format, example interceptor code, and edge case
  behavior notes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: IECspace <IECspace@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-04-25 09:18:13 +08:00
orbisai0security 950821cb9b fix: use subprocess instead of os.system in local_backend.py (#2494)
* fix: use subprocess instead of os.system in local_backend.py

The sandbox backend and skill evaluation scripts use subprocess

* fixing the failing test

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-25 08:59:31 +08:00
pyp0327 2bb1a2dfa2 feat(models): Provider for MindIE model engine (#2483)
* feat(models): 适配 MindIE引擎的模型

* test: add unit tests for MindIEChatModel adapter and fix PR review comments

* chore: update uv.lock with pytest-asyncio

* build: add pytest-asyncio to test dependencies

* fix: address PR review comments (lazy import, cache clients, safe newline escape, strict xml regex)

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-25 08:59:03 +08:00
DanielWalnut b970993425 fix: read lead agent options from context (#2515)
* fix: read lead agent options from context

* fix: validate runtime context config
2026-04-24 22:46:51 +08:00
DanielWalnut ec8a8cae38 fix: gate deferred MCP tool execution (#2513)
* fix: gate deferred MCP tool execution

* style: format deferred tool middleware

* fix: address deferred tool review feedback
2026-04-24 22:45:41 +08:00
DanielWalnut d78ed5c8f2 fix: inherit subagent skill allowlists (#2514) 2026-04-24 21:24:42 +08:00
Nan Gao f9ff3a698d fix(middleware): avoid rescuing non-skill tool outputs during summarization (#2458)
* fix(middelware): narrow skill rescue to skill-related tool outputs

* fix(summarization): address skill rescue review feedback

* fix: wire summarization skill rescue config

* fix: remove dead skill tool helper

* fix(lint): fix format

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-24 21:19:46 +08:00
Admire c2332bb790 fix memory settings layout overflow (#2420)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-24 20:29:55 +08:00
He Wang 3a61126824 fix: keep debug.py interactive terminal free from background log noise (#2466)
* fix(debug): keep terminal clean by redirecting all logs to file

- Redirect all logs to debug.log file to prevent background task logs
  from interfering with interactive terminal prompts
- Honor AppConfig.log_level setting instead of hard-coding to INFO
- Make logging setup idempotent by clearing pre-existing handlers
- Defer deerflow imports until after logging is configured to ensure
  import-time side effects are captured in debug.log
- Display active log level in startup banner
- Add prompt_toolkit installation tip for enhanced readline support

Made-with: Cursor

* attaching the file handler before importing/calling get_app_config()

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-24 17:09:41 +08:00
Airene Fang 11f557a2c6 feat(trace):Add run_name to the trace info for system agents. (#2492)
* feat(trace): Add `run_name` to the trace info for suggestions and memory.

before(in langsmith):
CodexChatModel
CodexChatModel
lead_agent
after:
suggest_agent
memory_agent
lead_agent

feat(trace): Add `run_name` to the trace info for suggestions and memory.

before(in langsmith):
CodexChatModel
CodexChatModel
lead_agent
after:
suggest_agent
memory_agent
lead_agent

* feat(trace): Add `run_name` to the trace info for system agents.

before(in langsmith):
CodexChatModel
CodexChatModel
CodexChatModel
CodexChatModel
lead_agent
after:
suggest_agent
title_agent
security_agent
memory_agent
lead_agent

* chore(code format):code format

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-24 17:06:55 +08:00
d 🔹 e8572b9d0c fix(jina): log transient failures at WARNING without traceback (#2484) (#2485)
The exception handler in JinaClient.crawl used logger.exception, which
emits an ERROR-level record with the full httpx/httpcore/anyio traceback
for every transient network failure (timeout, connection refused). Other
search/crawl providers in the project log the same class of recoverable
failures as a single line. One offline/slow-network session could produce
dozens of multi-frame ERROR stack traces, drowning out real problems.

Switch to logger.warning with a concise message that includes the
exception type and its str, matching the style used elsewhere for
recoverable transient failures (aio_sandbox, ddg, etc.). The exception
type now also surfaces into the returned "Error: ..." string so callers
retain diagnostic signal.

Adds a regression test that asserts the log record is WARNING, carries
no exc_info, and includes the exception class name.

Co-authored-by: voidborne-d <voidborne-d@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-24 16:00:14 +08:00
Willem Jiang 80a7446fd6 fix(backend): fix the unit test error in backend 2026-04-24 14:56:03 +08:00
Willem Jiang cd12821134 fix(backend): Updated the uv.lock with new added dependency 2026-04-24 14:55:13 +08:00
Xinmin Zeng 30d619de08 feat(subagents): support per-subagent skill loading and custom subagent types (#2253)
* feat(subagents): support per-subagent skill loading and custom subagent types (#2230)

Add per-subagent skill configuration and custom subagent type registration,
aligned with Codex's role-based config layering and per-session skill injection.

Backend:
- SubagentConfig gains `skills` field (None=all, []=none, list=whitelist)
- New CustomSubagentConfig for user-defined subagent types in config.yaml
- SubagentsAppConfig gains `custom_agents` section and `get_skills_for()`
- Registry resolves custom agents with three-layer config precedence
- SubagentExecutor loads skills per-session as conversation items (Codex pattern)
- task_tool no longer appends skills to system_prompt
- Lead agent system prompt dynamically lists all registered subagent types
- setup_agent tool accepts optional skills parameter
- Gateway agents API transparently passes skills in CRUD operations

Frontend:
- Agent/CreateAgentRequest/UpdateAgentRequest types include skills field
- Agent card displays skills as badges alongside tool_groups

Config:
- config.example.yaml documents custom_agents and per-agent skills override

Tests:
- 40 new tests covering all skill config, custom agents, and registry logic
- Existing tests updated for new get_skills_prompt_section signature

Closes #2230

* fix: address review feedback on skills PR

- Remove stale get_skills_prompt_section monkeypatches from test_task_tool_core_logic.py
  (task_tool no longer imports this function after skill injection moved to executor)
- Add key prefixes (tg:/sk:) to agent-card badges to prevent React key collisions
  between tool_groups and skills

* fix(ci): resolve lint and test failures

- Format agent-card.tsx with prettier (lint-frontend)
- Remove stale "Skills Appendix" system_prompt assertion — skills are now
  loaded per-session by SubagentExecutor, not appended to system_prompt

* fix(ci): sort imports in test_subagent_skills_config.py (ruff I001)

* fix(ci): use nullish coalescing in agent-card badge condition (eslint)

* fix: address review feedback on skills PR

- Use model_fields_set in AgentUpdateRequest to distinguish "field omitted"
  from "explicitly set to null" — fixes skills=None ambiguity where None
  means "inherit all" but was treated as "don't change"
- Move lazy import of get_subagent_config outside loop in
  _build_available_subagents_description to avoid repeated import overhead

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-23 23:59:47 +08:00
JerryChaox 4e72410154 fix(gateway): bound lifespan shutdown hooks to prevent worker hang under uvicorn reload (#2331)
* fix(gateway): bound lifespan shutdown hooks to prevent worker hang

Gateway worker can hang indefinitely in `uvicorn --reload` mode with
the listening socket still bound — all /api/* requests return 504,
and SIGKILL is the only recovery.

Root cause (py-spy dump from a reproduction showed 16+ stacked frames
of signal_handler -> Event.set -> threading.Lock.__enter__ on the
main thread): CPython's `threading.Event` uses `Condition(Lock())`
where the inner Lock is non-reentrant. uvicorn's BaseReload signal
handler calls `should_exit.set()` directly from signal context; if a
second signal (SIGTERM/SIGHUP from the reload supervisor, or
watchfiles-triggered reload) arrives while the first handler holds
the Lock, the reentrant call deadlocks on itself.

The reload supervisor keeps sending those signals only when the
worker fails to exit promptly. DeerFlow's lifespan currently awaits
`stop_channel_service()` with no timeout; if a channel's `stop()`
stalls (e.g. Feishu/Slack WebSocket waiting for an ack), the worker
can't exit, the supervisor keeps signaling, and the deadlock becomes
reachable.

This is a defense-in-depth fix — it does not repair the upstream
uvicorn/CPython issue, but it ensures DeerFlow's lifespan exits
within a bounded window so the supervisor has no reason to keep
firing signals. No behavior change on the happy path.

Wraps the shutdown hook in `asyncio.wait_for(timeout=5.0)` and logs
a warning on timeout before proceeding to worker exit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Update backend/app/gateway/app.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* style: apply make format (ruff) to test assertions

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-23 19:41:26 +08:00
He Wang c42ae3af79 feat: add optional prompt-toolkit support to debug.py (#2461)
* feat: add optional prompt-toolkit support to debug.py

Use PromptSession.prompt_async() for arrow-key navigation and input
history when prompt-toolkit is available, falling back to plain input()
with a helpful install tip otherwise.

Made-with: Cursor

* fix: handle EOFError gracefully in debug.py

Catch EOFError alongside KeyboardInterrupt so that Ctrl-D exits
cleanly instead of printing a traceback.

Made-with: Cursor
2026-04-23 17:49:18 +08:00
dependabot[bot] bd35cd39aa chore(deps): bump uuid from 13.0.0 to 14.0.0 in /frontend (#2467)
Bumps [uuid](https://github.com/uuidjs/uuid) from 13.0.0 to 14.0.0.
- [Release notes](https://github.com/uuidjs/uuid/releases)
- [Changelog](https://github.com/uuidjs/uuid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/uuidjs/uuid/compare/v13.0.0...v14.0.0)

---
updated-dependencies:
- dependency-name: uuid
  dependency-version: 14.0.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-23 14:47:15 +08:00
d 🔹 b90f219bd1 fix(skills): validate bundled SKILL.md front-matter in CI (fixes #2443) (#2457)
* fix(skills): validate bundled SKILL.md front-matter in CI (fixes #2443)

Adds a parametrized backend test that runs `_validate_skill_frontmatter`
against every bundled SKILL.md under `skills/public/`, so a broken
front-matter fails CI with a per-skill error message instead of
surfacing as a runtime gateway-load warning.

The new test caught two pre-existing breakages on `main` and fixes them:

* `bootstrap/SKILL.md`: the unquoted description had a second `:` mid-line
  ("Also trigger for updates: ..."), which YAML parses as a nested mapping
  ("mapping values are not allowed here"). Rewrites the description as a
  folded scalar (`>-`), which preserves the original wording (including the
  embedded colon, double quotes, and apostrophes) without further escaping.
  This complements PR #2436 (single-file colon→hyphen patch) with a more
  general convention that survives future edits.

* `chart-visualization/SKILL.md`: used `dependency:` which is not in
  `ALLOWED_FRONTMATTER_PROPERTIES`. Renamed to `compatibility:`, the
  documented field for "Required tools, dependencies" per skill-creator.
  No code reads `dependency` (verified by grep across backend/).

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Fix the lint error

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-04-23 14:06:14 +08:00
dependabot[bot] 96d00f6073 chore(deps): bump dompurify from 3.3.1 to 3.4.1 in /frontend (#2462)
Bumps [dompurify](https://github.com/cure53/DOMPurify) from 3.3.1 to 3.4.1.
- [Release notes](https://github.com/cure53/DOMPurify/releases)
- [Commits](https://github.com/cure53/DOMPurify/compare/3.3.1...3.4.1)

---
updated-dependencies:
- dependency-name: dompurify
  dependency-version: 3.4.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-23 12:18:59 +08:00
He Wang c43c803f66 fix: remove mismatched context param in debug.py to suppress Pydantic warning (#2446)
* fix: remove mismatched context param in debug.py to suppress Pydantic warning

The ainvoke call passed context={"thread_id": ...} but the agent graph
has no context_schema (ContextT defaults to None), causing a
PydanticSerializationUnexpectedValue warning on every invocation.

Align with the production run_agent path by injecting context via
Runtime into configurable["__pregel_runtime"] instead.

Closes #2445

Made-with: Cursor

* refactor: derive runtime thread_id from config to avoid duplication

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Made-with: Cursor

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-23 09:56:57 +08:00
dependabot[bot] dbd777fe62 chore(deps): bump python-dotenv from 1.2.1 to 1.2.2 in /backend (#2440)
Bumps [python-dotenv](https://github.com/theskumar/python-dotenv) from 1.2.1 to 1.2.2.
- [Release notes](https://github.com/theskumar/python-dotenv/releases)
- [Changelog](https://github.com/theskumar/python-dotenv/blob/main/CHANGELOG.md)
- [Commits](https://github.com/theskumar/python-dotenv/compare/v1.2.1...v1.2.2)

---
updated-dependencies:
- dependency-name: python-dotenv
  dependency-version: 1.2.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-22 16:48:09 +08:00
dependabot[bot] 1ca2621285 chore(deps): bump lxml from 6.0.2 to 6.1.0 in /backend (#2427)
Bumps [lxml](https://github.com/lxml/lxml) from 6.0.2 to 6.1.0.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](https://github.com/lxml/lxml/compare/lxml-6.0.2...lxml-6.1.0)

---
updated-dependencies:
- dependency-name: lxml
  dependency-version: 6.1.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-22 16:14:11 +08:00
Shawn Jasper 5ba1dacf25 fix: rename present_file to present_files in docs and prompts (#2393)
The tool is registered as `present_files` (plural) in present_file_tool.py,
but four references in documentation and prompt strings incorrectly used the
singular form `present_file`. This could cause confusion and potentially
lead to incorrect tool invocations.

Changed files:
- backend/docs/GUARDRAILS.md
- backend/docs/ARCHITECTURE.md
- backend/packages/harness/deerflow/agents/lead_agent/prompt.py (2 occurrences)
2026-04-21 16:10:14 +08:00
Reuben Bowlby 085c13edc7 fix: remove unnecessary f-string prefixes and unused import (#2352)
- Remove f-string prefix on 7 strings with no placeholders (F541)
  in analyze.py, aggregate_benchmark.py, run_loop.py, generate_review.py
- Remove unused `os` import in quick_validate.py (F401)

Found by ruff via HUMMBL Arbiter (https://hummbl.io/audit).
2026-04-21 09:53:18 +08:00
Copilot ef04174194 Fix invalid HTML nesting in reasoning trigger during complex task rendering (#2382)
* Initial plan

* fix(frontend): avoid invalid paragraph nesting in reasoning trigger

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/4c9eb0c2-ff29-4629-a61c-4e33d736d918

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

* test(frontend): strengthen reasoning trigger DOM nesting assertion

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/4c9eb0c2-ff29-4629-a61c-4e33d736d918

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
2026-04-21 09:41:28 +08:00
Ansel 6dce26a52e fix: resolve tool duplication and skill parser YAML inconsistencies (#1803) (#2107)
* Refactor tests for SKILL.md parser

Updated tests for SKILL.md parser to handle quoted names and descriptions correctly. Added new tests for parsing plain and single-quoted names, and ensured multi-line descriptions are processed properly.

* Implement tool name validation and deduplication

Add tool name mismatch warning and deduplication logic

* Refactor skill file parsing and error handling

* Add tests for tool name deduplication

Added tests for tool name deduplication in get_available_tools(). Ensured that duplicates are not returned, the first occurrence is kept, and warnings are logged for skipped duplicates.

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Update minimal config to include tools list

* Update test for nonexistent skill file

Ensure the test for nonexistent files checks for None.

* Refactor tool loading and add skill management support

Refactor tool loading logic to include skill management tools based on configuration and clean up comments.

* Enhance code comments for tool loading logic

Added comments to clarify the purpose of various code sections related to tool loading and configuration.

* Fix assertion for duplicate tool name warning

* Fix indentation issues in tools.py

* Fix the lint error of test_tool_deduplication

* Fix the lint error of tools.py

* Fix the lint error

* Fix the lint error

* make format

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-04-20 20:25:03 +08:00
imhaoran fc94e90f6c fix(setup-agent): prevent data loss when setup fails on existing agen… (#2254)
* fix(setup-agent): prevent data loss when setup fails on existing agent directory

Record whether the agent directory pre-existed before mkdir, and only
run shutil.rmtree cleanup when the directory was newly created during
this call. Previously, any failure would delete the entire directory
including pre-existing SOUL.md and config.yaml.

* fix: address PR review — init variables before try, remove unused result

* style: fix ruff I001 import block formatting in test file

* style: add missing blank lines between top-level definitions in test file
2026-04-20 20:17:30 +08:00
Eilen Shin f2013f47aa fix command palette hydration mismatch (#2301)
* fix command palette hydration mismatch

* style: format command dialog description
2026-04-20 11:36:16 +08:00
KiteEater 4be857f64b fix: use Apple Container image pull syntax (#2366) 2026-04-20 08:00:05 +08:00
Admire c99865f53d fix(token-usage): enable stream usage for openai-compatible models (#2217)
* fix(token-usage): enable stream usage for openai-compatible models

* fix(token-usage): narrow stream_usage default to ChatOpenAI
2026-04-19 22:42:55 +08:00
YYMa 05f1da03e5 fix(script): use portable locale for langgraph log pipeline on macOS (#2361) 2026-04-19 22:41:00 +08:00
Xun a62ca5dd47 fix: Catch httpx.ReadError in the error handling (#2309)
* fix: Catch httpx.ReadError in the error handling

* fix
2026-04-19 22:30:22 +08:00
Nan Gao f514e35a36 fix(backend): make clarification messages idempotent (#2350) (#2351) 2026-04-19 22:00:58 +08:00
Xun 7c87dc5bca fix(reasoning): prevent LLM-hallucinated HTML tags from rendering as DOM elements (#2321)
* fix

* add test

* fix
2026-04-19 19:27:34 +08:00
Hinotobi 80e210f5bb [security] fix(uploads): require explicit opt-in for host-side document conversion (#2332)
* fix: disable host-side upload conversion by default

* fix: address PR review comments on upload conversion gate
2026-04-18 22:47:42 +08:00
dependabot[bot] 5656f90792 chore(deps-dev): bump pytest from 9.0.2 to 9.0.3 in /backend (#2349)
Bumps [pytest](https://github.com/pytest-dev/pytest) from 9.0.2 to 9.0.3.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.2...9.0.3)

---
updated-dependencies:
- dependency-name: pytest
  dependency-version: 9.0.3
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-18 22:22:40 +08:00
Shawn Jasper 55474011c9 fix(subagent): inherit parent agent's tool_groups in task_tool (#2305)
* fix(subagent): inherit parent agent's tool_groups in task_tool

When a custom agent defines tool_groups (e.g. [file:read, file:write, bash]),
the restriction is correctly applied to the lead agent. However, when the lead
agent delegates work to a subagent via the task tool, get_available_tools() is
called without the groups parameter, causing the subagent to receive ALL tools
(including web_search, web_fetch, image_search, etc.) regardless of the parent
agent's configuration.

This fix propagates tool_groups through run metadata so that task_tool passes
the same group filter when building the subagent's tool set.

Changes:
- agent.py: include tool_groups in run metadata
- task_tool.py: read tool_groups from metadata and pass to get_available_tools()

* fix: initialize metadata before conditional block and update tests for tool_groups propagation

- Initialize metadata = {} before the 'if runtime is not None' block to
  avoid Ruff F821 (possibly-undefined variable) and simplify the
  parent_tool_groups expression.
- Update existing test assertion to expect groups=None in
  get_available_tools call signature.
- Add 3 new test cases:
  - test_task_tool_propagates_tool_groups_to_subagent
  - test_task_tool_no_tool_groups_passes_none
  - test_task_tool_runtime_none_passes_groups_none
2026-04-18 22:17:37 +08:00
imhaoran 24fe5fbd8c fix(mcp): prevent RuntimeError from escaping except block in get_cach… (#2252)
* fix(mcp): prevent RuntimeError from escaping except block in get_cached_mcp_tools

When `asyncio.get_event_loop()` raises RuntimeError and the fallback
`asyncio.run()` also fails, the exception escapes unhandled because
Python does not route exceptions raised inside an `except` block to
sibling `except` clauses. Wrap the fallback call in its own try/except
so failures are logged and the function returns [] as intended.

* fix: use logger.exception to preserve stack traces on MCP init failure
2026-04-18 21:07:30 +08:00
Willem Jiang be4663505a chroe(script): disable the color log of langgraph 2026-04-18 20:03:05 +08:00
dependabot[bot] aa6098e6a4 chore(deps): bump langsmith from 0.6.4 to 0.7.31 in /backend (#2291)
Bumps [langsmith](https://github.com/langchain-ai/langsmith-sdk) from 0.6.4 to 0.7.31.
- [Release notes](https://github.com/langchain-ai/langsmith-sdk/releases)
- [Commits](https://github.com/langchain-ai/langsmith-sdk/compare/v0.6.4...v0.7.31)

---
updated-dependencies:
- dependency-name: langsmith
  dependency-version: 0.7.31
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-18 19:54:21 +08:00
Airene Fang 1221448029 fix(scripts): Cloud Provider Reports Security Issue(aliyun could) (#2323)
ATT&CK矩阵ID:T1059.004
数据来源:进程启动触发检测
告警原因:该进程的命令行显示出反弹shelI的特征
命令行:timeout 1 bash -c exec 3<>/dev/tcp/127.0.0.1/2024
进程路径:/usr/bin/timeout
进程链:-[337650] /usr/sbin/sshd -D
-[397971] /usr/sbin/sshd -D -R
-[397977]-bash
-[398903] make dev
-[398920] bash ./scripts/serve.sh --dev
-[399037]bash ./scripts/wait-for-port.sh 2024 60 LangGraph
2026-04-18 19:33:32 +08:00
Jason 3b91df2b18 fix(frontend): add catch-all API rewrite for gateway routes (#2335)
When NEXT_PUBLIC_BACKEND_BASE_URL is unset, the frontend proxies API
requests to the gateway. Only /api/agents and /api/skills had rewrite
rules, causing 404s for /api/models, /api/threads, /api/memory,
/api/mcp, /api/suggestions, /api/runs, etc.

Add a catch-all /api/:path* rewrite that proxies all remaining gateway
API routes. The existing /api/langgraph rewrite takes priority because
it is pushed to the array first (Next.js checks rewrites in order).

Fixes #2327

Co-authored-by: JasonOA888 <JasonOA888@users.noreply.github.com>
2026-04-18 11:35:19 +08:00
Shawn Jasper ca1b7d5f48 fix(sandbox): add missing path masking in ls_tool output (#2317)
ls_tool was the only file-system tool that did not call
mask_local_paths_in_output() before returning its result, causing host
absolute paths (e.g. /Users/.../backend/.deer-flow/knowledge-base/...)
to leak to the LLM instead of the expected virtual paths
(/mnt/knowledge-base/...).

This patch:
- Adds the mask_local_paths_in_output() call to ls_tool, consistent
  with bash_tool, glob_tool and grep_tool.
- Initialises thread_data = None before the is_local_sandbox branch
  (same pattern as glob_tool) so the variable is always in scope.
- Adds three new tests covering user-data path masking, skills path
  masking and the empty-directory edge case.
2026-04-18 08:46:59 +08:00
yangzheli c6b0423558 feat(frontend): add Playwright E2E tests with CI workflow (#2279)
* feat(frontend): add Playwright E2E tests with CI workflow

Add end-to-end testing infrastructure using Playwright (Chromium only).
14 tests across 5 spec files cover landing page, chat workspace,
thread history, sidebar navigation, and agent chat — all with mocked
LangGraph/Backend APIs via network interception (zero backend dependency).

New files:
- playwright.config.ts — Chromium, 30s timeout, auto-start Next.js
- tests/e2e/utils/mock-api.ts — shared API mocks & SSE stream helpers
- tests/e2e/{landing,chat,thread-history,sidebar,agent-chat}.spec.ts
- .github/workflows/e2e-tests.yml — push main + PR trigger, paths filter

Updated: package.json, Makefile, .gitignore, CONTRIBUTING.md,
frontend/CLAUDE.md, frontend/AGENTS.md, frontend/README.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: apply Copilot suggestions

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-18 08:21:08 +08:00
DanielWalnut 898f4e8ac2 fix: Memory update system has cache corruption, data loss, and thread-safety bugs (#2251)
* fix(memory): cache corruption, thread-safety, and caller mutation bugs

Bug 1 (updater.py): deep-copy current_memory before passing to
_apply_updates() so a subsequent save() failure cannot leave a
partially-mutated object in the storage cache.

Bug 3 (storage.py): add _cache_lock (threading.Lock) to
FileMemoryStorage and acquire it around every read/write of
_memory_cache, fixing concurrent-access races between the background
timer thread and HTTP reload calls.

Bug 4 (storage.py): replace in-place mutation
  memory_data["lastUpdated"] = ...
with a shallow copy
  memory_data = {**memory_data, "lastUpdated": ...}
so save() no longer silently modifies the caller's dict.

Regression tests added for all three bugs in test_memory_storage.py
and test_memory_updater.py.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* style: format test_memory_updater.py with ruff

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* style: remove stale bug-number labels from code comments and docstrings

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 12:00:31 +08:00
dependabot[bot] 259a6844bf chore(deps): bump python-multipart from 0.0.22 to 0.0.26 in /backend (#2282)
Bumps [python-multipart](https://github.com/Kludex/python-multipart) from 0.0.22 to 0.0.26.
- [Release notes](https://github.com/Kludex/python-multipart/releases)
- [Changelog](https://github.com/Kludex/python-multipart/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Kludex/python-multipart/compare/0.0.22...0.0.26)

---
updated-dependencies:
- dependency-name: python-multipart
  dependency-version: 0.0.26
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-16 09:07:28 +08:00
d 🔹 a664d2f5c4 fix(checkpointer): create parent directory before opening SQLite in sync provider (#2272)
* fix(checkpointer): create parent directory before opening SQLite in sync provider

The sync checkpointer factory (_sync_checkpointer_cm) opens a SQLite
connection without first ensuring the parent directory exists.  The async
provider and both store providers already call ensure_sqlite_parent_dir(),
but this call was missing from the sync path.

When the deer-flow harness package is used from an external virtualenv
(where the .deer-flow directory is not pre-created), the missing parent
directory causes:

    sqlite3.OperationalError: unable to open database file

Add the missing ensure_sqlite_parent_dir() call in the sync SQLite
branch, consistent with the async provider, and add a regression test.

Closes #2259

* style: fix ruff format + add call-order assertion for ensure_parent_dir

- Fix formatting in test_checkpointer.py (ruff format)
- Add test_sqlite_ensure_parent_dir_before_connect to verify
  ensure_sqlite_parent_dir is called before from_conn_string
  (addresses Copilot review suggestion)

---------

Co-authored-by: voidborne-d <voidborne-d@users.noreply.github.com>
2026-04-16 09:06:38 +08:00
YuJitang 105db00987 feat: show token usage per assistant response (#2270)
* feat: show token usage per assistant response

* fix: align client models response with token usage

* fix: address token usage review feedback

* docs: clarify token usage config example

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-16 08:56:49 +08:00
Nan Gao 0e16a7fe55 fix(frontend): make Suggestion button opaque in dark mode (#2276)
* fix(frontend): make Suggestion button opaque in dark mode

The outline Button variant applies dark:bg-input/30, leaving Suggestion
pills ~70% transparent in dark mode. Scrolled chat content bled through
the buttons, making suggestion text unreadable. Override with
dark:bg-background so it matches the opaque light-mode appearance.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix the lint error of commit

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-16 08:55:16 +08:00
Nan Gao 4d3038a7b6 fix(frontend): stop artifact panel from auto-opening on rehydrated write_file (#2278)
After a page refresh, the artifact panel's autoOpen/autoSelect state is
reset to true. Submitting a new question flips thread.isLoading to true,
which message-list passes to every MessageGroup — including historical
ones. The previous response's last write_file step then satisfies the
auto-open condition and re-pops the stale artifact.

Gate the auto-open on the tool call having no result yet, so only a
write_file that is still streaming in the current response can trigger
it; rehydrated tool calls always carry a result and are now skipped.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-16 08:46:47 +08:00
Hinotobi 2176b2bbfc fix: validate bootstrap agent names before filesystem writes (#2274)
* fix: validate bootstrap agent names before filesystem writes

* fix: tighten bootstrap agent-name validation
2026-04-16 08:36:42 +08:00
Wen 8e3591312a test: add unit tests for ViewImageMiddleware (#2256)
* test: add unit tests for ViewImageMiddleware

- Add 33 test cases covering all 7 internal methods plus sync/async
  before_model hooks
- Cover normal path, edge cases (missing keys, empty base64, stale
  ToolMessages before assistant turn), and deduplication logic
- Related to Q2 Roadmap #1669

* test: add unit tests for ViewImageMiddleware

Add 35 test cases covering all internal methods, before_model hooks,
and edge cases (missing attrs, list-content dedup, stale ToolMessages).

Related to #1669
2026-04-15 23:54:30 +08:00
Willem Jiang 242c654075 fix(frontend):lint error of message-list-item.tsx 2026-04-15 23:35:50 +08:00
Willem Jiang 0c21cbf01f fix(frontend): lint error of frontend 2026-04-15 23:27:46 +08:00
Jason 772538ddba fix(frontend): add skills API rewrite rule to prevent HTML fallback (#2241)
Fixes #2203

When NEXT_PUBLIC_BACKEND_BASE_URL is not set, the frontend uses Next.js
rewrites to proxy API calls to the gateway. Skills API routes were missing
from the rewrite config, causing /api/skills to return the SPA HTML instead
of JSON, which produced 'Unexpected token <' errors in the skill settings page.

Co-authored-by: JasonOA888 <JasonOA888@users.noreply.github.com>
2026-04-15 23:21:40 +08:00
Jason 35fb3dd65a fix(frontend): resolve /mnt/ links in markdown to artifact API URLs (#2243)
* fix(gateway): forward agent_name and is_bootstrap from context to configurable

The frontend sends agent_name and is_bootstrap via the context field
in run requests, but services.py only forwards a hardcoded whitelist
of keys (_CONTEXT_CONFIGURABLE_KEYS) into the agent's configurable
dict.  Since agent_name was missing, custom agents never received
their name — make_lead_agent always fell back to the default lead
agent, skipping SOUL.md, per-agent config and skill filtering.

Similarly, is_bootstrap was dropped, so the bootstrap creation flow
could never activate the setup_agent tool path.

Add both keys to the whitelist so they reach make_lead_agent.

Fixes #2222

* fix(frontend): resolve /mnt/ links in markdown to artifact API URLs

AI agent messages contain links like /mnt/user-data/outputs/file.pdf
which were rendered as-is in the browser, resulting in 404 errors.
Images already got the correct treatment via MessageImage and
resolveArtifactURL, but anchor tags (<a>) were passed through
unchanged.

Add an 'a' component override in MessageContent_ that rewrites
/mnt/-prefixed hrefs to the artifact API endpoint, matching the
existing image handling pattern.

Fixes #2232

---------

Co-authored-by: JasonOA888 <JasonOA888@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-15 23:12:21 +08:00
Jason 692f79452d fix(gateway): forward agent_name and is_bootstrap from context to configurable (#2242)
The frontend sends agent_name and is_bootstrap via the context field
in run requests, but services.py only forwards a hardcoded whitelist
of keys (_CONTEXT_CONFIGURABLE_KEYS) into the agent's configurable
dict.  Since agent_name was missing, custom agents never received
their name — make_lead_agent always fell back to the default lead
agent, skipping SOUL.md, per-agent config and skill filtering.

Similarly, is_bootstrap was dropped, so the bootstrap creation flow
could never activate the setup_agent tool path.

Add both keys to the whitelist so they reach make_lead_agent.

Fixes #2222

Co-authored-by: JasonOA888 <JasonOA888@users.noreply.github.com>
2026-04-15 23:11:10 +08:00
DanielWalnut 8760937439 fix(memory): use asyncio.to_thread for blocking file I/O in aupdate_memory (#2220)
* fix(memory): use asyncio.to_thread for blocking file I/O in aupdate_memory

`_finalize_update` performs synchronous blocking operations (os.mkdir,
file open/write/rename/stat) that were called directly from the async
`aupdate_memory` method, causing `BlockingError` from blockbuster when
running under an ASGI server. Wrap the call with `asyncio.to_thread` to
offload all blocking I/O to a thread pool.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(memory): use unique temp filename to prevent concurrent write collision

`file_path.with_suffix(".tmp")` produces a fixed path — concurrent saves
for the same agent (now possible after wrapping _finalize_update in
asyncio.to_thread) would clobber the same temp file. Use a UUID-suffixed
temp file so each write is isolated.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(memory): also offload _prepare_update_prompt to thread pool

FileMemoryStorage.load() inside _prepare_update_prompt performs
synchronous stat() and file read, blocking the event loop just like
_finalize_update did. Wrap _prepare_update_prompt in asyncio.to_thread
for the same reason.

The async path now has no blocking file I/O on the event loop:
  to_thread(_prepare_update_prompt) → await model.ainvoke() → to_thread(_finalize_update)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 16:41:54 +08:00
DanielWalnut 4ba3167f48 feat: flush memory before summarization (#2176)
* feat: flush memory before summarization

* fix: keep agent-scoped memory on summarization flush

* fix: harden summarization hook plumbing

* fix: address summarization review feedback

* style: format memory middleware
2026-04-14 15:01:06 +08:00
Octopus e4f896e90d fix(todo-middleware): prevent premature agent exit with incomplete todos (#2135)
* fix(todo-middleware): prevent premature agent exit with incomplete todos

When plan mode is active (is_plan_mode=True), the agent occasionally
exits the loop and outputs a final response while todo items are still
incomplete. This happens because the routing edge only checks for
tool_calls, not todo completion state.

Fixes #2112

Add an after_model override to TodoMiddleware with
@hook_config(can_jump_to=["model"]). When the model produces a
response with no tool calls but there are still incomplete todos, the
middleware injects a todo_completion_reminder HumanMessage and returns
jump_to=model to force another model turn. A cap of 2 reminders
prevents infinite loops when the agent cannot make further progress.

Also adds _completion_reminder_count() helper and 14 new unit tests
covering all edge cases of the new after_model / aafter_model logic.

* Remove unnecessary blank line in test file

* Fix runtime argument annotation in before_model

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: octo-patch <octo-patch@github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-14 11:11:26 +08:00
luo jiyin 07fc25d285 feat: switch memory updater to async LLM calls (#2138)
* docs: mark memory updater async migration as completed

- Update TODO.md to mark the replacement of sync model.invoke()
  with async model.ainvoke() in title_middleware and memory updater
  as completed using [x] format

Addresses #2131

* feat: switch memory updater to async LLM calls

- Add async aupdate_memory() method using await model.ainvoke()
- Convert sync update_memory() to use async wrapper
- Add _run_async_update_sync() for nested loop context handling
- Maintain backward compatibility with existing sync API
- Add ThreadPoolExecutor for async execution from sync contexts

Addresses #2131

* test: add tests for async memory updater

- Add test_async_update_memory_uses_ainvoke() to verify async path
- Convert existing tests to use AsyncMock and ainvoke assertions
- Add test_sync_update_memory_wrapper_works_in_running_loop()
- Update all model mocks to use async await patterns

Addresses #2131

* fix: apply ruff formatting to memory updater

- Format multi-line expressions to single line
- Ensure code style consistency with project standards
- Fix lint issues caught by GitHub Actions

* test: add comprehensive tests for async memory updater

- Add test_async_update_memory_uses_ainvoke() to verify async path
- Convert existing tests to use AsyncMock and ainvoke assertions
- Add test_sync_update_memory_wrapper_works_in_running_loop()
- Update all model mocks to use async await patterns
- Ensure backward compatibility with sync API

* fix: satisfy ruff formatting in memory updater test

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-14 11:10:42 +08:00
Nan Gao 55bc09ac33 fix(backend): fix uploads for mounted sandbox providers (#2199)
* fix uploads for mounted sandbox providers

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-04-14 10:44:31 +08:00
dependabot[bot] c43a45ea40 chore(deps): bump pillow from 12.1.1 to 12.2.0 in /backend (#2206)
Bumps [pillow](https://github.com/python-pillow/Pillow) from 12.1.1 to 12.2.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](https://github.com/python-pillow/Pillow/compare/12.1.1...12.2.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-version: 12.2.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-14 10:35:59 +08:00
Admire 9cf7153b1d fix(check): windows pnpm version detection in check script (#2189)
* fix: resolve Windows pnpm detection in check script

* style: format check script regression test

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix: resolve corepack fallback on windows

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-04-14 10:29:44 +08:00
Octopus c91785dd68 fix(title): strip <think> tags from title model responses and assistant context (#1927)
* fix(title): strip <think> tags from title model responses and assistant context

Reasoning models (e.g. minimax M2.7, DeepSeek-R1) emit <think>...</think>
blocks before their actual output. When such a model is used as the title
model (or as the main agent), the raw thinking content leaked into the thread
title stored in state, so the chat list showed the internal monologue instead
of a meaningful title.

Fixes #1884

- Add `_strip_think_tags()` helper using a regex to remove all <think>...</think> blocks
- Apply it in `_parse_title()` so the title model response is always clean
- Apply it to the assistant message in `_build_title_prompt()` so thinking
  content from the first AI turn is not fed back to the title model
- Add four new unit tests covering: stripping in parse, think-only response,
  assistant prompt stripping, and end-to-end async flow with think tags

* Fix the lint error

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-14 09:51:39 +08:00
sqsge 053e18e1a6 fix(skills): avoid blocking custom skill deletion on readonly history writes (#2197) 2026-04-14 09:00:29 +08:00
Hinotobi a7e7c6d667 fix: disable custom-agent management API by default (#2161)
* fix: disable custom-agent management API by default

* style: format agents API hardening files

* fix: address review feedback for agents API hardening

* fix: add missing disabled API coverage
2026-04-14 00:03:38 +08:00
Nan Gao f4c17c66ce fix(middleware): fix present_files thread id fallback (#2181)
* fix present files thread id fallback

* fix: resolve present_files thread id from runtime config
2026-04-13 22:59:13 +08:00
lesliewangwyc-dev 1df389b9d0 fix: wrap blocking readability call with asyncio.to_thread in web_fetch (#2157)
* fix: wrap blocking readability call with asyncio.to_thread in web_fetch

The readability extractor internally spawns a Node.js subprocess via
readabilipy, which blocks the async event loop and causes a
BlockingError when web_fetch is invoked inside LangGraph's async
runtime.

Wrap the synchronous extract_article call with asyncio.to_thread to
offload it to a thread pool, unblocking the event loop.

Note: community/infoquest/tools.py has the same latent issue and
should be addressed in a follow-up PR.

Closes #2152

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: verify web_fetch offloads extraction via asyncio.to_thread

Add a regression test that monkeypatches asyncio.to_thread to confirm
readability extraction is offloaded to a worker thread, preventing
future refactors from reintroducing the blocking call.

Addresses Copilot review feedback on #2157.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-13 21:15:24 +08:00
5db71cb68c fix(middleware): repair dangling tool-call history after loop interru… (#2035)
* fix(middleware): repair dangling tool-call history after loop interruption (#2029)

* docs(backend): fix middleware chain ordering

---------

Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com>
2026-04-12 19:11:22 +08:00
yangzheli 4efc8d404f feat(frontend): set up Vitest frontend testing infrastructure with CI workflow (#2147)
* feat: set up Vitest frontend testing infrastructure with CI workflow

Migrate existing 4 frontend test files from Node.js native test runner
(node:test + node:assert/strict) to Vitest, reorganize test directory
structure under tests/unit/ mirroring src/ layout, and add a dedicated
CI workflow for frontend unit tests.

- Add vitest as devDependency, remove tsx
- Create vitest.config.ts with @/ path alias
- Migrate tests to Vitest API (test/expect/vi)
- Rename .mjs test files to .ts
- Move tests from src/ to tests/unit/ (mirrors src/ layout)
- Add frontend/Makefile `test` target
- Add .github/workflows/frontend-unit-tests.yml (parallel to backend)
- Update CONTRIBUTING.md, README.md, AGENTS.md, CLAUDE.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: fix the lint error

* style: fix the lint error

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-12 18:00:43 +08:00
Jin 4d4ddb3d3f feat(llm): introduce lightweight circuit breaker to prevent rate-limit bans and resource exhaustion (#2095) 2026-04-12 17:48:40 +08:00
luo jiyin 979a461af5 docs: move completed async migration to Completed Features (#2146)
- Move time.sleep() -> asyncio.sleep() from Planned to Completed Features
- Clean up duplicate entries in TODO.md

Ensures completed async optimizations are properly tracked.
2026-04-12 16:48:48 +08:00
Javen Fang ac04f2704f feat(subagents): allow model override per subagent in config.yaml (#2064)
* feat(subagents): allow model override per subagent in config.yaml

Wire the existing SubagentConfig.model field to config.yaml so users
can assign different models to different subagent types.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(subagents): cover model override in SubagentsAppConfig + registry

Addresses review feedback on #2064:

- registry.py: update stale inline comment — the block now applies
  timeout, max_turns AND model overrides, not just timeout.
- test_subagent_timeout_config.py: add coverage for model override
  resolution across SubagentOverrideConfig, SubagentsAppConfig
  (get_model_for + load), and registry.get_subagent_config:
  - per-agent model override is applied to registry-returned config
  - omitted `model` keeps the builtin value
  - explicit `model: null` in config.yaml is equivalent to omission
  - model override on one agent does not affect other agents
  - model override preserves all other fields (name, description,
    timeout_seconds, max_turns)
  - model override does not mutate BUILTIN_SUBAGENTS

Copilot's suggestion (3) "setting model to 'inherit' forces inheritance"
is skipped intentionally: there is no 'inherit' sentinel in the current
implementation — model is `str | None`, and None already means
"inherit from parent". Adding a sentinel would be a new feature, not
test coverage for this PR.

Tests run locally: 51 passed (37 existing + 14 new / expanded).

* test(subagents): reject empty-string model at config load time

Addresses WillemJiang's review comment on #2064 (empty-string edge case):

- subagents_config.py: add `min_length=1` to the `model` field on
  SubagentOverrideConfig. `model: ""` in config.yaml would otherwise
  bypass the `is not None` check and reach create_chat_model(name="")
  as a confusing runtime error. This is symmetric with the existing
  `ge=1` guards on timeout_seconds / max_turns, so the validation style
  stays consistent across all three override fields.
- test_subagent_timeout_config.py: add test_rejects_empty_model
  mirroring the existing test_rejects_zero / test_rejects_negative
  cases; update the docstring on test_model_accepts_any_string (now
  test_model_accepts_any_non_empty_string) to reflect the new guard.

Not addressing the first comment (validating `model` against the
`models:` section at load time) in this PR. `SubagentsAppConfig` is
scoped to the `subagents:` block and cannot see the sibling `models:`
section, so proper cross-section validation needs a second pass or a
structural change that is out of scope here — and the current behavior
is consistent with how timeout_seconds / max_turns work today. Happy to
track this as a follow-up issue covering cross-section validation
uniformly for all three fields.

Tests run locally: 52 passed in this file; 1847 passed, 18 skipped
across the full backend suite. Ruff check + format clean.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 16:40:21 +08:00
Matt Van Horn c4d273a68a feat(channels): add Discord channel integration (#1806)
* feat(channels): add Discord channel integration

Add a Discord bot channel following the existing Telegram/Slack pattern.
The bot listens for messages, creates conversation threads, and relays
responses back to Discord with 2000-char message splitting.

- DiscordChannel extends Channel base class
- Lazy imports discord.py with install hint
- Thread-based conversations (each Discord thread maps to a DeerFlow thread)
- Allowed guilds filter for access control
- File attachment support via discord.File
- Registered in service.py and manager.py

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(channels): address Copilot review suggestions for Discord integration

- Disable @everyone/@here mentions via AllowedMentions.none()
- Add 10s timeout to client close to prevent shutdown hangs
- Log publish_inbound errors via future callback instead of silently dropping
- Open file handle on caller thread to avoid cross-thread ownership issues
- Notify user in channel when thread creation fails

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(discord): resolve lint errors in Discord channel

- Replace asyncio.TimeoutError with builtin TimeoutError (UP041)
- Remove extraneous f-string prefix (F541)
- Apply ruff format

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(tests): remove fake langgraph_sdk shim from test_discord_channel

The module-level sys.modules.setdefault shim installed a fake
langgraph_sdk.errors.ConflictError during pytest collection. Because
pytest imports all test modules before running them, test_channels.py
then imported the fake ConflictError instead of the real one.

In test_handle_feishu_stream_conflict_sends_busy_message, the test
constructs ConflictError(message, response=..., body=...). The fake
only subclasses Exception (which takes no kwargs), so the construction
raised TypeError. The manager's _is_thread_busy_error check then saw a
TypeError instead of a ConflictError and fell through to the generic
'An error occurred' message.

langgraph_sdk is a real dependency, so the shim is unnecessary.
Removing it makes both test files import the same real ConflictError
and the full suite pass (1773 passed, 15 skipped).

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-11 17:48:04 +08:00
Jason dc50a7fdfb fix(sandbox): resolve paths in read_file/write_file content for LocalSandbox (#1935)
* fix(sandbox): resolve paths in read_file/write_file content for LocalSandbox

In LocalSandbox mode, read_file and write_file now transform
container paths in file content, matching the path handling
behavior of bash tool.

- write_file: resolves virtual paths in content to system paths
  before writing, so scripts with /mnt/user-data paths work
  when executed
- read_file: reverse-resolves system paths back to virtual
  paths in returned content for consistency

This fixes scenarios where agents write Python scripts with
virtual paths, then execute them via bash tool expecting the
paths to work.

Fixes #1778

* fix(sandbox): address Copilot review — dedicated content resolver + forward-slash safety + tests

- Extract _resolve_paths_in_content() separate from _resolve_paths_in_command()
  to decouple file-content path resolution from shell-command parsing
- Normalize resolved paths to forward slashes to avoid Windows backslash
  escape issues in source files (e.g. \U in Python string literals)
- Add 4 focused tests: write resolves content, forward-slash guarantee,
  read reverse-resolves content, and write→read roundtrip

* style: fix ruff lint — remove extraneous f-string prefix

* fix(sandbox): only reverse-resolve paths in agent-written files

read_file previously applied _reverse_resolve_paths_in_output to ALL
file content, which could silently rewrite paths in user uploads and
external tool output (Willem Jiang review on #1935).

Now tracks files written through write_file in _agent_written_paths.
Only those files get reverse-resolved on read. Non-agent files are
returned as-is.

---------

Co-authored-by: JasonOA888 <JasonOA888@users.noreply.github.com>
2026-04-11 17:41:36 +08:00
ZHANG Ning 5b633449f8 fix(middleware): add per-tool-type frequency detection to LoopDetectionMiddleware (#1988)
* fix(middleware): add per-tool-type frequency detection to LoopDetectionMiddleware

The existing hash-based loop detection only catches identical tool call
sets. When the agent calls the same tool type (e.g. read_file) on many
different files, each call produces a unique hash and bypasses detection.
This causes the agent to exhaust recursion_limit, consuming 150K-225K
tokens per failed run.

Add a second detection layer that tracks cumulative call counts per tool
type per thread. Warns at 30 calls (configurable) and forces stop at 50.
The hard stop message now uses the actual returned message instead of a
hardcoded constant, so both hash-based and frequency-based stops produce
accurate diagnostics.

Also fix _apply() to use the warning message returned by
_track_and_check() for hard stops, instead of always using _HARD_STOP_MSG.

Closes #1987

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(lint): remove unused imports and fix line length

- Remove unused _TOOL_FREQ_HARD_STOP_MSG and _TOOL_FREQ_WARNING_MSG
  imports from test file (F401)
- Break long _TOOL_FREQ_WARNING_MSG string to fit within 240 char limit (E501)

* style: apply ruff format

* test: add LRU eviction and per-thread reset coverage for frequency state

Address review feedback from @WillemJiang:
- Verify _tool_freq and _tool_freq_warned are cleaned on LRU eviction
- Add test for reset(thread_id=...) clearing only the target thread's
  frequency state while leaving others intact

* fix(makefile): route Windows shell-script targets through Git Bash (#2060)

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Asish Kumar <87874775+officialasishkumar@users.noreply.github.com>
2026-04-11 17:33:27 +08:00
yorick 02569136df fix(sandbox): improve sandbox security and preserve multimodal content (#2114)
* fix: improve sandbox security and preserve multimodal content

* Add unit test modifications for test_injects_uploaded_files_tag_into_list_content

* format updated_content

* Add regression tests for multimodal upload content and host bash default safety
2026-04-11 16:52:10 +08:00
dependabot[bot] 024ac0e464 chore(deps): bump langsmith from 0.5.2 to 0.5.18 in /frontend (#2110)
Bumps [langsmith](https://github.com/langchain-ai/langsmith-sdk) from 0.5.2 to 0.5.18.
- [Release notes](https://github.com/langchain-ai/langsmith-sdk/releases)
- [Commits](https://github.com/langchain-ai/langsmith-sdk/commits)

---
updated-dependencies:
- dependency-name: langsmith
  dependency-version: 0.5.18
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-11 14:51:21 +08:00
dependabot[bot] 19030928e0 chore(deps): bump langchain-core from 1.2.17 to 1.2.28 in /backend (#2109)
Bumps [langchain-core](https://github.com/langchain-ai/langchain) from 1.2.17 to 1.2.28.
- [Release notes](https://github.com/langchain-ai/langchain/releases)
- [Commits](https://github.com/langchain-ai/langchain/compare/langchain-core==1.2.17...langchain-core==1.2.28)

---
updated-dependencies:
- dependency-name: langchain-core
  dependency-version: 1.2.28
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-11 14:49:54 +08:00
Asish Kumar 092bf13f5e fix(makefile): route Windows shell-script targets through Git Bash (#2060) 2026-04-11 09:30:22 +08:00
JeffJiang fe2595a05c Update CMD to run uvicorn with --no-sync option (#2100) 2026-04-10 23:00:00 +08:00
Jin 718dddde75 fix(sandbox): prevent memory leak in file operation locks using WeakValueDictionary (#2096)
* fix(sandbox): prevent memory leak in file operation locks using WeakValueDictionary

* lint: fix lint issue in sandbox tools security
2026-04-10 22:55:53 +08:00
Willem Jiang 679ca657ee Add Contributor Covenant Code of Conduct
Added Contributor Covenant Code of Conduct to ensure a respectful and inclusive community.
2026-04-10 22:26:40 +08:00
Zic-Wang fa96acdf4b feat: add WeChat channel integration (#1869)
* feat: add WeChat channel integration

* fix(backend): recover stale channel threads and align upload artifact handling

* refactor(wechat): reduce scope and restore QR bootstrap

* fix(backend): sort manager imports for Ruff lint

* fix(tests): add missing patch import in test_channels.py

* Update backend/app/channels/wechat.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update backend/app/channels/manager.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(wechat): streamline allowed file extensions initialization and clean up test file

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-10 20:49:28 +08:00
Willem Jiang 90299e2710 feat(provisioner): add optional PVC support for sandbox volumes (#2020)
* feat(provisioner): add optional PVC support for sandbox volumes (#1978)

  Add SKILLS_PVC_NAME and USERDATA_PVC_NAME env vars to allow sandbox
  Pods to use PersistentVolumeClaims instead of hostPath volumes. This
  prevents data loss in production when pods are rescheduled across nodes.

  When USERDATA_PVC_NAME is set, a subPath of threads/{thread_id}/user-data
  is used so a single PVC can serve multiple threads. Falls back to hostPath
  when the new env vars are not set, preserving backward compatibility.

* add unit test for provisioner pvc volumes

* refactor: extract shared provisioner_module fixture to conftest.py

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/e7ccf708-c6ba-40e4-844a-b526bdb249dd

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JeffJiang <for-eleven@hotmail.com>
2026-04-10 20:40:30 +08:00
JeffJiang 7dc0c7d01f feat(blog): implement blog structure with post listing, tagging, and layout enhancements (#1962)
* feat(blog): implement blog structure with post listing and tagging functionality

* feat(blog): enhance blog layout and post metadata display with new components

* fix(blog): address PR #1962 review feedback and fix lint issues (#14)

* fix: format

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
2026-04-10 20:24:52 +08:00
JeffJiang 809b341350 Add TypeScript SDK path to code-workspace settings (#2052)
* Add TypeScript SDK path to code-workspace settings

Agent-Logs-Url: https://github.com/foreleven/deer-flow/sessions/7d99db18-eb9d-4798-b0a5-b33f6079cd1a

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

* Update deer-flow.code-workspace

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-10 18:20:08 +08:00
greatmengqi b1aabe88b8 fix(backend): stream DeerFlowClient AI text as token deltas (#1969) (#1974)
* fix(backend): stream DeerFlowClient AI text as token deltas (#1969)

DeerFlowClient.stream() subscribed to LangGraph stream_mode=["values",
"custom"] which only delivers full-state snapshots at graph-node
boundaries, so AI replies were dumped as a single messages-tuple event
per node instead of streaming token-by-token. `client.stream("hello")`
looked identical to `client.chat("hello")` — the bug reported in #1969.

Subscribe to "messages" mode as well, forward AIMessageChunk deltas as
messages-tuple events with delta semantics (consumers accumulate by id),
and dedup the values-snapshot path so it does not re-synthesize AI
text that was already streamed. Introduce a per-id usage_metadata
counter so the final AIMessage in the values snapshot and the final
"messages" chunk — which carry the same cumulative usage — are not
double-counted.

chat() now accumulates per-id deltas and returns the last message's
full accumulated text. Non-streaming mock sources (single event per id)
are a degenerate case of the same logic, keeping existing callers and
tests backward compatible.

Verified end-to-end against a real LLM: a 15-number count emits 35
messages-tuple events with BPE subword boundaries clearly visible
("eleven" -> "ele" / "ven", "twelve" -> "tw" / "elve"), 476ms across
the window, end-event usage matches the values-snapshot usage exactly
(not doubled). tests/test_client_live.py::TestLiveStreaming passes.

New unit tests:
- test_messages_mode_emits_token_deltas: 3 AIMessageChunks produce 3
  delta events with correct content/id/usage, values-snapshot does not
  duplicate, usage counted once.
- test_chat_accumulates_streamed_deltas: chat() rebuilds full text
  from deltas.
- test_messages_mode_tool_message: ToolMessage delivered via messages
  mode is not duplicated by the values-snapshot synthesis path.

The stream() docstring now documents why this client does not reuse
Gateway's run_agent() / StreamBridge pipeline (sync vs async, raw
LangChain objects vs serialized dicts, single caller vs HTTP fan-out).

Fixes #1969

* refactor(backend): simplify DeerFlowClient streaming helpers (#1969)

Post-review cleanup for the token-level streaming fix. No behavior
change for correct inputs; one efficiency regression fixed.

Fix: chat() O(n²) accumulator
-----------------------------
`chat()` accumulated per-id text via `buffers[id] = buffers.get(id,"") + delta`,
which is O(n) per concat → O(n²) total over a streamed response. At
~2 KB cumulative text this becomes user-visible; at 50 KB / 5000 chunks
it costs roughly 100-300 ms of pure copying. Switched to
`dict[str, list[str]]` + `"".join()` once at return.

Cleanup
-------
- Extract `_serialize_tool_calls`, `_ai_text_event`, `_ai_tool_calls_event`,
  and `_tool_message_event` static helpers. The messages-mode and
  values-mode branches previously repeated four inline dict literals each;
  they now call the same builders.
- `StreamEvent.type` is now typed as `Literal["values", "messages-tuple",
  "custom", "end"]` via a `StreamEventType` alias. Makes the closed set
  explicit and catches typos at type-check time.
- Direct attribute access on `AIMessage`/`AIMessageChunk`: `.usage_metadata`,
  `.tool_calls`, `.id` all have default values on the base class, so the
  `getattr(..., None)` fallbacks were dead code. Removed from the hot
  path.
- `_account_usage` parameter type loosened to `Any` so that LangChain's
  `UsageMetadata` TypedDict is accepted under strict type checking.
- Trimmed narrating comments on `seen_ids` / `streamed_ids` / the
  values-synthesis skip block; kept the non-obvious ones that document
  the cross-mode dedup invariant.

Net diff: -15 lines. All 132 unit tests + harness boundary test still
pass; ruff check and ruff format pass.

* docs(backend): add STREAMING.md design note (#1969)

Dedicated design document for the token-level streaming architecture,
prompted by the bug investigation in #1969.

Contents:
- Why two parallel streaming paths exist (Gateway HTTP/async vs
  DeerFlowClient sync/in-process) and why they cannot be merged.
- LangGraph's three-layer mode naming (Graph "messages" vs Platform
  SDK "messages-tuple" vs HTTP SSE) and why a shared string constant
  would be harmful.
- Gateway path: run_agent + StreamBridge + sse_consumer with a
  sequence diagram.
- DeerFlowClient path: sync generator + direct yield, delta semantics,
  chat() accumulator.
- Why the three id sets (seen_ids / streamed_ids / counted_usage_ids)
  each carry an independent invariant and cannot be collapsed.
- End-to-end sequence for a real conversation turn.
- Lessons from #1969: why mock-based tests missed the bug, why
  BPE subword boundaries in live output are the strongest
  correctness signal, and the regression test that locks it in.
- Source code location index.

Also:
- Link from backend/CLAUDE.md Embedded Client section.
- Link from backend/docs/README.md under Feature Documentation.

* test(backend): add refactor regression guards for stream() (#1969)

Three new tests in TestStream that lock the contract introduced by
PR #1974 so any future refactor (sync->async migration, sharing a
core with Gateway's run_agent, dedup strategy change) cannot
silently change behavior.

- test_dedup_requires_messages_before_values_invariant: canary that
  documents the order-dependence of cross-mode dedup. streamed_ids
  is populated only by the messages branch, so values-before-messages
  for the same id produces duplicate AI text events. Real LangGraph
  never inverts this order, but a refactor that does (or that makes
  dedup idempotent) must update this test deliberately.

- test_messages_mode_golden_event_sequence: locks the *exact* event
  sequence (4 events: 2 messages-tuple deltas, 1 values snapshot, 1
  end) for a canonical streaming turn. List equality gives a clear
  diff on any drift in order, type, or payload shape.

- test_chat_accumulates_in_linear_time: perf canary for the O(n^2)
  fix in commit 1f11ba10. 10,000 single-char chunks must accumulate
  in under 1s; the threshold is wide enough to pass on slow CI but
  tight enough to fail if buffer = buffer + delta is restored.

All three tests pass alongside the existing 12 TestStream tests
(15/15). ruff check + ruff format clean.

* docs(backend): clarify stream() docstring on JSON serialization (#1969)

Replace the misleading "raw LangChain objects (AIMessage,
usage_metadata as dataclasses), not dicts" claim in the
"Why not reuse Gateway's run_agent?" section. The implementation
already yields plain Python dicts (StreamEvent.data is dict, and
usage_metadata is a TypedDict), so the original wording suggested
a richer return type than the API actually delivers.

The corrected wording focuses on what is actually true and
relevant: this client skips the JSON/SSE serialization layer that
Gateway adds for HTTP wire transmission, and yields stream event
payloads directly as Python data structures.

Addresses Copilot review feedback on PR #1974.

* test(backend): document none-id messages dedup limitation (#1969)

Add test_none_id_chunks_produce_duplicates_known_limitation to
TestStream that explicitly documents and asserts the current
behavior when an LLM provider emits AIMessageChunk with id=None
(vLLM, certain custom backends).

The cross-mode dedup machinery cannot record a None id in
streamed_ids (guarded by ``if msg_id:``), so the values snapshot's
reassembled AIMessage with a real id falls through and synthesizes
a duplicate AI text event. The test asserts len == 2 and locks
this as a known limitation rather than silently letting future
contributors hit it without context.

Why this is documented rather than fixed:
* Falling back to ``metadata.get("id")`` does not help — LangGraph's
  messages-mode metadata never carries the message id.
* Synthesizing ``f"_synth_{id(msg_chunk)}"`` only helps if the
  values snapshot uses the same fallback, which it does not.
* A real fix requires provider cooperation (always emit chunk ids)
  or content-based dedup (false-positive risk), neither of which
  belongs in this PR.

If a real fix lands, replace this test with a positive assertion
that dedup works for None-id chunks.

Addresses Copilot review feedback on PR #1974 (client.py:515).

* fix(frontend): UI polish - fix CSS typo, dark mode border, and hardcoded colors (#1942)

- Fix `font-norma` typo to `font-normal` in message-list subtask count
- Fix dark mode `--border` using reddish hue (22.216) instead of neutral
- Replace hardcoded `rgb(184,184,192)` in hero with `text-muted-foreground`
- Replace hardcoded `bg-[#a3a1a1]` in streaming indicator with `bg-muted-foreground`
- Add missing `font-sans` to welcome description `<pre>` for consistency
- Make case-study-section padding responsive (`px-4 md:px-20`)

Closes #1940

* docs: clarify deployment sizing guidance (#1963)

* fix(frontend): prevent stale 'new' thread ID from triggering 422 history requests (#1960)

After history.replaceState updates the URL from /chats/new to
/chats/{UUID}, Next.js useParams does not update because replaceState
bypasses the router. The useEffect in useThreadChat would then set
threadIdFromPath ('new') as the threadId, causing the LangGraph SDK
to call POST /threads/new/history which returns HTTP 422 (Invalid
thread ID: must be a UUID).

This fix adds a guard to skip the threadId update when
threadIdFromPath is the literal string 'new', preserving the
already-correct UUID that was set when the thread was created.

* fix(frontend): avoid using route new as thread id (#1967)

Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com>

* Fix(subagent): Event loop conflict in SubagentExecutor.execute() (#1965)

* Fix event loop conflict in SubagentExecutor.execute()

When SubagentExecutor.execute() is called from within an already-running
event loop (e.g., when the parent agent uses async/await), calling
asyncio.run() creates a new event loop that conflicts with asyncio
primitives (like httpx.AsyncClient) that were created in and bound to
the parent loop.

This fix detects if we're already in a running event loop, and if so,
runs the subagent in a separate thread with its own isolated event loop
to avoid conflicts.

Fixes: sub-task cards not appearing in Ultra mode when using async parent agents

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(subagent): harden isolated event loop execution

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(backend): remove dead getattr in _tool_message_event

---------

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
Co-authored-by: Xinmin Zeng <135568692+fancyboi999@users.noreply.github.com>
Co-authored-by: 13ernkastel <LennonCMJ@live.com>
Co-authored-by: siwuai <458372151@qq.com>
Co-authored-by: 肖 <168966994+luoxiao6645@users.noreply.github.com>
Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com>
Co-authored-by: Saber <11769524+hawkli-1994@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-10 18:16:38 +08:00
KKK 654354c624 test(skills): add evaluation + trigger analysis for systematic-literature-review (#2061)
* test(skills): add trigger eval set for systematic-literature-review skill

20 eval queries (10 should-trigger, 10 should-not-trigger) for use with
skill-creator's run_eval.py. Includes real-world SLR queries contributed
by @VANDRANKI (issue #1862 author) and edge cases for routing
disambiguation with academic-paper-review.

* test(skills): add grader expectations for SLR skill evaluation

5 eval cases with 39 expectations covering:
- Standard SLR flow (APA/BibTeX/IEEE format selection)
- Keyword extraction and search behavior
- Subagent dispatch for metadata extraction
- Report structure (themes, convergences, gaps, per-paper annotations)
- Negative case: single-paper routing to academic-paper-review
- Edge case: implicit SLR without explicit keywords

* refactor(skills): shorten SLR description for better trigger rate

Reduce description from 833 to 344 chars. Key changes:
- Lead with "systematic literature review" as primary trigger phrase
- Strengthen single-paper exclusion: "Not for single-paper tasks"
- Remove verbose example patterns that didn't improve routing

Tested with run_eval.py (10 runs/query):
- False positive "best paper on RL": 67% → 20% (improved)
- True positive explicit SLR query: ~30% (unchanged)

Low recall is a routing-layer limitation, not a description issue —
see PR description for full analysis.

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-04-10 18:02:45 +08:00
DanielWalnut eef0a6e2da feat(dx): Setup Wizard + doctor command — closes #2030 (#2034) 2026-04-10 17:43:39 +08:00
Javen Fang b107444878 docs(api): document recursion_limit for LangGraph API runs (#1929)
The /api/langgraph/* endpoints proxy straight to the LangGraph server,
so clients inherit LangGraph's native recursion_limit default of 25
instead of the 100 that build_run_config sets for the Gateway and IM
channel paths. 25 is too low for plan-mode or subagent runs and
reliably triggers GraphRecursionError on the lead agent's final
synthesis step after subagents return.

Set recursion_limit: 100 in the Create Run example and the cURL
snippet, and add a short note explaining the discrepancy so users
following the docs don't hit the 25-step ceiling as a surprise.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 09:28:57 +08:00
KKK 16aa51c9b3 feat(skills): add systematic-literature-review skill for multi-paper SLR workflows (#2032)
* feat(skills): add systematic-literature-review skill for multi-paper SLR workflows

Adds a new skill that produces a structured systematic literature review (SLR)
across multiple academic papers on a topic. Addresses #1862 with a pure skill
approach: no new tools, no architectural changes, no new dependencies.

Skill layout:
- SKILL.md — 4+1 phase workflow (plan, search, extract, synthesize, present)
- scripts/arxiv_search.py — arXiv API client, stdlib only, with a
  requests->urllib fallback shim modeled after github-deep-research's
  github_api.py
- templates/{apa,ieee,bibtex}.md — citation format templates selected
  dynamically in Phase 4, mirroring podcast-generation's templates/ pattern

Design notes:
- Multi-paper synthesis uses the existing `task` tool to dispatch extraction
  subagents in parallel. SKILL.md's Phase 3 includes a fixed decision table
  for batch splitting to respect the runtime's MAX_CONCURRENT_SUBAGENTS = 3
  cap, and explicitly tells the agent to strip the "Task Succeeded. Result: "
  prefix before parsing subagent JSON output.
- arXiv only, by design. Semantic Scholar and PubMed adapters would push the
  scope toward a standalone MCP server (see #933) and are intentionally out
  of scope for this skill.
- Coexists with the existing `academic-paper-review` skill: this skill does
  breadth-first synthesis across many papers, academic-paper-review does
  single-paper peer review. The two are routed via distinct triggers and
  can compose (SLR on many + deep review on 1-2 important ones).
- Hard upper bound of 50 papers, tied to the Phase 3 concurrency strategy.
  Larger surveys degrade in synthesis quality and are better split by
  sub-topic.

BibTeX template explicitly uses @misc for arXiv preprints (not @article),
which is the most common mistake when generating BibTeX for arXiv papers.

arxiv_search.py was smoke-tested end-to-end against the live arXiv API with
two query shapes (relevance sort, submittedDate sort with category filter);
all returned JSON fields parse correctly (id normalization, Atom namespace
handling, URL encoding for multi-word queries).

* fix(skills): prevent LLM from saving intermediate search results to file

Adds an explicit "do not save" instruction at the end of Phase 2.
Observed during Test 1 with DeepSeek: the model saved search results
to a markdown file before proceeding to Phase 3, wasting 2-3 tool call
rounds and increasing the risk of hitting the graph recursion limit.
The search JSON should stay in context for Phase 3, not be persisted.

* fix(skills): use relevance+start-date instead of submittedDate sorting

Test 2 revealed that arXiv's submittedDate sorting returns the most
recently submitted papers in the category regardless of query relevance.
Searching "diffusion models" with sortBy=submittedDate in cs.CV returned
papers on spatial memory, Navier-Stokes, and photon-counting CT — none
about diffusion models. The LLM then retried with 4 different queries,
wasting tool calls and approaching the recursion limit.

Fix: always sort by relevance; when the user wants "recent" papers,
combine relevance sorting with --start-date to constrain the time window.
Also add an explicit "run the search exactly once" instruction to prevent
the retry loop.

* fix(skills): wrap multi-word arXiv queries in double quotes for phrase matching

Without quotes, `all:diffusion model` is parsed by arXiv's Lucene as
`all:diffusion OR model`, pulling in unrelated papers from physics
(thermal diffusion) and other fields. Wrapping in double quotes forces
phrase matching: `all:"diffusion model"`.

Also fixes date filtering: the previous bug caused 2011 papers to appear
in results despite --start-date 2024-04-09, because the unquoted query
words were OR'd with the date constraint.

Verified: "diffusion models" --category cs.CV --start-date 2024-04-09
now returns only relevant diffusion model papers published after April
2024.

* fix(skills): add query phrasing guide and enforce subagent delegation

Two fixes from Test 2 observations with DeepSeek:

1. Query phrasing: add a table showing good vs bad query examples.
   The script wraps multi-word queries in double quotes for phrase
   matching, so long queries like "diffusion models in computer vision"
   return 0 results. Guide the LLM to use 2-3 core keywords + --category
   instead.

2. Subagent enforcement: DeepSeek was extracting metadata inline via
   python -c scripts instead of using the task tool. Strengthen Phase 3
   to explicitly name the task tool, say "do not extract metadata
   yourself", and explain why (token budget, isolation). This is more
   direct than the previous natural-language-only approach while still
   providing the reasoning behind the constraint.

* fix(skills): strengthen search keyword guidance and subagent enforcement

Address two issues found during end-to-end testing with DeepSeek:

1. Search retry: LLM passed full topic descriptions as queries (e.g.
   "diffusion models in computer vision"), which returned 0 results due
   to exact phrase matching and triggered retries. Added explicit
   instruction to extract 2-3 core keywords before searching.

2. Subagent bypass: LLM used python -c to extract metadata instead of
   dispatching via task tool. Added explicit prohibition list (python -c,
   bash scripts, inline extraction) with  markers for clarity.

* fix(skills): address Copilot review feedback on SLR skill

- Fix legacy arXiv ID parsing: preserve archive prefix for pre-2007
  papers (e.g. hep-th/9901001 instead of just 9901001)
- Fix phase count: "four phases" -> "five phases"
- Add subagent_enabled prerequisite note to SKILL.md Notes section
- Remove PR-specific references ("PR 1") from ieee.md and bibtex.md
  templates, replace with workflow-scoped wording
- Fix script header: "stdlib only" -> "no additional dependencies
  required", fix relative path to github_api.py reference
- Remove reference to non-existent docs/enhancement/ path in header

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-10 08:54:28 +08:00
Javen Fang 133ffe7174 feat(models): add langchain-ollama for native Ollama thinking support (#2062)
Add langchain-ollama as an optional dependency and provide ChatOllama
config examples, enabling proper thinking/reasoning content preservation
for local Ollama models.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 08:38:31 +08:00
yangzheli f88970985a fix(frontend): replace invalid "context" select field with "metadata" in threads.search (#2053)
* fix(frontend): replace invalid "context" select field with "metadata" in threads.search

The LangGraph API server does not support "context" as a select field for
threads/search, causing a 422 Unprocessable Entity error introduced by
commit 60e0abf (#1771).

- Replace "context" with "metadata" in the default select list
- Persist agent_name into thread metadata on creation so search results
  carry the agent identity
- Update pathOfThread() to fall back to metadata.agent_name when
  context is unavailable from search results
- Add regression tests for metadata-based agent routing

Fixes #2037

Made-with: Cursor

* fix: apply Copilot suggestions

* style: fix the lint error
2026-04-10 08:35:07 +08:00
knukn 6572fa5b75 feat(smoke-test): add smoke test skill (#1947)
* feat(smoke-test): add end-to-end smoke test skill

* Update .agent/skills/smoke-test/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update .agent/skills/smoke-test/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update .agent/skills/smoke-test/references/SOP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update .agent/skills/smoke-test/scripts/check_local_env.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update .agent/skills/smoke-test/scripts/check_docker.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update .agent/skills/smoke-test/scripts/deploy_docker.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* refactor(smoke-test): optimize health check scripts and update document structure

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-09 18:56:28 +08:00
shivam johri 194bab4691 feat(config): add when_thinking_disabled support for model configs (#1970)
* feat(config): add when_thinking_disabled support for model configs

Allow users to explicitly configure what parameters are sent to the
model when thinking is disabled, via a new `when_thinking_disabled`
field in model config. This mirrors the existing `when_thinking_enabled`
pattern and takes full precedence over the hardcoded disable behavior
when set. Backwards compatible — existing configs work unchanged.

Closes #1675

* fix(config): address copilot review — gate when_thinking_disabled independently

- Switch truthiness check to `is not None` so empty dict overrides work
- Restructure disable path so when_thinking_disabled is gated independently
  of has_thinking_settings, allowing it to work without when_thinking_enabled
- Update test to reflect new behavior
2026-04-09 18:49:00 +08:00
luo jiyin 35f141fc48 feat: implement full checkpoint rollback on user cancellation (#1867)
* feat: implement full checkpoint rollback on user cancellation

- Capture pre-run checkpoint snapshot including checkpoint state, metadata, and pending_writes
- Add _rollback_to_pre_run_checkpoint() function to restore thread state
- Implement _call_checkpointer_method() helper to support both async and sync checkpointer methods
- Rollback now properly restores checkpoint, metadata, channel_versions, and pending_writes
- Remove obsolete TODO comment (Phase 2) as rollback is now complete

This resolves the TODO(Phase 2) comment and enables full thread state
restoration when a run is cancelled by the user.

* fix: address rollback review feedback

* fix: strengthen checkpoint rollback validation and error handling

- Validate restored_config structure and checkpoint_id before use
- Raise RuntimeError on malformed pending_writes instead of silent skip
- Normalize None checkpoint_ns to empty string instead of "None"
- Move delete_thread to only execute when pre_run_snapshot is None
- Add docstring noting non-atomic rollback as known limitation

This addresses review feedback on PR #1867 regarding data integrity
in the checkpoint rollback implementation.

* test: add comprehensive coverage for checkpoint rollback edge cases

- test_rollback_restores_snapshot_without_deleting_thread
- test_rollback_deletes_thread_when_no_snapshot_exists
- test_rollback_raises_when_restore_config_has_no_checkpoint_id
- test_rollback_normalizes_none_checkpoint_ns_to_root_namespace
- test_rollback_raises_on_malformed_pending_write_not_a_tuple
- test_rollback_raises_on_malformed_pending_write_non_string_channel
- test_rollback_propagates_aput_writes_failure

Covers all scenarios from PR #1867 review feedback.

* test: format rollback worker tests
2026-04-09 17:56:36 +08:00
Xinmin Zeng 0b6fa8b9e1 fix(sandbox): add startup reconciliation to prevent orphaned container leaks (#1976)
* fix(sandbox): add startup reconciliation to prevent orphaned container leaks

Sandbox containers were never cleaned up when the managing process restarted,
because all lifecycle tracking lived in in-memory dictionaries. This adds
startup reconciliation that enumerates running containers via `docker ps` and
either destroys orphans (age > idle_timeout) or adopts them into the warm pool.

Closes #1972

* fix(sandbox): address Copilot review — adopt-all strategy, improved error handling

- Reconciliation now adopts all containers into warm pool unconditionally,
  letting the idle checker decide cleanup. Avoids destroying containers
  that another concurrent process may still be using.
- list_running() logs stderr on docker ps failure and catches
  FileNotFoundError/OSError.
- Signal handler test restores SIGTERM/SIGINT in addition to SIGHUP.
- E2E test docstring corrected to match actual coverage scope.

* fix(sandbox): address maintainer review — batch inspect, lock tightening, import hygiene

- _reconcile_orphans(): merge check-and-insert into a single lock acquisition
  per container to eliminate the TOCTOU window.
- list_running(): batch the per-container docker inspect into a single call.
  Total subprocess calls drop from 2N+1 to 2 (one ps + one batch inspect).
  Parse port and created_at from the inspect JSON payload.
- Extract _parse_docker_timestamp() and _extract_host_port() as module-level
  pure helpers and test them directly.
- Move datetime/json imports to module top level.
- _make_provider_for_reconciliation(): document the __new__ bypass and the
  lockstep coupling to AioSandboxProvider.__init__.
- Add assertion that list_running() makes exactly ONE inspect call.
2026-04-09 17:21:23 +08:00
Admire 140907ce1d Fix abnormal preview of HTML files (#1986)
* Fix HTML artifact preview rendering

* Add after screenshot for HTML preview fix

* Add before screenshot for HTML preview fix

* Update before screenshot for HTML preview fix

* Update after screenshot for HTML preview fix

* Update before screenshot to Tsinghua homepage repro

* Update after screenshot to Tsinghua homepage preview

* Address PR review on HTML artifact preview

* Harden HTML artifact preview isolation
2026-04-09 16:32:01 +08:00
yangzheli 52718b0f23 fix(frontend): disable incomplete markdown parsing for human messages (#2014)
Streamdown's streaming safeguard appends closing markers (e.g. `*`) to
text with unmatched markdown syntax. This causes user messages containing
literal `*` (such as `99 * 87`) to display with a spurious trailing
asterisk. Human messages are always complete, so the incomplete-markdown
pre-processing is unnecessary.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-09 16:30:32 +08:00
Admire 563383c60f fix(agent): file-io path guidance in agent prompts (#2019)
* fix(prompt): guide workspace-relative file io

* Clarify bash agent file IO path guidance
2026-04-09 16:12:34 +08:00
Xun 1b74d84590 fix: resolve missing serialized kwargs in PatchedChatDeepSeek (#2025)
* add tests

* fix ci

* fix ci
2026-04-09 16:07:16 +08:00
Zhou 823f3af98c fix(docker): dev uv cache mounts on macOS (#2036) 2026-04-09 15:59:33 +08:00
Gao Mingfei 13664e99e7 fix(docker): nginx fails to start on hosts without IPv6 (#2027)
* fix(docker): nginx fails to start on hosts without IPv6

- Detect IPv6 support at runtime and remove `listen [::]` directive
  when unavailable, preventing nginx startup failure on non-IPv6 hosts
- Use `exec` to replace shell with nginx as PID 1 for proper signal
  handling (graceful shutdown on SIGTERM)
- Reformat command from YAML folded scalar to block scalar (no
  functional change)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(docker): harden nginx startup script (Copilot review feedback)

Add `set -e` so envsubst failures exit immediately instead of starting
nginx with an incomplete config. Narrow the sed pattern to match only
the `listen [::]:2026;` directive to avoid accidentally removing future
lines containing [::].

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-09 15:58:30 +08:00
60e0abfdb8 fix(frontend): preserve agent context in thread history routes (#1771)
* fix(frontend): preserve agent context in thread history routes

* fix(frontend): preserve agent thread fallback context

* style(frontend): format thread route utils test

---------

Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com>
2026-04-09 15:11:57 +08:00
Octopus 616caa92b1 fix(models): resolve duplicate keyword argument error when reasoning_effort appears in both config and kwargs (#2017)
When a model config includes `reasoning_effort` as an extra YAML field
(ModelConfig uses `extra="allow"`), and the thinking-disabled code path
also injects `reasoning_effort="minimal"` into kwargs, the previous
`model_class(**kwargs, **model_settings_from_config)` call raises:

  TypeError: got multiple values for keyword argument 'reasoning_effort'

Fix by merging the two dicts before instantiation, giving runtime kwargs
precedence over config values: `{**model_settings_from_config, **kwargs}`.

Fixes #1977

Co-authored-by: octo-patch <octo-patch@github.com>
2026-04-09 15:09:39 +08:00
knukn 31a3c9a3de feat(client): add thread query methods list_threads and get_thread (#1609)
* feat(client): add thread query methods `list_threads` and `get_thread`

Implemented two public API methods in `DeerFlowClient` to query threads using the underlying `checkpointer`.

* Update backend/packages/harness/deerflow/client.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update backend/packages/harness/deerflow/client.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update backend/tests/test_client.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update backend/packages/harness/deerflow/client.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(deerflow): Fix possible KeyError issue when sorting threads

* fix unit test

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-09 15:00:22 +08:00
Xinmin Zeng ad6d934a5f fix(middleware): handle string-serialized options in ClarificationMiddleware (#1997)
* fix(middleware): handle string-serialized options in ClarificationMiddleware (#1995)

Some models (e.g. Qwen3-Max) serialize array tool parameters as JSON
strings instead of native arrays. Add defensive type checking in
_format_clarification_message() to deserialize string options before
iteration, preventing per-character rendering.

* fix(middleware): normalize options after JSON deserialization

Address Copilot review feedback:
- Add post-deserialization normalization so options is always a list
  (handles json.loads returning a scalar string, dict, or None)
- Add test for JSON-encoded scalar string ("development")
- Fix test_json_string_with_mixed_types to use actual mixed types
2026-04-08 21:04:20 +08:00
hung_ng__ 5350b2fb24 feat(community): add Exa search as community tool provider (#1357)
* feat(community): add Exa search as community tool provider

Add Exa (exa.ai) as a new community search provider alongside Tavily,
Firecrawl, InfoQuest, and Jina AI. Exa is an AI-native search engine
with neural, keyword, and auto search types.

New files:
- community/exa/tools.py: web_search_tool and web_fetch_tool
- tests/test_exa_tools.py: 10 unit tests with mocked Exa client

Changes:
- pyproject.toml: add exa-py dependency
- config.example.yaml: add commented-out Exa configuration examples

Usage: set `use: deerflow.community.exa.tools:web_search_tool` in
config.yaml and provide EXA_API_KEY.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(community): address PR review comments for Exa tools

- Make _get_exa_client() accept tool_name param so web_fetch reads its own config
- Remove __init__.py to match namespace package pattern of other providers
- Add duplicate tool name warning in config.example.yaml
- Add regression tests for web_fetch config resolution

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Update revision in uv.lock to 3

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-08 17:13:39 +08:00
Gao Mingfei 29817c3b34 fix(backend): use timezone-aware UTC in memory modules (fix pytest DeprecationWarnings) (#1992)
* fix(backend): use timezone-aware UTC in memory modules

Replace datetime.utcnow() with datetime.now(timezone.utc) and a shared
utc_now_iso_z() helper so persisted ISO timestamps keep the trailing Z
suffix without triggering Python 3.12+ deprecation warnings.

Made-with: Cursor

* refactor(backend): use removesuffix for utc_now_iso_z suffix

Makes the +00:00 -> Z transform explicit for the trailing offset only
(Copilot review on PR #1992).

Made-with: Cursor

* style(backend): satisfy ruff UP017 with datetime.UTC in memory queue

Made-with: Cursor

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-08 16:28:00 +08:00
Saber e5b149068c Fix(subagent): Event loop conflict in SubagentExecutor.execute() (#1965)
* Fix event loop conflict in SubagentExecutor.execute()

When SubagentExecutor.execute() is called from within an already-running
event loop (e.g., when the parent agent uses async/await), calling
asyncio.run() creates a new event loop that conflicts with asyncio
primitives (like httpx.AsyncClient) that were created in and bound to
the parent loop.

This fix detects if we're already in a running event loop, and if so,
runs the subagent in a separate thread with its own isolated event loop
to avoid conflicts.

Fixes: sub-task cards not appearing in Ultra mode when using async parent agents

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(subagent): harden isolated event loop execution

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-08 11:46:06 +08:00
85b7ed3cec fix(frontend): avoid using route new as thread id (#1967)
Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com>
2026-04-08 10:08:55 +08:00
siwuai 24805200f0 fix(frontend): prevent stale 'new' thread ID from triggering 422 history requests (#1960)
After history.replaceState updates the URL from /chats/new to
/chats/{UUID}, Next.js useParams does not update because replaceState
bypasses the router. The useEffect in useThreadChat would then set
threadIdFromPath ('new') as the threadId, causing the LangGraph SDK
to call POST /threads/new/history which returns HTTP 422 (Invalid
thread ID: must be a UUID).

This fix adds a guard to skip the threadId update when
threadIdFromPath is the literal string 'new', preserving the
already-correct UUID that was set when the thread was created.
2026-04-08 10:03:07 +08:00
13ernkastel 722a9c4753 docs: clarify deployment sizing guidance (#1963) 2026-04-08 09:45:31 +08:00
Xinmin Zeng d1baf7212b fix(frontend): UI polish - fix CSS typo, dark mode border, and hardcoded colors (#1942)
- Fix `font-norma` typo to `font-normal` in message-list subtask count
- Fix dark mode `--border` using reddish hue (22.216) instead of neutral
- Replace hardcoded `rgb(184,184,192)` in hero with `text-muted-foreground`
- Replace hardcoded `bg-[#a3a1a1]` in streaming indicator with `bg-muted-foreground`
- Add missing `font-sans` to welcome description `<pre>` for consistency
- Make case-study-section padding responsive (`px-4 md:px-20`)

Closes #1940
2026-04-08 09:07:39 +08:00
Async23 0948c7a4e1 fix(provider): preserve streamed Codex output when response.completed.output is empty (#1928)
* fix: preserve streamed Codex output items

* fix: prefer completed Codex output over streamed placeholders
2026-04-07 18:21:22 +08:00
koppx c3170f22da fix(backend): make loop detection hash tool calls by stable keys (#1911)
* fix(backend): make loop detection hash tool calls by stable keys

The loop detection middleware previously hashed full tool call arguments,
which made repeated calls look different when only non-essential argument
details changed. In particular, `read_file` calls with nearby line ranges
could bypass repetition detection even when the agent was effectively
reading the same file region again and again.

- Hash tool calls using stable keys instead of the full raw args payload
- Bucket `read_file` line ranges so nearby reads map to the same region key
- Prefer stable identifiers such as `path`, `url`, `query`, or `command`
  before falling back to JSON serialization of args
- Keep hashing order-independent so the same tool call set produces the
  same hash regardless of call order

Fixes #1905

* fix(backend): harden loop detection hash normalization

- Normalize and parse stringified tool args defensively
- Expand stable key derivation to include pattern, glob, and cmd
- Normalize reversed read_file ranges before bucketing

Fixes #1905

* fix(backend): harden loop detection tool format

* exclude write_file and str_replace from the stable-key path — writing different content to the same file shouldn't be flagged.

---------

Co-authored-by: JeffJiang <for-eleven@hotmail.com>
2026-04-07 17:46:33 +08:00
Anson Li 1193ac64dc fix(frontend): unify local settings runtime state and remove sidebar layout from LocalSettings (#1879)
* fix(frontend): resolve layout flickering by migrating workspace sidebar state to cookie

* fix(frontend): unify local settings runtime state to fix state drift

* fix(frontend): only persist thread model on explicit context model updates
2026-04-07 17:41:34 +08:00
Admire ab41de2961 fix(frontend):keep DeerFlow chat thread ids in sync (#1931)
* fix: replay thread sync changes on top of main

* fix: avoid stale thread ids during stream startup
2026-04-07 17:15:46 +08:00
KKK 3b3e8e1b0b feat(sandbox): strengthen bash command auditing with compound splitting and expanded patterns (#1881)
* fix(sandbox): strengthen regex coverage in SandboxAuditMiddleware

Expand high-risk patterns from 6 to 13 and medium-risk from 4 to 6,
closing several bypass vectors identified by cross-referencing Claude
Code's BashSecurity validator chain against DeerFlow's threat model.

High-risk additions:
- Generalised pipe-to-sh (replaces narrow curl|sh rule)
- Targeted command substitution ($() / backtick with dangerous executables)
- base64 decode piped to execution
- Overwrite system binaries (/usr/bin/, /bin/, /sbin/)
- Overwrite shell startup files (~/.bashrc, ~/.profile, etc.)
- /proc/*/environ leakage
- LD_PRELOAD / LD_LIBRARY_PATH hijack
- /dev/tcp/ bash built-in networking

Medium-risk additions:
- sudo/su (no-op under Docker root, warn only)
- PATH= modification (long attack chain, warn only)

Design decisions:
- Command substitution uses targeted matching (curl/wget/bash/sh/python/
  ruby/perl/base64) rather than blanket block to avoid false positives
  on safe usage like $(date) or `whoami`.
- Skipped encoding/obfuscation checks (hex, octal, Unicode homoglyphs)
  as ROI is low in Docker sandbox — LLMs don't generate encoded commands
  and container isolation bounds the blast radius.
- Merged pip/pip3 into single pip3? pattern.

* feat(sandbox): compound command splitting and fork bomb detection

Split compound bash commands (&&, ||, ;) into sub-commands and classify
each independently — prevents dangerous commands hidden after safe
prefixes (e.g. "cd /workspace && rm -rf /") from bypassing detection.

- Add _split_compound_command() with shlex quote-aware splitting
- Add fork bomb detection patterns (classic and while-loop variants)
- Most severe verdict wins; block short-circuits
- 15 new tests covering compound commands, splitting, and fork bombs

* test(sandbox): add async tests for fork bomb and compound commands

Cover awrap_tool_call path for fork bomb detection (3 variants) and
compound command splitting (block/warn/pass scenarios).

* fix(sandbox): address Copilot review — no-whitespace operators, >>/etc/, whole-command scan

- _split_compound_command: replace shlex-based implementation with a
  character-by-character quote/escape-aware scanner. shlex.split only
  separates '&&' / '||' / ';' when they are surrounded by whitespace,
  so payloads like 'rm -rf /&&echo ok' or 'safe;rm -rf /' bypassed the
  previous splitter and therefore the per-sub-command classifier.
- _HIGH_RISK_PATTERNS: change r'>\s*/etc/' to r'>+\s*/etc/' so append
  redirection ('>>/etc/hosts') is also blocked.
- _classify_command: run a whole-command high-risk scan *before*
  splitting. Structural attacks like 'while true; do bash & done'
  span multiple shell statements — splitting on ';' destroys the
  pattern context, so the raw command must be scanned first.
- tests: add no-whitespace operator cases to TestSplitCompoundCommand
  and test_compound_command_classification to lock in the bypass fix.
2026-04-07 17:15:24 +08:00
Admire 4004fb849f Fix agent gallery after bootstrap creation 修复新建智能体后菜单仍为空的问题 (#1934)
* fix: persist agent before bootstrap chat

* style: normalize line endings for agent creation page

* fix: address review feedback for agent creation flow

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-07 17:10:08 +08:00
Henry Li f467e613b6 feat: add BytePlus logo (#1948) 2026-04-07 16:07:37 +08:00
lulusiyuyu f0dd8cb0d2 fix(subagents): add cooperative cancellation for subagent threads (#1873)
* fix(subagents): add cooperative cancellation for subagent threads

Subagent tasks run inside ThreadPoolExecutor threads with their own
event loop (asyncio.run). When a user clicks stop, RunManager cancels
the parent asyncio.Task, but Future.cancel() cannot terminate a running
thread and asyncio.Event does not propagate across event loops. This
causes subagent threads to keep executing (writing files, calling LLMs)
even after the user explicitly stops the run.

Fix: add a threading.Event (cancel_event) to SubagentResult and check
it cooperatively in _aexecute()'s astream iteration loop. On cancel,
request_cancel_background_task() sets the event, and the thread exits
at the next iteration boundary.

Changes:
- executor.py: Add cancel_event field to SubagentResult, check it in
  _aexecute loop, set it on timeout, add request_cancel_background_task
- task_tool.py: Call request_cancel_background_task on CancelledError

* fix(subagents): guard cancel status and add pre-check before astream

- Only overwrite status to FAILED when still RUNNING, preserving
  TIMED_OUT set by the scheduler thread.
- Add cancel_event pre-check before entering the astream loop so
  cancellation is detected immediately when already signalled.

* fix(subagents): guard status updates with lock to prevent race condition

Wrap the check-and-set on result.status in _aexecute with
_background_tasks_lock so the timeout handler in execute_async
cannot interleave between the read and write.

* fix(subagents): add dedicated CANCELLED status for user cancellation

Introduce SubagentStatus.CANCELLED to distinguish user-initiated
cancellation from actual execution failures.  Update _aexecute,
task_tool polling, cleanup terminal-status sets, and test fixtures.

* test(subagents): add cancellation tests and fix timeout regression test

- Add dedicated TestCooperativeCancellation test class with 6 tests:
  - Pre-set cancel_event prevents astream from starting
  - Mid-stream cancel_event returns CANCELLED immediately
  - request_cancel_background_task() sets cancel_event correctly
  - request_cancel on nonexistent task is a no-op
  - Real execute_async timeout does not overwrite CANCELLED (deterministic
    threading.Event sync, no wall-clock sleeps)
  - cleanup_background_task removes CANCELLED tasks

- Add task_tool cancellation coverage:
  - test_cancellation_calls_request_cancel: assert CancelledError path
    calls request_cancel_background_task(task_id)
  - test_task_tool_returns_cancelled_message: assert CANCELLED polling
    branch emits task_cancelled event and returns expected message

- Fix pre-existing test infrastructure issue: add deerflow.sandbox.security
  to _MOCKED_MODULE_NAMES (fixes ModuleNotFoundError for all executor tests)

- Add RUNNING guard to timeout handler in executor.py to prevent
  TIMED_OUT from overwriting CANCELLED status

- Add cooperative cancellation granularity comment documenting that
  cancellation is only detected at astream iteration boundaries

---------

Co-authored-by: lulusiyuyu <lulusiyuyu@users.noreply.github.com>
2026-04-07 11:12:25 +08:00
DanielWalnut 7643a46fca fix(skill): make skill prompt cache refresh nonblocking (#1924)
* fix: make skill prompt cache refresh nonblocking

* fix: harden skills prompt cache refresh

* chore: add timeout to skills cache warm-up
2026-04-07 10:50:34 +08:00
Markus Corazzione c4da0e8ca9 Move async SQLite mkdir off the event loop (#1921)
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
2026-04-07 10:47:20 +08:00
yangzheli 3acdf79beb fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities (#1904)
* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities

Fix `<button>` inside `<a>` invalid HTML in artifact components and add
missing `noopener,noreferrer` to `window.open` calls to prevent reverse
tabnabbing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(frontend): address Copilot review on tabnabbing and double-tab-open

Remove redundant parent onClick on web_fetch ChainOfThoughtStep to
prevent opening two tabs on link click, and explicitly null out
window.opener after window.open() for defensive tabnabbing hardening.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 09:44:17 +08:00
jie 2d068cc075 fix(docker): restore gateway env vars and fix langgraph empty arg issue (#1915)
Two production docker-compose.yaml bugs prevent `make up` from working:

1. Gateway missing DEER_FLOW_CONFIG_PATH and DEER_FLOW_EXTENSIONS_CONFIG_PATH
   environment overrides. Added in fb2d99f (#1836) but accidentally reverted
   by ca2fb95 (#1847). Without them, gateway reads host paths from .env via
   env_file, causing FileNotFoundError inside the container.

2. Langgraph command fails when LANGGRAPH_ALLOW_BLOCKING is unset (default).
   Empty $${allow_blocking} inserts a bare space between flags, causing
   ' --no-reload' to be parsed as unexpected extra argument. Fix by building
   args string first and conditionally appending --allow-blocking.

Co-authored-by: cooper <cooperfu@tencent.com>
2026-04-07 08:54:44 +08:00
JilongSun 88e535269e Feature/feishu receive file (#1608)
* feat(feishu): add channel file materialization hook for inbound messages

- Introduce Channel.receive_file(msg, thread_id) as a base method for file materialization; default is no-op.
- Implement FeishuChannel.receive_file to download files/images from Feishu messages, save to sandbox, and inject virtual paths into msg.text.
- Update ChannelManager to call receive_file for any channel if msg.files is present, enabling downstream model access to user-uploaded files.
- No impact on Slack/Telegram or other channels (they inherit the default no-op).

* style(backend): format code with ruff for lint compliance

- Auto-formatted packages/harness/deerflow/agents/factory.py and tests/test_create_deerflow_agent.py using `ruff format`
- Ensured both files conform to project linting standards
- Fixes CI lint check failures caused by code style issues

* fix(feishu): handle file write operation asynchronously to prevent blocking

* fix(feishu): rename GetMessageResourceRequest to _GetMessageResourceRequest and remove redundant code

* test(feishu): add tests for receive_file method and placeholder replacement

* fix(manager): remove unnecessary type casting for channel retrieval

* fix(feishu): update logging messages to reflect resource handling instead of image

* fix(feishu): sanitize filename by replacing invalid characters in file uploads

* fix(feishu): improve filename sanitization and reorder image key handling in message processing

* fix(feishu): add thread lock to prevent filename conflicts during file downloads

* fix(test): correct bad merge in test_feishu_parser.py

* chore: run ruff and apply formatting cleanup
fix(feishu): preserve rich-text attachment order and improve fallback filename handling
2026-04-06 22:14:12 +08:00
DanielWalnut 888f7bfb9d Implement skill self-evolution and skill_manage flow (#1874)
* chore: ignore .worktrees directory

* Add skill_manage self-evolution flow

* Fix CI regressions for skill_manage

* Address PR review feedback for skill evolution

* fix(skill-evolution): preserve history on delete

* fix(skill-evolution): tighten scanner fallbacks

* docs: add skill_manage e2e evidence screenshot

* fix(skill-manage): avoid blocking fs ops in session runtime

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-06 22:07:11 +08:00
KKK 055e4df049 fix(sandbox): add input sanitisation guard to SandboxAuditMiddleware (#1872)
* fix(sandbox): add L2 input sanitisation to SandboxAuditMiddleware

Add _validate_input() to reject malformed bash commands before regex
classification: empty commands, oversized commands (>10 000 chars), and
null bytes that could cause detection/execution layer inconsistency.

* fix(sandbox): address Copilot review — type guard, log truncation, reject reason

- Coerce None/non-string command to str before validation
- Truncate oversized commands in audit logs to prevent log amplification
- Propagate reject_reason through _pre_process() to block message
- Remove L2 label from comments and test class names

* fix(sandbox): isinstance type guard + async input sanitisation tests

Address review comments:
- Replace str() coercion with isinstance(raw_command, str) guard so
  non-string truthy values (0, [], False) fall back to empty string
  instead of passing validation as "0"/"[]"/"False".
- Add TestInputSanitisationBlocksInAwrapToolCall with 4 async tests
  covering empty, null-byte, oversized, and None command via
  awrap_tool_call path.
2026-04-06 17:21:58 +08:00
Zhou 1ced6e977c fix(backend): preserve viewed image reducer metadata (#1900)
Fix concurrent viewed_images state updates for multi-image input by preserving the reducer metadata in the vision middleware state schema.
2026-04-06 16:47:19 +08:00
Zhou f5088ed70d fix(frontend): artifact download action bounds and lint errors (#1899)
* fix: keep artifact download action in bounds

* fix: fix lint error
2026-04-06 16:34:40 +08:00
Zhou 55e78de6fc fix: wrap suggestion chips without overlapping input (#1895)
* fix: wrap suggestion chips without overlapping input

* fix: fix lint error
2026-04-06 16:30:57 +08:00
NmanQAQ dd30e609f7 feat(models): add vLLM provider support (#1860)
support for vLLM 0.19.0 OpenAI-compatible chat endpoints and fixes the Qwen reasoning toggle so flash mode can actually disable thinking.

Co-authored-by: NmanQAQ <normangyao@qq.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-06 15:18:34 +08:00
yangzheli 5fd2c581f6 fix: add output truncation to ls_tool to prevent context window overflow (#1896)
ls_tool was the only sandbox tool without output size limits, allowing
multi-MB results from large directories to blow up the model context
window. Add head-truncation (configurable via ls_output_max_chars,
default 20000) consistent with existing bash and read_file truncation.

Closes #1887

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-06 15:09:57 +08:00
Chincherry93 d7a3eff23e fix(docker): command syntax for LANGGRAPH_ALLOW_BLOCKING (#1891) 2026-04-06 15:02:29 +08:00
qqwas ee06440205 fix(frontend): Update route.ts default backend port(#1892) 2026-04-06 14:54:50 +08:00
7c68dd4ad4 Fix(#1702): stream resume run (#1858)
* fix: repair stream resume run metadata

# Conflicts:
#	backend/packages/harness/deerflow/runtime/stream_bridge/memory.py
#	frontend/src/core/threads/hooks.ts

* fix(stream): repair resumable replay validation

---------

Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-06 14:51:10 +08:00
suyua9 29575c32f9 fix: expose custom events from DeerFlowClient.stream() (#1827)
* fix: expose custom client stream events

Signed-off-by: suyua9 <1521777066@qq.com>

* fix(client): normalize streamed custom mode values

* test(client): satisfy backend ruff import ordering

---------

Signed-off-by: suyua9 <1521777066@qq.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-06 10:09:39 +08:00
amonduuuul ed90a2ee9d fix(docker): recover invalid .venv to prevent startup restart loops (#1871)
* fix(docker): recover invalid .venv before service startup

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-06 08:34:25 +08:00
Willem Jiang 993fb0ff9d fix: escape shell variables in production langgraph command (#1877) (#1880)
Escape  shell variables to prevent Docker Compose from attempting
substitution at parse time. Rename allow_blocking_flag to allow_blocking
for consistency with dev version.

Fixes the 'allow_blocking_flag not set' warning and enables --allow-blocking
flag to work correctly.
2026-04-06 08:24:51 +08:00
greatmengqi ca2fb95ee6 feat: unified serve.sh with gateway mode support (#1847) 2026-04-05 21:07:35 +08:00
Chris Z 117fa9b05d fix(channels): normalize slack allowed user ids (#1802)
* fix(channels): normalize slack allowed user ids

* style(channels): apply backend formatter

---------

Co-authored-by: haimingZZ <15558128926@qq.com>
Co-authored-by: suyua9 <1521777066@qq.com>
2026-04-05 18:04:21 +08:00
28474c47cb fix: avoid command palette hydration mismatch on macOS (#1563)
# Conflicts:
#	frontend/src/components/workspace/command-palette.tsx

Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-05 16:35:33 +08:00
thefoolgy 8049785de6 fix(memory): case-insensitive fact deduplication and positive reinforcement detection (#1804)
* fix(memory): case-insensitive fact deduplication and positive reinforcement detection

Two fixes to the memory system:

1. _fact_content_key() now lowercases content before comparison, preventing
   semantically duplicate facts like "User prefers Python" and "user prefers
   python" from being stored separately.

2. Adds detect_reinforcement() to MemoryMiddleware (closes #1719), mirroring
   detect_correction(). When users signal approval ("yes exactly", "perfect",
   "完全正确", etc.), the memory updater now receives reinforcement_detected=True
   and injects a hint prompting the LLM to record confirmed preferences and
   behaviors with high confidence.

   Changes across the full signal path:
   - memory_middleware.py: _REINFORCEMENT_PATTERNS + detect_reinforcement()
   - queue.py: reinforcement_detected field in ConversationContext and add()
   - updater.py: reinforcement_detected param in update_memory() and
     update_memory_from_conversation(); builds reinforcement_hint alongside
     the existing correction_hint

Tests: 11 new tests covering deduplication, hint injection, and signal
detection (Chinese + English patterns, window boundary, conflict with correction).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(memory): address Copilot review comments on reinforcement detection

- Tighten _REINFORCEMENT_PATTERNS: remove 很好, require punctuation/end-of-string boundaries on remaining patterns, split this-is-good into stricter variants
- Suppress reinforcement_detected when correction_detected is true to avoid mixed-signal noise
- Use casefold() instead of lower() for Unicode-aware fact deduplication
- Add missing test coverage for reinforcement_detected OR merge and forwarding in queue

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-05 16:23:00 +08:00
Evan Wu 9ca68ffaaa fix: preserve virtual path separator style (#1828)
* fix: preserve virtual path separator style

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-05 15:52:22 +08:00
Markus Corazzione 0ffe5a73c1 chroe(config):Increase subagent max-turn limits (#1852) 2026-04-05 15:41:00 +08:00
Echo-Nie d3b59a7931 docs: fix some broken links (#1864)
* Rename BACKEND_TODO.md to TODO.md in documentation

* Update MCP Setup Guide link in CONTRIBUTING.md

* Update reference to config.yaml path in documentation

* Fix config file path in TITLE_GENERATION_IMPLEMENTATION.md

Updated the path to the example config file in the documentation.
2026-04-05 15:35:42 +08:00
yangzheli e5416b539a fix(docker): use multi-stage build to remove build-essential from runtime image (#1846)
* fix(docker): use multi-stage build to remove build-essential from runtime image

The build-essential toolchain (~200 MB) was only needed for compiling
native Python extensions during `uv sync` but remained in the final
image, increasing size and attack surface. Split the Dockerfile into
a builder stage (with build-essential) and a clean runtime stage that
copies only the compiled artifacts, Node.js, Docker CLI, and uv.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(docker): add dev stage and pin docker:cli per review feedback

Address Copilot review comments:
- Add a `dev` build stage (FROM builder) that retains build-essential
  so startup-time `uv sync` in dev containers can compile from source
- Update docker-compose-dev.yaml to use `target: dev` for gateway and
  langgraph services
- Keep the clean runtime stage (no build-essential) as the default
  final stage for production builds

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-05 15:30:34 +08:00
SHIYAO ZHANG 72d4347adb fix(sandbox): guard against None runtime.context in sandbox tool helpers (#1853)
sandbox_from_runtime() and ensure_sandbox_initialized() write
sandbox_id into runtime.context after acquiring a sandbox. When
lazy_init=True and no context is supplied to the graph run,
runtime.context is None (the LangGraph default), causing a TypeError
on the assignment.

Add `if runtime.context is not None` guards at all three write sites.
Reads already had equivalent guards (e.g. `runtime.context.get(...) if
runtime.context else None`); this brings writes into line.
2026-04-05 10:58:38 +08:00
Octopus a283d4a02d fix: include soul field in GET /api/agents list response (fixes #1819) (#1863)
Previously, the list endpoint always returned soul=null because
_agent_config_to_response() was called without include_soul=True.
This caused confusion since PUT /api/agents/{name} and GET /api/agents/{name}
both returned the soul content, but the list endpoint silently omitted it.

Co-authored-by: octo-patch <octo-patch@users.noreply.github.com>
2026-04-05 10:49:58 +08:00
yangzheli 5f8dac66e6 chore(deps): update uv.lock (#1848)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-05 10:22:14 +08:00
Adem Akdoğan 8bb14fa1a7 feat(skills): add academic-paper-review, code-documentation, and newsletter-generation skills (#1861)
Add three new public skills to enhance DeerFlow's content creation capabilities:

- **academic-paper-review**: Structured peer-review-quality analysis of
  research papers following top-venue review standards (NeurIPS, ICML, ACL).
  Covers methodology assessment, contribution evaluation, literature
  positioning, and constructive feedback with a 3-phase workflow.

- **code-documentation**: Professional documentation generation for software
  projects, including README generation, API reference docs, architecture
  documentation with Mermaid diagrams, and inline code documentation
  supporting Python, TypeScript, Go, Rust, and Java conventions.

- **newsletter-generation**: Curated newsletter creation with research
  workflow, supporting daily digest, weekly roundup, deep-dive, and industry
  briefing formats. Includes audience-specific tone adaptation and
  multi-source content curation.

All skills:
- Follow the existing SKILL.md frontmatter convention (name + description)
- Pass the official _validate_skill_frontmatter() validation
- Use hyphen-case naming consistent with existing skills
- Contain only allowed frontmatter properties
- Include comprehensive examples, quality checklists, and output templates
2026-04-05 10:19:35 +08:00
DanielWalnut 2a150f5d4a fix: unblock concurrent threads and workspace hydration (#1839)
* fix: unblock concurrent threads and workspace hydration

* fix: restore async title generation

* fix: address PR review feedback

* style: format lead agent prompt
2026-04-04 21:19:35 +08:00
luobo 1c0051c1db fix(frontend): keep prompt attachments from breaking before upload (#1833)
* fix(frontend): preserve prompt attachment files during upload

* fix(frontend): harden prompt attachment fallback and tests

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-04 14:54:35 +08:00
luobo 144c9b2464 fix(frontend): block unsupported .app uploads (#1834)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-04 14:42:26 +08:00
SHIYAO ZHANG 163121d327 fix(uploads): handle split-bold headings and ** ** artefacts in extract_outline (#1838)
* feat(uploads): guide agent to use grep/glob/read_file for uploaded documents

Add workflow guidance to the <uploaded_files> context block so the agent
knows to use grep and glob (added in #1784) alongside read_file when
working with uploaded documents, rather than falling back to web search.

This is the final piece of the three-PR PDF agentic search pipeline:
- PR1 (#1727): pymupdf4llm converter produces structured Markdown with headings
- PR2 (#1738): document outline injected into agent context with line numbers
- PR3 (this):  agent guided to use outline + grep + read_file workflow

* feat(uploads): add file-first priority and fallback guidance to uploaded_files context

* fix(uploads): handle split-bold headings and ** ** artefacts in extract_outline

- Add _clean_bold_title() to merge adjacent bold spans (** **) produced
  by pymupdf4llm when bold text crosses span boundaries
- Add _SPLIT_BOLD_HEADING_RE (Style 3) to recognise **<num>** **<title>**
  headings common in academic papers; excludes pure-number table headers
  and rows with more than 4 bold blocks
- When outline is empty, read first 5 non-empty lines of the .md as a
  content preview and surface a grep hint in the agent context
- Update _format_file_entry to render the preview + grep hint instead of
  silently omitting the outline section
- Add 3 new extract_outline tests and 2 new middleware tests (65 total)

* fix(uploads): address Copilot review comments on extract_outline regex

- Replace ASCII [A-Za-z] guard with negative lookahead to support non-ASCII
  titles (e.g. **1** **概述**); pure-numeric/punctuation blocks still excluded
- Replace .+ with [^*]+ and cap repetition at {0,2} (four blocks total) to
  keep _SPLIT_BOLD_HEADING_RE linear and avoid ReDoS on malformed input
- Remove now-redundant len(blocks) <= 4 code-level check (enforced by regex)
- Log debug message with exc_info when preview extraction fails
2026-04-04 14:25:08 +08:00
fengxsong 19809800f1 feat: support wecom channel (#1390)
* feat: support wecom channel

* fix: sending file to client

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* test: add unit tests for wecom channel

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* docs: add example configs and setup docs

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* revert pypi default index setting

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* revert: keeping codes in harness untouched

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* fix: format issue

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

* fix: resolve Copilot comments

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>

---------

Signed-off-by: fengxusong <7008971+fengxsong@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-04 11:28:35 +08:00
Albert Zheng 6473d38917 fix(frontend): resolve button hydration mismatch with undefined variant/size (#1506)
Server-rendered data-variant={undefined} didn't match client hydration.
Now only render data-variant and data-size when explicitly set.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: JeffJiang <for-eleven@hotmail.com>
2026-04-04 11:21:04 +08:00
luobo 4ceb18c6e4 fix: use webpack for local frontend dev in serve.sh (#1832) 2026-04-04 11:12:25 +08:00
SHIYAO ZHANG bbd0866374 feat(uploads): guide agent using agentic search for uploaded documents (#1816)
* feat(uploads): guide agent to use grep/glob/read_file for uploaded documents

Add workflow guidance to the <uploaded_files> context block so the agent
knows to use grep and glob (added in #1784) alongside read_file when
working with uploaded documents, rather than falling back to web search.

This is the final piece of the three-PR PDF agentic search pipeline:
- PR1 (#1727): pymupdf4llm converter produces structured Markdown with headings
- PR2 (#1738): document outline injected into agent context with line numbers
- PR3 (this):  agent guided to use outline + grep + read_file workflow

* feat(uploads): add file-first priority and fallback guidance to uploaded_files context
2026-04-04 11:08:31 +08:00
Octopus fd310582bd fix: remove nginx Plus-only zone/resolve directives from nginx.conf (#1837)
* fix: add missing DEER_FLOW_CONFIG_PATH and DEER_FLOW_EXTENSIONS_CONFIG_PATH env vars to gateway service (fixes #1829)

The gateway service was missing these two environment variables that tell
it where to find the config files inside the container. Without them,
the gateway reads DEER_FLOW_CONFIG_PATH from the host's .env file (set
to a host filesystem path), which is not accessible inside the container,
causing FileNotFoundError on startup. The langgraph service already had
these variables set correctly.

* fix: remove nginx Plus-only zone/resolve directives from nginx.conf (fixes #1744)

The `zone` and `resolve` parameters in upstream server directives are
nginx Plus features not available in the standard `nginx:alpine` image.
This caused nginx to fail at startup with:

  [emerg] invalid parameter "resolve" in /etc/nginx/nginx.conf:25

Remove these directives so the config is compatible with open-source nginx.
Docker's internal DNS (127.0.0.11, already configured via `resolver`) handles
service name resolution. The `resolver` directive is kept for the provisioner
location which uses variable-based proxy_pass for optional-service support.
2026-04-04 11:03:22 +08:00
Octopus fb2d99fd86 fix: add missing DEER_FLOW_CONFIG_PATH and DEER_FLOW_EXTENSIONS_CONFIG_PATH env vars to gateway service (fixes #1829) (#1836)
The gateway service was missing these two environment variables that tell
it where to find the config files inside the container. Without them,
the gateway reads DEER_FLOW_CONFIG_PATH from the host's .env file (set
to a host filesystem path), which is not accessible inside the container,
causing FileNotFoundError on startup. The langgraph service already had
these variables set correctly.
2026-04-04 11:01:44 +08:00
ppyt db82b59254 fix(middleware): handle list-type AIMessage.content in LoopDetectionMiddleware (#1823)
* fix: inject longTermBackground into memory prompt

The format_memory_for_injection function only processed recentMonths and
earlierContext from the history section, silently dropping longTermBackground.

The LLM writes longTermBackground correctly and it persists to memory.json,
but it was never injected into the system prompt — making the user's
long-term background invisible to the AI.

Add the missing field handling and a regression test.

* fix(middleware): handle list-type AIMessage.content in LoopDetectionMiddleware

LangChain AIMessage.content can be str | list. When using providers that
return structured content blocks (e.g. Anthropic thinking mode, certain
OpenAI-compatible gateways), content is a list of dicts like
[{"type": "text", "text": "..."}].

The hard_limit branch in _apply() concatenated content with a string via
(last_msg.content or "") + f"\n\n{_HARD_STOP_MSG}", which raises
TypeError when content is a non-empty list (list + str is invalid).

Add _append_text() static method that:
- Returns the text directly when content is None
- Appends a {"type": "text"} block when content is a list
- Falls back to string concatenation when content is a str

This is consistent with how other modules in the project already handle
list content (client.py._extract_text, memory_middleware, executor.py).

* test(middleware): add unit tests for _append_text and list content hard stop

Add regression tests to verify LoopDetectionMiddleware handles list-type
AIMessage.content correctly during hard stop:

- TestAppendText: unit tests for the new _append_text() static method
  covering None, str, list (including empty list) content types
- TestHardStopWithListContent: integration tests verifying hard stop
  works correctly with list content (Anthropic thinking mode), None
  content, and str content

Requested by reviewer in PR #1823.

* fix(middleware): improve _append_text robustness and test isolation

- Add explicit isinstance(content, str) check with fallback for
  unexpected types (coerce to str) to prevent TypeError on edge cases
- Deep-copy list content in _make_state() test helper to prevent
  shared mutable references across test iterations
- Add test_unexpected_type_coerced_to_str: verify fallback for
  non-str/list/None content types
- Add test_list_content_not_mutated_in_place: verify _append_text
  does not modify the original list

* style: fix ruff format whitespace in test file

---------

Co-authored-by: ppyt <14163465+ppyt@users.noreply.github.com>
2026-04-04 10:38:22 +08:00
SHIYAO ZHANG ddfc988bef feat(uploads): add pymupdf4llm PDF converter with auto-fallback and async offload (#1727)
* feat(uploads): add pymupdf4llm PDF converter with auto-fallback and async offload

- Introduce pymupdf4llm as an optional PDF converter with better heading
  detection and table preservation than MarkItDown
- Auto mode: prefer pymupdf4llm when installed; fall back to MarkItDown
  when output is suspiciously sparse (image-based / scanned PDFs)
- Sparsity check uses chars-per-page (< 50 chars/page) rather than an
  absolute threshold, correctly handling both short and long documents
- Large files (> 1 MB) are offloaded to asyncio.to_thread() to avoid
  blocking the event loop (related: #1569)
- Add UploadsConfig with pdf_converter field (auto/pymupdf4llm/markitdown)
- Add pymupdf4llm as optional dependency: pip install deerflow-harness[pymupdf]
- Add 14 unit tests covering sparsity heuristic, routing logic, and async path

* fix(uploads): address Copilot review comments on PDF converter

- Fix docstring: MIN_CHARS_PYMUPDF -> _MIN_CHARS_PER_PAGE (typo)
- Fix file handle leak: wrap pymupdf.open in try/finally to ensure doc.close()
- Fix silent fallback gap: _convert_pdf_with_pymupdf4llm now catches all
  conversion exceptions (not just ImportError), so encrypted/corrupt PDFs
  fall back to MarkItDown instead of propagating
- Tighten type: pdf_converter field changed from str to Literal[auto|pymupdf4llm|markitdown]
- Normalize config value: _get_pdf_converter() strips and lowercases the raw
  config string, warns and falls back to 'auto' on unknown values
2026-04-03 21:59:45 +08:00
SHIYAO ZHANG 5ff230eafd feat(uploads): inject document outline into agent context for converted files (#1738)
* feat(uploads): inject document outline into agent context for converted files

Extract headings from converted .md files and inject them into the
<uploaded_files> context block so the agent can navigate large documents
by line number before reading.

- Add `extract_outline()` to `file_conversion.py`: recognises standard
  Markdown headings (#/##/###) and SEC-style bold structural headings
  (**ITEM N. BUSINESS**, **PART II**); caps at 50 entries; excludes
  cover-page boilerplate (WASHINGTON DC, CURRENT REPORT, SIGNATURES)
- Add `_extract_outline_for_file()` helper in `uploads_middleware.py`:
  looks for a sibling `.md` file produced by the conversion pipeline
- Update `UploadsMiddleware._create_files_message()` to render the outline
  under each file entry with `L{line}: {title}` format and a `read_file`
  prompt for range-based reading
- Tests: 10 new tests for `extract_outline()`, 4 new tests for outline
  injection in `UploadsMiddleware`; existing test updated for new `outline`
  field in `uploaded_files` state

Partially addresses #1647 (agent ignores uploaded files).

* fix(uploads): stream outline file reads and strip inline bold from heading titles

- Switch extract_outline() from read_text().splitlines() to open()+line iteration
  so large converted documents are not loaded into memory on every agent turn;
  exits as soon as MAX_OUTLINE_ENTRIES is reached (Copilot suggestion)
- Strip **...** wrapper from standard Markdown heading titles before appending
  to outline so agent context stays clean (e.g. "## **Overview**" → "Overview")
  (Copilot suggestion)
- Remove unused pathlib.Path import and fix import sort order in test_file_conversion.py
  to satisfy ruff CI lint

* fix(uploads): show truncation hint when outline exceeds MAX_OUTLINE_ENTRIES

When extract_outline() hits the cap it now appends a sentinel entry
{"truncated": True} instead of silently dropping the rest of the headings.
UploadsMiddleware reads the sentinel and renders a hint line:

  ... (showing first 50 headings; use `read_file` to explore further)

Without this the agent had no way to know the outline was incomplete and
would treat the first 50 headings as the full document structure.

* fix(uploads): fall back to configurable.thread_id when runtime.context lacks thread_id

runtime.context does not always carry thread_id (depends on LangGraph
invocation path). ThreadDataMiddleware already falls back to
get_config().configurable.thread_id — apply the same pattern so
UploadsMiddleware can resolve the uploads directory and attach outlines
in all invocation paths.

* style: apply ruff format

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-03 20:52:47 +08:00
SHIYAO ZHANG 46d0c329c1 fix(uploads): fall back to configurable.thread_id when runtime.context lacks thread_id (#1814)
* fix(uploads): fall back to configurable.thread_id when runtime.context lacks thread_id

runtime.context does not always carry thread_id depending on the
LangGraph invocation path. When absent, uploads_dir resolved to None
and the entire outline/historical-files attachment was silently skipped.

Apply the same fallback pattern already used by ThreadDataMiddleware:
try get_config().configurable.thread_id, with a RuntimeError guard for
test environments where get_config() is called outside a runnable context.

Discovered via live integration testing (curl against local LangGraph).
Unit tests inject uploads_dir directly and would not catch this.

* style: apply ruff format to uploads_middleware.py
2026-04-03 20:26:21 +08:00
Rain120 a2aba23962 fix: replace the offline link in the lead_agent prompt (#1800) 2026-04-03 20:19:23 +08:00
d 🔹 6dbdd4674f fix: guarantee END sentinel delivery when stream bridge queue is full (#1695)
When MemoryStreamBridge queue reaches capacity, publish_end() previously
used the same 30s timeout + drop strategy as regular events. If the END
sentinel was dropped, subscribe() would loop forever waiting for it,
causing the SSE connection to hang indefinitely and leaking _queues and
_counters resources for that run_id.

Changes:
- publish_end() now evicts oldest regular events when queue is full to
  guarantee END sentinel delivery — the sentinel is the only signal that
  allows subscribers to terminate
- Added per-run drop counters (_dropped_counts) with dropped_count() and
  dropped_total properties for observability
- cleanup() and close() now clear drop counters
- publish() logs total dropped count per run for easier debugging

Tests:
- test_end_sentinel_delivered_when_queue_full: verifies END arrives even
  with a completely full queue
- test_end_sentinel_evicts_oldest_events: verifies eviction behavior
- test_end_sentinel_no_eviction_when_space_available: no side effects
  when queue has room
- test_concurrent_tasks_end_sentinel: 4 concurrent producer/consumer
  pairs all terminate properly
- test_dropped_count_tracking, test_dropped_total,
  test_cleanup_clears_dropped_counts, test_close_clears_dropped_counts:
  drop counter coverage

Closes #1689

Co-authored-by: voidborne-d <voidborne-d@users.noreply.github.com>
2026-04-03 20:12:30 +08:00
Octopus 83039fa22c fix: use SystemMessage+HumanMessage for follow-up question generation (#1751)
* fix: use SystemMessage+HumanMessage for follow-up question generation (fixes #1697)

Some models (e.g. MiniMax-M2.7) require the system prompt and user
content to be passed as separate message objects rather than a single
combined string. Invoking with a plain string sends everything as a
HumanMessage, which causes these models to ignore the generation
instructions and fail to produce valid follow-up questions.

* test: verify model is invoked with SystemMessage and HumanMessage
2026-04-03 20:09:01 +08:00
Admire 3d4f9a88fe Add explicit save action for agent creation (#1798)
* Add explicit save action for agent creation

* Hide internal save prompts and retry agent reads

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-03 19:54:42 +08:00
finallylly 1694c616ef feat(sandbox): add read-only support for local sandbox path mappings (#1808) 2026-04-03 19:46:22 +08:00
DanielWalnut c6cdf200ce feat(sandbox): add built-in grep and glob tools (#1784)
* feat(sandbox): add grep and glob tools

* refactor(aio-sandbox): use native file search APIs

* fix(sandbox): address review issues in grep/glob tools

- aio_sandbox: use should_ignore_path() instead of should_ignore_name()
  for include_dirs=True branch to filter nested ignored paths correctly
- aio_sandbox: add early exit when max_results reached in glob loop
- aio_sandbox: guard entry.path.startswith(path) before stripping prefix
- aio_sandbox: validate regex locally before sending to remote API
- search: skip lines exceeding max_line_chars to prevent ReDoS
- search: remove resolve() syscall in os.walk loop
- tools: avoid double get_thread_data() call in glob_tool/grep_tool
- tests: add 6 new cases covering the above code paths
- tests: patch get_app_config in truncation test to isolate config

* Fix sandbox grep/glob review feedback

* Remove unrelated Langfuse RFC from PR
2026-04-03 16:03:06 +08:00
Admire 9735d73b83 fix(ui): avoid follow-up suggestion overlap (#1777)
* fix(ui): avoid follow-up suggestion overlap

* fix(ui): address followup review feedback

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-03 15:48:41 +08:00
Admire 48565664e0 fix ACP mcpServers payload (#1735)
* fix ACP mcpServers payload

* Handle invalid ACP MCP config
2026-04-03 15:28:56 +08:00
knukn 76fad8b08d feat(client): add available_skills parameter to DeerFlowClient (#1779)
* feat(client): add `available_skills` parameter to DeerFlowClient for dynamic runtime skill filtering

* Update backend/packages/harness/deerflow/client.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(client): include `agent_name` and `available_skills` in agent config cache key

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-03 11:22:58 +08:00
ppyt 5664b9d413 fix: inject longTermBackground into memory prompt (#1734)
The format_memory_for_injection function only processed recentMonths and
earlierContext from the history section, silently dropping longTermBackground.

The LLM writes longTermBackground correctly and it persists to memory.json,
but it was never injected into the system prompt — making the user's
long-term background invisible to the AI.

Add the missing field handling and a regression test.

Co-authored-by: ppyt <14163465+ppyt@users.noreply.github.com>
2026-04-03 11:21:58 +08:00
Subham Singhania 6de9c7b43f Improve Python reliability in channel retries and thread typing (#1776)
Agent-Logs-Url: https://github.com/0xxy0/deer-flow/sessions/95336da6-e16d-43b4-834a-e5534c9396c5

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-04-03 07:50:11 +08:00
JeffJiang c1366cf559 Add documents site (#1767)
* feat: add docs site

- Implemented dynamic routing for MDX documentation pages with language support.
- Created layout components for documentation with a header and footer.
- Added metadata for various documentation sections in English and Chinese.
- Developed initial content for the DeerFlow App and Harness documentation.
- Introduced i18n hooks and translations for English and Chinese languages.
- Enhanced header component to include navigation links for documentation and blog.
- Established a structure for tutorials and reference materials.
- Created a new translations file to manage locale-specific strings.

* feat: enhance documentation structure and content for application and harness sections

* feat: update .gitignore to include .playwright-mcp and remove obsolete Playwright YAML file

* fix(docs): correct punctuation and formatting in documentation files

* feat(docs): remove outdated index.mdx file from documentation

* fix(docs): update documentation links and improve Chinese description in index.mdx

* fix(docs): update title in Chinese for meta information in _meta.ts
2026-04-03 07:25:40 +08:00
ming1523 ef711a48b3 docs: sync README table of contents with current sections (#1774) 2026-04-02 20:21:41 +08:00
Admire 952059eb51 fix(ui): avoid over-segmenting cjk messages (#1726) 2026-04-02 19:45:43 +08:00
greatmengqi 8128a3bc57 fix: enable DanglingToolCallMiddleware for subagents (#1766) 2026-04-02 18:56:18 +08:00
yangzheli 636053fb6d fix(frontend): add missing rel="noopener noreferrer" to target="_blank" links (#1741)
* fix(frontend): add missing rel="noopener noreferrer" to target="_blank" links

Prevent tabnabbing attacks and referrer leakage by ensuring all
external links with target="_blank" include both noopener and
noreferrer in the rel attribute.

Made-with: Cursor

* style: fix code formatting
2026-04-02 17:32:52 +08:00
moose-lab f56d0b4869 fix(sandbox): exclude URL paths from absolute path validation (#1385) (#1419)
* fix(sandbox): URL路径被误判为不安全绝对路径 (#1385)

在本地沙箱模式下,bash工具对命令做绝对路径安全校验时,会把curl命令中的
HTTPS URL(如 https://example.com/api/v1/check)误识别为本地绝对路径并拦截。

根因:_ABSOLUTE_PATH_PATTERN 正则的负向后行断言 (?<![:\w]) 只排除了冒号和
单词字符,但 :// 中第二个斜杠前面是第一个斜杠(/),不在排除列表中,导致
//example.com/api/... 被匹配为绝对路径 /example.com/api/...。

修复:在负向后行断言中增加斜杠字符,改为 (?<![:\w/]),使得 :// 中的连续
斜杠不会触发绝对路径匹配。同时补充了URL相关的单元测试用例。

Signed-off-by: moose-lab <moose-lab@users.noreply.github.com>

* fix(sandbox): refine absolute path regex to preserve file:// defense-in-depth

Change lookbehind from (?<![:\w/]) to (?<![:\w])(?<!:/) so only the
second slash in :// sequences is excluded. This keeps URL paths from
false-positiving while still letting the regex detect /etc/passwd in
file:///etc/passwd. Also add explicit file:// URL blocking and tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Signed-off-by: moose-lab <moose-lab@users.noreply.github.com>
Co-authored-by: moose-lab <moose-lab@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-02 16:09:14 +08:00
Varian_米泽 a2cb38f62b fix: prevent concurrent subagent file write conflicts in sandbox tools (#1714)
* fix: prevent concurrent subagent file write conflicts

Serialize same-path str_replace operations in sandbox tools

Guard AioSandbox write_file/update_file with the existing sandbox lock

Add regression tests for concurrent str_replace and append races

Verify with backend full tests and ruff lint checks

* fix(sandbox): Fix the concurrency issue of file operations on the same path in isolated sandboxes.

Ensure that different sandbox instances use independent locks for file operations on the same virtual path to avoid concurrency conflicts. Change the lock key from a single path to a composite key of (sandbox.id, path), and add tests to verify the concurrent safety of isolated sandboxes.

* feat(sandbox): Extract file operation lock logic to standalone module and fix concurrency issues

Extract file operation lock related logic from tools.py into a separate file_operation_lock.py module.
Fix data race issues during concurrent str_replace and write_file operations.
2026-04-02 15:39:41 +08:00
dependabot[bot] 3aab2445a6 build(deps): bump aiohttp from 3.13.3 to 3.13.4 in /backend (#1750)
---
updated-dependencies:
- dependency-name: aiohttp
  dependency-version: 3.13.4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-02 15:26:51 +08:00
knukn f8fb8d6fb1 feat/per agent skill filter (#1650)
* feat(agent): 为AgentConfig添加skills字段并更新lead_agent系统提示

在AgentConfig中添加skills字段以支持配置agent可用技能
更新lead_agent的系统提示模板以包含可用技能信息

* fix: resolve agent skill configuration edge cases and add tests

* Update backend/packages/harness/deerflow/agents/lead_agent/prompt.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* refactor(agent): address PR review comments for skills configuration

- Add detailed docstring to `skills` field in `AgentConfig` to clarify the semantics of `None` vs `[]`.
- Add unit tests in `test_custom_agent.py` to verify `load_agent_config()` correctly parses omitted skills and explicit empty lists.
- Fix `test_make_lead_agent_empty_skills_passed_correctly` to include `agent_name` in the runtime config, ensuring it exercises the real code path.

* docs: 添加关于按代理过滤技能的配置说明

在配置示例文件和文档中添加说明,解释如何通过代理的config.yaml文件限制加载的技能

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-02 15:02:09 +08:00
totoyang 2d1f90d5dc feat(tracing): add optional Langfuse support (#1717)
* feat(tracing): add optional Langfuse support

* Fix tracing fail-fast behavior for explicitly enabled providers

* fix(lint)
2026-04-02 13:06:10 +08:00
3a672b39c7 Fix/1681 llm call retry handling (#1683)
* fix(runtime): handle llm call errors gracefully

* fix(runtime): preserve graph control flow in llm retry middleware

---------

Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com>
2026-04-02 10:12:17 +08:00
SHIYAO ZHANG df5339b5d0 feat(sandbox): truncate oversized bash and read_file tool outputs (#1677)
* feat(sandbox): truncate oversized bash and read_file tool outputs

Long tool outputs (large directory listings, multi-MB source files) can
overflow the model's context window. Two new configurable limits:

- bash_output_max_chars (default 20000): middle-truncates bash output,
  preserving both head and tail so stderr at the end is not lost
- read_file_output_max_chars (default 50000): head-truncates file output
  with a hint to use start_line/end_line for targeted reads

Both limits are enforced at the tool layer (sandbox/tools.py) rather
than middleware, so truncation is guaranteed regardless of call path.
Setting either limit to 0 disables truncation entirely.

Measured: read_file on a 250KB source file drops from 63,698 tokens to
19,927 tokens (69% reduction) with the default limit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(tests): remove unused pytest import and fix import sort order

* style: apply ruff format to sandbox/tools.py

* refactor(sandbox): address Copilot review feedback on truncation feature

- strict hard cap: while-loop ensures result (including marker) ≤ max_chars
- max_chars=0 now returns "" instead of original output
- get_app_config() wrapped in try/except with fallback to defaults
- sandbox_config.py: add ge=0 validation on truncation limit fields
- config.example.yaml: bump config_version 4→5
- tests: add len(result) <= max_chars assertions, edge-case (max=0, small
  max, various sizes) tests; fix skipped-count test for strict hard cap

* refactor(sandbox): replace while-loop truncation with fixed marker budget

Use a pre-allocated constant (_MARKER_MAX_LEN) instead of a convergence
loop to ensure result <= max_chars. Simpler, safer, and skipped-char
count in the marker is now an exact predictable value.

* refactor(sandbox): compute marker budget dynamically instead of hardcoding

* fix(sandbox): make max_chars=0 disable truncation instead of returning empty string

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: JeffJiang <for-eleven@hotmail.com>
2026-04-02 09:22:41 +08:00
Admire 0eb6550cf4 fix(frontend): persist model selection per thread (#1553)
* fix(frontend): persist model selection per thread

* fix(frontend): apply thread model override on fallback

* refactor(frontend): split thread settings hook

* fix frontend local storage guards
2026-04-01 23:27:03 +08:00
LYU Yichen 0a379602b8 fix: avoid treating Feishu file paths as commands (#1654)
Feishu channel classified any slash-prefixed text (including absolute
paths such as /mnt/user-data/...) as a COMMAND, causing them to be
misrouted through the command pipeline instead of the chat pipeline.

Fix by introducing a shared KNOWN_CHANNEL_COMMANDS frozenset in
app/channels/commands.py — the single authoritative source for the set
of supported slash commands.  Both the Feishu inbound parser and the
ChannelManager's unknown-command reply now derive from it, so adding
or removing a command requires only one edit.

Changes:
- app/channels/commands.py (new): defines KNOWN_CHANNEL_COMMANDS
- app/channels/feishu.py: replace local KNOWN_FEISHU_COMMANDS with the
  shared constant; _is_feishu_command() now gates on it
- app/channels/manager.py: import KNOWN_CHANNEL_COMMANDS and use it in
  the unknown-command fallback reply so the displayed list stays in sync
- tests/test_feishu_parser.py: parametrize over every entry in
  KNOWN_CHANNEL_COMMANDS (each must yield msg_type=command) and add
  parametrized chat cases for /unknown, absolute paths, etc.

Made with Cursor

Made-with: Cursor

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-01 23:23:00 +08:00
Jason 1fb5acee39 fix(gateway): prevent 400 error when client sends context with configurable (#1660)
* fix(gateway): prevent 400 error when client sends context with configurable

Fixes #1290

LangGraph >= 0.6.0 rejects requests that include both 'configurable' and
'context' in the run config. If the client (e.g. useStream hook) sends
a 'context' key, we now honour it and skip creating our own
'configurable' dict to avoid the conflict.

When no 'context' is provided, we fall back to the existing
'configurable' behaviour with thread_id.

* fix(gateway): address review feedback — warn on dual keys, fix runtime injection, add tests

- Log a warning when client sends both 'context' and 'configurable' so
  it's no longer silently dropped (reviewer feedback)
- Ensure thread_id is available in config['context'] when present so
  middlewares can find it there too
- Add test coverage for the context path, the both-keys-present case,
  passthrough of other keys, and the no-config fallback

* style: ruff format services.py

---------

Co-authored-by: JasonOA888 <JasonOA888@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-01 23:21:32 +08:00
Admire 82c3dbbc6b Fix Windows startup and dependency checks (#1709)
* windows check and dev fixes

* fix windows startup scripts

* fix windows startup scripts

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-01 23:13:00 +08:00
Alian e97c8c9943 fix(skills): support parsing multiline YAML strings in SKILL.md frontmatter (#1703)
* fix(skills): support parsing multiline YAML strings in SKILL.md frontmatter

* test(skills): add tests for multiline YAML descriptions
2026-04-01 23:08:30 +08:00
DanielWalnut 68d44f6755 fix: share .deer-flow in docker-compose-dev for uploads (#1718)
* fix: share dev thread data between gateway and langgraph

* refactor: drop redundant dev .deer-flow bind mounts
2026-04-01 21:04:13 +08:00
rayhpeng c2ff59a5b1 fix(gateway): merge context field into configurable for langgraph-compat runs (#1699) (#1707)
The langgraph-compat layer dropped the DeerFlow-specific `context` field
from run requests, causing agent config (subagent_enabled, is_plan_mode,
thinking_enabled, etc.) to fall back to defaults. Add `context` to
RunCreateRequest and merge allowlisted keys into config.configurable in
start_run, with existing configurable values taking precedence.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 17:17:09 +08:00
Shengyuan Wang 2f3744f807 refactor: replace sync requests with async httpx in Jina AI client (#1603)
* refactor: replace sync requests with async httpx in Jina AI client

Replace synchronous `requests.post()` with `httpx.AsyncClient` in
JinaClient.crawl() and make web_fetch_tool async. This is part of the
planned async concurrency optimization for the agent hot path
(see docs/TODO.md).

* fix: address Copilot review feedback on async Jina client

- Short-circuit error strings in web_fetch_tool before passing to
  ReadabilityExtractor, preventing misleading extraction results
- Log missing JINA_API_KEY warning only once per process to reduce
  noise under concurrent async fetching
- Use logger.exception instead of logger.error in crawl exception
  handler to preserve stack traces for debugging
- Add async web_fetch_tool tests and warn-once coverage

* fix: mock get_app_config in web_fetch_tool tests for CI

The web_fetch_tool tests failed in CI because get_app_config requires
a config.yaml file that isn't present in the test environment. Mock
the config loader to remove the filesystem dependency.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-01 17:02:39 +08:00
Llugaes 52c8c06cf2 fix: add --n-jobs-per-worker 10 to local dev Makefile (#1694)
#1623 added this flag to both Docker Compose files but missed the
backend Makefile used by `make dev`.  Without it `langgraph dev`
defaults to n_jobs_per_worker=1, so all conversation runs are
serialised and concurrent requests block.

This mirrors the Docker configuration.
2026-04-01 16:45:51 +08:00
AochenShen99 0cdecf7b30 feat(memory): structured reflection + correction detection in MemoryMiddleware (#1620) (#1668)
* feat(memory): add structured reflection and correction detection

* fix(memory): align sourceError schema and prompt guidance

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-04-01 16:45:29 +08:00
LYU Yichen 3e461d9d08 fix: use safe docker bind mount syntax for sandbox mounts (#1655)
Docker's -v host:container syntax is ambiguous for Windows drive-letter
paths (e.g. D:/...) because ':' is both the drive separator and the
volume separator, causing mount failures on Windows hosts.

Introduce _format_container_mount() which uses '--mount type=bind,...'
for Docker (unambiguous on all platforms) and keeps '-v' for Apple
Container runtime which does not support the --mount flag yet.

Adds unit tests covering Windows paths, read-only mounts, and Apple
Container pass-through.

Made-with: Cursor
2026-04-01 11:42:12 +08:00
JeffJiang cf43584d24 fix(artifact): enhance artifact content loading to include URL for non-write files (#1678) 2026-04-01 11:38:55 +08:00
d 🔹 6ff60f2af1 fix(gateway): forward assistant_id as agent_name in build_run_config (#1667)
* fix(gateway): forward assistant_id as agent_name in build_run_config

Fixes #1644

When the LangGraph Platform-compatible /runs endpoint receives a custom
assistant_id (e.g. 'finalis'), the Gateway's build_run_config() silently
ignored it — configurable['agent_name'] was never set, so make_lead_agent
fell through to the default lead agent and SOUL.md was never loaded.

Root cause (introduced in #1403):
  resolve_agent_factory() correctly falls back to make_lead_agent for all
  assistant_id values, but build_run_config() had no assistant_id parameter
  and never injected configurable['agent_name'].  The full call chain:

    POST /runs (assistant_id='finalis')
      → resolve_agent_factory('finalis')   # returns make_lead_agent ✓
      → build_run_config(thread_id, ...)   # no agent_name injected ✗
        → make_lead_agent(config)
          → cfg.get('agent_name') → None
            → load_agent_soul(None) → base SOUL.md (doesn't exist) → None

Fix:
- Add keyword-only  parameter to build_run_config().
- When assistant_id is set and differs from 'lead_agent', inject it as
  configurable['agent_name'] (matching the channel manager's existing
  _resolve_run_params() logic for IM channels).
- Honour an explicit configurable['agent_name'] in the request body;
  assistant_id mapping only fills the gap when it is absent.
- Remove stale log-only branch from resolve_agent_factory(); update
  docstring to explain the factory/configurable split.

Tests added (test_gateway_services.py):
- Custom assistant_id injects configurable['agent_name']
- 'lead_agent' assistant_id does NOT inject agent_name
- None assistant_id does NOT inject agent_name
- Explicit configurable['agent_name'] in request is not overwritten
- resolve_agent_factory returns make_lead_agent for all inputs

* style: format with ruff

* fix: validate and normalize assistant_id to prevent path traversal

Addresses Copilot review: strip/lowercase/replace underscores and
reject names that don't match [a-z0-9-]+, consistent with
ChannelManager._normalize_custom_agent_name().

---------

Co-authored-by: voidborne-d <voidborne-d@users.noreply.github.com>
2026-04-01 11:15:56 +08:00
Matt Van Horn a3bfea631c fix(sandbox): serialize concurrent exec_command calls in AioSandbox (#1435)
* fix(sandbox): serialize concurrent exec_command calls in AioSandbox

The AIO sandbox container maintains a single persistent shell session
that corrupts when multiple exec_command requests arrive concurrently
(e.g. when ToolNode issues parallel tool_calls). The corrupted session
returns 'ErrorObservation' strings as output, cascading into subsequent
commands.

Add a threading.Lock to AioSandbox to serialize shell commands. As a
secondary defense, detect ErrorObservation in output and retry with a
fresh session ID.

Fixes #1433

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(sandbox): address Copilot review findings

- Fix shell injection in list_dir: use shlex.quote(path) to escape
  user-provided paths in the find command
- Narrow ErrorObservation retry condition from broad substring match
  to the specific corruption signature to prevent false retries
- Improve test_lock_prevents_concurrent_execution: use threading.Barrier
  to ensure all workers contend for the lock simultaneously
- Improve test_list_dir_uses_lock: assert lock.locked() is True during
  exec_command to verify lock acquisition

* style: auto-format with ruff

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-31 22:33:35 +08:00
Admire aae59a8ba8 fix: surface configured sandbox mounts to agents (#1638)
* fix: surface configured sandbox mounts to agents

* fix: address PR review feedback

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-31 22:22:30 +08:00
Admire 3ff15423d6 fix Windows Docker sandbox path mounting (#1634)
* fix windows docker sandbox paths

* fix windows sandbox mount validation

* fix backend checks for windows sandbox path PR
2026-03-31 22:19:27 +08:00
SHIYAO ZHANG c2f7be37b3 fix(tools): move sandbox.tools import in view_image_tool to break circular import (#1674)
view_image_tool.py had a top-level import of deerflow.sandbox.tools, which
created a circular dependency chain:

  sandbox.tools
    -> deerflow.agents.thread_state (triggers agents/__init__.py)
      -> agents/factory.py
        -> tools/builtins/__init__.py
          -> view_image_tool.py
            -> deerflow.sandbox.tools  <-- circular!

This caused ImportError when any test directly imported sandbox.tools,
making test_sandbox_tools_security.py fail to collect since #1522.

Fix: move the sandbox.tools import inside the view_image_tool function body.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 22:05:23 +08:00
tryag 09a9209724 fix: improve Windows compatibility in dependency check (#1550)
* fix: improve Windows compatibility in dependency check

* fix: tolerate stdio reconfigure failures in check script

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-31 21:54:41 +08:00
Rosemary1812 b356a13da5 fix(frontend): improve network error message for agent name check (#1605)
* fix(frontend): distinguish CORS errors   from generic name check failures

* fix(frontend): improve network error message for agent name check

* Fix network error message in zh-CN locale

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-31 21:14:05 +08:00
1316151417 ac9a6ee6a2 fix(langgraph): correct config.yaml mount path in docker-compose (#1679)
Co-authored-by: zhoujie172 <zhoujie172@ke.com>
2026-03-31 19:40:49 +08:00
ZHANG Ning 64e0f5329a fix: remove LANGSMITH_TRACING override that ignores .env value (#1640)
The `environment` section in docker-compose.yaml set
`LANGSMITH_TRACING=${LANGSMITH_TRACING:-false}`, which always resolves
to `false` because Docker Compose evaluates `${}` substitutions from
the host shell environment, not from `env_file`.

Since `environment` entries take precedence over `env_file`, setting
`LANGSMITH_TRACING=true` in `.env` had no effect — tracing stayed
disabled despite following the documented instructions.

Remove the explicit `LANGSMITH_TRACING` from `environment` so the
value from `.env` (loaded via `env_file`) is used as intended.
2026-03-31 09:42:33 +08:00
Admire 9e3d484858 fix(frontend): route agent checks to gateway (#1572)
* fix(frontend): route agent checks to gateway

* fix(frontend): proxy langgraph requests locally

* fix(frontend): keep zh-CN text readable

* fix(frontend): add exact local api rewrites

* fix(frontend): support docker-safe internal rewrites

* Update frontend/src/core/agents/api.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-30 21:04:59 +08:00
Admire 4bb3c101a8 chore(uv): speed up Docker builds with mirrors (#1600)
* docker mirror defaults

* fix: make docker mirror defaults overridable

* fix docker compose default pypi index

* fix: restore upstream pypi defaults

* docs: remove misleading env example mirrors

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-30 20:16:44 +08:00
ZHANG Ning b21792d9be fix: run uv sync before dev services to keep venv up-to-date (#1626)
The dev Docker Compose uses named volumes (langgraph-venv, gateway-venv)
to persist .venv across container restarts. Docker only populates named
volumes from the image on first creation — subsequent rebuilds do NOT
refresh existing volume contents.

When new dependencies are added to packages/harness/pyproject.toml
(e.g. langchain-anthropic), the stale named volume still contains
the old .venv missing the new packages, causing ModuleNotFoundError
at runtime.

Add `uv sync` before launching both gateway and langgraph services.
When deps are already satisfied this is a no-op (~1s), but when the
volume is stale it installs missing packages before the service starts.

Fixes #1624

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-30 20:02:30 +08:00
ZHANG Ning 0f1b023a2a fix: add --n-jobs-per-worker 10 to langgraph dev command in Docker (#1623)
`langgraph dev` defaults `n_jobs_per_worker` to 1 when the flag is not
explicitly passed (see langgraph_api/cli.py), even though the
`N_JOBS_PER_WORKER` env-var default is 10.

This causes the LangGraph server to run with a single background worker,
meaning all conversation runs are processed serially. When one run is
busy (e.g. summarization or long tool-calling chains), all other threads
are blocked until it finishes.

Add `--n-jobs-per-worker 10` to both production and dev Docker Compose
files to match the intended default concurrency.
2026-03-30 19:50:02 +08:00
Admire 9a557751d6 feat: support memory import and export (#1521)
* feat: support memory import and export

* fix(memory): address review feedback

* style: format memory settings page

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-30 17:25:47 +08:00
JeffJiang 2330c38209 fix(config): update SSR fallback in getBaseOrigin function (#1617) 2026-03-30 16:13:32 +08:00
rayhpeng 34e835bc33 feat(gateway): implement LangGraph Platform API in Gateway, replace langgraph-cli (#1403)
* feat(gateway): implement LangGraph Platform API in Gateway, replace langgraph-cli

Implement all core LangGraph Platform API endpoints in the Gateway,
allowing it to fully replace the langgraph-cli dev server for local
development. This eliminates a heavyweight dependency and simplifies
the development stack.

Changes:
- Add runs lifecycle endpoints (create, stream, wait, cancel, join)
- Add threads CRUD and search endpoints
- Add assistants compatibility endpoints (search, get, graph, schemas)
- Add StreamBridge (in-memory pub/sub for SSE) and async provider
- Add RunManager with atomic create_or_reject (eliminates TOCTOU race)
- Add worker with interrupt/rollback cancel actions and runtime context injection
- Route /api/langgraph/* to Gateway in nginx config
- Skip langgraph-cli startup by default (SKIP_LANGGRAPH_SERVER=0 to restore)
- Add unit tests for RunManager, SSE format, and StreamBridge

* fix: drain bridge queue on client disconnect to prevent backpressure

When on_disconnect=continue, keep consuming events from the bridge
without yielding, so the worker is not blocked by a full queue.
Only on_disconnect=cancel breaks out immediately.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: remove pytest import

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: Fix default stream_mode to ["values", "messages-tuple"]

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: Remove unused if_exists field from ThreadCreateRequest

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: address review comments on gateway LangGraph API

- Mount runs.py router in app.py (missing include_router)
- Normalize interrupt_before/after "*" to node list before run_agent()
- Use entry.id for SSE event ID instead of counter
- Drain bridge queue on disconnect when on_disconnect=continue
- Reuse serialization helper in wait_run() for consistent wire format
- Reject unsupported multitask_strategy with 400
- Remove SKIP_LANGGRAPH_SERVER fallback, always use Gateway

* feat: extract app.state access into deps.py

Encapsulate read/write operations for singleton objects (RunManager,
StreamBridge, checkpointer) held in app.state into a shared utility,
reducing repeated access patterns across router modules.

* feat: extract deerflow.runtime.serialization module with tests

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: replace duplicated serialization with deerflow.runtime.serialization

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: extract app/gateway/services.py with run lifecycle logic

Create a service layer that centralizes SSE formatting, input/config
normalization, and run lifecycle management. Router modules will delegate
to these functions instead of using private cross-imported helpers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: wire routers to use services layer, remove cross-module private imports

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: apply ruff formatting to refactored files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(runtime): support LangGraph dev server and add compat route

- Enable official LangGraph dev server for local development workflow
- Decouple runtime components from agents package for better separation
- Provide gateway-backed fallback route when dev server is skipped
- Simplify lifecycle management using context manager in gateway

* feat(runtime): add Store providers with auto-backend selection

- Add async_provider.py and provider.py under deerflow/runtime/store/
- Support memory, sqlite, postgres backends matching checkpointer config
- Integrate into FastAPI lifespan via AsyncExitStack in deps.py
- Replace hardcoded InMemoryStore with config-driven factory

* refactor(gateway): migrate thread management from checkpointer to Store and resolve multiple endpoint failures

- Add Store-backed CRUD helpers (_store_get, _store_put, _store_upsert)
- Replace checkpoint-scanning search with two-phase strategy:
  phase 1 reads Store (O(threads)), phase 2 backfills from checkpointer
  for legacy/LangGraph Server threads with lazy migration
- Extend Store record schema with values field for title persistence
- Sync thread title from checkpoint to Store after run completion
- Fix /threads/{id}/runs/{run_id}/stream 405 by accepting both
  GET and POST methods; POST handles interrupt/rollback actions
- Fix /threads/{id}/state 500 by separating read_config and
  write_config, adding checkpoint_ns to configurable, and
  shallow-copying checkpoint/metadata before mutation
- Sync title to Store on state update for immediate search reflection
- Move _upsert_thread_in_store into services.py, remove duplicate logic
- Add _sync_thread_title_after_run: await run task, read final
  checkpoint title, write back to Store record
- Spawn title sync as background task from start_run when Store exists

* refactor(runtime): deduplicate store and checkpointer provider logic

Extract _ensure_sqlite_parent_dir() helper into checkpointer/provider.py
and use it in all three places that previously inlined the same mkdir logic.
Consolidate duplicate error constants in store/async_provider.py by importing
from store/provider.py instead of redefining them.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(runtime): move SQLite helpers to runtime/store, checkpointer imports from store

_resolve_sqlite_conn_str and _ensure_sqlite_parent_dir now live in
runtime/store/provider.py. agents/checkpointer/provider and
agents/checkpointer/async_provider import from there, reversing the
previous dependency direction (store → checkpointer becomes
checkpointer → store).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(runtime): extract SQLite helpers into runtime/store/_sqlite_utils.py

Move resolve_sqlite_conn_str and ensure_sqlite_parent_dir out of
checkpointer/provider.py into a dedicated _sqlite_utils module.
Functions are now public (no underscore prefix), making cross-module
imports semantically correct. All four provider files import from
the single shared location.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(gateway): use adelete_thread to fully remove thread checkpoints on delete

AsyncSqliteSaver has no adelete method — the previous hasattr check
always evaluated to False, silently leaving all checkpoint rows in the
database. Switch to adelete_thread(thread_id) which deletes every
checkpoint and pending-write row for the thread across all namespaces
(including sub-graph checkpoints).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(gateway): remove dead bridge_cm/ckpt_cm code and fix StrEnum lint

app.py had unreachable code after the async-with lifespan refactor:
bridge_cm and ckpt_cm were referenced but never defined (F821), and
the channel service startup/shutdown was outside the langgraph_runtime
block so it never ran. Move channel service lifecycle inside the
async-with block where it belongs.

Replace str+Enum inheritance in RunStatus and DisconnectMode with
StrEnum as suggested by UP042.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* style: format with ruff

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: JeffJiang <for-eleven@hotmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-30 16:02:23 +08:00
张凯强 7db95926b0 feat(feishu): add configurable domain for Lark international support (#1535)
The lark-oapi SDK defaults to open.feishu.cn (China), but apps on the
international Lark platform (open.larksuite.com) fail to connect with
error 1000040351 'Incorrect domain name'.

Changes:
- Add 'domain' config option to feishu channel (default: open.feishu.cn)
- Pass domain to both API client and WebSocket client
- Update config.example.yaml and all README files
2026-03-30 11:42:07 +08:00
d 🔹 9bcdba6038 fix: promote deferred tools after tool_search returns schema (#1570)
* fix: promote matched tools from deferred registry after tool_search returns schema

After tool_search returns a tool's full schema, the tool is promoted
(removed from the deferred registry) so DeferredToolFilterMiddleware
stops filtering it from bind_tools on subsequent LLM calls.

Without this, deferred tools are permanently filtered — the LLM gets
the schema from tool_search but can never invoke the tool because
the middleware keeps stripping it.

Fixes #1554

* test: add promote() and tool_search promotion tests

Tests cover:
- promote removes tools from registry
- promote nonexistent/empty is no-op
- search returns nothing after promote
- middleware passes promoted tools through
- tool_search auto-promotes matched tools (select + keyword)

* fix: address review — lint blank line + empty registry guard

- Add missing blank line between FakeRequest methods (E301)
- Use 'if not registry' to handle empty registries consistently

---------

Co-authored-by: d 🔹 <258577966+voidborne-d@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-30 11:23:15 +08:00
finallylly ef58bb8d3c fix(config): correct MiniMax M2.7 highspeed model name and add thinking support (#1596)
* fix(config): correct MiniMax M2.7 highspeed model name and add thinking support

- Rename minimax-m2.5-highspeed to minimax-m2.7-highspeed for CN region
- Add supports_thinking: true for both M2.7 and M2.7-highspeed models

* Add supports_thinking option to config examples

Added support_thinking configuration option in examples.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-30 11:13:47 +08:00
Jason c5034c03c7 fix(dev): exclude sandbox dirs from gateway hot-reload watcher (#1519)
* fix(dev): exclude sandbox dirs from gateway hot-reload watcher

The dev-mode gateway uses --reload which watches for file changes.
Sandbox containers mount the repo and write .pyc/__pycache__ during
execution, triggering spurious gateway restarts mid-request.

Add --reload-exclude for .pyc, __pycache__, and sandbox/ paths so
only actual source changes trigger a reload.

Fixes #1513

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: JasonOA888 <JasonOA888@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-30 09:51:07 +08:00
SHIYAO ZHANG 9aa3ff7c48 feat(sandbox): add SandboxAuditMiddleware for bash command security auditing (#1532)
* feat(sandbox): add SandboxAuditMiddleware for bash command security auditing

Addresses the LocalSandbox escape vector reported in #1224 where bash tool
calls can execute destructive commands against the host filesystem.

- Add SandboxAuditMiddleware with three-tier command classification:
  - High-risk (block): rm -rf /, curl|bash, dd if=, mkfs, /etc/shadow access
  - Medium-risk (warn): pip install, apt install, chmod 777
  - Safe (pass): normal workspace operations
- Register middleware after GuardrailMiddleware in _build_runtime_middlewares,
  applied to both lead agent and subagents
- Structured audit log via standard logger (visible in langgraph.log)
- Medium-risk commands execute but append a warning to the tool result,
  allowing the LLM to self-correct without blocking legitimate workflows
- High-risk commands return an error ToolMessage without calling the handler,
  so the agent loop continues gracefully

* fix(lint): sort imports in test_sandbox_audit_middleware

* refactor(sandbox-audit): address Copilot review feedback (3/5/6)

- Fix class docstring to match implementation: medium-risk commands are
  executed with a warning appended (not rejected), and cwd anchoring note
  removed (handled in a separate PR)
- Remove capsys.disabled() from benchmark test to avoid CI log noise;
  keep assertions for recall/precision targets
- Remove misleading 'cwd fix' from test module docstring

* test(sandbox-audit): add async tests for awrap_tool_call

* fix(sandbox-audit): address Copilot review feedback (1/2)

- Narrow rm high-risk regex to only block truly destructive targets
  (/, /*, ~, ~/*, /home, /root); legitimate workspace paths like
  /mnt/user-data/ are no longer false-positived
- Handle list-typed ToolMessage content in _append_warn_to_result;
  append a text block instead of str()-ing the list to avoid breaking
  structured content normalization

* style: apply ruff format to sandbox_audit_middleware files

* fix(sandbox-audit): update benchmark comment to match assert-based implementation

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-30 07:48:31 +08:00
Markus Corazzione 5ceb19f6f6 fix(oauth): Harden Claude OAuth cache-control handling (#1583) 2026-03-30 07:41:18 +08:00
Admire fc7de7fffe feat: support manual add and edit for memory facts (#1538)
* feat: support manual add and edit for memory facts

* fix: restore memory updater save helper

* fix: address memory fact review feedback

* fix: remove duplicate memory fact edit action

* docs: simplify memory fact review setup

* docs: relax memory review startup instructions

* fix: clear rebase marker in memory settings page

* fix: address memory fact review and format issues

* fix: address memory fact review feedback

* refactor: make memory fact updates explicit patch semantics

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-29 23:53:23 +08:00
SHIYAO ZHANG cdb2a3a017 fix(sandbox): anchor relative paths to thread workspace in local mode (#1522)
* fix(task_tool): fallback to configurable thread_id when context is missing

task_tool only read thread_id from runtime.context, but when invoked
via LangGraph Server, thread_id lives in config.configurable instead.
Add the same fallback that ThreadDataMiddleware uses (PR #1237).

Fixes subagent execution failure: 'Thread ID is required in runtime
context or config.configurable'

* remove debug logging from task_tool

* fix(sandbox): anchor relative paths to thread workspace in local mode

In local sandbox mode, bash commands using relative paths were resolved
against the langgraph server process cwd (backend/) instead of the
per-thread workspace directory. This allowed relative-path writes to
escape the thread isolation boundary.

Root cause: validate_local_bash_command_paths and
replace_virtual_paths_in_command only process absolute paths (scanning
for '/' prefix). Relative paths pass through untouched and inherit the
process cwd at subprocess.run time.

Fix: after virtual path translation, prepend `cd {workspace} &&` to
anchor the shell's cwd to the thread-isolated workspace directory before
execution. shlex.quote() ensures paths with spaces or special characters
are handled safely.

This mirrors the approach used by OpenHands (fixed cwd at execution
layer) and is the correct fix for local mode where each subprocess.run
is an independent process with no persistent shell session.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(sandbox): extract _apply_cwd_prefix and add unit tests

Extract the workspace cd-prefix logic from bash_tool into a dedicated
_apply_cwd_prefix() helper so it can be unit-tested in isolation.
Add four tests covering: normal prefix, no thread_data, missing
workspace_path, and paths with spaces (shlex.quote).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* revert: remove unrelated configurable thread_id fallback from sandbox/tools.py

This change belongs in a separate PR.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* style: remove trailing whitespace in test_sandbox_tools_security

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-29 23:21:06 +08:00
Sleepy Ranx 🌙 866cf4ef73 fix(frontend): prevent submit during IME composition (#1562) 2026-03-29 22:36:38 +08:00
Echo-Nie d475de7997 docs: fix some broken links (#1567)
* Fix path for TitleMiddleware implementation

* Fix link to Provisioner Setup Guide in CONFIGURATION.md

* Update file path for TitleMiddleware implementation

* Update image paths in Leica photography article
2026-03-29 21:52:28 +08:00
passer 75c4757f48 fix(nginx): re-resolve upstream DNS in Docker (#1517)
Enable runtime DNS re-resolution for docker-compose upstreams (gateway/langgraph/frontend) to avoid stale container IPs causing persistent 502s.
2026-03-29 21:47:26 +08:00
tryag 580920ef63 fix: use Git Bash for Windows local startup (#1551)
* fix: use Git Bash for Windows local startup

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-29 21:38:29 +08:00
Admire 68c9e09a7a fix: add Windows shell fallback for local sandbox (#1505)
* fix: add Windows shell fallback for local sandbox

* fix: handle PowerShell execution on Windows

* fix: handle Windows local shell execution

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-29 21:31:29 +08:00
13ernkastel 92c7a20cb7 [Security] Address critical host-shell escape in LocalSandboxProvider (#1547)
* fix(security): disable host bash by default in local sandbox

* fix(security): address review feedback for local bash hardening

* fix(ci): sort live test imports for lint

* style: apply backend formatter

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-29 21:03:58 +08:00
mlbo 8b6c333afc fix(docs): Correct security usage recommendations in README_zh.md (#1548)
* fix(docs): Correct security usage recommendations in README_zh.md

Fix formatting and punctuation in security usage recommendations.

* fix(docs): Correct security usage recommendations in README_zh.md
2026-03-29 19:43:03 +08:00
knukn 6091ba83c4 docs(config): add timeout and max_retries examples for model providers (#1549)
Added explicit timeout and retry configurations to `config.example.yaml` to help users properly configure their model connections.

Since different LangChain provider classes use different parameter names, this update maps the correct arguments for each:
- ChatOpenAI (OpenAI, MiniMax, Novita, OpenRouter): added `request_timeout` and `max_retries`
- ChatAnthropic (Claude): added `default_request_timeout` and `max_retries`
- ChatGoogleGenerativeAI (Gemini): added `timeout` and `max_retries`
- PatchedChatDeepSeek (Doubao, DeepSeek, Kimi): added `timeout` and `max_retries`

Default example values are set to 600.0 seconds for timeouts and 2 for max retries.
2026-03-29 19:29:55 +08:00
greatmengqi 70e9f2dd2c docs: add format step to contributing workflow (#1552)
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-03-29 18:39:10 +08:00
SHIYAO ZHANG 118485a7cb fix(sandbox): fall back to config.configurable for thread_id in lazy sandbox init (#1529)
* fix(sandbox): fall back to config.configurable for thread_id in lazy sandbox init

LangGraph Server injects thread_id via config["configurable"]["thread_id"],
not always via context["thread_id"]. Without the fallback, lazy sandbox
acquisition fails when context is empty.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(sandbox): align configurable fallback style with task_tool.py

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(sandbox): guard runtime.config None check for thread_id fallback

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 17:21:04 +08:00
DAN 9e5ba74ecd fix(sandbox): allow MCP filesystem server paths in local bash commands (#1527)
* feat/bug-fix: copy the allowed path configurations in MCP filesystem tools to bash tool. With updated unit test

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-29 17:10:27 +08:00
greatmengqi 25df82cbfd style: format unformatted files and add .omc/ to prettierignore (#1539)
Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-03-29 16:45:31 +08:00
greatmengqi 084dc7e748 ci: enforce code formatting checks for backend and frontend (#1536) 2026-03-29 15:34:38 +08:00
greatmengqi 06a623f9c8 feat: add create_deerflow_agent SDK entry point (Phase 1) (#1203) 2026-03-29 15:31:18 +08:00
Admire 7eb3a150b5 feat: add memory management actions and local filters in memory settings (#1467)
* Add MVP memory management actions

* Fix memory settings locale coverage

* Polish memory management interactions

* Add memory search and type filters

* Refine memory settings review feedback

* docs: simplify memory settings review setup

* fix: restore memory updater compatibility helpers

* fix: address memory settings review feedback

* docs: soften memory sample review wording

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: JeffJiang <for-eleven@hotmail.com>
2026-03-29 13:14:45 +08:00
knukn 481494b9c0 feat(client): support custom middleware injection (#1520)
* feat(client): support custom middleware injection

Add support for custom middleware, allowing custom middleware list to be passed when initializing DeerFlowClient. These middleware will be injected after the default middleware when creating the agent, extending the agent's functionality.

* feat: inject custom middlewares before ClarificationMiddleware to preserve ordering

- Add `custom_middlewares` param to `_build_middlewares`
- Inject custom middlewares right before `ClarificationMiddleware` to keep it as the last in the chain
- Remove unsafe `.extend()` in `client.py`
- Update tests in `test_client.py` and `test_lead_agent_model_resolution.py` to assert correct injection ordering
2026-03-29 11:24:46 +08:00
Nan Gao 89183ae76a fix(channel): reject concurrent same-thread runs (#1465) (#1475)
* fix(channel): reject concurrent same-thread runs (#1465)

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(lint): sort imports in manager.py and test_channels.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(channel): widen _is_thread_busy_error to BaseException and downgrade busy log to warning

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-29 09:55:47 +08:00
DanielWalnut 18e3487888 Support custom channel assistant IDs via lead_agent (#1500)
* Support custom channel assistant IDs via lead agent

* Normalize custom channel agent names
2026-03-28 19:07:38 +08:00
Nan Gao 520c0352b5 fix(middleware): fall back to configurable thread_id in MemoryMiddleware (#1425) (#1426)
* fix(middleware): fall back to configurable thread_id in MemoryMiddleware (#1425)

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-28 17:00:11 +08:00
SHIYAO ZHANG 690d80f46f fix(task_tool): fallback to configurable thread_id when context is mi… (#1343)
* fix(task_tool): fallback to configurable thread_id when context is missing

task_tool only read thread_id from runtime.context, but when invoked
via LangGraph Server, thread_id lives in config.configurable instead.
Add the same fallback that ThreadDataMiddleware uses (PR #1237).

Fixes subagent execution failure: 'Thread ID is required in runtime
context or config.configurable'

* remove debug logging from task_tool
2026-03-28 16:44:15 +08:00
DanielWalnut c2dd8937ed Fix IM channel backend URLs in Docker (#1497)
* Fix IM channel backend URLs in Docker

* Address Copilot review comments
2026-03-28 16:37:41 +08:00
zihao 9caea0266e fix(frontend): separate mock and default LangGraph clients (#1504)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-28 16:33:22 +08:00
7. Sun 49f2e38fbf fix: prevent SpeechRecognition instance leaks on render (#1369)
* fix: remove unstable dependencies from speech recognition effect

* fix: use refs to prevent stale closures in speech recognition

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-28 16:20:38 +08:00
JeffJiang d22cab8614 fix: refactor to use getBaseOrigin for URL construction in backend and LangGraph base URL functions (#1494) 2026-03-28 12:18:03 +08:00
taka6745 43ef3691a5 fix(oauth): inject billing header for Claude oAuth Models (#1442)
* fix(oauth): inject billing header for non-Haiku model access

The Anthropic Messages API requires a billing identification block
in the system prompt when using Claude Code OAuth tokens (sk-ant-oat*)
to access non-Haiku models (Opus, Sonnet). Without it, the API returns
a generic 400 "Error" with no actionable detail.

This was discovered by intercepting Claude Code CLI requests — the CLI
injects an `x-anthropic-billing-header` text block as the first system
prompt entry on every request. Third-party consumers of the same OAuth
tokens must do the same.

Changes:
- Add `_apply_oauth_billing()` to `ClaudeChatModel` that prepends the
  billing header block to the system prompt when `_is_oauth` is True
- Add `metadata.user_id` with device/session identifiers (required by
  the API alongside the billing header)
- Called from `_get_request_payload()` before prompt caching runs

Verified with Claude Max OAuth tokens against all three model tiers:
- claude-opus-4-6: 200 OK
- claude-sonnet-4-6: 200 OK
- claude-haiku-4-5-20251001: 200 OK (was already working)

Closes #1245

* fix(oauth): address review feedback on billing header injection

- Make OAUTH_BILLING_HEADER configurable via ANTHROPIC_BILLING_HEADER env var
- Normalize billing block to always be first in system list (strip + reinsert)
- Guard metadata with isinstance check for non-dict values
- Replace os.uname() with socket.gethostname() for Windows compat
- Fix docstrings to say "all OAuth requests" instead of "non-Haiku"
- Move inline imports to module level (fixes ruff I001)
- Add 9 unit tests for _apply_oauth_billing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-28 08:49:34 +08:00
luo jiyin ca20b48601 chore(ci): align workflow action versions (#1484) 2026-03-27 23:25:55 +08:00
moose-lab 03b144f9c9 fix: replace print() with logging across harness package (#1282)
Replace all bare print() calls with proper logging using Python's
standard logging module across the deerflow harness package.

Changes across 8 files (16 print statements replaced):

- agents/middlewares/clarification_middleware.py: use logger.info/debug
- agents/middlewares/memory_middleware.py: use logger.debug
- agents/middlewares/thread_data_middleware.py: use logger.debug
- agents/middlewares/view_image_middleware.py: use logger.debug
- agents/memory/queue.py: use logger.info/debug/warning/error
- agents/lead_agent/prompt.py: use logger.error
- skills/loader.py: use logger.warning
- skills/parser.py: use logger.error

Each file follows the established codebase convention:
  import logging
  logger = logging.getLogger(__name__)

Log levels chosen based on message semantics:
- debug: routine operational details (directory creation, timer resets)
- info: significant state changes (memory queued, updates processed)
- warning: recoverable issues (config load failures, skipped updates)
- error: unexpected failures (parsing errors, memory update errors)

Note: client.py is intentionally excluded as it uses print() for
CLI output, which is the correct behavior for a command-line client.

Co-authored-by: moose-lab <moose-lab@users.noreply.github.com>
2026-03-27 23:15:35 +08:00
luo jiyin 9a4e8f438a docs(frontend): update better-auth README notes (#1487) 2026-03-27 22:58:55 +08:00
Zeng Qingwen 6bf23ba0a3 docs(README): add missing cross-language README links (#1479)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-27 22:49:59 +08:00
luo jiyin 50db51d0fb chore(frontend): align format scripts with Makefile and README (#1481)
* chore(frontend): add format scripts

* docs(frontend): document format commands
2026-03-27 22:49:41 +08:00
Zeng Qingwen 18b0794125 docs(SETUP): correct setup documentation links (#1478) 2026-03-27 22:44:01 +08:00
Andrew Barnes 50f50d7654 test: add unit tests for skill frontmatter validation (#1309)
* test: add unit tests for skill frontmatter validation

Cover _validate_skill_frontmatter logic:
- Valid minimal and full-field skills
- Missing SKILL.md, missing frontmatter, invalid YAML
- Required field validation (name, description)
- Unexpected key rejection
- Name format: hyphen-case, no leading/trailing/consecutive hyphens
- Name and description length limits
- Angle bracket rejection in description

* test: fix unused variables flagged by ruff F841

Replace unused tuple elements with _ and add assertions on
msg/name return values in success-path tests.

* test: address review feedback on unused variables

* test: consolidate validation tests into single module

Move the UTF-8/windows-locale test from test_skills_router.py into
test_skills_validation.py and remove test_skills_router.py to eliminate
duplicated assertions and future maintenance drift.

* fix: match assertion strings to actual validation messages

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-27 20:20:31 +08:00
DanielWalnut 8590249db4 feat(acp): add env field to ACPAgentConfig for subprocess env injection (#1447)
Allow per-agent environment variables to be declared in config.yaml under
acp_agents.<name>.env. Values prefixed with $ are resolved from the host
environment at invocation time, consistent with other config fields.
Passes None to spawn_agent_process when env is empty so the subprocess
inherits the parent environment unchanged.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 20:03:30 +08:00
Admire 40a4acbbed fix(sandbox): Relax upload permissions for aio sandbox sync (#1409)
* Relax upload permissions for aio sandbox sync

* Harden upload permission sync checks

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-27 17:37:44 +08:00
Jason 4708700723 fix(middleware): return proper content format when no images viewed (#1454)
- Fix OpenAI BadRequestError: 'No images have been viewed.' was returned as
  a plain string array instead of a properly formatted content block
- The OpenAI API expects message content to be either a string or an array
  of objects with 'type' field, not an array of plain strings
- Changed return from ['No images have been viewed.'] to
  [{'type': 'text', 'text': 'No images have been viewed.'}]

Fixes #1441

Co-authored-by: JasonOA888 <noreply@github.com>
2026-03-27 17:33:17 +08:00
luo jiyin 43a19f9627 fix(task): avoid blocking in task tool polling (#1320)
* fix: avoid blocking in task tool polling

* test: adapt task tool polling tests for async tool

* fix: clean up cancelled task tool polling

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-27 17:12:40 +08:00
yangzheli a4e4bb21e3 docs: add LangSmith tracing configuration and documentation (#1414)
Add LangSmith tracing setup instructions across the project:
- .env.example: add LANGSMITH_* env vars (commented out)
- README.md + translations (zh/ja/fr/ru): add LangSmith Tracing section
  under Advanced with setup steps and env var reference
- backend/README.md: add detailed LangSmith Tracing section with setup,
  env var table, how-it-works explanation, and Docker notes
- docker-compose.yaml: update LANGCHAIN_TRACING_V2 to LANGSMITH_TRACING
  for naming consistency with the rest of the project

Made-with: Cursor

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-27 14:17:45 +08:00
Matt Van Horn 99965057c1 fix(config): add Docker service name guidance for channel URLs (#1437)
The channels config section uses localhost URLs by default, which don't
work inside Docker containers. Add inline comments showing the Docker
service names (langgraph, gateway) that match the docker-compose service
definitions.

Fixes #1421

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-27 14:15:48 +08:00
Kaushik Rajan 8ae023574e fix: add build-arg support for proxies and mirrors in Docker builds (#1346)
* fix: add build-arg support for proxies and mirrors in Docker builds (#1260)

Pin Debian images to bookworm, make UV source image configurable,
and pass APT_MIRROR/NPM_REGISTRY/UV_IMAGE through docker-compose.

* fix: ensure build args use consistent defaults across compose and Dockerfiles

UV_IMAGE: ${UV_IMAGE:-} resolved to empty when unset, overriding the
Dockerfile ARG default and breaking `FROM ${UV_IMAGE}`. Also configure
COREPACK_NPM_REGISTRY before pnpm download and propagate NPM_REGISTRY
into the prod stage.

* fix: dearmor NodeSource GPG key to resolve signing error

Pipe the downloaded key through gpg --dearmor so apt can verify
the repository signature (fixes NO_PUBKEY 2F59B5F99B1BE0B4).

---------

Co-authored-by: JeffJiang <for-eleven@hotmail.com>
2026-03-27 10:35:40 +08:00
SCPZ24 6b13f5c9fb feat: Support gitHub PAT configuration for higher github API accessing rate. (#1374)
* feat: Add github PAT configs, allowing larger github API rates.

* Update comment to English for better clarity

* fix: Remove unused config lines in config.example.yaml and unreferenced declarations in app_config. Fix lint issues and update documentation.

* fix: Remove unused imports, and passed the ruff check.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-27 09:54:14 +08:00
Ben Piper c13793386f Implement DuckDuckGo search (#1432)
* Implement DuckDuckGo search

* docs: add DuckDuckGo web search to config example
2026-03-27 09:20:22 +08:00
knukn 1c542ab7f1 feat(memory): Introduce configurable memory storage abstraction (#1353)
* feat(内存存储): 添加可配置的内存存储提供者支持

实现内存存储的抽象基类 MemoryStorage 和文件存储实现 FileMemoryStorage
重构内存数据加载和保存逻辑到存储提供者中
添加 storage_class 配置项以支持自定义存储提供者

* refactor(memory): 重构内存存储模块并更新相关测试

将内存存储逻辑从updater模块移动到独立的storage模块
使用存储接口模式替代直接文件操作
更新所有相关测试以使用新的存储接口

* Update backend/packages/harness/deerflow/agents/memory/storage.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update backend/packages/harness/deerflow/agents/memory/storage.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(内存存储): 添加线程安全锁并增加测试用例

添加线程锁确保内存存储单例初始化的线程安全
增加对无效代理名称的验证测试
补充单例线程安全性和异常处理的测试用例

* Update backend/tests/test_memory_storage.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(agents): 使用统一模式验证代理名称

修改代理名称验证逻辑以使用仓库中定义的AGENT_NAME_PATTERN模式,确保代码库一致性并防止路径遍历等安全问题。同时更新测试用例以覆盖更多无效名称情况。

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-27 07:41:06 +08:00
DanielWalnut e1853df06a docs: add install.md agent setup guide (#1402)
* docs: add install.md agent setup guide

* docs: tighten install.md setup flow

* docs: address copilot review comments
2026-03-26 21:39:34 +08:00
Henry Li f80d1743ab Add security alerts to documents (#1413) 2026-03-26 21:24:52 +08:00
7. Sun d7bdb1a4b9 fix: remove unused radix Icon import from suggestion (#1368)
* fix: use create_chat_model for summarization alias

* fix: remove unused radix Icon import from suggestion

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-26 21:14:56 +08:00
Henry Li 227967df3d feat: hide model ID for safety reason, only show the display_name (#1410)
Co-authored-by: Henry Li <lixin.henry@bytedance.com>
2026-03-26 21:13:32 +08:00
13ernkastel 0d3cefaa5a fix(gateway): enforce safe download for active artifact MIME types to mitigate stored XSS (#1389)
* docs: refocus security review on high-confidence artifact XSS

* fix(gateway): block inline active-content artifacts to mitigate XSS

* chore: remove security review markdown from PR

* Delete SECURITY_REVIEW.md

* fix(gateway): harden artifact attachment handling
2026-03-26 17:44:25 +08:00
Admire b9583f7204 Fix Windows backend test compatibility (#1384)
* Fix Windows backend test compatibility

* Preserve ACP path style on Windows

* Fix installer import ordering

* Address review comments for Windows fixes

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-26 17:39:16 +08:00
dependabot[bot] b3d3287b80 build(deps): bump requests from 2.32.5 to 2.33.0 in /backend (#1395)
Bumps [requests](https://github.com/psf/requests) from 2.32.5 to 2.33.0.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](https://github.com/psf/requests/compare/v2.32.5...v2.33.0)

---
updated-dependencies:
- dependency-name: requests
  dependency-version: 2.33.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-26 16:18:20 +08:00
RockeyDon c0a6b81852 Add packages section to pnpm-workspace.yaml (#1382)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-26 16:09:35 +08:00
JeffJiang 4d1a69a938 fix(config): return full URLs for backend and LangGraph base URLs (#1392) 2026-03-26 15:43:37 +08:00
Willem Jiang a087fe7bcc fix(LLM): fixing Gemini thinking + tool calls via OpenAI gateway (#1180) (#1205)
* fix(LLM): fixing Gemini thinking + tool calls via OpenAI gateway (#1180)

When using Gemini with thinking enabled through an OpenAI-compatible gateway,
the API requires that  fields on thinking content blocks are
preserved and echoed back verbatim in subsequent requests. Standard
 silently drops these signatures when serializing
messages, causing HTTP 400 errors:

Changes:
- Add PatchedChatOpenAI adapter that re-injects signed thinking blocks into
  request payloads, preserving the signature chain across multi-turn
  conversations with tool calls.
- Support two LangChain storage patterns: additional_kwargs.thinking_blocks
  and content list.
- Add 11 unit tests covering signed/unsigned blocks, storage patterns, edge
  cases, and precedence rules.
- Update config.example.yaml with Gemini + thinking gateway example.
- Update CONFIGURATION.md with detailed guidance and error explanation.

Fixes: #1180

* Updated the patched_openai.py with thought_signature of function call

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* docs: fix inaccurate thought_signature description in CONFIGURATION.md (#1220)

* Initial plan

* docs: fix CONFIGURATION.md wording for thought_signature - tool-call objects, not thinking blocks

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/360f5226-4631-48a7-a050-189094af8ffe

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
2026-03-26 15:07:05 +08:00
Admire 080a03f3bc fix(config): fix summarization model alias resolution (#1378)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-26 14:48:45 +08:00
xiangxiang-all-in-AI ae6a791c71 Update config.example.yaml (#1376)
使用deerpseek接口后会报错,因为max_token设置不对。所以从example里改

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-26 14:34:57 +08:00
DanielWalnut d119214fee feat(harness): integration ACP agent tool (#1344)
* refactor: extract shared utils to break harness→app cross-layer imports

Move _validate_skill_frontmatter to src/skills/validation.py and
CONVERTIBLE_EXTENSIONS + convert_file_to_markdown to src/utils/file_conversion.py.
This eliminates the two reverse dependencies from client.py (harness layer)
into gateway/routers/ (app layer), preparing for the harness/app package split.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: split backend/src into harness (deerflow.*) and app (app.*)

Physically split the monolithic backend/src/ package into two layers:

- **Harness** (`packages/harness/deerflow/`): publishable agent framework
  package with import prefix `deerflow.*`. Contains agents, sandbox, tools,
  models, MCP, skills, config, and all core infrastructure.

- **App** (`app/`): unpublished application code with import prefix `app.*`.
  Contains gateway (FastAPI REST API) and channels (IM integrations).

Key changes:
- Move 13 harness modules to packages/harness/deerflow/ via git mv
- Move gateway + channels to app/ via git mv
- Rename all imports: src.* → deerflow.* (harness) / app.* (app layer)
- Set up uv workspace with deerflow-harness as workspace member
- Update langgraph.json, config.example.yaml, all scripts, Docker files
- Add build-system (hatchling) to harness pyproject.toml
- Add PYTHONPATH=. to gateway startup commands for app.* resolution
- Update ruff.toml with known-first-party for import sorting
- Update all documentation to reflect new directory structure

Boundary rule enforced: harness code never imports from app.
All 429 tests pass. Lint clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: add harness→app boundary check test and update docs

Add test_harness_boundary.py that scans all Python files in
packages/harness/deerflow/ and fails if any `from app.*` or
`import app.*` statement is found. This enforces the architectural
rule that the harness layer never depends on the app layer.

Update CLAUDE.md to document the harness/app split architecture,
import conventions, and the boundary enforcement test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add config versioning with auto-upgrade on startup

When config.example.yaml schema changes, developers' local config.yaml
files can silently become outdated. This adds a config_version field and
auto-upgrade mechanism so breaking changes (like src.* → deerflow.*
renames) are applied automatically before services start.

- Add config_version: 1 to config.example.yaml
- Add startup version check warning in AppConfig.from_file()
- Add scripts/config-upgrade.sh with migration registry for value replacements
- Add `make config-upgrade` target
- Auto-run config-upgrade in serve.sh and start-daemon.sh before starting services
- Add config error hints in service failure messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix comments

* fix: update src.* import in test_sandbox_tools_security to deerflow.*

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: handle empty config and search parent dirs for config.example.yaml

Address Copilot review comments on PR #1131:
- Guard against yaml.safe_load() returning None for empty config files
- Search parent directories for config.example.yaml instead of only
  looking next to config.yaml, fixing detection in common setups

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: correct skills root path depth and config_version type coercion

- loader.py: fix get_skills_root_path() to use 5 parent levels (was 3)
  after harness split, file lives at packages/harness/deerflow/skills/
  so parent×3 resolved to backend/packages/harness/ instead of backend/
- app_config.py: coerce config_version to int() before comparison in
  _check_config_version() to prevent TypeError when YAML stores value
  as string (e.g. config_version: "1")
- tests: add regression tests for both fixes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: update test imports from src.* to deerflow.*/app.* after harness refactor

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(harness): add tool-first ACP agent invocation (#37)

* feat(harness): add tool-first ACP agent invocation

* build(harness): make ACP dependency required

* fix(harness): address ACP review feedback

* feat(harness): decouple ACP agent workspace from thread data

ACP agents (codex, claude-code) previously used per-thread workspace
directories, causing path resolution complexity and coupling task
execution to DeerFlow's internal thread data layout. This change:

- Replace _resolve_cwd() with a fixed _get_work_dir() that always uses
  {base_dir}/acp-workspace/, eliminating virtual path translation and
  thread_id lookups
- Introduce /mnt/acp-workspace virtual path for lead agent read-only
  access to ACP agent output files (same pattern as /mnt/skills)
- Add security guards: read-only validation, path traversal prevention,
  command path allowlisting, and output masking for acp-workspace
- Update system prompt and tool description to guide LLM: send
  self-contained tasks to ACP agents, copy results via /mnt/acp-workspace
- Add 11 new security tests for ACP workspace path handling

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(prompt): inject ACP section only when ACP agents are configured

The ACP agent guidance in the system prompt is now conditionally built
by _build_acp_section(), which checks get_acp_agents() and returns an
empty string when no ACP agents are configured. This avoids polluting
the prompt with irrelevant instructions for users who don't use ACP.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix lint

* fix(harness): address Copilot review comments on sandbox path handling and ACP tool

- local_sandbox: fix path-segment boundary bug in _resolve_path (== or startswith +"/")
  and add lookahead in _resolve_paths_in_command regex to prevent /mnt/skills matching
  inside /mnt/skills-extra
- local_sandbox_provider: replace print() with logger.warning(..., exc_info=True)
- invoke_acp_agent_tool: guard getattr(option, "optionId") with None default + continue;
  move full prompt from INFO to DEBUG level (truncated to 200 chars)
- sandbox/tools: fix _get_acp_workspace_host_path docstring to match implementation;
  remove misleading "read-only" language from validate_local_bash_command_paths

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(acp): thread-isolated workspaces, permission guardrail, and ContextVar registry

P1.1 – ACP workspace thread isolation
- Add `Paths.acp_workspace_dir(thread_id)` for per-thread paths
- `_get_work_dir(thread_id)` in invoke_acp_agent_tool now uses
  `{base_dir}/threads/{thread_id}/acp-workspace/`; falls back to
  global workspace when thread_id is absent or invalid
- `_invoke` extracts thread_id from `RunnableConfig` via
  `Annotated[RunnableConfig, InjectedToolArg]`
- `sandbox/tools.py`: `_get_acp_workspace_host_path(thread_id)`,
  `_resolve_acp_workspace_path(path, thread_id)`, and all callers
  (`replace_virtual_paths_in_command`, `mask_local_paths_in_output`,
  `ls_tool`, `read_file_tool`) now resolve ACP paths per-thread

P1.2 – ACP permission guardrail
- New `auto_approve_permissions: bool = False` field in `ACPAgentConfig`
- `_build_permission_response(options, *, auto_approve: bool)` now
  defaults to deny; only approves when `auto_approve=True`
- Document field in `config.example.yaml`

P2 – Deferred tool registry race condition
- Replace module-level `_registry` global with `contextvars.ContextVar`
- Each asyncio request context gets its own registry; worker threads
  inherit the context automatically via `loop.run_in_executor`
- Expose `get_deferred_registry` / `set_deferred_registry` /
  `reset_deferred_registry` helpers

Tests: 831 pass (57 for affected modules, 3 new tests)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(sandbox): mount /mnt/acp-workspace in docker sandbox container

The AioSandboxProvider was not mounting the ACP workspace into the
sandbox container, so /mnt/acp-workspace was inaccessible when the lead
agent tried to read ACP results in docker mode.

Changes:
- `ensure_thread_dirs`: also create `acp-workspace/` (chmod 0o777) so
  the directory exists before the sandbox container starts — required
  for Docker volume mounts
- `_get_thread_mounts`: add read-only `/mnt/acp-workspace` mount using
  the per-thread host path (`host_paths.acp_workspace_dir(thread_id)`)
- Update stale CLAUDE.md description (was "fixed global workspace")

Tests: `test_aio_sandbox_provider.py` (4 new tests)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(lint): remove unused imports in test_aio_sandbox_provider

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix config

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 14:20:18 +08:00
Hiren Thakore 792c49e6af fix: align config.example.yaml to use GEMINI_API_KEY (#1367)
The commented google_api_key example referenced $GOOGLE_API_KEY but the
codebase (.env.example, generate.py scripts) uses GEMINI_API_KEY.
Closes #1364
2026-03-26 08:34:25 +08:00
Andrew Barnes ac97dc6d42 test: add unit tests for TodoMiddleware (#1307)
* test: add unit tests for TodoMiddleware

Cover context-loss detection logic:
- _todos_in_messages and _reminder_in_messages helpers
- _format_todos formatting
- Reminder injection when write_todos truncated
- No-op when todos visible or reminder already present
- abefore_model async delegation

* test: fix event loop error in todo middleware async test

Use asyncio.run() instead of get_event_loop().run_until_complete()
to avoid RuntimeError on Python 3.12 where no default event loop
exists in the main thread.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-26 00:20:50 +08:00
Andrew Barnes 1f0ae64e02 test: add unit tests for DanglingToolCallMiddleware (#1305)
* test: add unit tests for DanglingToolCallMiddleware

Cover message patching logic for dangling tool calls:
- No-op when all tool calls have responses
- Synthetic ToolMessage insertion at correct positions
- Mixed responded/dangling scenarios
- wrap_model_call and awrap_model_call integration

* test: fix async tests and strengthen override assertions

- Use @pytest.mark.anyio + async def instead of deprecated
  asyncio.get_event_loop().run_until_complete() (fixes Py3.12 CI failure)
- Assert that override() receives the correct patched messages kwarg
  in both wrap_model_call and awrap_model_call tests

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-26 00:20:08 +08:00
offliner afe325d34e Fix command syntax for container image pull (#1349)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-26 00:14:08 +08:00
吴旭云 d7e510763d fix: add null checks for runtime.context and tighten langgraph constraint (#1326)
- Add null checks for runtime.context in uploads_middleware.py and
  sandbox/middleware.py to prevent NPE when langgraph runtime context is None
- Tighten langgraph version constraint from >=1.0.6 to >=1.0.6,<1.0.10
  to avoid context=None incompatibility with langgraph-api 0.7.x

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-25 21:01:10 +08:00
Simon Su adc51e541c fix(frontend): add stable ids for chat resizable panels (#1341)
Signed-off-by: sysusugan <sugan@foxmail.com>
2026-03-25 20:58:15 +08:00
zhoutianwang fdfe08d4aa Add user configuration template for China region (#1337)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-25 18:56:06 +08:00
Henry Li 12875664f1 docs: add domestic link of coding plan (#1340) 2026-03-25 18:53:31 +08:00
greatmengqi b8bc80d89b refactor: extract shared skill installer and upload manager to harness (#1202)
* refactor: extract shared skill installer and upload manager to harness

Move duplicated business logic from Gateway routers and Client into
shared harness modules, eliminating code duplication.

New shared modules:
- deerflow.skills.installer: 6 functions (zip security, extraction, install)
- deerflow.uploads.manager: 7 functions (normalize, deduplicate, validate,
  list, delete, get_uploads_dir, ensure_uploads_dir)

Key improvements:
- SkillAlreadyExistsError replaces stringly-typed 409 status routing
- normalize_filename rejects backslash-containing filenames
- Read paths (list/delete) no longer mkdir via get_uploads_dir
- Write paths use ensure_uploads_dir for explicit directory creation
- list_files_in_dir does stat inside scandir context (no re-stat)
- install_skill_from_archive uses single is_file() check (one syscall)
- Fix agent config key not reset on update_mcp_config/update_skill

Tests: 42 new (22 installer + 20 upload manager) + client hardening

* refactor: centralize upload URL construction and clean up installer

- Extract upload_virtual_path(), upload_artifact_url(), enrich_file_listing()
  into shared manager.py, eliminating 6 duplicated URL constructions across
  Gateway router and Client
- Derive all upload URLs from VIRTUAL_PATH_PREFIX constant instead of
  hardcoded "mnt/user-data/uploads" strings
- Eliminate TOCTOU pre-checks and double file read in installer — single
  ZipFile() open with exception handling replaces is_file() + is_zipfile()
  + ZipFile() sequence
- Add missing re-exports: ensure_uploads_dir in uploads/__init__.py,
  SkillAlreadyExistsError in skills/__init__.py
- Remove redundant .lower() on already-lowercase CONVERTIBLE_EXTENSIONS
- Hoist sandbox_uploads_dir(thread_id) before loop in uploads router

* fix: add input validation for thread_id and filename length

- Reject thread_id containing unsafe filesystem characters (only allow
  alphanumeric, hyphens, underscores, dots) — prevents 500 on inputs
  like <script> or shell metacharacters
- Reject filenames longer than 255 bytes (OS limit) in normalize_filename
- Gateway upload router maps ValueError to 400 for invalid thread_id

* fix: address PR review — symlink safety, input validation coverage, error ordering

- list_files_in_dir: use follow_symlinks=False to prevent symlink metadata
  leakage; check is_dir() instead of exists() for non-directory paths
- install_skill_from_archive: restore is_file() pre-check before extension
  validation so error messages match the documented exception contract
- validate_thread_id: move from ensure_uploads_dir to get_uploads_dir so
  all entry points (upload/list/delete) are protected
- delete_uploaded_file: catch ValueError from thread_id validation (was 500)
- requires_llm marker: also skip when OPENAI_API_KEY is unset
- e2e fixture: update TitleMiddleware exclusion comment (kept filtering —
  middleware triggers extra LLM calls that add non-determinism to tests)

* chore: revert uv.lock to main — no dependency changes in this PR

* fix: use monkeypatch for global config in e2e fixture to prevent test pollution

The e2e_env fixture was calling set_title_config() and
set_summarization_config() directly, which mutated global singletons
without automatic cleanup. When pytest ran test_client_e2e.py before
test_title_middleware_core_logic.py, the leaked enabled=False caused
5 title tests to fail in CI.

Switched to monkeypatch.setattr on the module-level private variables
so pytest restores the originals after each test.

* fix: address code review — URL encoding, API consistency, test isolation

- upload_artifact_url: percent-encode filename to handle spaces/#/?
- deduplicate_filename: mutate seen set in place (caller no longer
  needs manual .add() — less error-prone API)
- list_files_in_dir: document that size is int, enrich stringifies
- e2e fixture: monkeypatch _app_config instead of set_app_config()
  to prevent global singleton pollution (same pattern as title/summarization fix)
- _make_e2e_config: read LLM connection details from env vars so
  external contributors can override defaults
- Update tests to match new deduplicate_filename contract

* docs: rewrite RFC in English and add alternatives/breaking changes sections

* fix: address code review feedback on PR #1202

- Rename deduplicate_filename to claim_unique_filename to make
  the in-place set mutation explicit in the function name
- Replace PermissionError with PathTraversalError(ValueError) for
  path traversal detection — malformed input is 400, not 403

* fix: set _app_config_is_custom in e2e test fixture to prevent config.yaml lookup in CI

---------

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
2026-03-25 16:28:33 +08:00
Andrew Barnes ec46ae075d test: add unit tests for SubagentLimitMiddleware (#1306)
* test: add unit tests for SubagentLimitMiddleware

Cover subagent limit enforcement:
- _clamp_subagent_limit boundary clamping
- Task call truncation when exceeding limit
- Non-task tool calls preserved during truncation
- after_model/aafter_model delegation

* Update test_subagent_limit_middleware.py

* Fix import statement for MAX_CONCURRENT_SUBAGENTS

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-25 10:20:16 +08:00
Andrew Barnes afb0f66c73 test: add unit tests for skills parser (#1308)
Cover parse_skill_file logic:
- Valid SKILL.md parsing with all fields
- Missing required fields (name, description) return None
- Missing/wrong filename returns None
- Optional license field handling
- Custom and default relative_path behavior
- Colons in description values
- Empty front matter handling

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-25 10:17:40 +08:00
luo jiyin 97ad67db6b docs: fix typo and grammar issues in docs (#1315)
* docs: fix security policy wording

* docs: fix backend agents typo
2026-03-25 10:01:36 +08:00
Matthew 2eca58bd86 fix: add null checks for runtime.context in middlewares and tools (#1269)
Add defensive null checks before accessing runtime.context.get() to
prevent AttributeError when runtime.context is None. This affects:
- UploadsMiddleware
- MemoryMiddleware
- LoopDetectionMiddleware
- SandboxMiddleware
- sandbox tools
- setup_agent_tool
- present_file_tool
- task_tool

Also adds .env loading in serve.sh for environment variable support.

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-25 08:46:42 +08:00
Anna Terek f499f37e94 docs: add Russian README translation (#1311) 2026-03-25 08:39:38 +08:00
Emile Jouannet 21febe1cc9 docs: add French translation of README (#1303) 2026-03-25 08:24:02 +08:00
greatmengqi 16ed797e0e feat: add configurable log level and token usage tracking (#1301)
* feat: add configurable log level and token usage tracking

- Add `log_level` config to control deerflow module log level, synced
  to LangGraph Server via serve.sh `--server-log-level`
- Add `token_usage.enabled` config with TokenUsageMiddleware that logs
  input/output/total tokens per LLM call from usage_metadata
- Add .omc/ to .gitignore

* fix: use info level for token usage logs since feature has its own toggle

* fix: sort imports to pass lint check

---------

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-25 08:13:26 +08:00
d 🔹 77b8ef79ca fix(middleware): use HumanMessage in LoopDetectionMiddleware for Anthropic compat (#1300)
LoopDetectionMiddleware injected SystemMessage mid-conversation to warn
about repetitive tool calls. This crashes Anthropic models because
langchain_anthropic's _format_messages() requires system messages to
appear only at the start of the conversation — interleaved system
messages raise 'Received multiple non-consecutive system messages'.

Switch the warning injection from SystemMessage to HumanMessage, which
works with all providers (Anthropic, OpenAI, Google, etc.).

Fixes #1299

Co-authored-by: voidborne-d <voidborne-d@users.noreply.github.com>
2026-03-25 08:00:01 +08:00
Jason 067b19af00 fix: add Windows compatibility for make dev/start commands (#1297)
* fix: add Windows compatibility for make dev/start commands

On Windows with MinGW/Git Bash, the Makefile's direct shell script
execution fails with 'CreateProcess(NULL, env bash ...)' error.

This fix:
- Detects Windows via $(OS) == Windows_NT
- Uses explicit bash invocation on Windows
- Falls back to direct execution on Unix

Users need Git Bash installed (comes with Git for Windows).

Fixes #1288
Related #1278

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-24 23:01:45 +08:00
knukn a9940c391c fix(mcp): implement sync invocation wrapper for async MCP tools (#1287)
* fix(mcp): implement sync invocation wrapper for async MCP tools

Since DeerFlowClient streams synchronously, invoking async-only MCP tools
(loaded via langchain-mcp-adapters) resulted in a NotImplementedError.
This commit bridges the sync/async gap by dynamically injecting a `func`
wrapper into `StructuredTool` instances that only have a `coroutine`.

Key changes:
- Added `sync_wrapper` in `get_mcp_tools` to execute async tool calls.
- Handled nested event loops by delegating to a global `ThreadPoolExecutor`
  when an event loop is already running, avoiding `RuntimeError`.
- Added detailed error logging within the wrapper for better transparency.
- Added comprehensive test coverage in `test_mcp_sync_wrapper.py` verifying
  tool patching, event loop behavior, and exception propagation.

* refactor(mcp): extract sync wrapper to module level and fix test mocks

Addressed PR review comments:
- Extracted _make_sync_tool_wrapper to module level to avoid nested func definitions.
- Refactored tests to use the actual production helper instead of duplicating logic.
- Fixed AsyncMock patching for awaited dependencies in tests.
- Added atexit hook for graceful thread pool shutdown.
- Fixed PEP8 blank line formatting in tests.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-24 22:38:01 +08:00
kristoffern 6bf526748d fix(skills): follow symlinks when scanning custom skills directory (#1292)
os.walk() does not follow symbolic links by default. This means
custom skills installed as symlinks in skills/custom/ are discovered
as directories but never descended into, so their SKILL.md files
are never found and the skills silently fail to load.

Adding followlinks=True fixes this for users who symlink skill
directories from external projects into the custom skills folder.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-24 22:06:29 +08:00
orbisai0security 14a3fa5290 fix: use subprocess instead of os.system in analyze.py (#1289)
The data analysis skill executes shell commands using os
Resolves V-001

Co-authored-by: orbisai0security <orbisai0security@users.noreply.github.com>
2026-03-24 20:42:03 +08:00
evenboos 4b15f14647 fix: repair frontend check command and docs (#1281)
* fix: repair frontend check command and docs

* docs: 补充 Linux 下 Docker 权限排障说明
2026-03-24 17:02:54 +08:00
dependabot[bot] c5ddc6a171 build(deps): bump h3 from 1.15.5 to 1.15.10 in /frontend (#1280)
Bumps [h3](https://github.com/h3js/h3) from 1.15.5 to 1.15.10.
- [Release notes](https://github.com/h3js/h3/releases)
- [Changelog](https://github.com/h3js/h3/blob/v1.15.10/CHANGELOG.md)
- [Commits](https://github.com/h3js/h3/compare/v1.15.5...v1.15.10)

---
updated-dependencies:
- dependency-name: h3
  dependency-version: 1.15.10
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-24 14:39:57 +08:00
Willem Jiang d0049ad904 chron(ci):setup the lint check in frontend (#1276)
* chron(ci):setup the lint check in frontend

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(ci): correct lint-check.yml indentation, add Python 3.12 setup, upgrade checkout to v4 (#1277)

* Initial plan

* Fix lint-check.yml: fix steps indentation, add Python 3.12 setup, upgrade checkout to v4

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/7b4d4fad-f024-453a-9f93-5fc2dd83b471

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
2026-03-24 10:48:18 +08:00
Willem Jiang 48a197555b fix(frontend): fix the build error of i18n (#1274) 2026-03-24 09:55:39 +08:00
Gao Mingfei 0431a67b68 fix(frontend): filter task tool calls when rendering SubtaskCard (#1242)
Only tool calls with name === "task" should be rendered as SubtaskCard.
Previously all tool_calls were mapped to IDs, causing SubtaskCard to
render for non-task tool calls whose IDs were never registered in the
subtask context, resulting in a TypeError on task.status.

Signed-off-by: Gao Mingfei <g199209@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-24 09:44:36 +08:00
Matt Van Horn b40b05f623 feat(frontend): display token usage per conversation turn (#1229)
Surface the usage_metadata that PR #1218 added to the streaming API.
A compact indicator in the chat header shows cumulative tokens consumed
per thread, with a tooltip breakdown of input/output/total counts.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-24 08:59:35 +08:00
amdoi7. 8b0f3fe233 fix(threads): clean up local thread data after thread deletion (#1262)
* fix(threads): clean up local thread data after thread deletion

Delete DeerFlow-managed thread directories after the web UI removes a LangGraph thread.
This keeps local thread data in sync with conversation deletion and adds regression coverage for the cleanup flow.

* fix(threads): address thread cleanup review feedback

Encode thread cleanup URLs in the web client, keep cache updates explicit when no thread search data is cached, and return a generic 500 response from the cleanup endpoint while documenting the sanitized error behavior.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-24 00:36:08 +08:00
Jason 79acc3939a fix: add error handling for podcast generation failures (#1257)
* fix: add error handling for podcast generation failures

When TTS processing fails, the system was generating 0-second audio files
without any error indication. This fix adds:

1. Track failed TTS lines and log warning with indices
2. Raise ValueError when all TTS generation fails with helpful message
3. Check for empty audio output in mix_audio and raise error
4. Log success/failure ratio for debugging

Fixes #30

* fix: address Copilot review feedback

- Use `not audio` to catch both None and empty bytes
- Log failed lines with 1-based indices for user-friendly output
- Handle empty script case with clear error message
- Validate env vars before ThreadPoolExecutor for fast-fail on config errors

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-24 00:20:12 +08:00
Willem Jiang 3be1d841aa fix(hotkey):support to open settings with hotkey (#1259) 2026-03-23 18:53:06 +08:00
Matt Van Horn 48031e506b feat(frontend): add Cmd+K command palette and keyboard shortcuts (#1230)
* feat(frontend): add Cmd+K command palette and keyboard shortcuts

Wire up the existing shadcn/ui Command component as a global command
palette. Adds a useGlobalShortcuts hook for Cmd+K (palette), Cmd+Shift+N
(new chat), Cmd+, (settings), and Cmd+/ (shortcuts help overlay).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(frontend): address Copilot review feedback on command palette

- Normalize event.key with toLowerCase() for reliable Shift+key matching
- Replace dead deerflow:open-settings event with router.push navigation
- Use platform-appropriate Shift label (Shift+ on Windows/Linux, glyph on Mac)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-23 18:35:35 +08:00
Uchi Uchibeke a29134d7c9 feat(guardrails): add pre-tool-call authorization middleware with pluggable providers (#1240)
Add GuardrailMiddleware that evaluates every tool call before execution.
Three provider options: built-in AllowlistProvider (zero deps), OAP passport
providers (open standard), or custom providers loaded by class path.

- GuardrailProvider protocol with GuardrailRequest/Decision dataclasses
- GuardrailMiddleware (AgentMiddleware, position 5 in chain)
- AllowlistProvider for simple deny/allow by tool name
- GuardrailsConfig (Pydantic singleton, loaded from config.yaml)
- 25 tests covering allow/deny, fail-closed/open, async, GraphBubbleUp
- Comprehensive docs at backend/docs/GUARDRAILS.md

Closes #1213

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-23 18:07:33 +08:00
knukn fe75cb35ca feat(client): support agent_name injection to enable isolated memory and custom prompts (#1253)
* feat(client): 添加agent_name参数支持自定义代理名称

允许在初始化DeerFlowClient时指定代理名称,该名称将用于中间件构建和系统提示模板

* test: add coverage for agent_name parameter in DeerFlowClient

* fix(client): address PR review comments for agent_name injection

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-23 17:44:21 +08:00
infoquest-byteplus f6c54e0308 infoquest support image-search (#1255) 2026-03-23 17:06:56 +08:00
Ben Ghorbel Mohamed Aziz 38ace61617 feat(web): add conversation export as Markdown and JSON (#1002)
* feat(web): add conversation export as Markdown and JSON (#976)

Add the ability to export conversations in Markdown and JSON formats,
accessible from both the chat header and the sidebar context menu.

- Add export utility (formatThreadAsMarkdown, formatThreadAsJSON) with
  support for user/assistant messages, thinking blocks, and tool calls
- Add ExportTrigger component in chat header (appears when messages exist)
- Add Export submenu to sidebar dropdown (fetches full thread state on demand)
- Add i18n translations for en-US and zh-CN

Closes #976

Made-with: Cursor

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update thread creation timestamp to updated_at

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-23 08:21:54 +08:00
Jason 1c981ead2a fix: add ~/.codex and ~/.claude bind mounts to docker-compose-dev.yaml (#1247)
The dev compose file was missing CLI auth directory mounts that exist in
the production compose file. This caused CodexChatModel to fail with
'Codex CLI credential not found' error in dev mode.

Fixes #1246
2026-03-23 07:44:59 +08:00
Purricane 835ba041f8 feat: add Claude Code OAuth and Codex CLI as LLM providers (#1166)
* feat: add Claude Code OAuth and Codex CLI providers

Port of bytedance/deer-flow#1136 from @solanian's feat/cli-oauth-providers branch.\n\nCarries the feature forward on top of current main without the original CLA-blocked commit metadata, while preserving attribution in the commit message for review.

* fix: harden CLI credential loading

Align Codex auth loading with the current ~/.codex/auth.json shape, make Docker credential mounts directory-based to avoid broken file binds on hosts without exported credential files, and add focused loader tests.

* refactor: tighten codex auth typing

Replace the temporary Any return type in CodexChatModel._load_codex_auth with the concrete CodexCliCredential type after the credential loader was stabilized.

* fix: load Claude Code OAuth from Keychain

Match Claude Code's macOS storage strategy more closely by checking the Keychain-backed credentials store before falling back to ~/.claude/.credentials.json. Keep explicit file overrides and add focused tests for the Keychain path.

* fix: require explicit Claude OAuth handoff

* style: format thread hooks reasoning request

* docs: document CLI-backed auth providers

* fix: address provider review feedback

* fix: harden provider edge cases

* Fix deferred tools, Codex message normalization, and local sandbox paths

* chore: narrow PR scope to OAuth providers

* chore: remove unrelated frontend changes

* chore: reapply OAuth branch frontend scope cleanup

* fix: preserve upload guards with reasoning effort wiring

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-22 22:39:50 +08:00
mxyhi e119dc74ae feat(codex): support explicit OpenAI Responses API config (#1235)
* feat: support explicit OpenAI Responses API config

Co-authored-by: Codex <noreply@openai.com>

* Update backend/packages/harness/deerflow/config/model_config.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-22 20:39:26 +08:00
Gao Mingfei 644501ae07 fix(config): reload AppConfig when config path or mtime changes (#1239)
* fix(config): reload AppConfig when config path or mtime changes

- Track resolved path + mtime; invalidate cache on change
- Preserve set_app_config() injection behavior
- Add regression tests (test_app_config_reload.py)
- Document behavior in README and backend/CLAUDE.md

Signed-off-by: Gao Mingfei <g199209@gmail.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Signed-off-by: Gao Mingfei <g199209@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-22 20:34:01 +08:00
haoliangxu e6c6770b70 fix(middleware): fallback to configurable thread_id in thread data middleware (#1237)
Co-authored-by: Exploreunive <Exploreunive@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-22 20:14:51 +08:00
Ryanba 894875ab1b fix(gateway): accept output_text suggestion blocks (#1238)
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-22 19:59:54 +08:00
Chetan Sharma 7a90055ede fix(telegram): fix reply ordering race condition (#1231)
* fix(telegram): fix reply ordering race condition

* fix(telegram): address async race condition and add regression test

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-22 19:38:27 +08:00
Willem Jiang 72f01a1638 Update workflow to trigger on push to main
Add push trigger for unit tests on main branch
2026-03-22 17:57:06 +08:00
haoliangxu 3af709097e fix: normalize structured LLM content in serialization and memory updater (#1215)
* fix: normalize ToolMessage structured content in serialization

When models return ToolMessage content as a list of content blocks
(e.g. [{"type": "text", "text": "..."}]), the UI previously displayed
the raw Python repr string instead of the extracted text.

Replace str(msg.content) with the existing _extract_text() helper in
both _serialize_message() and stream() to properly normalize
list-of-blocks content to plain text.

Fixes #1149

Also fixes the same root cause as #1188 (characters displayed one per
line when tool response content is returned as structured blocks).

Added 11 regression tests covering string, list-of-blocks, mixed,
empty, and fallback content types.

* fix(memory): extract text from structured LLM responses in memory updater

When LLMs return response content as list of content blocks
(e.g. [{"type": "text", "text": "..."}]) instead of plain strings,
str() produces Python repr which breaks JSON parsing in the memory
updater. This caused memory updates to silently fail.

Changes:
- Add _extract_text() helper in updater.py for safe content normalization
- Use _extract_text() instead of str(response.content) in update_memory()
- Fix format_conversation_for_update() to handle plain strings in list content
- Fix subagent executor fallback path to extract text from list content
- Replace print() with structured logging (logger.info/warning/error)
- Add 13 regression tests covering _extract_text, format_conversation,
  and update_memory with structured LLM responses

* fix: address Copilot review - defensive text extraction + logger.exception

- client.py _extract_text: use block.get('text') + isinstance check (prevent KeyError/TypeError)
- prompt.py format_conversation_for_update: same defensive check for dict text blocks
- executor.py: type-safe text extraction in both code paths, fallback to placeholder instead of str(raw_content)
- updater.py: use logger.exception() instead of logger.error() for traceback preservation

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: preserve chunked structured content without spurious newlines

* fix: restore backend unit test compatibility

---------

Co-authored-by: Exploreunive <Exploreunive@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-22 17:29:29 +08:00
dependabot[bot] 9fad717977 build(deps): bump h3 from 1.15.5 to 1.15.9 in /frontend (#1234)
Bumps [h3](https://github.com/h3js/h3) from 1.15.5 to 1.15.9.
- [Release notes](https://github.com/h3js/h3/releases)
- [Changelog](https://github.com/h3js/h3/blob/v1.15.9/CHANGELOG.md)
- [Commits](https://github.com/h3js/h3/compare/v1.15.5...v1.15.9)

---
updated-dependencies:
- dependency-name: h3
  dependency-version: 1.15.9
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-22 09:17:31 +08:00
Ikko Eltociear Ashimine 9dbcca579d docs: add Japanese README (#1209)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-21 10:37:32 +08:00
haoliangxu 06cba217c3 feat: track token usage per conversation turn (#1218)
* feat: track token usage per conversation turn

Add token usage tracking to the streaming API so consumers can monitor
cost per turn without additional API calls.

Changes:

1. _serialize_message now includes usage_metadata for AI messages in
   values events, exposing input_tokens/output_tokens/total_tokens
   from LangChain's native metadata.

2. stream() accumulates token usage across all AI messages in a turn
   and emits the cumulative totals in the end event:
   {usage: {input_tokens: N, output_tokens: N, total_tokens: N}}

3. Each messages-tuple AI event with text content now includes a
   per-message usage_metadata field for granular tracking.

This enables the frontend to display token consumption per turn,
support cost-aware UX, and let users monitor API spending.

10 tests added covering serialization passthrough and cumulative
aggregation logic.

Co-Authored-By: OpenClaw <noreply@openclaw.ai>

* fix: address Copilot review - use Mapping access for usage_metadata

- Replace getattr(usage, 'input_tokens', 0) with usage.get('input_tokens', 0)
  since LangChain usage_metadata is a dict, not an object
- Remove unused 'import pytest' (fixes Ruff F401)
- Add proper stream() integration tests for cumulative usage in end event
  and per-message usage_metadata in messages-tuple events

---------

Co-authored-by: Exploreunive <Exploreunive@users.noreply.github.com>
Co-authored-by: OpenClaw <noreply@openclaw.ai>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-21 10:29:52 +08:00
Chetan Sharma e69dc2961f refactor: add channel-based streaming capability check (#1214) 2026-03-20 23:44:09 +08:00
dependabot[bot] 9a99485905 build(deps): bump kysely from 0.28.11 to 0.28.13 in /frontend (#1211)
Bumps [kysely](https://github.com/kysely-org/kysely) from 0.28.11 to 0.28.13.
- [Release notes](https://github.com/kysely-org/kysely/releases)
- [Commits](https://github.com/kysely-org/kysely/compare/v0.28.11...v0.28.13)

---
updated-dependencies:
- dependency-name: kysely
  dependency-version: 0.28.13
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-20 17:29:01 +08:00
Simon Su ceab7fac14 fix: improve MiniMax code plan integration (#1169)
This PR improves MiniMax Code Plan integration in DeerFlow by fixing three issues in the current flow: stream errors were not clearly surfaced in the UI, the frontend could not display the actual provider model ID, and MiniMax reasoning output could leak into final assistant content as inline <think>...</think>. The change adds a MiniMax-specific adapter, exposes real model IDs end-to-end, and adds a frontend fallback for historical messages.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-20 17:18:59 +08:00
knukn 3b235fd182 fix(feishu): support @bot message in topic groups (#1206)
* fix(feishu): support @bot message in topic groups

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix(feishu): preserve rich-text formatting and add parser unit tests

* chore(test): remove unused import to fix ruff lint error

* style: auto-format imports to satisfy ruff

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-20 17:03:39 +08:00
JilongSun c037ed6739 feat(manager): add bootstrap command to initialize soul.md in correct place (#1201)
* feat(manager): add bootstrap command to initialize soul.md in correct place

* feat(channels): add /bootstrap command to IM channels

Add a `/bootstrap` command that routes to the chat handler with
`is_bootstrap: True` in the run context, allowing the agent to invoke
its setup/initialization flow (e.g. `setup_agent`).

- The text after `/bootstrap` is forwarded as the chat message; when
  omitted a default "Initialize workspace" message is used.
- Feishu channels use the streaming path as with normal chat.
- No changes to ChannelStore — bootstrap is stateless and triggered
  purely by the command.
- Update /help output to include /bootstrap.
- Add 5 tests covering: text/no-text variants, Feishu streaming path,
  thread creation, and help text.

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix: accept copilot suggestion

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-20 16:54:11 +08:00
greatmengqi accf5b5f8e fix: add sync after_model to TitleMiddleware (#1190) 2026-03-19 15:46:31 +08:00
Ryanba f67c3d2c9e fix(harness): skip duplicate memory facts (#1193)
* fix(harness): skip duplicate memory facts

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* docs: note memory fact deduplication

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-18 22:41:13 +08:00
BillionToken 423ea59491 fix(scripts): handle docker-init failures gracefully for private registry (#1191)
* fix(scripts): handle docker-init failures gracefully for private registry

The make docker-init command was failing on Linux environments when users
could not access the private Volces container registry. This commonly
occurs in corporate intranet environments with proxies or for users
without registry credentials.

Changes:
- Detect sandbox mode from config.yaml before attempting image pull
- Skip image pull entirely for local sandbox mode (default)
- Gracefully handle pull failures with informative messages
- Update setup-sandbox Makefile target with same error handling

Fixes #1168

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: BillionClaw <billionclaw@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-18 22:06:35 +08:00
Ryanba 4c78188896 fix(gateway): remove generated markdown on upload delete (#1170)
* fix(gateway): remove generated markdown on upload delete

Keep thread upload storage consistent by deleting the generated markdown companion when the original convertible upload is removed.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-18 16:31:26 +08:00
Ryanba f737fbeae8 fix(frontend): block duplicate sends during uploads (#1165)
* fix(frontend): block duplicate sends during uploads

Expose pre-submit upload work as a busy state so the chat input does not allow a second send while the first attachment is still uploading.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* docs(frontend): document upload and stream ownership

Record that thread hooks own upload-before-submit state while the chat page owns composer busy wiring, so future changes do not reintroduce duplicate socket or upload state handling.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* fix(frontend): separate upload busy state from streaming

Keep uploads from reusing the streaming stop state so duplicate submits are blocked without turning the composer into a stop button during file uploads.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

---------

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-18 15:10:27 +08:00
dependabot[bot] beb0eab711 build(deps): bump pyasn1 from 0.6.2 to 0.6.3 in /backend (#1185)
Bumps [pyasn1](https://github.com/pyasn1/pyasn1) from 0.6.2 to 0.6.3.
- [Release notes](https://github.com/pyasn1/pyasn1/releases)
- [Changelog](https://github.com/pyasn1/pyasn1/blob/main/CHANGES.rst)
- [Commits](https://github.com/pyasn1/pyasn1/compare/v0.6.2...v0.6.3)

---
updated-dependencies:
- dependency-name: pyasn1
  dependency-version: 0.6.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-18 08:05:16 +08:00
dependabot[bot] 4977c43974 build(deps): bump next from 16.1.6 to 16.1.7 in /frontend (#1186)
Bumps [next](https://github.com/vercel/next.js) from 16.1.6 to 16.1.7.
- [Release notes](https://github.com/vercel/next.js/releases)
- [Changelog](https://github.com/vercel/next.js/blob/canary/release.js)
- [Commits](https://github.com/vercel/next.js/compare/v16.1.6...v16.1.7)

---
updated-dependencies:
- dependency-name: next
  dependency-version: 16.1.7
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-18 08:04:24 +08:00
dependabot[bot] 5b37de60b7 build(deps): bump flatted from 3.3.3 to 3.4.2 in /frontend (#1184)
Bumps [flatted](https://github.com/WebReflection/flatted) from 3.3.3 to 3.4.2.
- [Commits](https://github.com/WebReflection/flatted/compare/v3.3.3...v3.4.2)

---
updated-dependencies:
- dependency-name: flatted
  dependency-version: 3.4.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-18 07:50:32 +08:00
DanielWalnut feac03ecbc fix(harness): allow agent read access to /mnt/skills in local sandbox (#1178)
* fix(harness): allow agent read access to /mnt/skills in local sandbox

Skill files under /mnt/skills/ were blocked by the path validator,
preventing agents from reading skill definitions. This change:

- Refactors `resolve_local_tool_path` into `validate_local_tool_path`,
  a pure security gate that no longer resolves paths (left to the sandbox)
- Permits read-only access to the skills container path (/mnt/skills by
  default, configurable via config.skills.container_path)
- Blocks write access to skills paths (PermissionError)
- Allows /mnt/skills in bash command path validation
- Adds `LocalSandbox.update_path_mappings` and injects per-thread
  user-data mappings into the sandbox so all virtual-path resolution
  is handled uniformly by the sandbox layer
- Covers all new behaviour with tests

Fixes #1177

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(sandbox): unify all virtual path resolution in tools.py

Move skills path resolution from LocalSandbox into tools.py so that all
virtual-to-host path translation (user-data and skills) lives in one
layer.  LocalSandbox becomes a pure execution layer that receives only
real host paths — no more path_mappings, _resolve_path, or reverse
resolve logic.

This addresses architecture feedback that path resolution was split
across two layers (tools.py for user-data, LocalSandbox for skills),
making the flow hard to follow.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(sandbox): address Copilot review — cache-on-success and error path masking

- Replace @lru_cache with manual cache-on-success for _get_skills_container_path
  and _get_skills_host_path so transient failures at startup don't permanently
  disable skills access.
- Add _sanitize_error() helper that masks host filesystem paths in error
  messages via mask_local_paths_in_output before returning them to the agent.
- Apply _sanitize_error() to all catch-all (Exception/OSError) handlers in
  sandbox tool functions to prevent host path leakage in error output.
- Remove unused lru_cache import.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-17 21:44:36 +08:00
lhd 0091d9f071 feat(tools): add tool_search for deferred MCP tool loading (#1176)
* feat(tools): add tool_search for deferred MCP tool loading

When multiple MCP servers are enabled, total tool count can exceed 30-50,
causing context bloat and degraded tool selection accuracy. This adds a
deferred tool loading mechanism controlled by `tool_search.enabled` config.

- Add ToolSearchConfig with single `enabled` field
- Add DeferredToolRegistry with regex search (select:, +keyword, keyword)
- Add tool_search tool returning OpenAI-compatible function JSON
- Add DeferredToolFilterMiddleware to hide deferred schemas from bind_tools
- Add <available-deferred-tools> section to system prompt
- Enable MCP tool_name_prefix to prevent cross-server name collisions
- Add 34 unit tests covering registry, tool, prompt, and middleware

* fix: reset stale deferred registry and bump config_version

- Reset deferred registry upfront in get_available_tools() to prevent
  stale tool entries when MCP servers are disabled between calls
- Bump config_version to 2 for new tool_search config field

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(tests): mock get_app_config in prompt section tests for CI

CI has no config.yaml, causing TestDeferredToolsPromptSection to fail
with FileNotFoundError. Add autouse fixture to mock get_app_config.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-17 20:43:55 +08:00
Henry Li f29db80be7 docs: add coding plan from ByteDance Volcengine (#1174)
* docs: add coding plan

* docs: add coding plan
2026-03-17 14:33:47 +08:00
Henry Li cb4cae4064 docs: add README in Chinese (#1172)
Co-authored-by: Henry Li <lixin.henry@bytedance.com>
2026-03-17 13:51:01 +08:00
Karesansui 75c96300cf fix(scripts): add next-server to serve.sh cleanup trap (#1162)
The cleanup() trap kills "next dev" and "next start" but not
"next-server". Since "next start" forks a "next-server" child
process, killing the parent may leave the child running as a
zombie, holding port 3000. The startup teardown block (line 35)
already handles this, but the Ctrl+C / SIGTERM trap did not.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-17 10:07:16 +08:00
lailoo 9809af1f26 feat: add citation/reference support to deep research reports (#1143)
* feat: add citation/reference support to deep research reports (#1141)

- Enhance lead agent system prompt with mandatory citation requirements
  after web_search/web_fetch tool usage
- Add citation examples and best practices to GitHub Deep Research skill
- Add citation hints to report template (Executive Summary, Key Analysis)
- Style regular markdown links in frontend for visual distinction
  (color, underline, hover effect)
- Fix TitleMiddleware being registered when title generation is disabled

* fix: address PR review comments

- Revert TitleMiddleware conditional registration (agent.py) to avoid
  sync/async incompatibility with DeerFlowClient
- Fix markdown link rendering: merge classNames instead of overwriting,
  only set target=_blank for external http(s) URLs
- Remove unrelated package.json/pnpm-lock.yaml changes

* fix: use plain markdown links in Sources section for cleaner rendering

Inline citations in report body use [citation:Title](URL) for pill/badge style.
Sources section uses plain [Title](URL) for simple underlined link style.

* fix(frontend): render plain links as underlined text in artifact markdown

Only links with citation: prefix render as Badge pills.
Regular links in Sources section now render as underlined text links.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-17 09:51:08 +08:00
Ryanba b1913a1902 fix(harness): normalize structured content for titles (#1155)
* fix(harness): normalize structured content for titles

Flatten structured LangChain message content before prompting the title model so list/block payloads don't leak Python reprs into generated thread titles.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-17 09:10:09 +08:00
Karesansui ab0c10f002 fix(makefile): correct docker-init help description (#1163)
The help text described docker-init as "Build the custom k3s image"
but the actual implementation (scripts/docker.sh init) only pulls
the sandbox image. Updated to match the real behavior.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-16 21:53:58 +08:00
Matt Van Horn 609ff5849f fix(frontend): gracefully handle missing WebGL context (#1147)
Wrap the OGL Renderer instantiation in a try-catch so the app does not
crash when WebGL is unavailable (e.g. hardware acceleration disabled).
The Galaxy background simply does not render instead of taking down the
entire page.

Fixes #1144

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-16 21:22:17 +08:00
Karesansui 3212c7c5a2 fix(scripts): correct Makefile target name in docker.sh restart message (#1161)
docker.sh restart() tells users to run `make docker-dev-logs`, but
this target does not exist. The correct target is `make docker-logs`.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 19:58:59 +08:00
-Astraia- 191b60a326 fix: issue 1138 windows encoding (#1139)
* fix(windows): use utf-8 for text file operations

* fix(windows): normalize sandbox path masking

* fix(windows): preserve utf-8 handling after backend split
2026-03-16 16:53:12 +08:00
DanielWalnut 76803b826f refactor: split backend into harness (deerflow.*) and app (app.*) (#1131)
* refactor: extract shared utils to break harness→app cross-layer imports

Move _validate_skill_frontmatter to src/skills/validation.py and
CONVERTIBLE_EXTENSIONS + convert_file_to_markdown to src/utils/file_conversion.py.
This eliminates the two reverse dependencies from client.py (harness layer)
into gateway/routers/ (app layer), preparing for the harness/app package split.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: split backend/src into harness (deerflow.*) and app (app.*)

Physically split the monolithic backend/src/ package into two layers:

- **Harness** (`packages/harness/deerflow/`): publishable agent framework
  package with import prefix `deerflow.*`. Contains agents, sandbox, tools,
  models, MCP, skills, config, and all core infrastructure.

- **App** (`app/`): unpublished application code with import prefix `app.*`.
  Contains gateway (FastAPI REST API) and channels (IM integrations).

Key changes:
- Move 13 harness modules to packages/harness/deerflow/ via git mv
- Move gateway + channels to app/ via git mv
- Rename all imports: src.* → deerflow.* (harness) / app.* (app layer)
- Set up uv workspace with deerflow-harness as workspace member
- Update langgraph.json, config.example.yaml, all scripts, Docker files
- Add build-system (hatchling) to harness pyproject.toml
- Add PYTHONPATH=. to gateway startup commands for app.* resolution
- Update ruff.toml with known-first-party for import sorting
- Update all documentation to reflect new directory structure

Boundary rule enforced: harness code never imports from app.
All 429 tests pass. Lint clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: add harness→app boundary check test and update docs

Add test_harness_boundary.py that scans all Python files in
packages/harness/deerflow/ and fails if any `from app.*` or
`import app.*` statement is found. This enforces the architectural
rule that the harness layer never depends on the app layer.

Update CLAUDE.md to document the harness/app split architecture,
import conventions, and the boundary enforcement test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add config versioning with auto-upgrade on startup

When config.example.yaml schema changes, developers' local config.yaml
files can silently become outdated. This adds a config_version field and
auto-upgrade mechanism so breaking changes (like src.* → deerflow.*
renames) are applied automatically before services start.

- Add config_version: 1 to config.example.yaml
- Add startup version check warning in AppConfig.from_file()
- Add scripts/config-upgrade.sh with migration registry for value replacements
- Add `make config-upgrade` target
- Auto-run config-upgrade in serve.sh and start-daemon.sh before starting services
- Add config error hints in service failure messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix comments

* fix: update src.* import in test_sandbox_tools_security to deerflow.*

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: handle empty config and search parent dirs for config.example.yaml

Address Copilot review comments on PR #1131:
- Guard against yaml.safe_load() returning None for empty config files
- Search parent directories for config.example.yaml instead of only
  looking next to config.yaml, fixing detection in common setups

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: correct skills root path depth and config_version type coercion

- loader.py: fix get_skills_root_path() to use 5 parent levels (was 3)
  after harness split, file lives at packages/harness/deerflow/skills/
  so parent×3 resolved to backend/packages/harness/ instead of backend/
- app_config.py: coerce config_version to int() before comparison in
  _check_config_version() to prevent TypeError when YAML stores value
  as string (e.g. config_version: "1")
- tests: add regression tests for both fixes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: update test imports from src.* to deerflow.*/app.* after harness refactor

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 22:55:52 +08:00
YolenSong 9b49a80dda feat(feishu): stream updates on a single card (#1031)
* feat(feishu): stream updates on a single card

* fix(feishu): ensure final message on stream error and warn on missing card ID

- Wrap streaming loop in try/except/finally so a is_final=True outbound
  message is always published, even when the LangGraph stream breaks
  mid-way. This prevents _running_card_ids memory leaks and ensures the
  Feishu card shows a DONE reaction instead of hanging on "Working on it".
- Log a warning when _ensure_running_card gets no message_id back from
  the Feishu reply API, making silent fallback to new-card behavior
  visible in logs.
- Add test_handle_feishu_stream_error_still_sends_final to cover the
  error path.
- Reformat service.py dict comprehension (ruff format, no logic change).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Avoid blocking inbound on Feishu card creation

---------

Co-authored-by: songyaolun <songyaolun@bytedance.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-14 22:24:35 +08:00
virtaava d18a9ae5aa feat: add LoopDetectionMiddleware to break repetitive tool call loops (#1056)
* feat: add LoopDetectionMiddleware to break repetitive tool call loops

Adds a new AgentMiddleware that detects when the agent is stuck calling
the same tools with the same arguments repeatedly, which currently runs
until the recursion limit kills the run.

Detection: per-thread sliding window of tool call hashes (name + args).
- Warn threshold (default 3): injects a "wrap up" system message
- Hard limit (default 5): strips tool_calls, forcing final text output

Includes 13 unit tests covering hashing, thresholds, window sliding,
reset, and edge cases.

Closes #1055

* fix: address PR #1056 review feedback for LoopDetectionMiddleware

- Remove unused imports (Awaitable, Callable, ModelCallResult,
  ModelRequest, ModelResponse, AIMessage) from loop_detection_middleware
- Remove unused pytest import from test file
- Fix _hash_tool_calls sort key: sort by (name, serialized args) for
  deterministic hashing when multiple calls share the same tool name
- Revert subagent_enabled default to False in agent.py to match
  DeerFlowClient and channel defaults
- Remove unrelated SearxNG tools and Next.js rewrite changes from PR

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address 2nd round review feedback on PR #1056

- Inject loop warning only once per thread (prevents context bloat)
- Add threading.Lock for thread-safe history mutations
- Use runtime.context thread_id instead of workspace_path
- Add LRU eviction for per-thread history (max 100 threads)
- Add 5 new tests covering warn-once, LRU eviction, thread isolation,
  fallback thread_id, and lock presence

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: resolve lint errors in loop detection middleware tests

Sort imports (I001) and remove unused _WARNING_MSG import (F401)
to fix ruff lint failures in CI.

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-14 22:17:54 +08:00
Octopus bbd87df6eb Add MiniMax as an OpenAI-compatible model provider (#1120)
* Add MiniMax as an OpenAI-compatible model provider

MiniMax offers high-performance LLMs (M2.5, M2.5-highspeed) with
204K context windows. This commit adds MiniMax as a selectable
provider in the configuration system.

Changes:
- Add MiniMax to SUPPORTED_MODELS with model definitions
- Add MiniMax provider configuration in conf/config.yaml
- Update documentation with MiniMax setup instructions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Update README to remove MiniMax API details

Removed mention of MiniMax API usage and configuration examples.

---------

Co-authored-by: octo-patch <octo-patch@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-14 22:05:18 +08:00
dependabot[bot] cc192a9846 build(deps): bump pyjwt from 2.10.1 to 2.12.0 in /backend (#1135)
Bumps [pyjwt](https://github.com/jpadilla/pyjwt) from 2.10.1 to 2.12.0.
- [Release notes](https://github.com/jpadilla/pyjwt/releases)
- [Changelog](https://github.com/jpadilla/pyjwt/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/jpadilla/pyjwt/compare/2.10.1...2.12.0)

---
updated-dependencies:
- dependency-name: pyjwt
  dependency-version: 2.12.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-14 10:03:09 +08:00
dependabot[bot] 9983f9d296 build(deps): bump flatted from 3.3.3 to 3.4.1 in /frontend (#1134)
Bumps [flatted](https://github.com/WebReflection/flatted) from 3.3.3 to 3.4.1.
- [Commits](https://github.com/WebReflection/flatted/compare/v3.3.3...v3.4.1)

---
updated-dependencies:
- dependency-name: flatted
  dependency-version: 3.4.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-14 10:00:51 +08:00
Matt(허철진) d197d50146 fix: preserve conversation context in Telegram private chats (#1105)
* fix: preserve conversation context in Telegram private chats

In private (1-on-1) chats, set topic_id=None so all messages map to a
single DeerFlow thread per chat instead of creating a new thread for
every message. Also fix _cmd_generic to use topic_id=None in private
chats so /new correctly targets the default thread.

Group chat behavior is unchanged (reply_to or msg_id as topic_id).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: preserve conversation context in Telegram private chats

Fixes #1101

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: mirror _on_text reply logic in _cmd_generic for group chats

_cmd_generic now prefers reply_to_message.message_id over msg_id in
group/supergroup chats, consistent with _on_text. This ensures commands
like /new and /status target the correct conversation thread when sent
as a reply in group chats.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: JeffJiang <for-eleven@hotmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-14 09:47:24 +08:00
dependabot[bot] d6bfadab12 build(deps): bump orjson from 3.11.5 to 3.11.6 in /backend (#1133)
Bumps [orjson](https://github.com/ijl/orjson) from 3.11.5 to 3.11.6.
- [Release notes](https://github.com/ijl/orjson/releases)
- [Changelog](https://github.com/ijl/orjson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ijl/orjson/compare/3.11.5...3.11.6)

---
updated-dependencies:
- dependency-name: orjson
  dependency-version: 3.11.6
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
2026-03-13 23:03:23 +08:00
Willem Jiang 253fe4d87f feat(sandbox): harden local file access and mask host paths (#983)
* feat(sandbox): harden local file access and mask host paths

- enforce local sandbox file tools to only accept /mnt/user-data paths
- add path traversal checks against thread workspace/uploads/outputs roots
- preserve requested virtual paths in tool error messages (no host path leaks)
- mask local absolute paths in bash output back to virtual sandbox paths
- update bash tool guidance to prefer thread-local venv + python -m pip
- add regression tests for path mapping, masking, and access restrictions

Fixes #968

* feat(sandbox): restrict risky absolute paths in local bash commands

- validate absolute path usage in local-mode bash commands
- allow only /mnt/user-data virtual paths for user data access
- keep a small allowlist for system executable/device paths
- return clear permission errors for unsafe command paths
- add regression tests for bash path validation rules

* test(sandbox): add success path test for resolve_local_tool_path (#992)

* Initial plan

* test(sandbox): add success path test for resolve_local_tool_path

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

* fix(sandbox): reject bare virtual root early with clear error in resolve_local_tool_path (#991)

* Initial plan

* fix(sandbox): reject bare virtual root early with clear error in resolve_local_tool_path

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
2026-03-13 22:38:32 +08:00
Frank 918ba6b5bf docs: clarify OpenRouter configuration (#1123)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-13 22:12:30 +08:00
Ryanba 5a8481416f fix(frontend): surface upload API error details (#1113)
Preserve backend upload/list/delete error details in the frontend API layer so users see the actual server failure instead of a generic message.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-13 21:55:33 +08:00
Willem Jiang a79d414695 fix: make check/config cross-platform for Windows (#1080) (#1093)
* fix: make check/config cross-platform for Windows (#1080)

- replace shell-based check/config recipes with Python entrypoints
- add a cross-platform dependency checker script
- add a cross-platform config bootstrap script
- route make targets through a Python variable for consistent invocation
- preserve existing config-abort behavior when config file already exists

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-13 21:33:12 +08:00
Ryanba b155923ab0 fix(gateway): ignore archive metadata wrappers (#1108)
* fix(gateway): ignore archive metadata wrappers

Treat top-level __MACOSX and dotfile entries as packaging metadata so valid .skill archives still resolve to their real skill directory.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-13 21:27:54 +08:00
Ryanba cda9fb7bca fix(gateway): allow standard skill frontmatter metadata (#1103)
* fix(gateway): allow standard skill frontmatter metadata

Accept standard optional frontmatter fields during .skill installs so external skills with version, author, or compatibility metadata do not fail validation.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* docs: sync skill installer metadata behavior

Document the skill install allowlist so user-facing and backend contributor docs match the gateway validation contract.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-13 21:23:35 +08:00
Ryanba 03cafea715 fix(gateway): normalize suggestion response content (#1098)
* fix(gateway): normalize suggestion response content

Handle list-style model content before JSON parsing so provider wrappers do not silently drop follow-up suggestions.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* docs: sync suggestions endpoint behavior

Document the rich-content normalization path so the README and backend gateway notes stay aligned with the current router contract.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-13 21:20:15 +08:00
Willem Jiang b5fcb1334a fix(memory): inject stored facts into system prompt memory context (#1083)
* fix(memory): inject stored facts into system prompt memory context

- add Facts section rendering in format_memory_for_injection
- rank facts by confidence and coerce confidence values safely
- enforce max token budget while appending fact lines
- add regression tests for fact inclusion, ordering, and budget behavior

Fixes #1059

* Update the document with the latest status

* fix(memory): harden fact injection — NaN/inf confidence, None content, incremental token budget (#1090)

* Initial plan

* fix(memory): address review feedback on confidence coercion, None content, and token budget

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
2026-03-13 14:37:40 +08:00
Liu Jice 3521cc2668 fix(middleware): degrade tool-call exceptions to error tool messages (#1110)
* fix(middleware): degrade tool-call exceptions to error tool messages

* update script

* fix(middleware): preserve LangGraph control-flow exceptions in tool error handling
2026-03-13 09:41:59 +08:00
JeffJiang 08ea9d3038 feat: enhance Docker support with production setup and deployment script (#1086)
* feat: add `make start` command for local previewing

* Update Makefile

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: update help text for `make dev` and `make start` commands

* feat: enhance Docker support with production setup and deployment script

* feat: add production commands to Makefile

* feat: remove PostgreSQL and Redis services from Docker Compose and update deploy script

* fix: address Copilot review suggestions from Docker production PR #1086 (#10)

* Initial plan

* fix: address all review suggestions from PR #1086

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

* Update docker/docker-compose.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* feat: remove deprecated Dockerfile.langgraph to clean up repository

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-12 22:18:18 +08:00
JeffJiang fdacb1c3a5 fix(chat): update navigation method to prevent state loss during thread remount (#1107) 2026-03-12 14:57:17 +08:00
Willem Jiang e5a21b9ba0 fix(makefile):quick fix of the makefile formate error (#1085) 2026-03-11 16:00:40 +08:00
Ryanba 4bae3c724c fix(client): Harden upload validation and conversion flow (#989)
* fix(client): Harden upload validation and conversion flow

* test(client): cover event-loop upload conversion reuse

* test(client): remove unused import in upload regression coverage

* fix(client): load optional shared checkpointer fallback

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

* docs(backend): document config preflight and IM channels

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-03-11 15:17:31 +08:00
Orion a0c38a5cf3 feat: add dev-daemon target for background development mode (#1047)
* feat: add dev-daemon target for background development mode

Add a new make dev-daemon target that allows running DeerFlow services
in background mode without keeping the terminal connection.

Following the pattern of PR #1042, the implementation uses a dedicated
shell script (scripts/start-daemon.sh) for better maintainability.

- Create scripts/start-daemon.sh for daemon mode startup
- Add dev-daemon target to Makefile
- Each service writes logs to separate files (langgraph, gateway, frontend, nginx)
- Services can be stopped with make stop
- Use nohup for proper daemon process detachment
- Add cleanup on failure when services fail to start
- Use more specific pkill pattern to avoid killing unrelated nginx processes

* refactor: use wait-for-port.sh instead of hardcoded sleep in daemon script

* refactor: use specific nginx process pattern to avoid killing unrelated processes

* Revert "refactor: use specific nginx process pattern to avoid killing unrelated processes"

This reverts commit 4c369155bfc91ccce347876a8982f955fa039da8.

* refactor: use consistent nginx kill pattern across all scripts

* chore(daemon): add trap for cleanup on interrupt signals

* fix(daemon): pass repo root as positional argument to nginx command

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-11 15:10:40 +08:00
LofiSu 5d4fd9cf72 fix(frontend): fix new-chat navigation stale state issue (#1077)
- Use router.replace() instead of history.replaceState() so Next.js
  router's internal state is updated on chat start. This ensures
  subsequent "New Chat" clicks are treated as a real cross-route
  navigation (actual-id → "new") rather than a no-op same-path
  navigation, which was causing stale content to persist.

- In ChatLayout, increment the SubtasksProvider key only when
  navigating TO "new" from a non-"new" route. This forces a full
  remount for a fresh new-chat state without remounting when the URL
  transitions from "new" → actual-id (which would interrupt streaming).

Made-with: Cursor

Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
2026-03-11 13:51:51 +08:00
JeffJiang 2e7964d0aa feat: add make start command for local previewing (#1078)
* feat: add `make start` command for local previewing

* Update Makefile

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update Makefile

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update scripts/check.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: update help text for `make dev` and `make start` commands

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-11 13:46:05 +08:00
Willem Jiang 06c4b0c828 chore(issue):create an issue template to provide runtime information (#1069) 2026-03-11 10:28:07 +08:00
Willem Jiang 96dbee00e3 fix(tracing): support LANGCHAIN_* env fallback for LangSmith config (#1065)
* fix(tracing): support LANGCHAIN_* env fallback for LangSmith config

- add backward-compatible env parsing in tracing_config.py
- support fallback keys:
   LANGCHAIN_TRACING_V2 / LANGCHAIN_TRACING
   LANGCHAIN_API_KEY
   LANGCHAIN_PROJECT
   LANGCHAIN_ENDPOINT
- keep LANGSMITH_* as preferred source when both are present
- add regression tests in test_tracing_config.py

* fix(tracing): correct LANGSMITH_* precedence over LANGCHAIN_* for enabled flag (#1067)

* Initial plan

* fix(tracing): use first-present-wins logic for enabled flag, add precedence docs and test

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
2026-03-11 10:26:56 +08:00
JeffJiang f836d8e17c chore(docker): Refactor sandbox state management and improve Docker integration (#1068)
* Refactor sandbox state management and improve Docker integration

- Removed FileSandboxStateStore and SandboxStateStore classes for a cleaner architecture.
- Enhanced LocalContainerBackend to handle port allocation retries and introduced environment variable support for sandbox host configuration.
- Updated Paths class to include host_base_dir for Docker volume mounts and ensured proper permissions for sandbox directories.
- Modified ExtensionsConfig to improve error handling when loading configuration files and adjusted environment variable resolution.
- Updated sandbox configuration to include a replicas option for managing concurrent sandbox containers.
- Improved logging and context management in SandboxMiddleware for better sandbox lifecycle handling.
- Enhanced network port allocation logic to bind to 0.0.0.0 for compatibility with Docker.
- Updated Docker Compose files to ensure proper volume management and environment variable configuration.
- Created scripts to ensure necessary configuration files are present before starting services.
- Cleaned up unused MCP server configurations in extensions_config.example.json.

* Address Copilot review suggestions from PR #1068 (#9)

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
2026-03-11 10:03:01 +08:00
aworki 6ae7f0c0ee fix: load all thread pages in thread lists (#1044)
* fix(frontend): load all thread pages in thread lists

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-10 23:11:08 +08:00
Xinmin Zeng d5135ab757 fix(frontend): sanitize unsupported langgraph stream modes (#1050)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-10 18:56:19 +08:00
-Astraia- 19604e7f47 fix: improve port detection in WSL (#1061) 2026-03-10 18:52:25 +08:00
JeffJiang f5bd691172 feat(middleware): introduce TodoMiddleware for context-loss detection in todo management (#1041)
* feat(middleware): introduce TodoMiddleware for context-loss detection in todo management

* Address PR #1041 review suggestions: todo reminder dedup, thread switching, artifact deselect, debug log (#8)

* Initial plan

* Handle all suggestions from PR #1041 review

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

* fix(chat-box): prevent automatic deselection of artifacts when switching threads
fix(hooks): reset thread state on new thread creation

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>
2026-03-10 11:24:53 +08:00
Willem Jiang cf1c4a68ea docs: fix stream_mode examples for runs stream (#1033) (#1039) 2026-03-10 11:19:31 +08:00
DanielWalnut 33f086b612 feat(channels): upload file attachments via IM channels (Slack, Telegram, Feishu) (#1040) 2026-03-10 09:11:57 +08:00
momorebi 0409f8cefd fix(subagents): cleanup background tasks after completion to prevent memory leak (#1030)
* fix(subagents): cleanup background tasks after completion to prevent memory leak

Added cleanup_background_task() function to remove completed subagent results
from the global _background_tasks dict. Found a small issue: completed tasks
were never removed, causing memory to grow indefinitely with each subagent
execution.

Alternative approaches considered:
- Future + SubagentHandle pattern: Not chosen due to requiring refactoring

Chose the simple cleanup approach for minimal code changes while effectively
resolving the memory leak.

Changes:
- Add cleanup_background_task() in executor.py
- Call cleanup in all task_tool return paths (completed, failed, timed out)

* fix(subagents): prevent race condition in background task cleanup

Address Copilot review feedback on memory leak fix:

- Add terminal state check in cleanup_background_task() to only remove
  tasks that are COMPLETED/FAILED/TIMED_OUT or have completed_at set
- Remove cleanup call from polling safety-timeout branch in task_tool
  since the task may still be running
- Add comprehensive tests for cleanup behavior including:
  - Verification that cleanup is called on terminal states
  - Verification that cleanup is NOT called on polling timeout
  - Tests for terminal state check logic in executor

This prevents KeyError when the background executor tries to update
a task that was prematurely removed from _background_tasks.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-10 07:41:48 +08:00
JeffJiang f6508e0677 feat(dev): refactor service startup to use dedicated start script (#1042) 2026-03-10 07:38:19 +08:00
Willem Jiang 46918f0786 Revert "feat(threads): paginate full history via summaries endpoint (#1022)" (#1037)
This reverts commit 2f47f1ced2.
2026-03-09 16:25:08 +08:00
aworki 2f47f1ced2 feat(threads): paginate full history via summaries endpoint (#1022)
* feat(threads): add paginated summaries API and load full history

* fix(threads): address summaries review feedback

- validate summaries sort params and log gateway failures
- page frontend thread summaries without stale query keys or silent truncation
- export router modules and tighten thread list typing

Refs: 2901804166, 2901804176, 2901804179, 2901804180, 2901804183, 2901804187, 2901804191

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-09 16:08:02 +08:00
lailoo 959b4f2b09 fix(checkpointer): return InMemorySaver instead of None when not configured (#1016) (#1019)
* fix(checkpointer): return InMemorySaver instead of None when not configured (#1016)

* fix(checkpointer): also fix get_checkpointer() to return InMemorySaver

Make all three checkpointer functions consistent:
- make_checkpointer() (async) → InMemorySaver
- checkpointer_context() (sync) → InMemorySaver
- get_checkpointer() (sync singleton) → InMemorySaver

This ensures DeerFlowClient always has a valid checkpointer.

* fix: address CI failure and Copilot review feedback

- Fix import order in test_checkpointer_none_fix.py (I001 ruff error)
- Fix type annotation: _checkpointer should be Checkpointer | None
- Update docstring: change "None if not configured" to "InMemorySaver if not configured"
- Ensure app config is loaded before checking checkpointer config to prevent incorrect InMemorySaver fallback

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-09 15:48:27 +08:00
Ashwek-Werghi 4f0a8da2ee fix(frontend): enable HTML preview for generated artifacts using srcDoc (#1001)
* fix(frontend): enable HTML preview for generated artifacts using srcDoc

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-09 15:34:44 +08:00
aworki ac1e1915ef feat(channels): make mobile session settings configurable by channel and user (#1021) 2026-03-08 22:19:40 +08:00
DanielWalnut 8871fca5cb feat: add claude-to-deerflow skill for DeerFlow API integration (#1024)
* feat: add claude-to-deerflow skill for DeerFlow API integration

Add a new skill that enables Claude Code to interact with the DeerFlow
AI agent platform via its HTTP API, including chat streaming and status
checking capabilities.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: fix telegram channel

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 22:06:24 +08:00
JeffJiang 3721c82ba8 Update Nginx configuration for uploads and improve thread ID handling (#1023)
* fix: update nginx configuration for uploads endpoint and improve thread ID handling in hooks

* Update docker/nginx/nginx.local.conf

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update frontend/src/core/threads/hooks.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-08 21:15:03 +08:00
Willem Jiang 6b5c4fe6dd fix(dev): improve gateway startup diagnostics for config errors (#1020) 2026-03-08 21:06:57 +08:00
JeffJiang cf9af1fe75 Enhance chat UI and compatible anthropic thinking messages (#1018) 2026-03-08 20:19:31 +08:00
JeffJiang 3512279ce3 feat: add thinking settings to compatible anthropic api (#1017) 2026-03-08 20:18:21 +08:00
Jason 511e9eaf5e fix(docker): remove cache_from to prevent missing cache warnings (#1013)
Partially addresses #1011

The cache_from options reference /tmp/docker-cache-* directories
that don't exist by default, causing WARN messages on startup:

WARN local cache import at /tmp/docker-cache-gateway not found
WARN local cache import at /tmp/docker-cache-langgraph not found

Fix: Comment out cache_from with setup instructions.

To re-enable caching, create the directories:
  mkdir -p /tmp/docker-cache-gateway /tmp/docker-cache-langgraph

Note: This PR only fixes the cache warnings. The main NoneType error
in #1011 requires further investigation.
2026-03-08 19:47:46 +08:00
DanielWalnut 75b7302000 feat: add IM channels for Feishu, Slack, and Telegram (#1010)
* feat: add IM channels system for Feishu, Slack, and Telegram integration

Bridge external messaging platforms to DeerFlow via LangGraph Server with
async message bus, thread management, and per-channel configuration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address review comments on IM channels system

Fix topic_id handling in store remove/list_entries and manager commands,
correct Telegram reply threading, remove unused imports/variables, update
docstrings and docs to match implementation, and prevent config mutation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* update skill creator

* fix im reply text

* fix comments

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:21:18 +08:00
JeffJiang d664ae5a4b Support langgraph checkpointer (#1005)
* Add checkpointer configuration to config.example.yaml

- Introduced a new section for checkpointer configuration to enable state persistence for the embedded DeerFlowClient.
- Documented supported types: memory, sqlite, and postgres, along with examples for each.
- Clarified that the LangGraph Server manages its own state persistence separately.

* refactor(checkpointer): streamline checkpointer initialization and logging

* fix(uv.lock): update revision and add new wheel URLs for brotlicffi package

* feat: add langchain-anthropic dependency and update related configurations

* Fix checkpointer lifecycle, docstring, and path resolution bugs from PR #1005 review (#4)

* Initial plan

* Address all review suggestions from PR #1005

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

* Fix resolve_path to always return real Path; move SQLite special-string handling to callers

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>
2026-03-07 21:07:21 +08:00
Xinmin Zeng 09325ca28f fix: normalize presented artifact paths (#998)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-06 22:51:27 +08:00
null4536251 9d2144d431 feat: may_ask (#981)
* feat: u may ask

* chore: adjust code according to CR

* chore: adjust code according to CR

* ut: test for suggestions.py

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-06 22:39:58 +08:00
Willem Jiang 2e90101be8 chore(config):comment out the LLM model setting by default (#975)
* chore(config):comment out the LLM model setting by default

* config: update the configure of the LLM models
2026-03-06 17:47:01 +08:00
Willem Jiang cfae751902 chore(ci):add copilot instructions file (#996) 2026-03-06 17:45:02 +08:00
infoquest-byteplus 28e1257e1e support infoquest (#960)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-06 15:32:13 +08:00
FangHao 3e4a24f48b fix(subagent): support async MCP tools in subagent executor (#917)
* fix(subagent): support async MCP tools in subagent executor

SubagentExecutor.execute() was synchronous and could not handle async-only                                                                                                                                  tools like MCP tools. This caused failures when trying to use MCP tools within subagents.

Changes:
- Add _aexecute() async method using agent.astream() for async execution
- Refactor execute() to use asyncio.run() wrapping _aexecute()
- This allows subagents to use async tools (MCP) within ThreadPoolExecutor

* test(subagent): add unit tests for executor async/sync paths

Add comprehensive tests covering:
- Async _aexecute() with success/error cases
- Sync execute() wrapper using asyncio.run()
- Async tool (MCP) support verification
- Thread pool execution safety

* fix(subagent): subagent-test-circular-depend

- Use session-scoped fixture with delayed import to handle circular dependencies
    without affecting other test modules

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-06 14:40:56 +08:00
Willem Jiang 3a5e0b935d fix(backend): upgrade langgraph-api to 0.7 and stabilize memory path tests (#984)
- replace  with explicit runtime deps:
- regenerate  after dependency changes
- make  deterministic by patching
  to avoid leaked global  affecting expected paths
2026-03-06 09:44:40 +08:00
Xinmin Zeng 0c7c96d75e fix(nginx): use cross-platform local paths for pid and logs (#977)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-05 17:50:02 +08:00
JeffJiang 1b3939cb78 fix(chat): handle empty uploaded files case and improve artifact selection logic (#979)
* fix(chat): handle empty uploaded files case and improve artifact selection logic

* Update frontend/src/components/workspace/chats/chat-box.tsx

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: address code review suggestions from PR #979 (#3)

* Initial plan

* fix: address PR #979 review suggestions

- utils.ts: scope (empty) check inside <uploaded_files> tag content
- chat-box.tsx: remove stale `artifacts` from useEffect deps
- context.tsx: wrap select/deselect with useCallback for stable refs
- test: add test_empty_new_files_produces_empty_marker

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>
2026-03-05 17:45:25 +08:00
JeffJiang b17c087174 Implement optimistic UI for file uploads and enhance message handling (#967)
* feat(upload): implement optimistic UI for file uploads and enhance message handling

* feat(middleware): enhance file handling by collecting historical uploads from directory

* feat(thread-title): update page title handling for new threads and improve loading state

* feat(uploads-middleware): enhance file extraction by verifying file existence in uploads directory

* feat(thread-stream): update file path reference to use virtual_path for uploads

* feat(tests): add core behaviour tests for UploadsMiddleware

* feat(tests): remove unused pytest import from test_uploads_middleware_core_logic.py

* feat: enhance file upload handling and localization support

- Update UploadsMiddleware to validate filenames more robustly.
- Modify MessageListItem to parse uploaded files from raw content for backward compatibility.
- Add localization for uploading messages in English and Chinese.
- Introduce parseUploadedFiles utility to extract uploaded files from message content.
2026-03-05 11:16:34 +08:00
DanielWalnut 3ada4f98b1 fix(memory): prevent file upload events from persisting in long-term memory (#971)
* fix(memory): prevent file upload events from persisting in long-term memory

Uploaded files are session-scoped and unavailable in future sessions.
Previously, upload interactions were recorded in memory, causing the
agent to search for non-existent files in subsequent conversations.

Changes:
- memory_middleware: skip human messages containing <uploaded_files>
  and their paired AI responses from the memory queue
- updater: post-process generated memory to strip upload mentions
  before saving to file
- prompt: instruct the memory LLM to ignore file upload events

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(memory): address Copilot review feedback on upload filtering

- memory_middleware: strip <uploaded_files> block from human messages
  instead of dropping the entire turn; only skip the turn (and paired
  AI response) when nothing remains after stripping
- updater: narrow the upload-scrubbing regex to explicit upload events
  (avoids false-positive removal of "User works with CSV files" etc.);
  also filter upload-event facts from the facts array
- prompt: move `import re` to module scope; skip upload-only human
  messages (empty after stripping) rather than appending "User: "

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(memory): allow optional words between 'upload' and 'file' in scrub regex

The previous pattern required 'uploading file' with no intervening words,
so 'uploading a test file' was not matched and leaked into long-term memory.
Allow up to 3 modifier words between the verb and noun (e.g. 'uploading a
test file', 'uploaded the attachment').

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(memory): add unit tests for upload filtering in memory pipeline

Covers _filter_messages_for_memory and _strip_upload_mentions_from_memory
per Copilot review suggestion.  15 test cases verify:

- Upload-only turns (and paired AI responses) are excluded from memory queue
- User's real question is preserved when combined with an upload block
- Upload file paths are never present in filtered message content
- Intermediate tool messages are always excluded
- Multi-turn conversations: only the upload turn is dropped
- Multimodal (list-content) human messages are handled
- Upload-event sentences are removed from summaries and facts
- Legitimate file-related facts (CSV preferences, PDF exports) are preserved
- "uploading a test file" (words between verb and noun) is caught by regex

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-05 11:14:34 +08:00
Tao 6ac0042cfe fix(readme): correct typo Offiical to Official (#972) 2026-03-05 07:25:30 +08:00
Chris Chen 7149f0c9b5 Add CORS_ORIGINS to .env.example for custom frontend port support (#969)
Fixes issue #47: CORS error when frontend port isn't 3000

Users running the frontend on a port other than 3000 need to set
CORS_ORIGINS to allow cross-origin requests. This addition to
.env.example makes this configuration option visible.

Co-authored-by: GitHub Agent <agent@example.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-04 20:07:37 +08:00
Chetan Sharma cff78206da refactor: reduce repeated configurable dict lookups (#970)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-04 20:06:28 +08:00
Stable Genius 452595255e docs: add make install step before local dev (#955) (#963)
Co-authored-by: Stable Genius <259448942+stablegenius49@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-04 18:29:11 +08:00
Willem Jiang a3c8efb00b fix(make):added make config command in make file (#964) 2026-03-04 09:51:15 +08:00
JeffJiang 14d1e01149 Refactor hooks and improve error handling in chat functionality (#962)
* refactor: update useThreadChat and useThreadStream hooks for improved state management

* fix: improve error handling in agent configuration loading and enhance chat page functionality

* fix: enhance error handling in agent configuration loading

* Update frontend/src/app/workspace/agents/[agent_name]/chats/[thread_id]/page.tsx

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-04 09:50:45 +08:00
JeffJiang 7de94394d4 feat(agent):Supports custom agent and chat experience with refactoring (#957)
* feat: add agent management functionality with creation, editing, and deletion

* feat: enhance agent creation and chat experience

- Added AgentWelcome component to display agent description on new thread creation.
- Improved agent name validation with availability check during agent creation.
- Updated NewAgentPage to handle agent creation flow more effectively, including enhanced error handling and user feedback.
- Refactored chat components to streamline message handling and improve user experience.
- Introduced new bootstrap skill for personalized onboarding conversations, including detailed conversation phases and a structured SOUL.md template.
- Updated localization files to reflect new features and error messages.
- General code cleanup and optimizations across various components and hooks.

* Refactor workspace layout and agent management components

- Updated WorkspaceLayout to use useLayoutEffect for sidebar state initialization.
- Removed unused AgentFormDialog and related edit functionality from AgentCard.
- Introduced ArtifactTrigger component to manage artifact visibility.
- Enhanced ChatBox to handle artifact selection and display.
- Improved message list rendering logic to avoid loading states.
- Updated localization files to remove deprecated keys and add new translations.
- Refined hooks for local settings and thread management to improve performance and clarity.
- Added temporal awareness guidelines to deep research skill documentation.

* feat: refactor chat components and introduce thread management hooks

* feat: improve artifact file detail preview logic and clean up console logs

* feat: refactor lead agent creation logic and improve logging details

* feat: validate agent name format and enhance error handling in agent setup

* feat: simplify thread search query by removing unnecessary metadata

* feat: update query key in useDeleteThread and useRenameThread for consistency

* feat: add isMock parameter to thread and artifact handling for improved testing

* fix: reorder import of setup_agent for consistency in builtins module

* feat: append mock parameter to thread links in CaseStudySection for testing purposes

* fix: update load_agent_soul calls to use cfg.name for improved clarity

* fix: update date format in apply_prompt_template for consistency

* feat: integrate isMock parameter into artifact content loading for enhanced testing

* docs: add license section to SKILL.md for clarity and attribution

* feat(agent): enhance model resolution and agent configuration handling

* chore: remove unused import of _resolve_model_name from agents

* feat(agent): remove unused field

* fix(agent): set default value for requested_model_name in _resolve_model_name function

* feat(agent): update get_available_tools call to handle optional agent_config and improve middleware function signature

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-03 21:32:01 +08:00
Xinmin Zeng 8342e88534 fix(models): handle google provider import errors and add dependency (#952)
* fix(models): improve provider import guidance and add google provider dep

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(reflection): prefer provider install hint on transitive import errors

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-03 14:56:54 +08:00
Xinmin Zeng 7754c49217 feat(skills): support recursive nested skill loading (#950)
* feat(skills): support recursive nested skill loading

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-02 21:02:03 +08:00
Zhiyunyao a138d5388a feat: add reasoning_effort configuration support for Doubao/GPT-5 models (#947)
* feat: Add reasoning effort configuration support

* Add `reasoning_effort` parameter to model config and agent initialization
* Support reasoning effort levels (minimal/low/medium/high) for Doubao/GPT-5 models
* Add UI controls in input box for reasoning effort selection
* Update doubao-seed-1.8 example config with reasoning effort support

Fixes & Cleanup:
* Ensure UTF-8 encoding for file operations
* Remove unused imports

* fix: set reasoning_effort to None for unsupported models

* fix: unit test error

* Update frontend/src/components/workspace/input-box.tsx

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-02 20:49:41 +08:00
haibow e399d09e8f Fix line numbering (#954)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-02 20:43:58 +08:00
Zhiyunyao 72df234636 refactor(frontend): optimize network queries and improve code readability (#919)
* refactor(frontend): optimize network queries and improve code readability
- useThreadStream: Add useStream with fetchStateHistory: {limit: 1}
- useThreads: Add select fields and disable refetchOnWindowFocus
- useModels: Disable refetchOnWindowFocus
- ChatPage and titleOfThread: Improve thread title logic
- loadModels: Refactor code for better readability

* fix: address the review comments of Copilot

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-02 20:35:46 +08:00
Willem Jiang a2f91c7594 feat(mcp): add OAuth support for HTTP/SSE MCP servers (#908)
add oauth schema to MCP server config (extensions_config.json)
support client_credentials and refresh_token grants
implement token manager with caching and pre-expiry refresh
inject OAuth Authorization header for MCP tool discovery and tool calls
extend MCP gateway config models to read/write OAuth settings
update docs and examples for OAuth configuration
add unit tests for token fetch/cache and header injection
2026-03-01 22:38:58 +08:00
エイカク 80316c131e fix(backend): Fix readability extraction crash when Node parser fails (#937)
* Fix readability fallback when Node extraction fails

* Narrow readability fallback errors and enrich logs

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-03-01 22:24:02 +08:00
atian8179 d728bb26d5 fix: use shell fallback instead of hardcoded /bin/zsh in LocalSandbox (#939)
* fix: use shell fallback instead of hardcoded /bin/zsh in LocalSandbox

Replace hardcoded /bin/zsh executable with dynamic shell detection
that falls back through /bin/zsh → /bin/bash → /bin/sh. This fixes
skill execution failures in Docker containers (python:3.12-slim)
where zsh is not available.

Closes #935

* Update backend/src/sandbox/local/local_sandbox.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: atian8179 <atian8179@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-01 22:08:07 +08:00
Willem Jiang 8c6dd9e264 fix(uploads): persist thread uploads canonically and fail fast on upload errors (#943)
* fix(uploads): persist thread uploads canonically and fail fast on upload errors

 - write uploads to thread-scoped storage first to guarantee agent visibility
 - sync files to sandbox virtual path only for non-local sandboxes
 - fix markdown conversion flow to operate on canonical saved files and sync converted files when needed
 - prevent silent attachment upload failures in frontend submit flow (show error + abort submit)
 - add regression tests for local vs non-local upload behavior
 - update upload docs with thread-first persistence and troubleshooting notes

* Update frontend/src/core/threads/hooks.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(uploads): reject "." and ".." filenames in upload sanitization (#944)

* Initial plan

* fix(uploads): reject '.' and '..' filenames in upload sanitization

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
2026-03-01 15:35:30 +08:00
layla 5a1ac6287e Fix typo: Offiical to Official (#942) 2026-03-01 13:09:57 +08:00
YolenSong 3d3ea84a57 test(backend): add core logic unit tests for task/title/mcp (#936)
* test(backend): add core logic unit tests for task/title/mcp

* test(backend): fix lint issues in client test modules

---------

Co-authored-by: songyaolun <songyaolun@bytedance.com>
2026-03-01 12:36:09 +08:00
Henry Li f2123efdb9 docs: #1 on GitHub Trending (#932) 2026-02-28 16:20:26 +08:00
greatmengqi 30d948711f test: add Gateway conformance tests for DeerFlowClient (#931)
Validate that all dict-returning client methods conform to Gateway
Pydantic response models (ModelsListResponse, ModelResponse,
SkillsListResponse, SkillResponse, SkillInstallResponse,
McpConfigResponse, UploadResponse, MemoryConfigResponse,
MemoryStatusResponse). Pydantic ValidationError in CI catches
schema drift between client and Gateway with zero production coupling.

Also includes prior review fixes: enhanced client methods, expanded
unit tests (67→77), live integration test improvements, and updated
documentation.

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-02-28 16:08:04 +08:00
greatmengqi 9d48c42a20 feat: add DeerFlowClient for embedded programmatic access (#926)
Add `DeerFlowClient` class that provides direct in-process access to
DeerFlow's agent and Gateway capabilities without requiring LangGraph
Server or Gateway API processes. This enables users to import and use
DeerFlow as a Python library.

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-02-28 14:38:15 +08:00
Willem Jiang 5ad8a657f4 fix(git):add .gitattributes to avoid 'bash\r' issue (#924) 2026-02-28 10:32:56 +08:00
Alex e62b3d4167 feat: add Novita AI as optional LLM provider (#910)
* feat: add Novita AI as optional LLM provider

Adds Novita AI (https://novita.ai) as an optional, OpenAI-compatible
LLM provider. 

Changes:
- Added Novita model configuration example in config.example.yaml
- Added NOVITA_API_KEY to .env.example

Usage: Set NOVITA_API_KEY in your environment and use novita-gpt-4
as the model name.

* update correct model info

* Update README.md

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-02-27 11:16:31 +08:00
Xinmin Zeng e9adaab7a6 fix(i18n): normalize locale and prevent undefined translations (#914)
* fix(i18n): guard locale input and add safe translation fallback

* refactor(i18n): isolate locale utils and normalize server cookie decode

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-02-27 08:10:38 +08:00
Salman Chishti 902ff3b9f3 Upgrade GitHub Actions to latest versions (#913)
Signed-off-by: Salman Muin Kayser Chishti <13schishti@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-02-26 22:49:32 +08:00
Salman Chishti 32a22069e9 Upgrade GitHub Actions for Node 24 compatibility (#912)
Signed-off-by: Salman Muin Kayser Chishti <13schishti@gmail.com>
2026-02-26 20:23:57 +08:00
Xinmin Zeng 6a55860a15 fix: recover from stale model context when configured models change (#898)
* fix: recover from stale model context after config model changes

* fix: fail fast on missing model config and expand model resolution tests

* fix: remove duplicate get_app_config imports

* fix: align model resolution tests with runtime imports

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: remove duplicate model resolution test case

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-26 13:54:29 +08:00
dependabot[bot] 3e6e4b0b5f build(deps): bump minimatch from 3.1.2 to 3.1.4 in /frontend (#906)
Bumps [minimatch](https://github.com/isaacs/minimatch) from 3.1.2 to 3.1.4.
- [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md)
- [Commits](https://github.com/isaacs/minimatch/compare/v3.1.2...v3.1.4)

---
updated-dependencies:
- dependency-name: minimatch
  dependency-version: 3.1.4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-02-25 22:34:42 +08:00
DanielWalnut d27a7a5f54 fix(middleware): fix DanglingToolCallMiddleware inserting patches at wrong position (#904)
Previously used before_model which returned {"messages": patches}, causing
LangGraph's add_messages reducer to append patches at the end of the message
list. This resulted in invalid ordering (ToolMessage after a HumanMessage)
that LLMs reject with tool call ID mismatch errors.

Switch to wrap_model_call/awrap_model_call to insert synthetic ToolMessages
immediately after each dangling AIMessage before the request reaches the LLM,
without persisting the patches to state.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 22:29:33 +08:00
JeffJiang 33595f0bac fix(skill): enhance data authenticity protocols and clarify reporting guidelines (#905) 2026-02-25 22:25:23 +08:00
JeffJiang 3a7251c95e fix(docker): update nginx configuration and simplify docker script (#903) 2026-02-25 22:16:43 +08:00
JeffJiang d24a66ffd3 Refactor base paths with centralized path management (#901)
* Initial plan

* refactor: centralize path management and improve memory storage configuration

* fix: update memory storage path in config.example.yaml for clarity

* Initial plan

* Address PR #901 review comments: security fixes and documentation improvements

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>
2026-02-25 21:30:33 +08:00
JeffJiang adfe5c4b44 Enforces config env var checks and improves startup handling (#892)
* Enforces config env var checks and improves startup handling

Ensures critical environment variables are validated during config resolution,
raising clear errors if missing. Improves server startup reliability by
verifying that backend services are listening and by terminating on
misconfiguration at launch. Adds more robust feedback to developers when
API startup fails, reducing silent misconfigurations and speeding up
troubleshooting.

* Initial plan

* Implement suggestions from PR #892: fix env var checks and improve error logging

Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-02-25 16:12:59 +08:00
dependabot[bot] 6d1878fb1a build(deps): bump cryptography from 46.0.3 to 46.0.5 in /backend (#896)
Bumps [cryptography](https://github.com/pyca/cryptography) from 46.0.3 to 46.0.5.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/46.0.3...46.0.5)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-version: 46.0.5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-02-25 09:42:39 +08:00
DanielWalnut faa422072c feat(subagents): make subagent timeout configurable via config.yaml (#897)
* feat(subagents): make subagent timeout configurable via config.yaml

- Add SubagentsAppConfig supporting global and per-agent timeout_seconds
- Load subagents config section in AppConfig.from_file()
- Registry now applies config.yaml overrides without mutating builtin defaults
- Polling safety-net in task_tool is now dynamic (execution timeout + 60s buffer)
- Document subagents section in config.example.yaml
- Add make test command and enforce TDD policy in CLAUDE.md
- Add 38 unit tests covering config validation, timeout resolution, registry
  override behavior, and polling timeout formula

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(subagents): add logging for subagent timeout config and execution

- Log loaded timeout config (global default + per-agent overrides) on startup
- Log debug message in registry when config.yaml overrides a builtin timeout
- Include timeout in executor's async execution start log
- Log effective timeout and polling limit when a task is dispatched
- Fix UnboundLocalError: move max_poll_count assignment before logger.info

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* ci(backend): add lint step and run all unit tests via Makefile

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix lint

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 08:39:29 +08:00
dependabot[bot] 310d54e443 build(deps): bump python-multipart from 0.0.21 to 0.0.22 in /backend (#895)
Bumps [python-multipart](https://github.com/Kludex/python-multipart) from 0.0.21 to 0.0.22.
- [Release notes](https://github.com/Kludex/python-multipart/releases)
- [Changelog](https://github.com/Kludex/python-multipart/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Kludex/python-multipart/compare/0.0.21...0.0.22)

---
updated-dependencies:
- dependency-name: python-multipart
  dependency-version: 0.0.22
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-24 23:11:46 +08:00
dependabot[bot] af374cd40b build(deps): bump pillow from 12.1.0 to 12.1.1 in /backend (#894)
Bumps [pillow](https://github.com/python-pillow/Pillow) from 12.1.0 to 12.1.1.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](https://github.com/python-pillow/Pillow/compare/12.1.0...12.1.1)

---
updated-dependencies:
- dependency-name: pillow
  dependency-version: 12.1.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-02-24 22:58:54 +08:00
dependabot[bot] 50d9174123 build(deps): bump protobuf from 6.33.4 to 6.33.5 in /backend (#893)
Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 6.33.4 to 6.33.5.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Commits](https://github.com/protocolbuffers/protobuf/commits)

---
updated-dependencies:
- dependency-name: protobuf
  dependency-version: 6.33.5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-24 22:53:45 +08:00
Willem Jiang 03705acf3a fix(sandbox):deer-flow-provisioner container fails to start in local execution mode (#889) 2026-02-24 08:31:52 +08:00
Willem Jiang b5c11baece docs(config):updated the configuration of deepseek-v3 (#885) 2026-02-21 22:06:01 +08:00
CHANGXUBO 85af540076 feat: add LangSmith tracing integration (#878)
* feat: add LangSmith tracing integration

Add optional LangSmith tracing support that can be enabled via environment
variables (LANGSMITH_TRACING, LANGSMITH_API_KEY, LANGSMITH_PROJECT,
LANGSMITH_ENDPOINT). When enabled, a LangChainTracer callback is attached
to chat models and run metadata is injected for trace tagging.

Co-Authored-By: Claude <noreply@anthropic.com>

* Update backend/src/config/tracing_config.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update backend/src/agents/lead_agent/agent.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update backend/src/agents/lead_agent/agent.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update backend/src/models/factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add threading lock to ensure thread-safe access to tracing configuration

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-21 16:41:34 +08:00
Zhiyunyao 75226b2fe6 docs: make README easier to follow and update related docs (#884) 2026-02-21 07:48:20 +08:00
Zhiyunyao 0d7c0826f0 chore: add a Makefile command to create all required local configuration files (#883)
* fix: polish the makefile to provide config command for local config setup

* docs: polish the instructions in README
2026-02-19 09:04:37 +08:00
Willem Jiang ea4e0139af docs: Update Quick Start instructions in README (#881)
Fixes the issue #880
2026-02-18 19:29:25 +08:00
CHANGXUBO 9f74589d09 fix: HTML artifact preview renders blank in preview mode (#876)
The condition guarding ArtifactFilePreview only allowed markdown files
through, which prevented HTML files from reaching the preview component.
Added `language === "html"` to the condition so HTML artifacts render
correctly in preview mode.

Fixes #873

Co-authored-by: Claude <noreply@anthropic.com>
2026-02-18 10:06:21 +08:00
CHANGXUBO 67dbb10c2a fix: use /tmp/nginx.pid to avoid permission denied errors (#877)
Set pid directive to /tmp/nginx.pid in nginx.conf and nginx.local.conf
to prevent permission denied errors when running nginx as a non-root user.

Co-authored-by: Claude <noreply@anthropic.com>
2026-02-18 10:01:51 +08:00
Willem Jiang 79c841db9e chore: Ignore legacy web folder in .gitignore (#866)
Add rule to ignore the legacy 'web' folder
2026-02-15 10:45:59 +08:00
Henry Li 2d3a22aeb0 docs: add videos and official website (#865)
* docs: add videos and official website

* docs: use public video URL
2026-02-14 23:48:29 +08:00
Willem Jiang d796c5a328 docs:Add security policy documentation (#864) 2026-02-14 21:34:41 +08:00
Willem Jiang 8039da2fc4 docs:Update README.md (#863)
Updated the link for the original Deep Research framework to point to the correct branch.
2026-02-14 21:30:37 +08:00
Willem Jiang da1bcf0573 Merge remote-tracking branch 'deer-flow-2/experimental' into main-2.x 2026-02-14 16:29:38 +08:00
Willem Jiang a66d8c94fa Prepare to merge deer-flow-2 2026-02-14 16:28:12 +08:00
Willem Jiang 06248fa6f1 chore: Polishing the license headers work (#860)
* feat(tool): Adding license header check and apply tool

* Update docs/LICENSE_HEADERS.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update scripts/license_header.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* chore: Polishing the license headers work

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-14 16:24:44 +08:00
大猫子 13a25112b1 fix: move Key Citations to early position in reporter prompt to reduce URL hallucination (#859)
* fix: move Key Citations to early position in reporter prompt to reduce URL hallucination

Move the Key Citations section from position 6 (end of report) to position 2
(immediately after title) in the reporter prompt. When citations are placed at
the end of a long report, LLMs tend to forget real URLs from source material
and fabricate plausible-looking but non-existent URLs.

Changes to src/prompts/reporter.md:
- Move Key Citations from section 6 to section 2 (right after Title)
- Add explicit anti-hallucination instructions: only use URLs from provided
  source material, never fabricate or guess URLs
- Keep a repeated citation list at the end (section 7) for completeness
- Renumber all subsequent sections accordingly
- Update Notes section to reflect new structure

Tested with real DeerFlow backend + DuckDuckGo search:
- Before: multiple hallucinated URLs in report citations
- After: hallucinated URLs reduced significantly

Closes #825

* fix: move citations after observations in reporter_node to reduce URL hallucination

Previously, the citation message was appended BEFORE observation messages,
meaning it got buried under potentially thousands of chars of research data.
By the time the LLM reached the end of the context to generate the report,
it had 'forgotten' the real URLs and fabricated plausible-looking ones.

Now citations are appended AFTER compressed observations, placing them
closest to the LLM's generation point for maximum recall accuracy.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-02-14 15:21:24 +08:00
大猫子 c95b2711c3 docs: clarify .env configuration for Docker Compose deployment (#858)
- Expand Docker Compose section in README.md and README_zh.md with:
  - Explicit note that only root .env is used (not web/.env)
  - Instructions to update NEXT_PUBLIC_API_URL for remote/LAN deployment
  - Explanation that NEXT_PUBLIC_API_URL is a build-time variable
- Improve NEXT_PUBLIC_API_URL comments in root .env.example

Closes #527
2026-02-14 11:38:21 +08:00
Henry Li 88e89921b9 docs: update LICENSE 2026-02-13 11:49:51 +08:00
Willem Jiang 56b8c3a496 feat(tool): Adding license header check and apply tool (#857)
* feat(tool): Adding license header check and apply tool

* Update docs/LICENSE_HEADERS.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update scripts/license_header.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-13 11:23:41 +08:00
Rin ba45c1a3a9 security: patch orjson DoS and harden container/frontend (#852) 2026-02-13 10:15:39 +08:00
Henry Li 8f44ca595b docs: update README.md 2026-02-13 09:38:35 +08:00
Henry Li 15df224856 docs: update README.md 2026-02-13 09:25:15 +08:00
JeffJiang 4d5fdcb8db Consolidates market and data analysis skills; adds chart viz (#36)
Unifies market analysis, data analysis, and consulting reporting into a comprehensive consulting-analysis skill, enabling a two-phase workflow from analysis framework design to professional report generation. Introduces a DuckDB-based data analysis utility for Excel/CSV files and a chart-visualization skill with a flexible JS interface and extensive chart type documentation. Removes the legacy market analysis skill to streamline report generation and improve extensibility for consulting and data-driven workflows.
2026-02-12 11:08:09 +08:00
JeffJiang 300e5a519a Adds Kubernetes sandbox provisioner support (#35)
* Adds Kubernetes sandbox provisioner support

* Improves Docker dev setup by standardizing host paths

Replaces hardcoded host paths with a configurable root directory,
making the development environment more portable and easier to use
across different machines. Automatically sets the root path if not
already defined, reducing manual setup steps.
2026-02-12 11:02:09 +08:00
hetao e87fd74e17 docs(ppt-generation): enforce sequential slide image generation
Explicitly prohibit parallel image generation to ensure each slide
can use the previous slide as a reference image for visual consistency.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-11 15:36:51 +08:00
hetao 770d92fe36 feat: make max concurrent subagents configurable via runtime config
Support configuring max_concurrent_subagents (2-4, default 3) through
config.configurable, with automatic clamping and dynamic prompt generation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 11:51:11 +08:00
hetao 4a85c5de7b feat: enable skills support for subagents
Extract get_skills_prompt_section() from apply_prompt_template() so
subagents can also receive the available skills list in their system
prompt. This allows subagents to discover and load skills via read_file,
just like the lead agent.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 11:04:50 +08:00
Henry Li ebf4ec2786 chore: add pnpm-workspace.yaml 2026-02-10 22:07:33 +08:00
Henry Li eb287f095a chore: add .npmrc back 2026-02-10 22:07:25 +08:00
Henry Li 595dba6c35 chore: upgrade langchain and langgraph 2026-02-10 22:07:17 +08:00
Henry Li aeadc00f96 Merge remote-tracking branch 'refs/remotes/origin/experimental' into experimental 2026-02-10 17:30:57 +08:00
LofiSu 192078e50e Merge pull request #32 from LofiSu/experimental
Experimental
2026-02-10 12:42:38 +08:00
LofiSu c8f7bc28e1 docs: 添加技能名称冲突修复的详细文档
- 记录 public 和 custom 技能同名冲突问题的解决方案
- 详细说明所有代码改动(后端配置、API、前端)
- 包含配置格式变更、API 变更说明
- 标注已知问题暂时保留,待后续版本修复
- 提供测试建议和回滚方案

相关改动:
- 使用组合键 {category}:{name} 存储配置
- API 支持可选的 category 查询参数
- 添加类别内重复技能名称检查
- 前端传递 category 参数确保唯一性

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-10 12:41:58 +08:00
LofiSu b3a1f018ab fix: 修复新建技能后输入框无法编辑的问题
问题:点击新建技能按钮后,对话框中预设的文字无法删除或修改
原因:useEffect 依赖项包含 promptInputController.textInput,该对象在每次输入时都会重新创建,导致 useEffect 重复执行并覆盖用户输入
解决:使用 useRef 保存 setInput 方法,并跟踪已设置的初始值,确保 useEffect 只在初始值变化时执行一次

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-10 12:39:44 +08:00
LofiSu 3b2001d89d Merge pull request #31 from LofiSu/experimental
Experimental
2026-02-10 12:31:55 +08:00
LofiSu cc88823a64 fix:memory 为空时i18n字体显示 2026-02-10 12:29:14 +08:00
LofiSu f87d5678f3 feat: 改进设置页面UI和国际化支持 / Improve settings pages UI and i18n support
- 添加 rehype-raw 依赖以支持在 markdown 中渲染 HTML
  Add rehype-raw dependency to support HTML rendering in markdown

- 重构 memory-settings-page,提取 formatMemorySection 函数减少重复代码
  Refactor memory-settings-page by extracting formatMemorySection function to reduce code duplication

- 改进空状态显示,使用 HTML span 标签替代 markdown 斜体,提供更好的样式控制
  Improve empty state display by using HTML span tags instead of markdown italics for better style control

- 为 skill-settings-page 添加完整的国际化支持,替换硬编码的英文文本
  Add complete i18n support for skill-settings-page, replacing hardcoded English text

- 更新国际化文件,添加技能设置页面的空状态文本(中英文)
  Update i18n files with empty state text for skill settings page (both Chinese and English)

- 在 streamdown 插件配置中添加 rehypeRaw 以支持 HTML 渲染
  Add rehypeRaw to streamdown plugins configuration to support HTML rendering

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-10 12:15:37 +08:00
LofiSu 76292125ff Merge pull request #30 from LofiSu/experimental
fix: citations prompt
2026-02-10 11:54:41 +08:00
LofiSu 6109216d54 fix: citations prompt 2026-02-10 00:18:33 +08:00
Henry Li 13b3032d02 fix: eslint 2026-02-09 23:28:36 +08:00
Henry Li df3668ecd5 fix: eslint 2026-02-09 23:28:36 +08:00
LofiSu 11489ae4da Strip citation prefix in citation badges 2026-02-09 22:57:54 +08:00
LofiSu f8e4fe05b2 Strip citation prefix in citation badges 2026-02-09 22:57:54 +08:00
LofiSu d51e6e2f43 Merge branch 'experimental' of github.com:hetaoBackend/deer-flow into feat/citations 2026-02-09 22:05:44 +08:00
LofiSu 1af14bf7e4 Merge branch 'experimental' of github.com:hetaoBackend/deer-flow into feat/citations 2026-02-09 22:05:44 +08:00
JeffJiang 7b7e32f262 Add Kubernetes-based sandbox provider for multi-instance support (#19)
* feat: adds docker-based dev environment

* docs: updates Docker command help

* fix local dev

* feat(sandbox): add Kubernetes-based sandbox provider for multi-instance support

* fix: skills path in k8s

* feat: add example config for k8s sandbox

* fix: docker config

* fix: load skills on docker dev

* feat: support sandbox execution to Kubernetes Deployment model

* chore: rename web service name
2026-02-09 21:59:13 +08:00
JeffJiang b6da3a219e Add Kubernetes-based sandbox provider for multi-instance support (#19)
* feat: adds docker-based dev environment

* docs: updates Docker command help

* fix local dev

* feat(sandbox): add Kubernetes-based sandbox provider for multi-instance support

* fix: skills path in k8s

* feat: add example config for k8s sandbox

* fix: docker config

* fix: load skills on docker dev

* feat: support sandbox execution to Kubernetes Deployment model

* chore: rename web service name
2026-02-09 21:59:13 +08:00
LofiSu c89bd9edc9 Merge upstream/experimental: resolve conflicts (keep feat/citations)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 21:56:02 +08:00
LofiSu 8a2cac7b5a Merge upstream/experimental: resolve conflicts (keep feat/citations)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 21:56:02 +08:00
dependabot[bot] b92ad7e39a build(deps): bump protobuf from 6.32.1 to 6.33.5 (#855)
Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 6.32.1 to 6.33.5.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Commits](https://github.com/protocolbuffers/protobuf/commits)

---
updated-dependencies:
- dependency-name: protobuf
  dependency-version: 6.33.5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-09 21:54:08 +08:00
hobostay 7607e14088 fix: Add None check for db_uri in ChatStreamManager (#854)
This commit fixes a potential AttributeError in ChatStreamManager.__init__().

Problem:
- When checkpoint_saver=True and db_uri=None, the code attempts to call
  self.db_uri.startswith() on line 56, which raises AttributeError
- Line 56: if self.db_uri.startswith("mongodb://"):
  This fails with "AttributeError: 'NoneType' object has no attribute 'startswith'"

Root Cause:
- The __init__ method accepts db_uri: Optional[str] = None
- If None is passed, self.db_uri is set to None
- The code doesn't check if db_uri is None before calling .startswith()

Solution:
- Add explicit None check before string prefix checks
- Provide clear warning message when db_uri is None but checkpoint_saver is enabled
- This prevents AttributeError and helps users understand the configuration issue

Changes:
```python
# Before:
if self.checkpoint_saver:
    if self.db_uri.startswith("mongodb://"):

# After:
if self.checkpoint_saver:
    if self.db_uri is None:
        self.logger.warning(
            "Checkpoint saver is enabled but db_uri is None. "
            "Please provide a valid database URI or disable checkpoint saver."
        )
    elif self.db_uri.startswith("mongodb://"):
```

This makes the error handling more robust and provides better user feedback.

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-02-09 21:44:01 +08:00
LofiSu 2f50e5d969 feat(citations): inline citation links with [citation:Title](URL)
- Backend: add citation format to lead_agent and general_purpose prompts
- Add CitationLink component (Badge + HoverCard) for citation cards
- MarkdownContent: detect citation: prefix in link text, render CitationLink
- Message/artifact/subtask: use MarkdownContent or Streamdown with CitationLink
- message-list-item: pass img via components prop (remove isHuman/img)
- message-group, subtask-card: drop unused imports; fix import order (lint)

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 21:40:20 +08:00
Henry Li 69c8b41186 feat: basic implmenetation 2026-02-09 19:02:21 +08:00
Henry Li 554ec7a91e feat: basic implmenetation 2026-02-09 19:02:21 +08:00
LofiSu 715d7436f1 Merge pull request #28 from LofiSu/experimental
chore: 移除所有 Citations 相关逻辑,为后续重构做准备
2026-02-09 16:28:37 +08:00
LofiSu 47bceca87c Merge pull request #28 from LofiSu/experimental
chore: 移除所有 Citations 相关逻辑,为后续重构做准备
2026-02-09 16:28:37 +08:00
LofiSu 6a540d8408 Merge upstream/experimental: resolve conflict in lead_agent/prompt.py
- Keep upstream subagent HARD LIMIT (max 3 task calls, batching) in subagent_reminder
- Keep our removal of Citations: do not add back 'Citations when synthesizing'

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 16:27:36 +08:00
ruitanglin 1860e10451 Merge upstream/experimental: resolve conflict in lead_agent/prompt.py
- Keep upstream subagent HARD LIMIT (max 3 task calls, batching) in subagent_reminder
- Keep our removal of Citations: do not add back 'Citations when synthesizing'

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 16:27:36 +08:00
LofiSu 46048c76ce chore: 移除所有 Citations 相关逻辑,为后续重构做准备
- Backend: 删除 lead_agent / general_purpose 中的 citations_format 与引用相关 reminder;artifacts 下载不再对 markdown 做 citation 清洗,统一走 FileResponse,保留 Response 用于二进制 inline
- Frontend: 删除 core/citations 模块、inline-citation、safe-citation-content;新增 MarkdownContent 仅做 Markdown 渲染;消息/artifact 预览与复制均使用原始 content
- i18n: 移除 citations 命名空间(loadingCitations、loadingCitationsWithCount)
- 技能与 demo: 措辞改为 references,demo 数据去掉 <citations> 块
- 文档: 更新 CLAUDE/AGENTS/README 描述,新增按文件 diff 的代码变更总结

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 16:24:01 +08:00
ruitanglin 8747873b8d chore: 移除所有 Citations 相关逻辑,为后续重构做准备
- Backend: 删除 lead_agent / general_purpose 中的 citations_format 与引用相关 reminder;artifacts 下载不再对 markdown 做 citation 清洗,统一走 FileResponse,保留 Response 用于二进制 inline
- Frontend: 删除 core/citations 模块、inline-citation、safe-citation-content;新增 MarkdownContent 仅做 Markdown 渲染;消息/artifact 预览与复制均使用原始 content
- i18n: 移除 citations 命名空间(loadingCitations、loadingCitationsWithCount)
- 技能与 demo: 措辞改为 references,demo 数据去掉 <citations> 块
- 文档: 更新 CLAUDE/AGENTS/README 描述,新增按文件 diff 的代码变更总结

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 16:24:01 +08:00
LofiSu 0f0346f73e Merge pull request #27 from LofiSu/experimental
Experimental
2026-02-09 15:59:41 +08:00
LofiSu 0093134db0 Merge pull request #27 from LofiSu/experimental
Experimental
2026-02-09 15:59:41 +08:00
LofiSu cef8d389fd refactor(frontend): consolidate citation logic, slim exports and impl
- SafeCitationContent: add loadingOnly and renderBody props.
  - loadingOnly: show only loading indicator or null (e.g. write_file step).
  - renderBody(parsed): custom body renderer (e.g. artifact preview).

- message-group write_file: use SafeCitationContent(content, isLoading,
  rehypePlugins, loadingOnly) instead of local useParsedCitations +
  shouldShowCitationLoading + CitationsLoadingIndicator. Pass rehypePlugins
  into ToolCall.

- artifact-file-detail markdown preview: use SafeCitationContent with
  renderBody((p) => <ArtifactFilePreview ... cleanContent={p.cleanContent}
  citationMap={p.citationMap} />). Remove local shouldShowCitationLoading
  and CitationsLoadingIndicator branch.

- core/citations: inline buildCitationMap into use-parsed-citations, remove
  from utils; stop exporting hasCitationsBlock (internal to shouldShowCitationLoading).

- inline-citation: make InlineCitationCard, InlineCitationCardBody,
  InlineCitationSource file-private (no longer exported).

Co-authored-by: Cursor <cursoragent@cursor.com>

---
refactor(前端): 收拢引用逻辑、精简导出与实现

- SafeCitationContent 新增 loadingOnly、renderBody。
  - loadingOnly:仅显示加载或 null(如 write_file 步骤)。
  - renderBody(parsed):自定义正文渲染(如 artifact 预览)。

- message-group write_file:改用 SafeCitationContent(loadingOnly),去掉
  本地 useParsedCitations + shouldShowCitationLoading + CitationsLoadingIndicator,
  并向 ToolCall 传入 rehypePlugins。

- artifact-file-detail 的 markdown 预览:改用 SafeCitationContent +
  renderBody 渲染 ArtifactFilePreview,去掉本地加载判断与
  CitationsLoadingIndicator 分支。

- core/citations:buildCitationMap 内联到 use-parsed-citations 并从 utils
  删除;hasCitationsBlock 不再导出(仅 shouldShowCitationLoading 内部使用)。

- inline-citation:InlineCitationCard/Body/Source 改为文件内私有,不再导出。
2026-02-09 15:58:59 +08:00
ruitanglin 59c8fec7e7 refactor(frontend): consolidate citation logic, slim exports and impl
- SafeCitationContent: add loadingOnly and renderBody props.
  - loadingOnly: show only loading indicator or null (e.g. write_file step).
  - renderBody(parsed): custom body renderer (e.g. artifact preview).

- message-group write_file: use SafeCitationContent(content, isLoading,
  rehypePlugins, loadingOnly) instead of local useParsedCitations +
  shouldShowCitationLoading + CitationsLoadingIndicator. Pass rehypePlugins
  into ToolCall.

- artifact-file-detail markdown preview: use SafeCitationContent with
  renderBody((p) => <ArtifactFilePreview ... cleanContent={p.cleanContent}
  citationMap={p.citationMap} />). Remove local shouldShowCitationLoading
  and CitationsLoadingIndicator branch.

- core/citations: inline buildCitationMap into use-parsed-citations, remove
  from utils; stop exporting hasCitationsBlock (internal to shouldShowCitationLoading).

- inline-citation: make InlineCitationCard, InlineCitationCardBody,
  InlineCitationSource file-private (no longer exported).

Co-authored-by: Cursor <cursoragent@cursor.com>

---
refactor(前端): 收拢引用逻辑、精简导出与实现

- SafeCitationContent 新增 loadingOnly、renderBody。
  - loadingOnly:仅显示加载或 null(如 write_file 步骤)。
  - renderBody(parsed):自定义正文渲染(如 artifact 预览)。

- message-group write_file:改用 SafeCitationContent(loadingOnly),去掉
  本地 useParsedCitations + shouldShowCitationLoading + CitationsLoadingIndicator,
  并向 ToolCall 传入 rehypePlugins。

- artifact-file-detail 的 markdown 预览:改用 SafeCitationContent +
  renderBody 渲染 ArtifactFilePreview,去掉本地加载判断与
  CitationsLoadingIndicator 分支。

- core/citations:buildCitationMap 内联到 use-parsed-citations 并从 utils
  删除;hasCitationsBlock 不再导出(仅 shouldShowCitationLoading 内部使用)。

- inline-citation:InlineCitationCard/Body/Source 改为文件内私有,不再导出。
2026-02-09 15:58:59 +08:00
dependabot[bot] fb2f54c4ac build(deps): bump next from 15.5.9 to 15.5.10 in /web (#853)
Bumps [next](https://github.com/vercel/next.js) from 15.5.9 to 15.5.10.
- [Release notes](https://github.com/vercel/next.js/releases)
- [Changelog](https://github.com/vercel/next.js/blob/canary/release.js)
- [Commits](https://github.com/vercel/next.js/compare/v15.5.9...v15.5.10)

---
updated-dependencies:
- dependency-name: next
  dependency-version: 15.5.10
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-09 15:43:04 +08:00
LofiSu d9a86c10e8 fix(frontend): no half-finished citations, correct state when SSE ends
Citations:
- In shouldShowCitationLoading, treat any unreplaced [cite-N] in cleanContent
  as show-loading (no body). Fixes Ultra and other modes when refs arrive
  before the <citations> block in the stream.
- Single rule: hasUnreplacedCitationRefs(cleanContent) => true forces loading;
  then isLoading && hasCitationsBlock(rawContent) for streaming indicator.

SSE end state:
- When stream finishes, SDK may set isLoading=false before client state has
  the final message content, so UI stayed wrong until refresh.
- Store onFinish(state) as finalState in chat page; clear when stream starts.
- Pass messagesOverride={finalState.messages} to MessageList when not loading
  so the list uses server-complete messages right after SSE ends (no refresh).

Chore:
- Stop tracking .githooks/pre-commit; add .githooks/ to .gitignore (local only).

Co-authored-by: Cursor <cursoragent@cursor.com>

---
fix(前端): 杜绝半成品引用,SSE 结束时展示正确状态

引用:
- shouldShowCitationLoading 中只要 cleanContent 仍含未替换的 [cite-N] 就
  只显示加载、不渲染正文,解决流式时引用块未到就出现 [cite-1] 的问题。
- 规则:hasUnreplacedCitationRefs(cleanContent) 为真则一律显示加载;
  此外 isLoading && hasCitationsBlock 用于流式时显示「正在整理引用」。

SSE 结束状态:
- 流结束时 SDK 可能先置 isLoading=false,客户端 messages 尚未包含
  最终内容,导致需刷新才显示正确。
- 在对话页保存 onFinish(state) 为 finalState,流开始时清空。
- 非加载时向 MessageList 传入 messagesOverride={finalState.messages},
  列表在 SSE 结束后立即用服务端完整消息渲染,无需刷新。

杂项:
- 取消跟踪 .githooks/pre-commit,.gitignore 增加 .githooks/(仅本地)。
2026-02-09 15:15:20 +08:00
ruitanglin 53509eaeb1 fix(frontend): no half-finished citations, correct state when SSE ends
Citations:
- In shouldShowCitationLoading, treat any unreplaced [cite-N] in cleanContent
  as show-loading (no body). Fixes Ultra and other modes when refs arrive
  before the <citations> block in the stream.
- Single rule: hasUnreplacedCitationRefs(cleanContent) => true forces loading;
  then isLoading && hasCitationsBlock(rawContent) for streaming indicator.

SSE end state:
- When stream finishes, SDK may set isLoading=false before client state has
  the final message content, so UI stayed wrong until refresh.
- Store onFinish(state) as finalState in chat page; clear when stream starts.
- Pass messagesOverride={finalState.messages} to MessageList when not loading
  so the list uses server-complete messages right after SSE ends (no refresh).

Chore:
- Stop tracking .githooks/pre-commit; add .githooks/ to .gitignore (local only).

Co-authored-by: Cursor <cursoragent@cursor.com>

---
fix(前端): 杜绝半成品引用,SSE 结束时展示正确状态

引用:
- shouldShowCitationLoading 中只要 cleanContent 仍含未替换的 [cite-N] 就
  只显示加载、不渲染正文,解决流式时引用块未到就出现 [cite-1] 的问题。
- 规则:hasUnreplacedCitationRefs(cleanContent) 为真则一律显示加载;
  此外 isLoading && hasCitationsBlock 用于流式时显示「正在整理引用」。

SSE 结束状态:
- 流结束时 SDK 可能先置 isLoading=false,客户端 messages 尚未包含
  最终内容,导致需刷新才显示正确。
- 在对话页保存 onFinish(state) 为 finalState,流开始时清空。
- 非加载时向 MessageList 传入 messagesOverride={finalState.messages},
  列表在 SSE 结束后立即用服务端完整消息渲染,无需刷新。

杂项:
- 取消跟踪 .githooks/pre-commit,.gitignore 增加 .githooks/(仅本地)。
2026-02-09 15:15:20 +08:00
LofiSu 4f9d1d524e feat(frontend): unify citation logic and prevent half-finished citations
- Add SafeCitationContent as single component for citation-aware body:
  useParsedCitations + shouldShowCitationLoading; show loading until
  citations complete, then render body with createCitationMarkdownComponents.
  Supports optional remarkPlugins, rehypePlugins, isHuman, img.

- Refactor MessageListItem: assistant message body now uses
  SafeCitationContent only; remove duplicate useParsedCitations,
  shouldShowCitationLoading, createCitationMarkdownComponents and
  CitationsLoadingIndicator logic. Human messages keep plain
  AIElementMessageResponse (no citation parsing).

- Use SafeCitationContent for clarification, present-files (message-list),
  thinking steps and write_file loading (message-group), subtask result
  (subtask-card). Artifact markdown preview keeps same guard
  (shouldShowCitationLoading) with ArtifactFilePreview.

- Unify loading condition: shouldShowCitationLoading(rawContent,
  cleanContent, isLoading) is the single source of truth. Show loading when
  (isLoading && hasCitationsBlock(rawContent)) or when
  (hasCitationsBlock(rawContent) && hasUnreplacedCitationRefs(cleanContent))
  so Pro/Ultra modes also show "loading citations" and half-finished
  [cite-N] never appear.

- message-group write_file: replace hasCitationsBlock + threadIsLoading
  with shouldShowCitationLoading(fileContent, cleanContent,
  threadIsLoading && isLast) for consistency.

- citations/utils: parse incomplete <citations> during streaming;
  remove isCitationsBlockIncomplete; keep hasUnreplacedCitationRefs
  internal; document display rule in file header.

Co-authored-by: Cursor <cursoragent@cursor.com>

---
feat(前端): 统一引用逻辑并杜绝半成品引用

- 新增 SafeCitationContent 作为引用正文的唯一出口:内部使用
  useParsedCitations + shouldShowCitationLoading,在引用未就绪时只显示
  「正在整理引用」,就绪后用 createCitationMarkdownComponents 渲染正文;
  支持可选 remarkPlugins、rehypePlugins、isHuman、img。

- 重构 MessageListItem:助手消息正文仅通过 SafeCitationContent 渲染,
  删除重复的 useParsedCitations、shouldShowCitationLoading、
  createCitationMarkdownComponents、CitationsLoadingIndicator 等逻辑;
  用户消息仍用 AIElementMessageResponse,不做引用解析。

- 澄清、present-files(message-list)、思考步骤与 write_file 加载
  (message-group)、子任务结果(subtask-card)均使用
  SafeCitationContent;Artifact 的 markdown 预览仍用同一 guard
  shouldShowCitationLoading,正文由 ArtifactFilePreview 渲染。

- 统一加载条件:shouldShowCitationLoading(rawContent, cleanContent,
  isLoading) 为唯一判断。在「流式中且已有引用块」或「有引用块且
  cleanContent 中仍有未替换的 [cite-N]」时仅显示加载,从而在 Pro/Ultra
  下也能看到「正在整理引用」,且永不出现半成品 [cite-N]。

- message-group 的 write_file:用 shouldShowCitationLoading(
  fileContent, cleanContent, threadIsLoading && isLast) 替代
  hasCitationsBlock + threadIsLoading,与其他场景一致。

- citations/utils:流式时解析未闭合的 <citations>;移除
  isCitationsBlockIncomplete;hasUnreplacedCitationRefs 保持内部使用;
  在文件头注释中说明展示规则。
2026-02-09 15:01:51 +08:00
ruitanglin a4268cb6d3 feat(frontend): unify citation logic and prevent half-finished citations
- Add SafeCitationContent as single component for citation-aware body:
  useParsedCitations + shouldShowCitationLoading; show loading until
  citations complete, then render body with createCitationMarkdownComponents.
  Supports optional remarkPlugins, rehypePlugins, isHuman, img.

- Refactor MessageListItem: assistant message body now uses
  SafeCitationContent only; remove duplicate useParsedCitations,
  shouldShowCitationLoading, createCitationMarkdownComponents and
  CitationsLoadingIndicator logic. Human messages keep plain
  AIElementMessageResponse (no citation parsing).

- Use SafeCitationContent for clarification, present-files (message-list),
  thinking steps and write_file loading (message-group), subtask result
  (subtask-card). Artifact markdown preview keeps same guard
  (shouldShowCitationLoading) with ArtifactFilePreview.

- Unify loading condition: shouldShowCitationLoading(rawContent,
  cleanContent, isLoading) is the single source of truth. Show loading when
  (isLoading && hasCitationsBlock(rawContent)) or when
  (hasCitationsBlock(rawContent) && hasUnreplacedCitationRefs(cleanContent))
  so Pro/Ultra modes also show "loading citations" and half-finished
  [cite-N] never appear.

- message-group write_file: replace hasCitationsBlock + threadIsLoading
  with shouldShowCitationLoading(fileContent, cleanContent,
  threadIsLoading && isLast) for consistency.

- citations/utils: parse incomplete <citations> during streaming;
  remove isCitationsBlockIncomplete; keep hasUnreplacedCitationRefs
  internal; document display rule in file header.

Co-authored-by: Cursor <cursoragent@cursor.com>

---
feat(前端): 统一引用逻辑并杜绝半成品引用

- 新增 SafeCitationContent 作为引用正文的唯一出口:内部使用
  useParsedCitations + shouldShowCitationLoading,在引用未就绪时只显示
  「正在整理引用」,就绪后用 createCitationMarkdownComponents 渲染正文;
  支持可选 remarkPlugins、rehypePlugins、isHuman、img。

- 重构 MessageListItem:助手消息正文仅通过 SafeCitationContent 渲染,
  删除重复的 useParsedCitations、shouldShowCitationLoading、
  createCitationMarkdownComponents、CitationsLoadingIndicator 等逻辑;
  用户消息仍用 AIElementMessageResponse,不做引用解析。

- 澄清、present-files(message-list)、思考步骤与 write_file 加载
  (message-group)、子任务结果(subtask-card)均使用
  SafeCitationContent;Artifact 的 markdown 预览仍用同一 guard
  shouldShowCitationLoading,正文由 ArtifactFilePreview 渲染。

- 统一加载条件:shouldShowCitationLoading(rawContent, cleanContent,
  isLoading) 为唯一判断。在「流式中且已有引用块」或「有引用块且
  cleanContent 中仍有未替换的 [cite-N]」时仅显示加载,从而在 Pro/Ultra
  下也能看到「正在整理引用」,且永不出现半成品 [cite-N]。

- message-group 的 write_file:用 shouldShowCitationLoading(
  fileContent, cleanContent, threadIsLoading && isLast) 替代
  hasCitationsBlock + threadIsLoading,与其他场景一致。

- citations/utils:流式时解析未闭合的 <citations>;移除
  isCitationsBlockIncomplete;hasUnreplacedCitationRefs 保持内部使用;
  在文件头注释中说明展示规则。
2026-02-09 15:01:51 +08:00
Henry Li cbe0f3b32f feat: update translations 2026-02-09 13:57:46 +08:00
Henry Li 738c509c7e feat: update translations 2026-02-09 13:57:46 +08:00
hetao f68b3c26c3 feat: enforce subagent concurrency limit of 3 per turn with batch execution
Strengthen the SUBAGENT_SECTION prompt to prevent the model from launching
more than 3 subagents in a single response. When >3 sub-tasks are needed,
the model is now explicitly instructed to plan and execute in sequential
batches of ≤3. Reinforced at three prompt injection points: thinking style,
main subagent instructions, and critical reminders.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 13:50:54 +08:00
hetao 3aa45ff035 feat: enforce subagent concurrency limit of 3 per turn with batch execution
Strengthen the SUBAGENT_SECTION prompt to prevent the model from launching
more than 3 subagents in a single response. When >3 sub-tasks are needed,
the model is now explicitly instructed to plan and execute in sequential
batches of ≤3. Reinforced at three prompt injection points: thinking style,
main subagent instructions, and critical reminders.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 13:50:54 +08:00
LofiSu 804d988002 chore: add pre-commit hook to reject *@bytedance.com author/committer email
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 13:46:18 +08:00
ruitanglin 79c85d6410 chore: add pre-commit hook to reject *@bytedance.com author/committer email
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 13:46:18 +08:00
hetao caf12da0f2 feat: add DanglingToolCallMiddleware and SubagentLimitMiddleware
Add two new middlewares to improve robustness of the agent pipeline:
- DanglingToolCallMiddleware injects placeholder ToolMessages for
  interrupted tool calls, preventing LLM errors from malformed history
- SubagentLimitMiddleware truncates excess parallel task tool calls at
  the model response level, replacing the runtime check in task_tool

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 13:22:49 +08:00
hetao 48e3039055 feat: add DanglingToolCallMiddleware and SubagentLimitMiddleware
Add two new middlewares to improve robustness of the agent pipeline:
- DanglingToolCallMiddleware injects placeholder ToolMessages for
  interrupted tool calls, preventing LLM errors from malformed history
- SubagentLimitMiddleware truncates excess parallel task tool calls at
  the model response level, replacing the runtime check in task_tool

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 13:22:49 +08:00
LofiSu 2c3ddbb9e5 Merge pull request #26 from LofiSu/experimental
引用(Citations)优化、Gateway 路径工具抽离、模式悬停说明与中英文国际化
2026-02-09 13:10:37 +08:00
LofiSu dda6b57b46 Merge pull request #26 from LofiSu/experimental
引用(Citations)优化、Gateway 路径工具抽离、模式悬停说明与中英文国际化
2026-02-09 13:10:37 +08:00
LofiSu da9b8e3333 Merge upstream/experimental, resolve conflicts in input-box, message-group, en-US, zh-CN
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 13:10:14 +08:00
ruitanglin 2bac0b904f Merge upstream/experimental, resolve conflicts in input-box, message-group, en-US, zh-CN
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 13:10:14 +08:00
ruitanglin 412923708f Merge upstream/experimental, resolve conflicts in input-box, message-group, en-US, zh-CN
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 13:10:14 +08:00
LofiSu 1c9a969a70 i18n(zh-CN): keep Pro and Ultra as English in mode labels
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 13:02:54 +08:00
ruitanglin 9af66f384b i18n(zh-CN): keep Pro and Ultra as English in mode labels
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 13:02:54 +08:00
LofiSu 0fff2880d3 Merge upstream/experimental and resolve conflicts; citations + path_utils + mode-hover
## 冲突解决 (Resolve conflicts)
- input-box.tsx: 保留 ModeHoverGuide 包裹的模式选择器(PR #26 的 mode-hover-guide)
- message-group.tsx: 保留 getCleanContent / hasCitationsBlock / useParsedCitations
- message-list-item.tsx: 保留 useParsedCitations,移除重复的 MessageLink(使用 CitationAwareLink)
- artifact-file-detail.tsx: 保留 CitationAwareLink、useParsedCitations、contentWithoutCitationsFromParsed
- artifacts.py: 保留 path_utils 与 _extract_citation_urls + remove_citations_block 精简实现
- citations/index.ts: 保留并补充 contentWithoutCitationsFromParsed 导出
- en-US.ts: 保留 Ultra 模式描述 "Reasoning, planning and execution with subagents..."
- zh-CN.ts: 保留「超级」标签,描述保留「思考、计划并执行,可调用子代理分工协作...」

## PR #26 代码改动汇总

### 1. Citations(引用)
- lead_agent prompt: 增加 Web search 与子代理合成时的 citation 提示
- general_purpose: 子代理 system prompt 增加 <citations_format> 说明
- frontend utils: 新增 contentWithoutCitationsFromParsed,removeAllCitations 基于单次解析
- frontend artifact: 使用 contentWithoutCitationsFromParsed(parsed) 避免对同一内容解析两次
- backend artifacts: _extract_citation_urls + remove_citations_block,json 提到顶部

### 2. path_utils(路径解析)
- 新增 backend/src/gateway/path_utils.py:resolve_thread_virtual_path,防 path traversal
- artifacts.py / skills.py:删除内联路径解析,统一使用 path_utils

### 3. Mode hover guide
- input-box: 模式选择器外包 ModeHoverGuide,悬停展示模式说明

### 4. i18n
- en: ultraModeDescription 与 zh: ultraMode / ultraModeDescription 与上游对齐并保留 PR 文案

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 13:01:01 +08:00
ruitanglin 58e10b0bca Merge upstream/experimental and resolve conflicts; citations + path_utils + mode-hover
## 冲突解决 (Resolve conflicts)
- input-box.tsx: 保留 ModeHoverGuide 包裹的模式选择器(PR #26 的 mode-hover-guide)
- message-group.tsx: 保留 getCleanContent / hasCitationsBlock / useParsedCitations
- message-list-item.tsx: 保留 useParsedCitations,移除重复的 MessageLink(使用 CitationAwareLink)
- artifact-file-detail.tsx: 保留 CitationAwareLink、useParsedCitations、contentWithoutCitationsFromParsed
- artifacts.py: 保留 path_utils 与 _extract_citation_urls + remove_citations_block 精简实现
- citations/index.ts: 保留并补充 contentWithoutCitationsFromParsed 导出
- en-US.ts: 保留 Ultra 模式描述 "Reasoning, planning and execution with subagents..."
- zh-CN.ts: 保留「超级」标签,描述保留「思考、计划并执行,可调用子代理分工协作...」

## PR #26 代码改动汇总

### 1. Citations(引用)
- lead_agent prompt: 增加 Web search 与子代理合成时的 citation 提示
- general_purpose: 子代理 system prompt 增加 <citations_format> 说明
- frontend utils: 新增 contentWithoutCitationsFromParsed,removeAllCitations 基于单次解析
- frontend artifact: 使用 contentWithoutCitationsFromParsed(parsed) 避免对同一内容解析两次
- backend artifacts: _extract_citation_urls + remove_citations_block,json 提到顶部

### 2. path_utils(路径解析)
- 新增 backend/src/gateway/path_utils.py:resolve_thread_virtual_path,防 path traversal
- artifacts.py / skills.py:删除内联路径解析,统一使用 path_utils

### 3. Mode hover guide
- input-box: 模式选择器外包 ModeHoverGuide,悬停展示模式说明

### 4. i18n
- en: ultraModeDescription 与 zh: ultraMode / ultraModeDescription 与上游对齐并保留 PR 文案

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 13:01:01 +08:00
LofiSu 2a39947830 feat: citations prompts, path_utils, and citation code cleanup
- Prompt: add citation reminders for web_search and subagent synthesis (lead_agent, general_purpose)
- Gateway: add path_utils for shared thread virtual path resolution; refactor artifacts and skills to use it
- Citations: simplify removeAllCitations (single parse); backend _extract_citation_urls and remove_citations_block cleanup

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 12:55:12 +08:00
ruitanglin eb5782b93b feat: citations prompts, path_utils, and citation code cleanup
- Prompt: add citation reminders for web_search and subagent synthesis (lead_agent, general_purpose)
- Gateway: add path_utils for shared thread virtual path resolution; refactor artifacts and skills to use it
- Citations: simplify removeAllCitations (single parse); backend _extract_citation_urls and remove_citations_block cleanup

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 12:55:12 +08:00
LofiSu 8168ea47b3 chore(frontend): remove unused Citation UI components from inline-citation
- Remove InlineCitation, InlineCitationText, InlineCitationCardTrigger
- Remove InlineCitationCarousel and all Carousel subcomponents (Content, Item, Header, Index, Prev, Next)
- Remove InlineCitationQuote
- Drop Carousel/carousel and ArrowLeft/ArrowRight icon imports; keep only CitationLink, CitationAwareLink, CitationsLoadingIndicator and their dependencies

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 12:49:31 +08:00
ruitanglin 2b10b97bb9 chore(frontend): remove unused Citation UI components from inline-citation
- Remove InlineCitation, InlineCitationText, InlineCitationCardTrigger
- Remove InlineCitationCarousel and all Carousel subcomponents (Content, Item, Header, Index, Prev, Next)
- Remove InlineCitationQuote
- Drop Carousel/carousel and ArrowLeft/ArrowRight icon imports; keep only CitationLink, CitationAwareLink, CitationsLoadingIndicator and their dependencies

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 12:49:31 +08:00
LofiSu d265bdb245 feat(frontend): add mode hover guide and adjust mode i18n
## 中文

### 代码改动
- **新增** `frontend/src/components/workspace/mode-hover-guide.tsx`
  - 新增 ModeHoverGuide 组件:接收 mode (flash/thinking/pro/ultra) 与 children,用 Tooltip 包裹
  - hover 时展示该模式名称与简介,支持 showTitle 控制是否显示模式名
  - 文案通过 useI18n 从 inputBox 的 *Mode / *ModeDescription 读取,中英文已支持
- **修改** `frontend/src/components/workspace/input-box.tsx`
  - 在模式选择器触发按钮外包一层 ModeHoverGuide,悬停当前模式时显示说明
- **修改** `frontend/src/core/i18n/locales/zh-CN.ts`
  - ultraModeDescription:改为完整描述「思考、计划并执行,可调用子代理分工协作,适合复杂多步骤任务,能力最强」(不再仅写「专业模式加子代理」)
  - proMode / ultraMode:中文环境下保留英文原文 "Pro"、"Ultra",不再翻译为「专业」「超级」
- **修改** `frontend/src/core/i18n/locales/en-US.ts`
  - ultraModeDescription:改为 "Reasoning, planning and execution with subagents to divide work; best for complex multi-step tasks"

### 说明
为 Flash / 思考 / Pro / Ultra 四种模式增加 hover 说明,并统一超级模式文案与 Pro/Ultra 在中文下的展示。

Co-authored-by: Cursor <cursoragent@cursor.com>

---

## English

### Code changes
- **Add** `frontend/src/components/workspace/mode-hover-guide.tsx`
  - New ModeHoverGuide component: takes mode (flash/thinking/pro/ultra) and children, wraps in Tooltip
  - On hover shows mode name and short description; showTitle toggles mode name in tooltip
  - Copy from useI18n (inputBox *Mode / *ModeDescription), i18n in zh-CN and en-US
- **Update** `frontend/src/components/workspace/input-box.tsx`
  - Wrap mode selector trigger with ModeHoverGuide so hovering shows current mode description
- **Update** `frontend/src/core/i18n/locales/zh-CN.ts`
  - ultraModeDescription: full description (reasoning, planning, execution, subagents, complex tasks); no longer "Pro + subagents" only
  - proMode / ultraMode: keep English "Pro" and "Ultra" in zh locale instead of "专业" / "超级"
- **Update** `frontend/src/core/i18n/locales/en-US.ts`
  - ultraModeDescription: "Reasoning, planning and execution with subagents to divide work; best for complex multi-step tasks"

### Summary
Hover guide for all four modes (Flash / Reasoning / Pro / Ultra); clearer Ultra copy and Pro/Ultra labels in Chinese.
2026-02-09 12:33:16 +08:00
ruitanglin 5e000f1a99 feat(frontend): add mode hover guide and adjust mode i18n
## 中文

### 代码改动
- **新增** `frontend/src/components/workspace/mode-hover-guide.tsx`
  - 新增 ModeHoverGuide 组件:接收 mode (flash/thinking/pro/ultra) 与 children,用 Tooltip 包裹
  - hover 时展示该模式名称与简介,支持 showTitle 控制是否显示模式名
  - 文案通过 useI18n 从 inputBox 的 *Mode / *ModeDescription 读取,中英文已支持
- **修改** `frontend/src/components/workspace/input-box.tsx`
  - 在模式选择器触发按钮外包一层 ModeHoverGuide,悬停当前模式时显示说明
- **修改** `frontend/src/core/i18n/locales/zh-CN.ts`
  - ultraModeDescription:改为完整描述「思考、计划并执行,可调用子代理分工协作,适合复杂多步骤任务,能力最强」(不再仅写「专业模式加子代理」)
  - proMode / ultraMode:中文环境下保留英文原文 "Pro"、"Ultra",不再翻译为「专业」「超级」
- **修改** `frontend/src/core/i18n/locales/en-US.ts`
  - ultraModeDescription:改为 "Reasoning, planning and execution with subagents to divide work; best for complex multi-step tasks"

### 说明
为 Flash / 思考 / Pro / Ultra 四种模式增加 hover 说明,并统一超级模式文案与 Pro/Ultra 在中文下的展示。

Co-authored-by: Cursor <cursoragent@cursor.com>

---

## English

### Code changes
- **Add** `frontend/src/components/workspace/mode-hover-guide.tsx`
  - New ModeHoverGuide component: takes mode (flash/thinking/pro/ultra) and children, wraps in Tooltip
  - On hover shows mode name and short description; showTitle toggles mode name in tooltip
  - Copy from useI18n (inputBox *Mode / *ModeDescription), i18n in zh-CN and en-US
- **Update** `frontend/src/components/workspace/input-box.tsx`
  - Wrap mode selector trigger with ModeHoverGuide so hovering shows current mode description
- **Update** `frontend/src/core/i18n/locales/zh-CN.ts`
  - ultraModeDescription: full description (reasoning, planning, execution, subagents, complex tasks); no longer "Pro + subagents" only
  - proMode / ultraMode: keep English "Pro" and "Ultra" in zh locale instead of "专业" / "超级"
- **Update** `frontend/src/core/i18n/locales/en-US.ts`
  - ultraModeDescription: "Reasoning, planning and execution with subagents to divide work; best for complex multi-step tasks"

### Summary
Hover guide for all four modes (Flash / Reasoning / Pro / Ultra); clearer Ultra copy and Pro/Ultra labels in Chinese.
2026-02-09 12:33:16 +08:00
LofiSu 30e1760211 refactor(frontend): simplify and deduplicate Citation-related code
- Extract removeCitationsBlocks in utils, reuse in parseCitations and removeAllCitations
- Add hasCitationsBlock; isCitationsBlockIncomplete now uses it
- Add useParsedCitations hook (parseCitations + buildCitationMap) for message/artifact
- Add CitationAwareLink to unify link rendering (message-list-item + artifact-file-detail)
- Add getCleanContent helper; message-group uses it and useParsedCitations
- ArtifactFileDetail: single useParsedCitations, pass cleanContent/citationMap to Preview
- Stop exporting buildCitationMap and removeCitationsBlocks from citations index
- Remove duplicate MessageLink and inline link logic in artifact preview

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 12:13:06 +08:00
ruitanglin 175c1d2e3b refactor(frontend): simplify and deduplicate Citation-related code
- Extract removeCitationsBlocks in utils, reuse in parseCitations and removeAllCitations
- Add hasCitationsBlock; isCitationsBlockIncomplete now uses it
- Add useParsedCitations hook (parseCitations + buildCitationMap) for message/artifact
- Add CitationAwareLink to unify link rendering (message-list-item + artifact-file-detail)
- Add getCleanContent helper; message-group uses it and useParsedCitations
- ArtifactFileDetail: single useParsedCitations, pass cleanContent/citationMap to Preview
- Stop exporting buildCitationMap and removeCitationsBlocks from citations index
- Remove duplicate MessageLink and inline link logic in artifact preview

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 12:13:06 +08:00
LofiSu f0423b88ac Merge branch 'hetaoBackend:experimental' into experimental 2026-02-09 11:58:56 +08:00
LofiSu 302211696e Merge branch 'hetaoBackend:experimental' into experimental 2026-02-09 11:58:56 +08:00
Henry Li 8b053a4415 feat: update workspace header to conditionally render title based on environment variable 2026-02-09 09:20:32 +08:00
Henry Li fd4f6c679a feat: update workspace header to conditionally render title based on environment variable 2026-02-09 09:20:32 +08:00
Henry Li 3ad2cd936f feat: update workspace header to conditionally render title based on environment variable 2026-02-09 09:20:32 +08:00
Henry Li 305e8969ef feat: make it golden 2026-02-09 09:15:39 +08:00
Henry Li 189fcab4c5 feat: make it golden 2026-02-09 09:15:39 +08:00
Henry Li e6261469ef feat: make it golden 2026-02-09 09:15:39 +08:00
Henry Li ddbda4e38f feat: make the title golden in Ultra mode 2026-02-09 08:59:40 +08:00
Henry Li db79ab27f4 feat: make the title golden in Ultra mode 2026-02-09 08:59:40 +08:00
Henry Li 76cdb0e16e feat: make the title golden in Ultra mode 2026-02-09 08:59:40 +08:00
LofiSu 2d70aaa969 fix(frontend): citations display + refactor link/citation utils
- Citations: no underline while streaming (message links); artifact markdown external links as citation cards
- Refactor: add isExternalUrl, syntheticCitationFromLink in core/citations; shared externalLinkClass in lib/utils; simplify message-list-item and artifact-file-detail link rendering

修复引用展示并抽离链接/引用工具
- 引用:流式输出时链接不这下划线;Artifact 内 Markdown 外链以引用卡片展示
- 重构:core/citations 新增 isExternalUrl、syntheticCitationFromLink;lib/utils 共享 externalLinkClass;精简消息与 Artifact 中的链接渲染逻辑

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 04:03:15 +08:00
ruitanglin 509ea874f7 fix(frontend): citations display + refactor link/citation utils
- Citations: no underline while streaming (message links); artifact markdown external links as citation cards
- Refactor: add isExternalUrl, syntheticCitationFromLink in core/citations; shared externalLinkClass in lib/utils; simplify message-list-item and artifact-file-detail link rendering

修复引用展示并抽离链接/引用工具
- 引用:流式输出时链接不这下划线;Artifact 内 Markdown 外链以引用卡片展示
- 重构:core/citations 新增 isExternalUrl、syntheticCitationFromLink;lib/utils 共享 externalLinkClass;精简消息与 Artifact 中的链接渲染逻辑

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 04:03:15 +08:00
LofiSu d72aad8063 fix(frontend): build + remove hover tooltips in step links
- Fix Turbopack build: replace raw-loader .md import with inlined about-content.ts; drop raw-loader from next.config and package.json
- Remove all hover tooltips on step-area links (web_fetch, read_file, ls, bash, write_file, web_search) so hidden steps no longer show popups

修复:构建错误与步骤链接悬停提示
- 修复 Turbopack 构建:用内联 about-content.ts 替代 raw-loader 导入 about.md,并移除 next.config 与 package.json 中的 raw-loader
- 移除步骤区域内所有链接的悬停提示(查看网页、读文件、列目录、bash、写文件、网页搜索),隐藏步骤悬停不再弹出内容

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 03:42:16 +08:00
ruitanglin 8cb14ad4fb fix(frontend): build + remove hover tooltips in step links
- Fix Turbopack build: replace raw-loader .md import with inlined about-content.ts; drop raw-loader from next.config and package.json
- Remove all hover tooltips on step-area links (web_fetch, read_file, ls, bash, write_file, web_search) so hidden steps no longer show popups

修复:构建错误与步骤链接悬停提示
- 修复 Turbopack 构建:用内联 about-content.ts 替代 raw-loader 导入 about.md,并移除 next.config 与 package.json 中的 raw-loader
- 移除步骤区域内所有链接的悬停提示(查看网页、读文件、列目录、bash、写文件、网页搜索),隐藏步骤悬停不再弹出内容

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 03:42:16 +08:00
LofiSu fe06be8258 Revert "fix(frontend): Turbopack about page + remove hover on web search/citations"
This reverts commit 7e9e061f20fcec1f1a9c35be40d9407c05ed82be.
2026-02-09 03:23:51 +08:00
ruitanglin f577ff115b Revert "fix(frontend): Turbopack about page + remove hover on web search/citations"
This reverts commit 7e9e061f20fcec1f1a9c35be40d9407c05ed82be.
2026-02-09 03:23:51 +08:00
LofiSu 842c4ecac0 fix(frontend): Turbopack about page + remove hover on web search/citations
- About: use aboutMarkdown from about-content.ts instead of raw-loader for
  about.md (fixes Turbopack 'Cannot find module raw-loader')
- Web search: remove Tooltip from web_search and web_fetch result links
- Citations: remove HoverCard from CitationLink so no hover popup on badges

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 03:23:02 +08:00
ruitanglin 77859d01b8 fix(frontend): Turbopack about page + remove hover on web search/citations
- About: use aboutMarkdown from about-content.ts instead of raw-loader for
  about.md (fixes Turbopack 'Cannot find module raw-loader')
- Web search: remove Tooltip from web_search and web_fetch result links
- Citations: remove HoverCard from CitationLink so no hover popup on badges

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 03:23:02 +08:00
Henry Li cebf2599c9 feat: add mode in welcome 2026-02-09 00:41:25 +08:00
Henry Li 143f9f1f4d feat: add mode in welcome 2026-02-09 00:41:25 +08:00
Henry Li d197ee8f28 feat: add mode in welcome 2026-02-09 00:41:25 +08:00
Henry Li 25b60e732f feat: set golden color for ultra 2026-02-09 00:30:20 +08:00
Henry Li 9da3a1dcb2 feat: set golden color for ultra 2026-02-09 00:30:20 +08:00
Henry Li d9b60778a9 feat: set golden color for ultra 2026-02-09 00:30:20 +08:00
Henry Li f146e35ee7 feat: rewording 2026-02-08 23:44:36 +08:00
Henry Li a4e1e1a95e feat: rewording 2026-02-08 23:44:36 +08:00
Henry Li eb9af00d1d feat: rewording 2026-02-08 23:44:36 +08:00
hetaoBackend 6eb4cdd3ec feat: disallow present_files tool in subagents and add market-analysis skill
Add "present_files" to disallowed_tools for bash and general-purpose
subagents to prevent them from presenting files directly. Also add the
new market-analysis skill for generating consulting-grade reports.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 23:38:55 +08:00
hetaoBackend f9b769b5c3 feat: disallow present_files tool in subagents and add market-analysis skill
Add "present_files" to disallowed_tools for bash and general-purpose
subagents to prevent them from presenting files directly. Also add the
new market-analysis skill for generating consulting-grade reports.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 23:38:55 +08:00
hetaoBackend 54f2f1bd3a feat: disallow present_files tool in subagents and add market-analysis skill
Add "present_files" to disallowed_tools for bash and general-purpose
subagents to prevent them from presenting files directly. Also add the
new market-analysis skill for generating consulting-grade reports.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 23:38:55 +08:00
Henry Li 8a2351593c feat: add special effect for Ultra mode 2026-02-08 23:22:51 +08:00
Henry Li d36fbcdfc1 feat: add special effect for Ultra mode 2026-02-08 23:22:51 +08:00
Henry Li 0d55230016 feat: add special effect for Ultra mode 2026-02-08 23:22:51 +08:00
hetaoBackend 2703eb0b22 docs: revise backend README and CLAUDE.md to reflect full architecture
Updated documentation to accurately cover all backend subsystems including
subagents, memory, middleware chain, sandbox, MCP, skills, and gateway API.
Fixed broken MCP_SETUP.md link in root README.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 22:50:42 +08:00
hetaoBackend fdd25c1bb8 docs: revise backend README and CLAUDE.md to reflect full architecture
Updated documentation to accurately cover all backend subsystems including
subagents, memory, middleware chain, sandbox, MCP, skills, and gateway API.
Fixed broken MCP_SETUP.md link in root README.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 22:50:42 +08:00
hetaoBackend d891a8a37c docs: revise backend README and CLAUDE.md to reflect full architecture
Updated documentation to accurately cover all backend subsystems including
subagents, memory, middleware chain, sandbox, MCP, skills, and gateway API.
Fixed broken MCP_SETUP.md link in root README.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 22:50:42 +08:00
Henry Li 010aba1e28 feat: add realtime subagent status report 2026-02-08 22:43:51 +08:00
Henry Li 7ed1be32fd feat: add realtime subagent status report 2026-02-08 22:43:51 +08:00
Henry Li 7d4b5eb3ca feat: add realtime subagent status report 2026-02-08 22:43:51 +08:00
hetaoBackend 808e028338 feat: limit concurrent subagents to 3 per turn
Prevent resource exhaustion by capping the number of parallel subagents.
Adds runtime enforcement in task_tool and updates prompts/examples accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 22:12:21 +08:00
hetaoBackend 9e2b3f1f39 feat: limit concurrent subagents to 3 per turn
Prevent resource exhaustion by capping the number of parallel subagents.
Adds runtime enforcement in task_tool and updates prompts/examples accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 22:12:21 +08:00
hetaoBackend faa327b3cd feat: limit concurrent subagents to 3 per turn
Prevent resource exhaustion by capping the number of parallel subagents.
Adds runtime enforcement in task_tool and updates prompts/examples accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 22:12:21 +08:00
hetaoBackend 96bace7ab6 feat: add real-time streaming of subagent AI messages
Enable task tool to capture and stream AI messages as they are generated by subagents. This replaces simple polling status updates with detailed message-level progress updates.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-08 21:29:45 +08:00
hetaoBackend 0a27a7561a feat: add real-time streaming of subagent AI messages
Enable task tool to capture and stream AI messages as they are generated by subagents. This replaces simple polling status updates with detailed message-level progress updates.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-08 21:29:45 +08:00
hetaoBackend 54772947cb feat: add real-time streaming of subagent AI messages
Enable task tool to capture and stream AI messages as they are generated by subagents. This replaces simple polling status updates with detailed message-level progress updates.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-08 21:29:45 +08:00
Henry Li 0355493a16 feat: rewording and add initial animation 2026-02-08 21:24:17 +08:00
Henry Li 2b3dc96e40 feat: rewording and add initial animation 2026-02-08 21:24:17 +08:00
Henry Li ff7437f830 feat: rewording and add initial animation 2026-02-08 21:24:17 +08:00
hetaoBackend 17365e40d5 fix: fix sub agent timeout 2026-02-08 21:09:18 +08:00
hetaoBackend 19a1d03fc8 fix: fix sub agent timeout 2026-02-08 21:09:18 +08:00
hetaoBackend f01c470e64 fix: fix sub agent timeout 2026-02-08 21:09:18 +08:00
Chaos_Ljc 5b29c9f70a docs: fix typo and grammar in readme (#851) 2026-02-08 10:39:57 +08:00
Henry Li 5d4cecbb84 refactor: optimize task handling in message list 2026-02-07 18:42:24 +08:00
Henry Li 4f9150229c refactor: optimize task handling in message list 2026-02-07 18:42:24 +08:00
Henry Li 542b04588a refactor: optimize task handling in message list 2026-02-07 18:42:24 +08:00
Henry Li de8ff9d336 feat: add ambilight 2026-02-07 18:42:08 +08:00
Henry Li 01aa035905 feat: add ambilight 2026-02-07 18:42:08 +08:00
Henry Li a4e89cc96b feat: add ambilight 2026-02-07 18:42:08 +08:00
Henry Li d9a52f07e7 feat: add handling for task timeout and enhance Streamdown plugin for word animation 2026-02-07 18:06:22 +08:00
Henry Li 99e8f22d1d feat: add handling for task timeout and enhance Streamdown plugin for word animation 2026-02-07 18:06:22 +08:00
Henry Li 0810917b69 feat: add handling for task timeout and enhance Streamdown plugin for word animation 2026-02-07 18:06:22 +08:00
Henry Li 260953fb81 feat: adjust position 2026-02-07 18:00:24 +08:00
Henry Li dce82c1db4 feat: adjust position 2026-02-07 18:00:24 +08:00
Henry Li 4dc3cdac48 feat: adjust position 2026-02-07 18:00:24 +08:00
Henry Li b135449c07 fix: adjust suggestion positioning and height for improved UI layout 2026-02-07 17:56:06 +08:00
Henry Li 2510991698 fix: adjust suggestion positioning and height for improved UI layout 2026-02-07 17:56:06 +08:00
Henry Li 17b2630b73 fix: adjust suggestion positioning and height for improved UI layout 2026-02-07 17:56:06 +08:00
hetaoBackend f41d9b3be5 refactor: optimize task tool parameter order and improve task tracking
- Reorder task tool parameters to prioritize description first for better usability
- Add tool_call_id injection for better task traceability
- Use tool_call_id as task_id in executor for consistent tracking
- Simplify event messages by removing redundant task_type field
- Update task examples in prompt to reflect new parameter order

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-07 16:56:13 +08:00
hetaoBackend 1425294f9b refactor: optimize task tool parameter order and improve task tracking
- Reorder task tool parameters to prioritize description first for better usability
- Add tool_call_id injection for better task traceability
- Use tool_call_id as task_id in executor for consistent tracking
- Simplify event messages by removing redundant task_type field
- Update task examples in prompt to reflect new parameter order

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-07 16:56:13 +08:00
hetaoBackend a6db74baba refactor: optimize task tool parameter order and improve task tracking
- Reorder task tool parameters to prioritize description first for better usability
- Add tool_call_id injection for better task traceability
- Use tool_call_id as task_id in executor for consistent tracking
- Simplify event messages by removing redundant task_type field
- Update task examples in prompt to reflect new parameter order

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-07 16:56:13 +08:00
Henry Li 3e2883e2a3 feat: support subtasks 2026-02-07 16:14:48 +08:00
Henry Li a016332a37 feat: support subtasks 2026-02-07 16:14:48 +08:00
Henry Li 46798c0931 feat: support subtasks 2026-02-07 16:14:48 +08:00
Henry Li 39a5d8dc30 Merge remote-tracking branch 'refs/remotes/origin/experimental' into experimental 2026-02-07 16:13:48 +08:00
Henry Li 617559a900 Merge remote-tracking branch 'refs/remotes/origin/experimental' into experimental 2026-02-07 16:13:48 +08:00
Henry Li e04e70c7a8 Merge remote-tracking branch 'refs/remotes/origin/experimental' into experimental 2026-02-07 16:13:48 +08:00
LofiSu e4eb4a65cf Merge pull request #25 from LofiSu/feat/citations
feat(citations): add shared citation components and optimize code
2026-02-07 12:13:32 +08:00
LofiSu afb7a36739 Merge pull request #25 from LofiSu/feat/citations
feat(citations): add shared citation components and optimize code
2026-02-07 12:13:32 +08:00
LofiSu 9f8d9e4da2 Merge pull request #25 from LofiSu/feat/citations
feat(citations): add shared citation components and optimize code
2026-02-07 12:13:32 +08:00
Henry Li 91a05acdf8 feat: enhance workspace navigation menu with conditional rendering and mounted state 2026-02-07 11:10:24 +08:00
Henry Li 4ac637a0eb feat: enhance workspace navigation menu with conditional rendering and mounted state 2026-02-07 11:10:24 +08:00
Henry Li e7cd5287f1 feat: enhance workspace navigation menu with conditional rendering and mounted state 2026-02-07 11:10:24 +08:00
Henry Li 60be7ee20d docs: update description for surprise-me skill to enhance clarity 2026-02-07 10:51:43 +08:00
Henry Li 86ad92a1a6 docs: update description for surprise-me skill to enhance clarity 2026-02-07 10:51:43 +08:00
Henry Li 85767c8470 docs: update description for surprise-me skill to enhance clarity 2026-02-07 10:51:43 +08:00
Henry Li c758a28a3e styles: format 2026-02-07 10:50:08 +08:00
Henry Li 70191783ad styles: format 2026-02-07 10:50:08 +08:00
Henry Li 1d3ed9f43b styles: format 2026-02-07 10:50:08 +08:00
Henry Li a122f76e36 feat: add animations 2026-02-07 10:30:35 +08:00
Henry Li a2af464a6f feat: add animations 2026-02-07 10:30:35 +08:00
Henry Li fc543a9b30 feat: add animations 2026-02-07 10:30:35 +08:00
LofiSu f0075e0d64 Merge upstream/experimental into feat/citations
Resolved conflicts:
- backend/src/gateway/routers/artifacts.py: Keep citations block removal for markdown downloads
- frontend/src/components/workspace/messages/message-list-item.tsx: Keep improved citation handling with rehypePlugins, humanMessagePlugins, and CitationsLoadingIndicator

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-07 00:53:16 +08:00
ruitanglin 2331c67446 Merge upstream/experimental into feat/citations
Resolved conflicts:
- backend/src/gateway/routers/artifacts.py: Keep citations block removal for markdown downloads
- frontend/src/components/workspace/messages/message-list-item.tsx: Keep improved citation handling with rehypePlugins, humanMessagePlugins, and CitationsLoadingIndicator

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-07 00:53:16 +08:00
ruitanglin ea543ce1f4 Merge upstream/experimental into feat/citations
Resolved conflicts:
- backend/src/gateway/routers/artifacts.py: Keep citations block removal for markdown downloads
- frontend/src/components/workspace/messages/message-list-item.tsx: Keep improved citation handling with rehypePlugins, humanMessagePlugins, and CitationsLoadingIndicator

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-07 00:53:16 +08:00
Willem Jiang f21bc6b83f fix(server): graceful stream termination on cancellation (issue #847) (#850)
* fix(server): graceful stream termination on cancellation (issue #847)

* Update the code with review suggestion
2026-02-06 23:41:23 +08:00
Henry Li 5ed15d79c9 fix: fix markdown table 2026-02-06 22:00:55 +08:00
Henry Li 8f1a42a8e0 fix: fix markdown table 2026-02-06 22:00:55 +08:00
Henry Li c3f9089e95 fix: fix markdown table 2026-02-06 22:00:55 +08:00
Henry Li 6b56e68ff2 Merge pull request #24 from LofiSu/fix/upload-files-alignment
fix: 修复用户消息中上传文件的右对齐显示
2026-02-06 21:53:01 +08:00
Henry Li 537687c2c5 Merge pull request #24 from LofiSu/fix/upload-files-alignment
fix: 修复用户消息中上传文件的右对齐显示
2026-02-06 21:53:01 +08:00
Henry Li 5016a5f7d9 Merge pull request #24 from LofiSu/fix/upload-files-alignment
fix: 修复用户消息中上传文件的右对齐显示
2026-02-06 21:53:01 +08:00
hetao 9e4f2512f3 fix: fix subagent prompt 2026-02-06 20:32:15 +08:00
hetao a423dfb9fd fix: fix subagent prompt 2026-02-06 20:32:15 +08:00
hetao d1d275bb81 fix: fix subagent prompt 2026-02-06 20:32:15 +08:00
hetao 9bf3a12c30 feat: send custom event 2026-02-06 17:48:15 +08:00
hetao 172813720a feat: send custom event 2026-02-06 17:48:15 +08:00
hetao 4f15670455 feat: send custom event 2026-02-06 17:48:15 +08:00
hetao 9f367b5563 feat: fix task polling issue 2026-02-06 17:48:15 +08:00
hetao 41d8d2fd5c feat: fix task polling issue 2026-02-06 17:48:15 +08:00
hetao 498c8b3ec0 feat: fix task polling issue 2026-02-06 17:48:15 +08:00
hetao 449ffbad75 feat: add ultra mode 2026-02-06 17:48:14 +08:00
hetao 926c322c36 feat: add ultra mode 2026-02-06 17:48:14 +08:00
hetao 96baab12a2 feat: add ultra mode 2026-02-06 17:48:14 +08:00
LofiSu 5484233548 fix(citations): hide citations block in reasoning/thinking content
The reasoning content in message-group.tsx was not being processed
through parseCitations, causing raw <citations> blocks to be visible.
Now reasoning content is parsed to remove citations blocks.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 16:12:43 +08:00
ruitanglin ca6bcaa31c fix(citations): hide citations block in reasoning/thinking content
The reasoning content in message-group.tsx was not being processed
through parseCitations, causing raw <citations> blocks to be visible.
Now reasoning content is parsed to remove citations blocks.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 16:12:43 +08:00
ruitanglin 50ced32722 fix(citations): hide citations block in reasoning/thinking content
The reasoning content in message-group.tsx was not being processed
through parseCitations, causing raw <citations> blocks to be visible.
Now reasoning content is parsed to remove citations blocks.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 16:12:43 +08:00
LofiSu 582bfaee39 fix(citations): only citation links in citationMap render as badges
Revert streaming logic - only links that are actually in citationMap
should render as badges. This prevents project URLs and other regular
links from being incorrectly rendered as citation badges.

During streaming, links may initially appear as plain links until the
citations block is fully parsed, then they will update to badge style.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 16:10:29 +08:00
ruitanglin 666b747b8a fix(citations): only citation links in citationMap render as badges
Revert streaming logic - only links that are actually in citationMap
should render as badges. This prevents project URLs and other regular
links from being incorrectly rendered as citation badges.

During streaming, links may initially appear as plain links until the
citations block is fully parsed, then they will update to badge style.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 16:10:29 +08:00
ruitanglin e8ee19821d fix(citations): only citation links in citationMap render as badges
Revert streaming logic - only links that are actually in citationMap
should render as badges. This prevents project URLs and other regular
links from being incorrectly rendered as citation badges.

During streaming, links may initially appear as plain links until the
citations block is fully parsed, then they will update to badge style.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 16:10:29 +08:00
LofiSu e7ea0fc551 fix(citations): render external links as badges during streaming
During streaming when citations are still loading (isLoadingCitations=true),
all external links should be rendered as badges since we don't know yet
which links are citations. After streaming completes, only links in
citationMap are rendered as badges.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 16:09:03 +08:00
ruitanglin 697c683dfa fix(citations): render external links as badges during streaming
During streaming when citations are still loading (isLoadingCitations=true),
all external links should be rendered as badges since we don't know yet
which links are citations. After streaming completes, only links in
citationMap are rendered as badges.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 16:09:03 +08:00
ruitanglin e444817c5d fix(citations): render external links as badges during streaming
During streaming when citations are still loading (isLoadingCitations=true),
all external links should be rendered as badges since we don't know yet
which links are citations. After streaming completes, only links in
citationMap are rendered as badges.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 16:09:03 +08:00
LofiSu f1c3f908c9 fix(citations): parse citations in reasoning content
When only reasoning content exists (no main content), the citations
block was not being parsed and removed. Now reasoning content also
goes through parseCitations to hide the raw citations block.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 16:04:49 +08:00
ruitanglin 579dccbdce fix(citations): parse citations in reasoning content
When only reasoning content exists (no main content), the citations
block was not being parsed and removed. Now reasoning content also
goes through parseCitations to hide the raw citations block.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 16:04:49 +08:00
ruitanglin e9648b11cd fix(citations): parse citations in reasoning content
When only reasoning content exists (no main content), the citations
block was not being parsed and removed. Now reasoning content also
goes through parseCitations to hide the raw citations block.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 16:04:49 +08:00
LofiSu 7c21d8f3a6 fix(artifacts): only render citation badges for links in citationMap
Same fix as message-list-item: project URLs and regular links in
artifact file preview should be rendered as plain links, not badges.
Only actual citations (in citationMap) should be rendered as badges.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 15:55:53 +08:00
ruitanglin 365e3f4304 fix(artifacts): only render citation badges for links in citationMap
Same fix as message-list-item: project URLs and regular links in
artifact file preview should be rendered as plain links, not badges.
Only actual citations (in citationMap) should be rendered as badges.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 15:55:53 +08:00
ruitanglin 0cf8ba86d1 fix(artifacts): only render citation badges for links in citationMap
Same fix as message-list-item: project URLs and regular links in
artifact file preview should be rendered as plain links, not badges.
Only actual citations (in citationMap) should be rendered as badges.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 15:55:53 +08:00
Henry Li 70989a949e feat: add 'about' page 2026-02-06 15:18:37 +08:00
Henry Li 44742c6353 feat: add 'about' page 2026-02-06 15:18:37 +08:00
Henry Li f9811671d8 feat: add 'about' page 2026-02-06 15:18:37 +08:00
Henry Li bc7837ed6f docs: rewording 2026-02-06 15:18:19 +08:00
Henry Li dd4a7aae36 docs: rewording 2026-02-06 15:18:19 +08:00
Henry Li ee41324887 docs: rewording 2026-02-06 15:18:19 +08:00
LofiSu 5d8c08d3ba fix(citations): only render citation badges for links in citationMap
Project URLs and regular links should be rendered as plain underlined
links, not as citation badges. Only links that are actual citations
(present in citationMap) should be rendered as badges.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 15:15:45 +08:00
ruitanglin 1ce154fa71 fix(citations): only render citation badges for links in citationMap
Project URLs and regular links should be rendered as plain underlined
links, not as citation badges. Only links that are actual citations
(present in citationMap) should be rendered as badges.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 15:15:45 +08:00
ruitanglin 7a3a5f5196 fix(citations): only render citation badges for links in citationMap
Project URLs and regular links should be rendered as plain underlined
links, not as citation badges. Only links that are actual citations
(present in citationMap) should be rendered as badges.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 15:15:45 +08:00
LofiSu 49f7cf1662 fix(citations): use markdown link text as fallback for display
When citation data is not available, use the markdown link text
(children) as display text instead of just the domain. This ensures
that links like [OpenJudge](github.com/...) show 'OpenJudge' instead
of just 'github.com'.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 15:06:51 +08:00
ruitanglin acbf2fb453 fix(citations): use markdown link text as fallback for display
When citation data is not available, use the markdown link text
(children) as display text instead of just the domain. This ensures
that links like [OpenJudge](github.com/...) show 'OpenJudge' instead
of just 'github.com'.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 15:06:51 +08:00
ruitanglin c87f176fac fix(citations): use markdown link text as fallback for display
When citation data is not available, use the markdown link text
(children) as display text instead of just the domain. This ensures
that links like [OpenJudge](github.com/...) show 'OpenJudge' instead
of just 'github.com'.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 15:06:51 +08:00
Henry Li 23c082f05d docs: add CLAUDE.md 2026-02-06 14:40:52 +08:00
Henry Li a711c5f310 docs: add CLAUDE.md 2026-02-06 14:40:52 +08:00
Henry Li 8bd20ab4e6 docs: add CLAUDE.md 2026-02-06 14:40:52 +08:00
LofiSu a91302ac72 fix(prompt): clarify citation link format must include URL
AI was outputting bare brackets like [arXiv:xxx] without URLs,
which do not render as links. Updated prompt to explicitly show
correct vs wrong formats and require complete markdown links.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 14:38:31 +08:00
ruitanglin f43522bd27 fix(prompt): clarify citation link format must include URL
AI was outputting bare brackets like [arXiv:xxx] without URLs,
which do not render as links. Updated prompt to explicitly show
correct vs wrong formats and require complete markdown links.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 14:38:31 +08:00
ruitanglin b46a19e116 fix(prompt): clarify citation link format must include URL
AI was outputting bare brackets like [arXiv:xxx] without URLs,
which do not render as links. Updated prompt to explicitly show
correct vs wrong formats and require complete markdown links.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 14:38:31 +08:00
Henry Li 78b6164770 docs: add AGENTS.md 2026-02-06 14:32:48 +08:00
Henry Li 5b33a62f05 docs: add AGENTS.md 2026-02-06 14:32:48 +08:00
Henry Li 30cd2387f2 docs: add AGENTS.md 2026-02-06 14:32:48 +08:00
LofiSu 738b71be47 fix(messages): prevent URL autolink bleeding into adjacent text
For human messages, disable remark-gfm autolink feature to prevent
URLs from incorrectly including adjacent text (especially Chinese
characters) as part of the link. This ensures that when users input
"https://example.com 帮我分析", only the URL becomes a link.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 14:30:57 +08:00
ruitanglin c8c4d2fc95 fix(messages): prevent URL autolink bleeding into adjacent text
For human messages, disable remark-gfm autolink feature to prevent
URLs from incorrectly including adjacent text (especially Chinese
characters) as part of the link. This ensures that when users input
"https://example.com 帮我分析", only the URL becomes a link.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 14:30:57 +08:00
ruitanglin 34a199c6f3 fix(messages): prevent URL autolink bleeding into adjacent text
For human messages, disable remark-gfm autolink feature to prevent
URLs from incorrectly including adjacent text (especially Chinese
characters) as part of the link. This ensures that when users input
"https://example.com 帮我分析", only the URL becomes a link.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 14:30:57 +08:00
LofiSu 6f968242d6 fix(citations): only render CitationLink badges for AI messages
Human messages should display links as plain underlined text,
not as citation badges. This preserves the original user input
appearance when users paste URLs in their messages.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 14:28:28 +08:00
ruitanglin 1b0c016093 fix(citations): only render CitationLink badges for AI messages
Human messages should display links as plain underlined text,
not as citation badges. This preserves the original user input
appearance when users paste URLs in their messages.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 14:28:28 +08:00
ruitanglin bcbbf9cf3f fix(citations): only render CitationLink badges for AI messages
Human messages should display links as plain underlined text,
not as citation badges. This preserves the original user input
appearance when users paste URLs in their messages.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 14:28:28 +08:00
Henry Li b74cf65275 feat: update surprise-me functionality with localization support 2026-02-06 14:21:03 +08:00
Henry Li 765e35fc70 feat: update surprise-me functionality with localization support 2026-02-06 14:21:03 +08:00
Henry Li bbb1a731a5 feat: update surprise-me functionality with localization support 2026-02-06 14:21:03 +08:00
Henry Li 22dea3fd43 feat: add surprise-me 2026-02-06 14:04:15 +08:00
Henry Li 26e078df7d feat: add surprise-me 2026-02-06 14:04:15 +08:00
Henry Li 697ea8e845 feat: add surprise-me 2026-02-06 14:04:15 +08:00
dependabot[bot] ec46937384 build(deps): bump python-multipart from 0.0.20 to 0.0.22 (#849)
Bumps [python-multipart](https://github.com/Kludex/python-multipart) from 0.0.20 to 0.0.22.
- [Release notes](https://github.com/Kludex/python-multipart/releases)
- [Changelog](https://github.com/Kludex/python-multipart/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Kludex/python-multipart/compare/0.0.20...0.0.22)

---
updated-dependencies:
- dependency-name: python-multipart
  dependency-version: 0.0.22
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-02-06 10:51:24 +08:00
dependabot[bot] ee73886c9c build(deps): bump jspdf from 4.0.0 to 4.1.0 in /web (#848)
Bumps [jspdf](https://github.com/parallax/jsPDF) from 4.0.0 to 4.1.0.
- [Release notes](https://github.com/parallax/jsPDF/releases)
- [Changelog](https://github.com/parallax/jsPDF/blob/master/RELEASE.md)
- [Commits](https://github.com/parallax/jsPDF/compare/v4.0.0...v4.1.0)

---
updated-dependencies:
- dependency-name: jspdf
  dependency-version: 4.1.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-06 10:29:43 +08:00
Henry Li f391060573 feat: adjust position 2026-02-06 09:39:20 +08:00
Henry Li 254efe7391 feat: adjust position 2026-02-06 09:39:20 +08:00
Henry Li dedfa1bfb5 feat: adjust position 2026-02-06 09:39:20 +08:00
hetao 85128f5f14 feat: add configuration to enable/disable subagents
Add subagents.enabled flag in config.yaml to control subagent feature:
- When disabled, task/task_status tools are not loaded
- When disabled, system prompt excludes subagent documentation
- Default is enabled for backward compatibility

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-05 20:49:02 +08:00
hetao b7bf027aa5 feat: add configuration to enable/disable subagents
Add subagents.enabled flag in config.yaml to control subagent feature:
- When disabled, task/task_status tools are not loaded
- When disabled, system prompt excludes subagent documentation
- Default is enabled for backward compatibility

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-05 20:49:02 +08:00
hetao b7ba237c36 feat: add configuration to enable/disable subagents
Add subagents.enabled flag in config.yaml to control subagent feature:
- When disabled, task/task_status tools are not loaded
- When disabled, system prompt excludes subagent documentation
- Default is enabled for backward compatibility

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-05 20:49:02 +08:00
hetao ef379a3100 feat: support sub agent mechanism 2026-02-05 19:59:25 +08:00
hetao cbd2fe66de feat: support sub agent mechanism 2026-02-05 19:59:25 +08:00
hetao 6e3f43c943 feat: support sub agent mechanism 2026-02-05 19:59:25 +08:00
Henry Li 43ebce3b37 feat: remove demo 2026-02-05 09:46:05 +08:00
Henry Li c31175defd feat: remove demo 2026-02-05 09:46:05 +08:00
Henry Li 118fc00368 feat: remove demo 2026-02-05 09:46:05 +08:00
hetao db0461142e feat: enhance memory system with tiktoken and improved prompt guidelines
Add accurate token counting using tiktoken library and significantly enhance
memory update prompts with detailed section guidelines, multilingual support,
and improved fact extraction. Update deep-research skill to be more proactive
for research queries.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-04 20:44:26 +08:00
hetao 0d245d6e31 feat: enhance memory system with tiktoken and improved prompt guidelines
Add accurate token counting using tiktoken library and significantly enhance
memory update prompts with detailed section guidelines, multilingual support,
and improved fact extraction. Update deep-research skill to be more proactive
for research queries.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-04 20:44:26 +08:00
hetao df1191c90a feat: enhance memory system with tiktoken and improved prompt guidelines
Add accurate token counting using tiktoken library and significantly enhance
memory update prompts with detailed section guidelines, multilingual support,
and improved fact extraction. Update deep-research skill to be more proactive
for research queries.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-04 20:44:26 +08:00
LofiSu 2debcf421c fix(citations): improve citation link rendering and copy behavior
- Use citation.title for display text in CitationLink to ensure correct
  titles show during streaming (instead of generic "Source" text)
- Render all external links as CitationLink badges for consistent styling
  during streaming output
- Add removeAllCitations when copying message content to clipboard
- Simplify citations_format prompt for cleaner AI output

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-04 16:34:12 +08:00
ruitanglin f6e625ec3b fix(citations): improve citation link rendering and copy behavior
- Use citation.title for display text in CitationLink to ensure correct
  titles show during streaming (instead of generic "Source" text)
- Render all external links as CitationLink badges for consistent styling
  during streaming output
- Add removeAllCitations when copying message content to clipboard
- Simplify citations_format prompt for cleaner AI output

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-04 16:34:12 +08:00
ruitanglin 0f9e3d508b fix(citations): improve citation link rendering and copy behavior
- Use citation.title for display text in CitationLink to ensure correct
  titles show during streaming (instead of generic "Source" text)
- Render all external links as CitationLink badges for consistent styling
  during streaming output
- Add removeAllCitations when copying message content to clipboard
- Simplify citations_format prompt for cleaner AI output

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-04 16:34:12 +08:00
LofiSu 644229f968 feat(citations): add shared citation components and optimize code
## New Features
- Add `CitationLink` shared component for rendering citation hover cards
- Add `CitationsLoadingIndicator` component for showing loading state
- Add `removeAllCitations` utility to strip all citations from content
- Add backend support for removing citations when downloading markdown files
- Add i18n support for citation loading messages (en-US, zh-CN)

## Code Optimizations
- Remove duplicate `ExternalLinkBadge` component, reuse `CitationLink` instead
- Consolidate `remarkPlugins` config in `streamdownPlugins` to avoid duplication
- Remove unused imports: `Citation`, `buildCitationMap`, `extractDomainFromUrl`, etc.
- Remove unused `messages` parameter from `ToolCall` component
- Remove unused `isWriteFile` parameter from `ArtifactFilePreview` component
- Remove unused `useI18n` hook from `MessageContent` component

## Bug Fixes
- Fix `remarkGfm` plugin configuration that prevented table rendering
- Fix React Hooks rule violation: move `useMemo` to component top level
- Replace `||` with `??` for nullish coalescing in clipboard data

## Code Cleanup
- Remove debug console.log/info statements from:
  - `threads/hooks.ts`
  - `notification/hooks.ts`
  - `memory-settings-page.tsx`
- Fix import order in `message-group.tsx`

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-04 11:56:10 +08:00
ruitanglin c67f1af889 feat(citations): add shared citation components and optimize code
## New Features
- Add `CitationLink` shared component for rendering citation hover cards
- Add `CitationsLoadingIndicator` component for showing loading state
- Add `removeAllCitations` utility to strip all citations from content
- Add backend support for removing citations when downloading markdown files
- Add i18n support for citation loading messages (en-US, zh-CN)

## Code Optimizations
- Remove duplicate `ExternalLinkBadge` component, reuse `CitationLink` instead
- Consolidate `remarkPlugins` config in `streamdownPlugins` to avoid duplication
- Remove unused imports: `Citation`, `buildCitationMap`, `extractDomainFromUrl`, etc.
- Remove unused `messages` parameter from `ToolCall` component
- Remove unused `isWriteFile` parameter from `ArtifactFilePreview` component
- Remove unused `useI18n` hook from `MessageContent` component

## Bug Fixes
- Fix `remarkGfm` plugin configuration that prevented table rendering
- Fix React Hooks rule violation: move `useMemo` to component top level
- Replace `||` with `??` for nullish coalescing in clipboard data

## Code Cleanup
- Remove debug console.log/info statements from:
  - `threads/hooks.ts`
  - `notification/hooks.ts`
  - `memory-settings-page.tsx`
- Fix import order in `message-group.tsx`

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-04 11:56:10 +08:00
ruitanglin 1e2675beb3 feat(citations): add shared citation components and optimize code
## New Features
- Add `CitationLink` shared component for rendering citation hover cards
- Add `CitationsLoadingIndicator` component for showing loading state
- Add `removeAllCitations` utility to strip all citations from content
- Add backend support for removing citations when downloading markdown files
- Add i18n support for citation loading messages (en-US, zh-CN)

## Code Optimizations
- Remove duplicate `ExternalLinkBadge` component, reuse `CitationLink` instead
- Consolidate `remarkPlugins` config in `streamdownPlugins` to avoid duplication
- Remove unused imports: `Citation`, `buildCitationMap`, `extractDomainFromUrl`, etc.
- Remove unused `messages` parameter from `ToolCall` component
- Remove unused `isWriteFile` parameter from `ArtifactFilePreview` component
- Remove unused `useI18n` hook from `MessageContent` component

## Bug Fixes
- Fix `remarkGfm` plugin configuration that prevented table rendering
- Fix React Hooks rule violation: move `useMemo` to component top level
- Replace `||` with `??` for nullish coalescing in clipboard data

## Code Cleanup
- Remove debug console.log/info statements from:
  - `threads/hooks.ts`
  - `notification/hooks.ts`
  - `memory-settings-page.tsx`
- Fix import order in `message-group.tsx`

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-04 11:56:10 +08:00
hetao 5959ef87b8 feat: add Apple Container support with comprehensive documentation and dev tools
Add native Apple Container support for better performance on macOS while
maintaining full Docker compatibility. Enhance documentation with memory system
details, development guidelines, and sandbox setup instructions. Improve dev
experience with container image pre-pulling and unified cleanup tools.

Key changes:
- Auto-detect and prefer Apple Container on macOS with Docker fallback
- Add APPLE_CONTAINER.md with complete usage and troubleshooting guide
- Document memory system architecture in CLAUDE.md
- Add make setup-sandbox for pre-pulling container images
- Create cleanup-containers.sh for cross-runtime container cleanup
- Update all related documentation (README, SETUP, config examples)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-03 20:41:36 +08:00
hetao 70a27b49c0 feat: add Apple Container support with comprehensive documentation and dev tools
Add native Apple Container support for better performance on macOS while
maintaining full Docker compatibility. Enhance documentation with memory system
details, development guidelines, and sandbox setup instructions. Improve dev
experience with container image pre-pulling and unified cleanup tools.

Key changes:
- Auto-detect and prefer Apple Container on macOS with Docker fallback
- Add APPLE_CONTAINER.md with complete usage and troubleshooting guide
- Document memory system architecture in CLAUDE.md
- Add make setup-sandbox for pre-pulling container images
- Create cleanup-containers.sh for cross-runtime container cleanup
- Update all related documentation (README, SETUP, config examples)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-03 20:41:36 +08:00
hetao ef10f3ba41 feat: add Apple Container support with comprehensive documentation and dev tools
Add native Apple Container support for better performance on macOS while
maintaining full Docker compatibility. Enhance documentation with memory system
details, development guidelines, and sandbox setup instructions. Improve dev
experience with container image pre-pulling and unified cleanup tools.

Key changes:
- Auto-detect and prefer Apple Container on macOS with Docker fallback
- Add APPLE_CONTAINER.md with complete usage and troubleshooting guide
- Document memory system architecture in CLAUDE.md
- Add make setup-sandbox for pre-pulling container images
- Create cleanup-containers.sh for cross-runtime container cleanup
- Update all related documentation (README, SETUP, config examples)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-03 20:41:36 +08:00
hetao b773bae407 fix: fix frontend rendering issue 2026-02-03 19:38:10 +08:00
hetao d670cc0ab1 fix: fix frontend rendering issue 2026-02-03 19:38:10 +08:00
hetao 8f8637c3c4 fix: fix frontend rendering issue 2026-02-03 19:38:10 +08:00
LofiSu 3b411fe499 fix: 修复用户消息中上传文件的右对齐显示
在 UploadedFilesList 组件中添加 justify-end 类,确保上传的文件卡片在用户消息中保持右对齐显示,与消息气泡对齐一致。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-03 19:13:17 +08:00
ruitanglin 1fac83eafa fix: 修复用户消息中上传文件的右对齐显示
在 UploadedFilesList 组件中添加 justify-end 类,确保上传的文件卡片在用户消息中保持右对齐显示,与消息气泡对齐一致。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-03 19:13:17 +08:00
ruitanglin 901772136e fix: 修复用户消息中上传文件的右对齐显示
在 UploadedFilesList 组件中添加 justify-end 类,确保上传的文件卡片在用户消息中保持右对齐显示,与消息气泡对齐一致。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-03 19:13:17 +08:00
Henry Li 6b53456b39 feat: add memory settings page 2026-02-03 18:18:56 +08:00
Henry Li 94acb15c0c feat: add memory settings page 2026-02-03 18:18:56 +08:00
Henry Li 552d1c3a9a feat: add memory settings page 2026-02-03 18:18:56 +08:00
Henry Li 4d650f35f8 chore: add /api/memory 2026-02-03 15:21:15 +08:00
Henry Li b8c325eb3a chore: add /api/memory 2026-02-03 15:21:15 +08:00
Henry Li 1cf081120e chore: add /api/memory 2026-02-03 15:21:15 +08:00
hetaoBackend 2c32e8a461 fix: add file mtime-based cache invalidation for memory data
Implement automatic cache invalidation based on file modification time to ensure memory data consistency across Gateway API and agent prompts. The cache now automatically reloads when the memory file is updated externally.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-03 13:50:01 +08:00
hetaoBackend 9e15e609ec fix: add file mtime-based cache invalidation for memory data
Implement automatic cache invalidation based on file modification time to ensure memory data consistency across Gateway API and agent prompts. The cache now automatically reloads when the memory file is updated externally.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-03 13:50:01 +08:00
hetaoBackend 5682f7b67d fix: add file mtime-based cache invalidation for memory data
Implement automatic cache invalidation based on file modification time to ensure memory data consistency across Gateway API and agent prompts. The cache now automatically reloads when the memory file is updated externally.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-03 13:50:01 +08:00
hetaoBackend 3b30913e10 feat: add memory API and optimize memory middleware
- Add memory API endpoints for retrieving memory data:
  - GET /api/memory - get current memory data
  - POST /api/memory/reload - reload from file
  - GET /api/memory/config - get memory configuration
  - GET /api/memory/status - get config and data together
- Optimize MemoryMiddleware to only use user inputs and final
  assistant responses, filtering out intermediate tool calls
- Add memory configuration example to config.example.yaml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 13:41:04 +08:00
hetaoBackend 7b7a7abaf2 feat: add memory API and optimize memory middleware
- Add memory API endpoints for retrieving memory data:
  - GET /api/memory - get current memory data
  - POST /api/memory/reload - reload from file
  - GET /api/memory/config - get memory configuration
  - GET /api/memory/status - get config and data together
- Optimize MemoryMiddleware to only use user inputs and final
  assistant responses, filtering out intermediate tool calls
- Add memory configuration example to config.example.yaml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 13:41:04 +08:00
hetaoBackend 74d47ad87f feat: add memory API and optimize memory middleware
- Add memory API endpoints for retrieving memory data:
  - GET /api/memory - get current memory data
  - POST /api/memory/reload - reload from file
  - GET /api/memory/config - get memory configuration
  - GET /api/memory/status - get config and data together
- Optimize MemoryMiddleware to only use user inputs and final
  assistant responses, filtering out intermediate tool calls
- Add memory configuration example to config.example.yaml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 13:41:04 +08:00
hetaoBackend 0ea666e0cf feat: add global memory mechanism for personalized conversations
Implement a memory system that stores user context and conversation history
in memory.json, uses LLM to summarize conversations, and injects relevant
context into system prompts for personalized responses.

Key components:
- MemoryConfig for configuration management
- MemoryUpdateQueue with debounce for batch processing
- MemoryUpdater for LLM-based memory extraction
- MemoryMiddleware to queue conversations after agent execution
- Memory injection into lead agent system prompt

Note: Add memory section to config.yaml to enable (see config.example.yaml)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 13:31:05 +08:00
hetaoBackend 18d85ab6e5 feat: add global memory mechanism for personalized conversations
Implement a memory system that stores user context and conversation history
in memory.json, uses LLM to summarize conversations, and injects relevant
context into system prompts for personalized responses.

Key components:
- MemoryConfig for configuration management
- MemoryUpdateQueue with debounce for batch processing
- MemoryUpdater for LLM-based memory extraction
- MemoryMiddleware to queue conversations after agent execution
- Memory injection into lead agent system prompt

Note: Add memory section to config.yaml to enable (see config.example.yaml)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 13:31:05 +08:00
hetaoBackend ffd07bbafe feat: add global memory mechanism for personalized conversations
Implement a memory system that stores user context and conversation history
in memory.json, uses LLM to summarize conversations, and injects relevant
context into system prompts for personalized responses.

Key components:
- MemoryConfig for configuration management
- MemoryUpdateQueue with debounce for batch processing
- MemoryUpdater for LLM-based memory extraction
- MemoryMiddleware to queue conversations after agent execution
- Memory injection into lead agent system prompt

Note: Add memory section to config.yaml to enable (see config.example.yaml)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 13:31:05 +08:00
Henry Li 86255511e1 docs: add README.md 2026-02-02 23:24:58 +08:00
Henry Li 0baa8a733a docs: add README.md 2026-02-02 23:24:58 +08:00
Henry Li 4fd9a2de8e docs: add README.md 2026-02-02 23:24:58 +08:00
huangkevin-apr fab1d39323 Fix a11y: add accessible name for icon button (#844)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-02-02 21:56:34 +08:00
Willem Jiang e3e7a83f40 fix(node):deal with the plan_data content with multipmodal message (#846)
* fix(node):deal with the plan_data content with multipmodal message

* Update the code with review comments
2026-02-02 20:31:58 +08:00
Henry Li e84715831f fix: fix position 2026-02-02 16:40:43 +08:00
Henry Li 03f84f2b76 fix: fix position 2026-02-02 16:40:43 +08:00
Henry Li 268b7f911c fix: fix position 2026-02-02 16:40:43 +08:00
Henry Li 018241c203 fix: set default state for todo list collapse to true 2026-02-02 14:46:26 +08:00
Henry Li 35c5b6ba6b fix: set default state for todo list collapse to true 2026-02-02 14:46:26 +08:00
Henry Li 8bc9d1b226 fix: set default state for todo list collapse to true 2026-02-02 14:46:26 +08:00
Henry Li 6f6d799051 fix: set default state for todo list collapse to false 2026-02-02 14:46:10 +08:00
Henry Li a745b824d5 fix: set default state for todo list collapse to false 2026-02-02 14:46:10 +08:00
Henry Li e01127eec9 fix: set default state for todo list collapse to false 2026-02-02 14:46:10 +08:00
Henry Li f4f16bfa5c feat: enhance welcome component and input box with skill mode handling and localization updates 2026-02-02 14:44:23 +08:00
Henry Li 010eadecca feat: enhance welcome component and input box with skill mode handling and localization updates 2026-02-02 14:44:23 +08:00
Henry Li 26acd6f3ad feat: enhance welcome component and input box with skill mode handling and localization updates 2026-02-02 14:44:23 +08:00
Henry Li ccf21238af feat: update button in skill settings to include icon 2026-02-02 14:31:44 +08:00
Henry Li 67451df910 feat: update button in skill settings to include icon 2026-02-02 14:31:44 +08:00
Henry Li 9cc41139cb feat: update button in skill settings to include icon 2026-02-02 14:31:44 +08:00
Henry Li efd56fdf51 feat: use list of links 2026-02-02 13:25:21 +08:00
Henry Li a5a0222963 feat: use list of links 2026-02-02 13:25:21 +08:00
Henry Li 207cb2b98d feat: use list of links 2026-02-02 13:25:21 +08:00
Henry Li b7c9bf557b feat: update button styling for artifacts tooltip 2026-02-02 11:43:02 +08:00
Henry Li 44daeaf37d feat: update button styling for artifacts tooltip 2026-02-02 11:43:02 +08:00
Henry Li b5e9eeea99 feat: update button styling for artifacts tooltip 2026-02-02 11:43:02 +08:00
Henry Li 3067f8dd03 feat: add suggestions 2026-02-02 11:21:30 +08:00
Henry Li 154fbb0ba3 feat: add suggestions 2026-02-02 11:21:30 +08:00
Henry Li e673405c00 feat: add suggestions 2026-02-02 11:21:30 +08:00
Henry Li 6c0e5fffd0 feat: integrate PromptInputProvider into ChatLayout and utilize prompt input controller in ChatPage 2026-02-02 10:18:02 +08:00
Henry Li f287022ac0 feat: integrate PromptInputProvider into ChatLayout and utilize prompt input controller in ChatPage 2026-02-02 10:18:02 +08:00
Henry Li b1227bb911 feat: integrate PromptInputProvider into ChatLayout and utilize prompt input controller in ChatPage 2026-02-02 10:18:02 +08:00
Henry Li 867749d7a3 feat: add file icons 2026-02-02 10:02:31 +08:00
Henry Li c587460dbc feat: add file icons 2026-02-02 10:02:31 +08:00
Henry Li f1db301d77 feat: add file icons 2026-02-02 10:02:31 +08:00
Henry Li 37dcee41c0 feat: add file icon 2026-02-02 09:49:44 +08:00
Henry Li 8bb4c35416 feat: add file icon 2026-02-02 09:49:44 +08:00
Henry Li 02400e0e8c feat: add file icon 2026-02-02 09:49:44 +08:00
Henry Li 51b4ed3124 feat: adjust tooltips 2026-02-02 09:32:18 +08:00
Henry Li 7274f9a6ae feat: adjust tooltips 2026-02-02 09:32:18 +08:00
Henry Li 0091da1aee feat: adjust tooltips 2026-02-02 09:32:18 +08:00
Henry Li 6d31c1c5cf feat: wrap path and command in Tooltip for enhanced user experience 2026-02-02 09:23:36 +08:00
Henry Li cb494fe4df feat: wrap path and command in Tooltip for enhanced user experience 2026-02-02 09:23:36 +08:00
Henry Li 076c1f0985 feat: wrap path and command in Tooltip for enhanced user experience 2026-02-02 09:23:36 +08:00
Henry Li a66f76f43d fix: update TooltipContent component to handle sideOffset correctly and add shadow styling 2026-02-02 09:23:24 +08:00
Henry Li ccab24983e fix: update TooltipContent component to handle sideOffset correctly and add shadow styling 2026-02-02 09:23:24 +08:00
Henry Li 33e82a7abe fix: update TooltipContent component to handle sideOffset correctly and add shadow styling 2026-02-02 09:23:24 +08:00
Henry Li 90104291ae docs: add comments 2026-02-02 09:11:05 +08:00
Henry Li 68df848b82 docs: add comments 2026-02-02 09:11:05 +08:00
Henry Li ac16a73a47 docs: add comments 2026-02-02 09:11:05 +08:00
Henry Li 54277b9d9e feat: add skeleton 2026-02-02 09:05:33 +08:00
Henry Li b797ef8168 feat: add skeleton 2026-02-02 09:05:33 +08:00
Henry Li 7da0a03dd0 feat: add skeleton 2026-02-02 09:05:33 +08:00
Henry Li a0a3a3fc02 feat: dynamic title 2026-02-02 09:05:24 +08:00
Henry Li be65130a06 feat: dynamic title 2026-02-02 09:05:24 +08:00
Henry Li 1eb4da6c75 feat: dynamic title 2026-02-02 09:05:24 +08:00
Henry Li b540ad4505 feat: use create skill as title 2026-02-02 08:43:37 +08:00
Henry Li dc1190b228 feat: use create skill as title 2026-02-02 08:43:37 +08:00
Henry Li b50fbf83d0 feat: use create skill as title 2026-02-02 08:43:37 +08:00
hetaoBackend f082ef3d87 feat: add find-skills skill for discovering agent skills
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 23:54:08 +08:00
hetaoBackend e4939216fd feat: add find-skills skill for discovering agent skills
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 23:54:08 +08:00
hetaoBackend 7fd5ba258d feat: add find-skills skill for discovering agent skills
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 23:54:08 +08:00
hetaoBackend 9043c964ca docs: add comprehensive backend documentation
- Add README.md with project overview, quick start, and API reference
- Add CONTRIBUTING.md with development setup and contribution guidelines
- Add docs/ARCHITECTURE.md with detailed system architecture diagrams
- Add docs/API.md with complete API reference for LangGraph and Gateway
- Add docs/README.md as documentation index
- Update CLAUDE.md with improved structure and new features
- Update docs/TODO.md to reflect current status
- Update pyproject.toml description

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 22:18:25 +08:00
hetaoBackend 68c3e3341a docs: add comprehensive backend documentation
- Add README.md with project overview, quick start, and API reference
- Add CONTRIBUTING.md with development setup and contribution guidelines
- Add docs/ARCHITECTURE.md with detailed system architecture diagrams
- Add docs/API.md with complete API reference for LangGraph and Gateway
- Add docs/README.md as documentation index
- Update CLAUDE.md with improved structure and new features
- Update docs/TODO.md to reflect current status
- Update pyproject.toml description

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 22:18:25 +08:00
hetaoBackend 4f4b7cde2e docs: add comprehensive backend documentation
- Add README.md with project overview, quick start, and API reference
- Add CONTRIBUTING.md with development setup and contribution guidelines
- Add docs/ARCHITECTURE.md with detailed system architecture diagrams
- Add docs/API.md with complete API reference for LangGraph and Gateway
- Add docs/README.md as documentation index
- Update CLAUDE.md with improved structure and new features
- Update docs/TODO.md to reflect current status
- Update pyproject.toml description

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 22:18:25 +08:00
Henry Li 9b77070406 feat: update skills 2026-02-01 11:12:08 +08:00
Henry Li 7e11f28d55 feat: update skills 2026-02-01 11:12:08 +08:00
Henry Li 890a8379ce feat: update skills 2026-02-01 11:12:08 +08:00
Henry Li ec444e1f8b docs: update artifacts 2026-02-01 11:05:00 +08:00
Henry Li 37e9810191 docs: update artifacts 2026-02-01 11:05:00 +08:00
Henry Li e28d5d2cf9 docs: update artifacts 2026-02-01 11:05:00 +08:00
Henry Li 22ef5fb5ba feat: add new demo 2026-02-01 10:58:27 +08:00
Henry Li d131a497d7 feat: add new demo 2026-02-01 10:58:27 +08:00
Henry Li 88e1c7c0b3 feat: add new demo 2026-02-01 10:58:27 +08:00
Henry Li f206a574c5 feat: update github-deep-research skill 2026-02-01 10:55:21 +08:00
Henry Li 8c37c9c755 feat: update github-deep-research skill 2026-02-01 10:55:21 +08:00
Henry Li f656fd0768 feat: update github-deep-research skill 2026-02-01 10:55:21 +08:00
Henry Li e1ecf62afa feat: add tooltip for installation 2026-02-01 10:55:08 +08:00
Henry Li 4721f1a890 feat: add tooltip for installation 2026-02-01 10:55:08 +08:00
Henry Li a1267875fa feat: add tooltip for installation 2026-02-01 10:55:08 +08:00
Henry Li 46feff6c16 feat: add github-deep-research skill 2026-02-01 10:54:19 +08:00
Henry Li 16122dd92d feat: add github-deep-research skill 2026-02-01 10:54:19 +08:00
Henry Li 469e044935 feat: add github-deep-research skill 2026-02-01 10:54:19 +08:00
Henry Li f5b1412ac0 fix: add translations 2026-01-31 22:49:59 +08:00
Henry Li 8a2fb353c6 fix: add translations 2026-01-31 22:49:59 +08:00
Henry Li 45fab66a7d fix: add translations 2026-01-31 22:49:59 +08:00
Henry Li ca83ed00f8 docs: rephrasing 2026-01-31 22:42:17 +08:00
Henry Li f3d7fea9ce docs: rephrasing 2026-01-31 22:42:17 +08:00
Henry Li 7d3e7eb1c9 docs: rephrasing 2026-01-31 22:42:17 +08:00
Henry Li bdd2e25e14 feat: implement create skill 2026-01-31 22:31:25 +08:00
Henry Li 8639dde3ad feat: implement create skill 2026-01-31 22:31:25 +08:00
Henry Li 67ec1162cb feat: implement create skill 2026-01-31 22:31:25 +08:00
hetaoBackend 06511f38e1 feat: add .skill file preview support
Enable previewing .skill files (ZIP archives) by extracting and displaying
their SKILL.md content. Add caching to avoid repeated ZIP extraction.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 22:27:06 +08:00
hetaoBackend f31258dd10 feat: add .skill file preview support
Enable previewing .skill files (ZIP archives) by extracting and displaying
their SKILL.md content. Add caching to avoid repeated ZIP extraction.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 22:27:06 +08:00
hetaoBackend 41f8b931c9 feat: add .skill file preview support
Enable previewing .skill files (ZIP archives) by extracting and displaying
their SKILL.md content. Add caching to avoid repeated ZIP extraction.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 22:27:06 +08:00
hetaoBackend a9e11f6341 feat: add skill installation API endpoint
Add POST /api/skills/install endpoint to install .skill files from
thread's user-data directory. The endpoint extracts the ZIP archive,
validates SKILL.md frontmatter, and installs to skills/custom/.

Frontend Install buttons now call the API instead of downloading.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 22:10:05 +08:00
hetaoBackend 624f758163 feat: add skill installation API endpoint
Add POST /api/skills/install endpoint to install .skill files from
thread's user-data directory. The endpoint extracts the ZIP archive,
validates SKILL.md frontmatter, and installs to skills/custom/.

Frontend Install buttons now call the API instead of downloading.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 22:10:05 +08:00
hetaoBackend 5834b15af7 feat: add skill installation API endpoint
Add POST /api/skills/install endpoint to install .skill files from
thread's user-data directory. The endpoint extracts the ZIP archive,
validates SKILL.md frontmatter, and installs to skills/custom/.

Frontend Install buttons now call the API instead of downloading.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 22:10:05 +08:00
hetaoBackend 1899333b95 refactor: update skills XML format in prompt template
Change skills rendering from attribute-based format to nested element format
with <available_skills>, <skill>, <name>, <description>, and <location> tags
for better readability and structure.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 21:54:29 +08:00
hetaoBackend 3c3bf901e7 refactor: update skills XML format in prompt template
Change skills rendering from attribute-based format to nested element format
with <available_skills>, <skill>, <name>, <description>, and <location> tags
for better readability and structure.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 21:54:29 +08:00
hetaoBackend c76481d8f7 refactor: update skills XML format in prompt template
Change skills rendering from attribute-based format to nested element format
with <available_skills>, <skill>, <name>, <description>, and <location> tags
for better readability and structure.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 21:54:29 +08:00
Henry Li d3ff5f9d3c fix: fix eslint errors and warnings 2026-01-31 21:46:31 +08:00
Henry Li 718bb947d0 fix: fix eslint errors and warnings 2026-01-31 21:46:31 +08:00
Henry Li 8ecb6b3d1d fix: fix eslint errors and warnings 2026-01-31 21:46:31 +08:00
Henry Li cf961328a9 feat: preview the message if possible 2026-01-31 20:22:15 +08:00
Henry Li 20a023ee90 feat: preview the message if possible 2026-01-31 20:22:15 +08:00
Henry Li 9c3b928f1d feat: preview the message if possible 2026-01-31 20:22:15 +08:00
Henry Li e858ef0250 fix: fix eslint errors 2026-01-31 11:11:13 +08:00
Henry Li b8281be892 fix: fix eslint errors 2026-01-31 11:11:13 +08:00
Henry Li 2ba687b239 fix: fix eslint errors 2026-01-31 11:11:13 +08:00
Henry Li 5295f5b5b9 feat: add notification 2026-01-31 11:08:27 +08:00
Henry Li 47fe2f8195 feat: add notification 2026-01-31 11:08:27 +08:00
Henry Li c62caf95c4 feat: add notification 2026-01-31 11:08:27 +08:00
Henry Li 835fd4d0c7 feat: change email 2026-01-30 22:32:47 +08:00
Henry Li cb660c2643 feat: change email 2026-01-30 22:32:47 +08:00
Henry Li 4e0571f3b3 feat: change email 2026-01-30 22:32:47 +08:00
hetaoBackend 43ee8a2968 fix: fix aio sandbox shutdown bug 2026-01-30 22:02:07 +08:00
hetaoBackend 733c020c58 fix: fix aio sandbox shutdown bug 2026-01-30 22:02:07 +08:00
hetaoBackend 8182ed3737 fix: fix aio sandbox shutdown bug 2026-01-30 22:02:07 +08:00
Henry Li c07c0228f6 fix: fix condition of displaying artifacts 2026-01-30 21:51:18 +08:00
Henry Li 697f094ba9 fix: fix condition of displaying artifacts 2026-01-30 21:51:18 +08:00
Henry Li 21e12d91eb fix: fix condition of displaying artifacts 2026-01-30 21:51:18 +08:00
Henry Li c1182c680c feat: support Github Flavored Markdown 2026-01-30 16:41:18 +08:00
Henry Li 618b3e1e8f feat: support Github Flavored Markdown 2026-01-30 16:41:18 +08:00
Henry Li 1bb91bb267 feat: support Github Flavored Markdown 2026-01-30 16:41:18 +08:00
Henry Li 7d024326dc chore: remove 2026-01-30 11:01:09 +08:00
Henry Li 3339e70c25 chore: remove 2026-01-30 11:01:09 +08:00
Henry Li 05794e29d1 chore: remove 2026-01-30 11:01:09 +08:00
Xun 3adb4e90cb fix: improve JSON repair handling for markdown code blocks (#841)
* fix: improve JSON repair handling for markdown code blocks

* unified import path

* compress_crawl_udf

* fix

* reverse
2026-01-30 08:47:23 +08:00
Henry Li 4dffad89ca feat: re-arrange icons 2026-01-29 16:17:41 +08:00
Henry Li cbcbbbe0a8 feat: re-arrange icons 2026-01-29 16:17:41 +08:00
Henry Li 939745d027 feat: re-arrange icons 2026-01-29 16:17:41 +08:00
Henry Li a4f749f939 fix: add max width 2026-01-29 16:12:30 +08:00
Henry Li c265f5410d fix: add max width 2026-01-29 16:12:30 +08:00
Henry Li 66deedf3b2 fix: add max width 2026-01-29 16:12:30 +08:00
Henry Li a135ddfa48 feat: display mode 2026-01-29 15:57:08 +08:00
Henry Li 86ed750a38 feat: display mode 2026-01-29 15:57:08 +08:00
Henry Li 79955d2e6c feat: display mode 2026-01-29 15:57:08 +08:00
Henry Li 62ac3b6b03 feat: use "mode" instead of "thinking_enabled" and "is_plan_mode" 2026-01-29 15:48:50 +08:00
Henry Li 7bf15cb777 feat: use "mode" instead of "thinking_enabled" and "is_plan_mode" 2026-01-29 15:48:50 +08:00
Henry Li 98e08a85c9 feat: use "mode" instead of "thinking_enabled" and "is_plan_mode" 2026-01-29 15:48:50 +08:00
Henry Li 4411af68f5 fix: fix renaming 2026-01-29 15:31:56 +08:00
Henry Li caf469d2ab fix: fix renaming 2026-01-29 15:31:56 +08:00
Henry Li 0ba82a9fd7 fix: fix renaming 2026-01-29 15:31:56 +08:00
Henry Li 9d889434c4 feat: add placeholder for image 2026-01-29 15:01:18 +08:00
Henry Li 4fc54a7408 feat: add placeholder for image 2026-01-29 15:01:18 +08:00
Henry Li 16a9626d54 feat: add placeholder for image 2026-01-29 15:01:18 +08:00
hetao 2c7a56dd33 feat: optimize vision tools and image handling
- Add model-aware vision tool loading based on supports_vision flag
- Move view_image_tool from config to builtin tools for dynamic inclusion
- Add timeout to image search to prevent hanging requests
- Optimize image search results format using thumbnails
- Add image validation for reference images in generation
- Improve error handling with detailed messages

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-29 14:57:26 +08:00
hetao 314ea41781 feat: optimize vision tools and image handling
- Add model-aware vision tool loading based on supports_vision flag
- Move view_image_tool from config to builtin tools for dynamic inclusion
- Add timeout to image search to prevent hanging requests
- Optimize image search results format using thumbnails
- Add image validation for reference images in generation
- Improve error handling with detailed messages

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-29 14:57:26 +08:00
hetao 7aa10b980f feat: optimize vision tools and image handling
- Add model-aware vision tool loading based on supports_vision flag
- Move view_image_tool from config to builtin tools for dynamic inclusion
- Add timeout to image search to prevent hanging requests
- Optimize image search results format using thumbnails
- Add image validation for reference images in generation
- Improve error handling with detailed messages

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-29 14:57:26 +08:00
hetao 75801d9817 fix: fix frontend bug 2026-01-29 13:55:30 +08:00
hetao 2c6dbbe065 fix: fix frontend bug 2026-01-29 13:55:30 +08:00
hetao 3cbf54b2eb fix: fix frontend bug 2026-01-29 13:55:30 +08:00
hetao 09d9c18a28 feat: add view_image tool and optimize web fetch tools
Add image viewing capability for vision-enabled models with ViewImageMiddleware and view_image_tool. Limit web_fetch tool output to 4096 characters to prevent excessive content. Update model config to support vision capability flag.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-29 13:44:39 +08:00
hetao 7414947cc6 feat: add view_image tool and optimize web fetch tools
Add image viewing capability for vision-enabled models with ViewImageMiddleware and view_image_tool. Limit web_fetch tool output to 4096 characters to prevent excessive content. Update model config to support vision capability flag.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-29 13:44:39 +08:00
hetao 9dc2405555 feat: add view_image tool and optimize web fetch tools
Add image viewing capability for vision-enabled models with ViewImageMiddleware and view_image_tool. Limit web_fetch tool output to 4096 characters to prevent excessive content. Update model config to support vision capability flag.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-29 13:44:39 +08:00
Henry Li 73a5a7972e Merge pull request #22 from LofiSu/experimental
feat: add inline citations and thread management features
2026-01-29 13:06:00 +08:00
Henry Li d77e7c32b2 Merge pull request #22 from LofiSu/experimental
feat: add inline citations and thread management features
2026-01-29 13:06:00 +08:00
Henry Li 294a1614b8 Merge pull request #22 from LofiSu/experimental
feat: add inline citations and thread management features
2026-01-29 13:06:00 +08:00
LofiSu 588673d043 merge: upstream/experimental with citations feature
- Merge upstream changes including image search, tooltips, and UI improvements
- Keep citations feature with inline hover cards
- Resolve conflict in message-list-item.tsx: use upstream img max-width (90%) while preserving citations logic
- Maintain file upload improvements with citations support

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29 12:55:43 +08:00
ruitanglin ac283b92aa merge: upstream/experimental with citations feature
- Merge upstream changes including image search, tooltips, and UI improvements
- Keep citations feature with inline hover cards
- Resolve conflict in message-list-item.tsx: use upstream img max-width (90%) while preserving citations logic
- Maintain file upload improvements with citations support

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29 12:55:43 +08:00
ruitanglin 5120022d6d merge: upstream/experimental with citations feature
- Merge upstream changes including image search, tooltips, and UI improvements
- Keep citations feature with inline hover cards
- Resolve conflict in message-list-item.tsx: use upstream img max-width (90%) while preserving citations logic
- Maintain file upload improvements with citations support

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29 12:55:43 +08:00
LofiSu 849cc4d771 feat: improve file upload message handling and UI
Backend:
- Handle both string and list format for message content in uploads middleware
- Extract text content from structured message blocks
- Add logging for debugging file upload flow

Frontend:
- Separate file display from message bubble for human messages
- Show uploaded files outside the message bubble for cleaner layout
- Improve file card border styling with subtle border color
- Add debug logging for message submission with files

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29 12:51:21 +08:00
ruitanglin ce9731c10a feat: improve file upload message handling and UI
Backend:
- Handle both string and list format for message content in uploads middleware
- Extract text content from structured message blocks
- Add logging for debugging file upload flow

Frontend:
- Separate file display from message bubble for human messages
- Show uploaded files outside the message bubble for cleaner layout
- Improve file card border styling with subtle border color
- Add debug logging for message submission with files

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29 12:51:21 +08:00
ruitanglin 341397562a feat: improve file upload message handling and UI
Backend:
- Handle both string and list format for message content in uploads middleware
- Extract text content from structured message blocks
- Add logging for debugging file upload flow

Frontend:
- Separate file display from message bubble for human messages
- Show uploaded files outside the message bubble for cleaner layout
- Improve file card border styling with subtle border color
- Add debug logging for message submission with files

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29 12:51:21 +08:00
Henry Li eff241f9f2 feat: enable images in content 2026-01-29 12:51:04 +08:00
Henry Li f809b67c47 feat: enable images in content 2026-01-29 12:51:04 +08:00
LofiSu e2e0fbf114 fix: hide incomplete citations block during streaming
Improve UX by hiding citations block while it's being streamed:
- Remove complete citations blocks (existing logic)
- Also remove incomplete citations blocks during streaming
- Prevents flickering of raw citations XML in the UI

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29 12:50:09 +08:00
ruitanglin 6ae4868780 fix: hide incomplete citations block during streaming
Improve UX by hiding citations block while it's being streamed:
- Remove complete citations blocks (existing logic)
- Also remove incomplete citations blocks during streaming
- Prevents flickering of raw citations XML in the UI

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29 12:50:09 +08:00
ruitanglin 2ec506d590 fix: hide incomplete citations block during streaming
Improve UX by hiding citations block while it's being streamed:
- Remove complete citations blocks (existing logic)
- Also remove incomplete citations blocks during streaming
- Prevents flickering of raw citations XML in the UI

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29 12:50:09 +08:00
LofiSu c14378a312 feat: refine citations format and improve content presentation
Backend:
- Simplify citations prompt format and rules
- Add clear distinction between chat responses and file content
- Enforce full URL usage in markdown links, prohibit [cite-1] format
- Require content-first approach: write full content, then add citations at end

Frontend:
- Hide <citations> block in both chat messages and markdown preview
- Remove top-level Citations/Sources list for cleaner UI
- Auto-remove <citations> block in code editor view for markdown files
- Keep inline citation hover cards for reference details

This ensures citations are presented like Claude: clean content with inline reference badges.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29 12:29:13 +08:00
ruitanglin 4b63e70b7e feat: refine citations format and improve content presentation
Backend:
- Simplify citations prompt format and rules
- Add clear distinction between chat responses and file content
- Enforce full URL usage in markdown links, prohibit [cite-1] format
- Require content-first approach: write full content, then add citations at end

Frontend:
- Hide <citations> block in both chat messages and markdown preview
- Remove top-level Citations/Sources list for cleaner UI
- Auto-remove <citations> block in code editor view for markdown files
- Keep inline citation hover cards for reference details

This ensures citations are presented like Claude: clean content with inline reference badges.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29 12:29:13 +08:00
ruitanglin e8a8b5e56b feat: refine citations format and improve content presentation
Backend:
- Simplify citations prompt format and rules
- Add clear distinction between chat responses and file content
- Enforce full URL usage in markdown links, prohibit [cite-1] format
- Require content-first approach: write full content, then add citations at end

Frontend:
- Hide <citations> block in both chat messages and markdown preview
- Remove top-level Citations/Sources list for cleaner UI
- Auto-remove <citations> block in code editor view for markdown files
- Keep inline citation hover cards for reference details

This ensures citations are presented like Claude: clean content with inline reference badges.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-29 12:29:13 +08:00
Henry Li 6b030d7589 feat: add tooltips 2026-01-29 09:23:06 +08:00
Henry Li e4d373541f feat: add tooltips 2026-01-29 09:23:06 +08:00
Henry Li c700bd6841 feat: enhance search_image 2026-01-29 09:19:43 +08:00
Henry Li f7ec116c26 feat: enhance search_image 2026-01-29 09:19:43 +08:00
Henry Li 8359d842b5 feat: support image_search 2026-01-29 08:59:55 +08:00
Henry Li d787b1ca54 feat: support image_search 2026-01-29 08:59:55 +08:00
Henry Li 7decdbcc83 fix: improve hasPresentFiles function to check for multiple tool calls 2026-01-29 08:59:45 +08:00
Henry Li 946031b79f fix: improve hasPresentFiles function to check for multiple tool calls 2026-01-29 08:59:45 +08:00
hetaoBackend 1926c58cf2 feat: add image search builtin tool 2026-01-29 08:23:50 +08:00
hetaoBackend 5e62471312 feat: add image search builtin tool 2026-01-29 08:23:50 +08:00
hetaoBackend 248ffe61bc feat: modernize PPT styles and add deep-research skill
Update presentation generation with contemporary design styles
(glassmorphism, dark-premium, neo-brutalist, etc.) and add a new
deep-research skill to guide thorough web research before content
generation tasks.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 01:54:57 +08:00
hetaoBackend af18df480b feat: modernize PPT styles and add deep-research skill
Update presentation generation with contemporary design styles
(glassmorphism, dark-premium, neo-brutalist, etc.) and add a new
deep-research skill to guide thorough web research before content
generation tasks.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 01:54:57 +08:00
hetaoBackend d4bfed271b feat: display ask_clarification tool messages directly in frontend
Simplify clarification message handling by having the frontend detect and
display ask_clarification tool messages directly, instead of relying on
backend to add an extra AIMessage.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 01:25:31 +08:00
hetaoBackend 73a1d32a5b feat: display ask_clarification tool messages directly in frontend
Simplify clarification message handling by having the frontend detect and
display ask_clarification tool messages directly, instead of relying on
backend to add an extra AIMessage.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 01:25:31 +08:00
Willem Jiang 756421c3ac fix(mcp-tool): using the async invocation for MCP tools (#840) 2026-01-28 21:25:16 +08:00
LofiSu ad85b72064 feat: add inline citations and thread management features
Citations:
- Add citations parsing utilities for extracting source references from AI responses
- Render inline citations as hover card badges in message content
- Display citation cards with title, URL, and description on hover
- Add citation badge rendering in artifact markdown preview
- Update prompt to guide AI to output citations in correct format

Thread Management:
- Add rename functionality for chat threads with dialog UI
- Add share functionality to copy thread link to clipboard
- Share links use Vercel URL for production accessibility
- Add useRenameThread hook for thread title updates

i18n:
- Add translations for rename, share, cancel, save, and linkCopied

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-28 19:15:11 +08:00
ruitanglin 33c47e0c56 feat: add inline citations and thread management features
Citations:
- Add citations parsing utilities for extracting source references from AI responses
- Render inline citations as hover card badges in message content
- Display citation cards with title, URL, and description on hover
- Add citation badge rendering in artifact markdown preview
- Update prompt to guide AI to output citations in correct format

Thread Management:
- Add rename functionality for chat threads with dialog UI
- Add share functionality to copy thread link to clipboard
- Share links use Vercel URL for production accessibility
- Add useRenameThread hook for thread title updates

i18n:
- Add translations for rename, share, cancel, save, and linkCopied

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-28 19:15:11 +08:00
ruitanglin f8d2d88727 feat: add inline citations and thread management features
Citations:
- Add citations parsing utilities for extracting source references from AI responses
- Render inline citations as hover card badges in message content
- Display citation cards with title, URL, and description on hover
- Add citation badge rendering in artifact markdown preview
- Update prompt to guide AI to output citations in correct format

Thread Management:
- Add rename functionality for chat threads with dialog UI
- Add share functionality to copy thread link to clipboard
- Share links use Vercel URL for production accessibility
- Add useRenameThread hook for thread title updates

i18n:
- Add translations for rename, share, cancel, save, and linkCopied

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-01-28 19:15:11 +08:00
Henry Li a010953880 feat: update notes 2026-01-28 14:42:13 +08:00
Henry Li 453efa1a1d feat: update notes 2026-01-28 14:42:13 +08:00
Henry Li efb0db53bc Merge remote-tracking branch 'refs/remotes/origin/experimental' into experimental 2026-01-28 14:20:24 +08:00
Henry Li 3293dda5db Merge remote-tracking branch 'refs/remotes/origin/experimental' into experimental 2026-01-28 14:20:24 +08:00
Henry Li dd2c2011f1 feat: update a new demo 2026-01-28 14:19:00 +08:00
Henry Li c0980bfa82 feat: update a new demo 2026-01-28 14:19:00 +08:00
hetaoBackend 49f6c001c3 feat: modify the config example yaml 2026-01-28 14:06:38 +08:00
hetaoBackend 055ab1fb04 feat: modify the config example yaml 2026-01-28 14:06:38 +08:00
hetaoBackend fa9fba3f8e fix: preserve reasoning_content in multi-turn conversations
When using thinking-enabled models (like Kimi K2.5, DeepSeek), the API
expects reasoning_content on all assistant messages. The original
ChatDeepSeek stores reasoning_content in additional_kwargs but doesn't
include it when making subsequent API calls, causing "reasoning_content
is missing" errors.

This adds PatchedChatDeepSeek which overrides _get_request_payload to
restore reasoning_content from additional_kwargs into the payload.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-28 14:04:35 +08:00
hetaoBackend 9d0a0ea022 fix: preserve reasoning_content in multi-turn conversations
When using thinking-enabled models (like Kimi K2.5, DeepSeek), the API
expects reasoning_content on all assistant messages. The original
ChatDeepSeek stores reasoning_content in additional_kwargs but doesn't
include it when making subsequent API calls, causing "reasoning_content
is missing" errors.

This adds PatchedChatDeepSeek which overrides _get_request_payload to
restore reasoning_content from additional_kwargs into the payload.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-28 14:04:35 +08:00
Henry Li d84a34b7cd feat: add Leica demo 2026-01-28 13:52:41 +08:00
Henry Li 486a06d772 feat: add Leica demo 2026-01-28 13:52:41 +08:00
Henry Li d075e7a234 feat: fallback to error reporting 2026-01-28 12:23:29 +08:00
Henry Li 5980bbde02 feat: fallback to error reporting 2026-01-28 12:23:29 +08:00
Henry Li a249b7178a feat: add another Kimi K2.5 demo 2026-01-28 12:17:21 +08:00
Henry Li 5d5aec43d3 feat: add another Kimi K2.5 demo 2026-01-28 12:17:21 +08:00
Henry Li e2bcc70a84 feat: add kimi-k2.5 demo with vercel deployment 2026-01-28 10:13:32 +08:00
Henry Li ade5426d9e feat: add kimi-k2.5 demo with vercel deployment 2026-01-28 10:13:32 +08:00
Henry Li dab4093103 feat: fallback to textarea when loading 2026-01-28 10:13:17 +08:00
Henry Li 86c8f1a25e feat: fallback to textarea when loading 2026-01-28 10:13:17 +08:00
Henry Li 90782f29a2 chore: remove 2026-01-27 13:37:09 +08:00
Henry Li 6b3e101b66 chore: remove 2026-01-27 13:37:09 +08:00
Henry Li 28361ca03c feat: add scroll indicator 2026-01-27 13:15:49 +08:00
Henry Li 7c42fa5162 feat: add scroll indicator 2026-01-27 13:15:49 +08:00
Henry Li ed31dc6aab fix: hide chats when sidebar is not open 2026-01-27 10:41:08 +08:00
Henry Li ec31e61f95 fix: hide chats when sidebar is not open 2026-01-27 10:41:08 +08:00
Henry Li cc1fe4e50e fix: eslint 2026-01-27 10:39:44 +08:00
Henry Li 7928a6f2e1 fix: eslint 2026-01-27 10:39:44 +08:00
Henry Li eca2b139cc fix: bugfix 2026-01-27 09:50:42 +08:00
Henry Li 0bcbaebb7e fix: bugfix 2026-01-27 09:50:42 +08:00
Xun ee02b9f637 feat: Generate a fallback report upon recursion limit hit (#838)
* finish handle_recursion_limit_fallback

* fix

* renmae test file

* fix

* doc

---------

Co-authored-by: lxl0413 <lixinling2021@gmail.com>
2026-01-26 21:10:18 +08:00
hetao b8c33e342b feat: add firecrawl community package with web_search and web_fetch tools
Add web_search_tool and web_fetch_tool implementations using the official
firecrawl-py SDK as an alternative to Tavily/Jina AI integrations.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 19:58:08 +08:00
hetao ce7f7258ba feat: add firecrawl community package with web_search and web_fetch tools
Add web_search_tool and web_fetch_tool implementations using the official
firecrawl-py SDK as an alternative to Tavily/Jina AI integrations.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 19:58:08 +08:00
hetao 9215c9cce7 feat: add ppt-generation skill
Creates presentations by generating AI images for each slide and composing
them into PPTX files. Features include:
- Multiple presentation styles (business, academic, minimal, keynote, creative)
- Visual consistency through reference image chaining (each slide uses the
  previous slide as reference)
- Speaker notes from presentation plan

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 19:35:29 +08:00
hetao 0cc7cc08e9 feat: add ppt-generation skill
Creates presentations by generating AI images for each slide and composing
them into PPTX files. Features include:
- Multiple presentation styles (business, academic, minimal, keynote, creative)
- Visual consistency through reference image chaining (each slide uses the
  previous slide as reference)
- Speaker notes from presentation plan

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 19:35:29 +08:00
Henry Li 3ce4968e95 feat: auto select the first model as default model 2026-01-26 17:13:34 +08:00
Henry Li 574dfd2b05 feat: auto select the first model as default model 2026-01-26 17:13:34 +08:00
hetao 22004406a7 perf: parallelize TTS generation in podcast skill
Use ThreadPoolExecutor to generate audio for multiple script lines
concurrently, significantly speeding up podcast generation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 14:50:56 +08:00
hetao ff7065b085 perf: parallelize TTS generation in podcast skill
Use ThreadPoolExecutor to generate audio for multiple script lines
concurrently, significantly speeding up podcast generation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 14:50:56 +08:00
hetao dddd745b5b refactor: simplify podcast-generation to use direct JSON script input
- Remove LLM script generation from Python script, model now generates
  JSON script directly (similar to image-generation skill)
- Add --transcript-file option to generate markdown transcript
- Add optional "title" field in JSON for transcript heading
- Remove dependency on OPENAI_API_KEY for podcast generation
- Update SKILL.md with new workflow and JSON format documentation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 14:01:48 +08:00
hetao 68a3be1491 refactor: simplify podcast-generation to use direct JSON script input
- Remove LLM script generation from Python script, model now generates
  JSON script directly (similar to image-generation skill)
- Add --transcript-file option to generate markdown transcript
- Add optional "title" field in JSON for transcript heading
- Remove dependency on OPENAI_API_KEY for podcast generation
- Update SKILL.md with new workflow and JSON format documentation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-26 14:01:48 +08:00
hetaoBackend 9f5658fa0e feat: add podcast generation skill
- Add podcast-generation skill for creating tech explainer podcasts
- Include generate.py script with TTS synthesis capabilities
- Add tech-explainer template for structured podcast content
- Increase sandbox command timeout from 30s to 600s to support
  longer-running skill scripts
2026-01-26 13:16:35 +08:00
hetaoBackend 3fa16467a2 feat: add podcast generation skill
- Add podcast-generation skill for creating tech explainer podcasts
- Include generate.py script with TTS synthesis capabilities
- Add tech-explainer template for structured podcast content
- Increase sandbox command timeout from 30s to 600s to support
  longer-running skill scripts
2026-01-26 13:16:35 +08:00
hetaoBackend 139063283f fix: ensure MCP and skills config changes are immediately reflected
- Use ExtensionsConfig.from_file() instead of cached config to always
  read latest configuration from disk in LangGraph Server
- Add mtime-based cache invalidation for MCP tools to detect config
  file changes made through Gateway API
- Call reload_extensions_config() in Gateway API after updates to
  refresh the global cache
- Remove unnecessary MCP initialization from Gateway startup since
  MCP tools are only used by LangGraph Server

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 22:40:09 +08:00
hetaoBackend 038f5d44f4 fix: ensure MCP and skills config changes are immediately reflected
- Use ExtensionsConfig.from_file() instead of cached config to always
  read latest configuration from disk in LangGraph Server
- Add mtime-based cache invalidation for MCP tools to detect config
  file changes made through Gateway API
- Call reload_extensions_config() in Gateway API after updates to
  refresh the global cache
- Remove unnecessary MCP initialization from Gateway startup since
  MCP tools are only used by LangGraph Server

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 22:40:09 +08:00
Henry Li f629e134d4 feat: adjust button 2026-01-25 22:10:50 +08:00
Henry Li 044e38aec6 feat: adjust button 2026-01-25 22:10:50 +08:00
Henry Li 598fed797f fix: many minor fixes 2026-01-25 21:57:57 +08:00
Henry Li 756b396a64 fix: many minor fixes 2026-01-25 21:57:57 +08:00
Henry Li ae0e7de3b7 feat: add image and video generation skills 2026-01-25 21:57:44 +08:00
Henry Li b53a2ea5e1 feat: add image and video generation skills 2026-01-25 21:57:44 +08:00
Henry Li af4fc800ee feat: update demo 2026-01-25 21:57:22 +08:00
Henry Li 90c30c8485 feat: update demo 2026-01-25 21:57:22 +08:00
Xun 9a34e32252 chore : Improved citation system (#834)
* improve: Improved citation system

* fix

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-01-25 15:49:45 +08:00
Henry Li 87200d1ad1 feat: update translations 2026-01-25 11:54:49 +08:00
Henry Li e6cac2cae4 feat: update translations 2026-01-25 11:54:49 +08:00
Henry Li c82f705541 fix: fix artifacts in demo mode 2026-01-25 11:42:25 +08:00
Henry Li fecc5faacf fix: fix artifacts in demo mode 2026-01-25 11:42:25 +08:00
Henry Li 74dd09b364 feat: update demos 2026-01-25 11:41:48 +08:00
Henry Li e84fb705ce feat: update demos 2026-01-25 11:41:48 +08:00
dependabot[bot] 31624b64b8 build(deps): bump starlette from 0.46.1 to 0.49.1 (#837)
Bumps [starlette](https://github.com/Kludex/starlette) from 0.46.1 to 0.49.1.
- [Release notes](https://github.com/Kludex/starlette/releases)
- [Changelog](https://github.com/Kludex/starlette/blob/main/docs/release-notes.md)
- [Commits](https://github.com/Kludex/starlette/compare/0.46.1...0.49.1)

---
updated-dependencies:
- dependency-name: starlette
  dependency-version: 0.49.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-25 11:10:58 +08:00
dependabot[bot] b71ec515ab build(deps): bump pyasn1 from 0.6.1 to 0.6.2 (#836)
Bumps [pyasn1](https://github.com/pyasn1/pyasn1) from 0.6.1 to 0.6.2.
- [Release notes](https://github.com/pyasn1/pyasn1/releases)
- [Changelog](https://github.com/pyasn1/pyasn1/blob/main/CHANGES.rst)
- [Commits](https://github.com/pyasn1/pyasn1/compare/v0.6.1...v0.6.2)

---
updated-dependencies:
- dependency-name: pyasn1
  dependency-version: 0.6.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-25 10:37:57 +08:00
Henry Li 78bba47769 feat: add Titanic ADA demo 2026-01-25 00:35:42 +08:00
Henry Li c6dbd9fbf4 feat: add Titanic ADA demo 2026-01-25 00:35:42 +08:00
Henry Li 3ac6e58d4f fix: remove tooltip 2026-01-25 00:06:49 +08:00
Henry Li 9501ec5eed fix: remove tooltip 2026-01-25 00:06:49 +08:00
Henry Li 03b380cb8b fix: fix auto select first artifact 2026-01-24 23:59:41 +08:00
Henry Li 1e2855b533 fix: fix auto select first artifact 2026-01-24 23:59:41 +08:00
Henry Li 35f2aea510 feat: add new demo 2026-01-24 23:51:38 +08:00
Henry Li 03311d43da feat: add new demo 2026-01-24 23:51:38 +08:00
Henry Li a83e5e238d feat: auto expand in demo mode 2026-01-24 23:51:11 +08:00
Henry Li 099fb727cc feat: auto expand in demo mode 2026-01-24 23:51:11 +08:00
dependabot[bot] 939a83af81 build(deps): bump urllib3 from 2.3.0 to 2.6.3 (#835)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.3.0 to 2.6.3.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.3.0...2.6.3)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.6.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-24 22:43:15 +08:00
Henry Li 2698c26768 chore: update translation 2026-01-24 22:41:40 +08:00
Henry Li 2eb9bb2167 chore: update translation 2026-01-24 22:41:40 +08:00
hetao 6e147a772e feat: add environment variable injection for Docker sandbox
- Add environment field to sandbox config for injecting env vars into container
- Support $VAR syntax to resolve values from host environment variables
- Refactor frontend API modules to use centralized getBackendBaseURL()
- Improve Doraemon skill with explicit input/output path arguments
- Add .env.example file

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 22:36:05 +08:00
hetao 5671642dbe feat: add environment variable injection for Docker sandbox
- Add environment field to sandbox config for injecting env vars into container
- Support $VAR syntax to resolve values from host environment variables
- Refactor frontend API modules to use centralized getBackendBaseURL()
- Improve Doraemon skill with explicit input/output path arguments
- Add .env.example file

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 22:36:05 +08:00
Henry Li 869af570c9 feat: add i18n 2026-01-24 22:19:37 +08:00
Henry Li 48b5428000 feat: add i18n 2026-01-24 22:19:37 +08:00
Henry Li 2bdda01657 Merge remote-tracking branch 'refs/remotes/origin/experimental' into experimental 2026-01-24 22:05:15 +08:00
Henry Li 481396210b Merge remote-tracking branch 'refs/remotes/origin/experimental' into experimental 2026-01-24 22:05:15 +08:00
Henry Li 5a27a3beeb feat: expand by default in demo mode 2026-01-24 22:03:38 +08:00
Henry Li 56b21e00bf feat: expand by default in demo mode 2026-01-24 22:03:38 +08:00
JeffJiang 38081306fe feat: adds docker-based dev environment (#18)
* feat: adds docker-based dev environment

* docs: updates Docker command help

* fix local dev
2026-01-24 22:01:00 +08:00
JeffJiang 400349c3e0 feat: adds docker-based dev environment (#18)
* feat: adds docker-based dev environment

* docs: updates Docker command help

* fix local dev
2026-01-24 22:01:00 +08:00
dependabot[bot] f299792ce1 build(deps): bump mcp from 1.12.2 to 1.23.0 (#833)
Bumps [mcp](https://github.com/modelcontextprotocol/python-sdk) from 1.12.2 to 1.23.0.
- [Release notes](https://github.com/modelcontextprotocol/python-sdk/releases)
- [Changelog](https://github.com/modelcontextprotocol/python-sdk/blob/main/RELEASE.md)
- [Commits](https://github.com/modelcontextprotocol/python-sdk/compare/v1.12.2...v1.23.0)

---
updated-dependencies:
- dependency-name: mcp
  dependency-version: 1.23.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-24 21:57:39 +08:00
Henry Li c468381064 feat: add Doraemon Skill 2026-01-24 21:54:01 +08:00
Henry Li ee9950d6aa feat: add Doraemon Skill 2026-01-24 21:54:01 +08:00
Henry Li cae7e67a1f feat: remove over-scroll 2026-01-24 21:14:33 +08:00
Henry Li 291b899486 feat: remove over-scroll 2026-01-24 21:14:33 +08:00
Henry Li 72e0f3d081 feat: add new demo 2026-01-24 20:59:06 +08:00
Henry Li 2c2a177186 feat: add new demo 2026-01-24 20:59:06 +08:00
Henry Li 08f1af00b6 feat: support absolute path as image src 2026-01-24 20:58:56 +08:00
Henry Li 4aef821344 feat: support absolute path as image src 2026-01-24 20:58:56 +08:00
Henry Li cdcadc3fe3 style: update tooltip background class for consistency 2026-01-24 20:58:36 +08:00
Henry Li c47455e1eb style: update tooltip background class for consistency 2026-01-24 20:58:36 +08:00
Henry Li 6485ed2a50 chore: add new demo 2026-01-24 19:44:06 +08:00
Henry Li cced422e9d chore: add new demo 2026-01-24 19:44:06 +08:00
Henry Li e88b34c0cb chore: delete 2026-01-24 19:43:44 +08:00
Henry Li 2487d7684f chore: delete 2026-01-24 19:43:44 +08:00
Henry Li 2c8a41dc6c style: reformat 2026-01-24 19:40:44 +08:00
Henry Li 931421f57b style: reformat 2026-01-24 19:40:44 +08:00
Henry Li 72e3ba9b79 feat: add new demo 2026-01-24 19:38:17 +08:00
Henry Li 373fe0cd3c feat: add new demo 2026-01-24 19:38:17 +08:00
Henry Li 27df1b5f73 feat: add uploads 2026-01-24 19:38:08 +08:00
Henry Li 1f4591a4d1 feat: add uploads 2026-01-24 19:38:08 +08:00
Henry Li a3eb03b105 chore: add new demo 2026-01-24 18:53:13 +08:00
Henry Li db27ca4ae0 chore: add new demo 2026-01-24 18:53:13 +08:00
Henry Li 930e6bd46f feat: remove background 2026-01-24 18:48:35 +08:00
Henry Li 0f1bfc3403 feat: remove background 2026-01-24 18:48:35 +08:00
Henry Li 6f24a71e1e feat: update save-demo 2026-01-24 18:33:30 +08:00
Henry Li 3ea1dcac11 feat: update save-demo 2026-01-24 18:33:30 +08:00
Henry Li 584c88f0dd feat: add more links 2026-01-24 18:21:50 +08:00
Henry Li 3c40446ade feat: add more links 2026-01-24 18:21:50 +08:00
dependabot[bot] 3cd0c3ae9d build(deps): bump aiohttp from 3.12.14 to 3.13.3 (#832)
---
updated-dependencies:
- dependency-name: aiohttp
  dependency-version: 3.13.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-01-24 18:08:25 +08:00
dependabot[bot] 47bd1815c7 build(deps): bump h11 from 0.14.0 to 0.16.0 (#831)
Bumps [h11](https://github.com/python-hyper/h11) from 0.14.0 to 0.16.0.
- [Commits](https://github.com/python-hyper/h11/compare/v0.14.0...v0.16.0)

---
updated-dependencies:
- dependency-name: h11
  dependency-version: 0.16.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-24 18:03:54 +08:00
Henry Li cd63f41b4c feat: support static website 2026-01-24 18:01:27 +08:00
Henry Li ebda30c7cf feat: support static website 2026-01-24 18:01:27 +08:00
LoftyComet b7f0f54aa0 feat: add citation support in research report block and markdown
* feat: add citation support in research report block and markdown

- Enhanced ResearchReportBlock to fetch citations based on researchId and pass them to the Markdown component.
- Introduced CitationLink component to display citation metadata on hover for links in markdown.
- Implemented CitationCard and CitationList components for displaying citation details and lists.
- Updated Markdown component to handle citation links and inline citations.
- Created HoverCard component for displaying citation information in a tooltip-like manner.
- Modified store to manage citations, including setting and retrieving citations for ongoing research.
- Added CitationsEvent type to handle citations in chat events and updated Message type to include citations.

* fix(log): Enable the logging level  when enabling the DEBUG environment variable (#793)

* fix(frontend): render all tool calls in the frontend #796 (#797)

* build(deps): bump jspdf from 3.0.4 to 4.0.0 in /web (#798)

Bumps [jspdf](https://github.com/parallax/jsPDF) from 3.0.4 to 4.0.0.
- [Release notes](https://github.com/parallax/jsPDF/releases)
- [Changelog](https://github.com/parallax/jsPDF/blob/master/RELEASE.md)
- [Commits](https://github.com/parallax/jsPDF/compare/v3.0.4...v4.0.0)

---
updated-dependencies:
- dependency-name: jspdf
  dependency-version: 4.0.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(frontend):added the display of the 'analyst' message #800 (#801)

* fix: migrate from deprecated create_react_agent to langchain.agents.create_agent (#802)

* fix: migrate from deprecated create_react_agent to langchain.agents.create_agent

Fixes #799

- Replace deprecated langgraph.prebuilt.create_react_agent with
  langchain.agents.create_agent (LangGraph 1.0 migration)
- Add DynamicPromptMiddleware to handle dynamic prompt templates
  (replaces the 'prompt' callable parameter)
- Add PreModelHookMiddleware to handle pre-model hooks
  (replaces the 'pre_model_hook' parameter)
- Update AgentState import from langchain.agents in template.py
- Update tests to use the new API

* fix:update the code with review comments

* fix: Add runtime parameter to compress_messages method(#803) 

* fix: Add runtime parameter to compress_messages method(#803)

    The compress_messages method was being called by PreModelHookMiddleware
    with both state and runtime parameters, but only accepted state parameter.
    This caused a TypeError when the middleware executed the pre_model_hook.

    Added optional runtime parameter to compress_messages signature to match
    the expected interface while maintaining backward compatibility.

* Update the code with the review comments

* fix: Refactor citation handling and add comprehensive tests for citation features

* refactor: Clean up imports and formatting across citation modules

* fix: Add monkeypatch to clear AGENT_RECURSION_LIMIT in recursion limit tests

* feat: Enhance citation link handling in Markdown component

* fix: Exclude citations from finish reason handling in mergeMessage function

* fix(nodes): update message handling

* fix(citations): improve citation extraction and handling in event processing

* feat(citations): enhance citation extraction and handling with improved merging and normalization

* fix(reporter): update citation formatting instructions for clarity and consistency

* fix(reporter): prioritize using Markdown tables for data presentation and comparison

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: LoftyComet <1277173875@qq。>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-24 17:49:13 +08:00
Willem Jiang 612bddd3fb feat(server): add MCP server configuration validation (#830)
* feat(server): add MCP server configuration validation

Add comprehensive validation for MCP server configurations,
inspired by Flowise's validateMCPServerConfig implementation.

MCPServerConfig checks implemented:
- Command allowlist validation (node, npx, python, docker, uvx, etc.)
- Path traversal prevention (blocks ../, absolute paths, ~/)
- Shell command injection prevention (blocks ; & | ` $ etc.)
- Dangerous environment variable blocking (PATH, LD_PRELOAD, etc.)
- URL validation for SSE/HTTP transports (scheme, credentials)
- HTTP header injection prevention (blocks newlines)

* fix the unit test error of test_chat_request

* Added the related path cases as reviewer commented

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-24 17:32:17 +08:00
Henry Li 3ffce7667c Merge pull request #16 from amszuidas/experimental
fix: refactor env var resolution to support complex structures and fix in-place mutation bug
2026-01-24 10:14:20 +08:00
Henry Li c66995bcc0 Merge pull request #16 from amszuidas/experimental
fix: refactor env var resolution to support complex structures and fix in-place mutation bug
2026-01-24 10:14:20 +08:00
Henry Li b1e7028ea0 Merge pull request #17 from amszuidas/fix/tavily-api-key-config
fix: support loading tavily ak from config.yaml
2026-01-24 10:14:04 +08:00
Henry Li 9498e783f1 Merge pull request #17 from amszuidas/fix/tavily-api-key-config
fix: support loading tavily ak from config.yaml
2026-01-24 10:14:04 +08:00
amszuidas d6176e86d6 fix: support loading tavily ak from config.yaml 2026-01-23 23:50:40 +08:00
amszuidas c1c8942491 fix: support loading tavily ak from config.yaml 2026-01-23 23:50:40 +08:00
amszuidas 3972485fe0 fix: use return value of resolve_env_variables in config loading 2026-01-23 21:51:48 +08:00
amszuidas 761cb6a7f5 fix: use return value of resolve_env_variables in config loading 2026-01-23 21:51:48 +08:00
hetao f6a20a69e3 feat: implement file upload feature 2026-01-23 18:47:39 +08:00
hetao 1fe37fdb6c feat: implement file upload feature 2026-01-23 18:47:39 +08:00
amszuidas eb802361e1 fix: correct spelling 2026-01-23 18:29:20 +08:00
amszuidas 2ef320f107 fix: correct spelling 2026-01-23 18:29:20 +08:00
amszuidas 82a6ae81bd fix: robust environment variable resolution in config 2026-01-23 17:01:38 +08:00
amszuidas 303e0252ce fix: robust environment variable resolution in config 2026-01-23 17:01:38 +08:00
Henry Li 3f4bcd9433 feat: implement the first version of landing page 2026-01-23 13:24:03 +08:00
Henry Li 0908127bd7 feat: implement the first version of landing page 2026-01-23 13:24:03 +08:00
Xun c0849af37e feat(context): decrease token in web_search AIMessage (#827)
This PR addresses token limit issues when web_search is enabled with include_raw_content by implementing a two-pronged approach: changing the default behavior to exclude raw content and adding compression logic for when raw content is included.
2026-01-23 08:31:48 +08:00
Henry Li 307972f93e feat: implement the first section of landing page 2026-01-23 00:15:21 +08:00
Henry Li b69c13a3e5 feat: implement the first section of landing page 2026-01-23 00:15:21 +08:00
Henry Li 65cdc182d3 docs: add notes for v2.0 (#828) 2026-01-22 20:08:59 +08:00
Henry Li 459d9d0287 fix: fix menu item in side bar collapsed mode 2026-01-22 15:18:42 +08:00
Henry Li 6e1f63e47f fix: fix menu item in side bar collapsed mode 2026-01-22 15:18:42 +08:00
Henry Li e9ab427326 feat: adjust styles 2026-01-22 14:28:10 +08:00
Henry Li dc9d28018c feat: adjust styles 2026-01-22 14:28:10 +08:00
Henry Li c48a3f499d docs: rewording 2026-01-22 14:21:03 +08:00
Henry Li 9df56299c1 docs: rewording 2026-01-22 14:21:03 +08:00
Henry Li e0f491dcdb feat: add main menu 2026-01-22 14:19:54 +08:00
Henry Li e1378123f5 feat: add main menu 2026-01-22 14:19:54 +08:00
Henry Li 80b07bcac0 feat: update opacities 2026-01-22 13:50:09 +08:00
Henry Li cb996f0858 feat: update opacities 2026-01-22 13:50:09 +08:00
Henry Li 8c994293a8 feat: make reasoning mode as default 2026-01-22 13:46:43 +08:00
Henry Li 99eb2474b3 feat: make reasoning mode as default 2026-01-22 13:46:43 +08:00
Henry Li ec4b3a0ead docs: update description 2026-01-22 13:46:31 +08:00
Henry Li b938f40e4c docs: update description 2026-01-22 13:46:31 +08:00
Henry Li 7d4d706738 feat: put all options into '+' 2026-01-22 13:43:45 +08:00
Henry Li 8ef89b3004 feat: put all options into '+' 2026-01-22 13:43:45 +08:00
hetao 31bf49917c feat: add unified development environment with nginx proxy
Add a root-level Makefile to manage frontend, backend, and nginx services:
- `make check` validates required dependencies (Node.js 22+, pnpm, uv, nginx)
- `make install` installs all project dependencies
- `make dev` starts all services with unified port 2026
- `make stop` and `make clean` for cleanup

Update nginx configuration:
- Change port from 8000 to 2026
- Add frontend upstream and routing (port 3000)
- Add /api/langgraph/* routing with path rewriting to LangGraph server
- Keep other /api/* routes to Gateway API
- Route non-API requests to frontend

Update frontend configuration:
- Use relative URLs through nginx proxy by default
- Support environment variables for direct backend access
- Construct full URL for LangGraph SDK compatibility

Clean up backend Makefile by removing nginx and serve targets.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-22 12:00:46 +08:00
hetao 2fac72601e feat: add unified development environment with nginx proxy
Add a root-level Makefile to manage frontend, backend, and nginx services:
- `make check` validates required dependencies (Node.js 22+, pnpm, uv, nginx)
- `make install` installs all project dependencies
- `make dev` starts all services with unified port 2026
- `make stop` and `make clean` for cleanup

Update nginx configuration:
- Change port from 8000 to 2026
- Add frontend upstream and routing (port 3000)
- Add /api/langgraph/* routing with path rewriting to LangGraph server
- Keep other /api/* routes to Gateway API
- Route non-API requests to frontend

Update frontend configuration:
- Use relative URLs through nginx proxy by default
- Support environment variables for direct backend access
- Construct full URL for LangGraph SDK compatibility

Clean up backend Makefile by removing nginx and serve targets.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-22 12:00:46 +08:00
hetao c00f780501 fix: fix nginx conf 2026-01-22 12:00:46 +08:00
hetao 50c25f5c4d fix: fix nginx conf 2026-01-22 12:00:46 +08:00
Henry Li 16a499190b feat: show in-progress 2026-01-22 11:56:45 +08:00
Henry Li 93f70893fc feat: show in-progress 2026-01-22 11:56:45 +08:00
Henry Li aa7436db2f feat: adjust input background in light mode 2026-01-22 11:51:56 +08:00
Henry Li 4f712861a3 feat: adjust input background in light mode 2026-01-22 11:51:56 +08:00
Henry Li 93842e81a4 feat: adjust styles 2026-01-22 11:42:25 +08:00
Henry Li aed2f7ce67 feat: adjust styles 2026-01-22 11:42:25 +08:00
Henry Li 54710960cb docs: remove '/' 2026-01-22 11:31:23 +08:00
Henry Li 3774d0453c docs: remove '/' 2026-01-22 11:31:23 +08:00
Henry Li 11918b5270 fix: update summarization configuration values 2026-01-22 10:36:19 +08:00
Henry Li bd33f72017 fix: update summarization configuration values 2026-01-22 10:36:19 +08:00
Henry Li e8e522c2fe feat: add animations 2026-01-22 09:41:01 +08:00
Henry Li 9e72dc4f63 feat: add animations 2026-01-22 09:41:01 +08:00
Willem Jiang 546e2e6234 Fixes(unit-test): the unit tests error of recent change of #816 (#826)
* feat: Implement DeerFlow API server with chat streaming, Langgraph orchestration, and various content generation capabilities.

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* - Use MongoDB `$push` with `$each` to append new messages to existing threads
- Use PostgreSQL jsonb concatenation operator to merge messages instead of overwriting
- Update comments to reflect append behavior in both database implementations

* fix: updated the unit tests with the recent changes

---------

Co-authored-by: Bink <992359580@qq.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: YikB <54528024+Bin1783@users.noreply.github.com>
2026-01-22 09:17:14 +08:00
YikB 7cd2265272 append messages to chat_streams table (#816)
* feat: Implement DeerFlow API server with chat streaming, Langgraph orchestration, and various content generation capabilities.

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* - Use MongoDB `$push` with `$each` to append new messages to existing threads
- Use PostgreSQL jsonb concatenation operator to merge messages instead of overwriting
- Update comments to reflect append behavior in both database implementations

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-22 09:09:15 +08:00
Henry Li 37e2c3d3c9 feat: update skill settings 2026-01-22 00:38:20 +08:00
Henry Li b630e1846a feat: update skill settings 2026-01-22 00:38:20 +08:00
Henry Li 1e4e51a80c feat: add Todos 2026-01-22 00:26:11 +08:00
Henry Li 44850d9a61 feat: add Todos 2026-01-22 00:26:11 +08:00
hetao 08101aa432 refactor: refine skills 2026-01-21 21:22:56 +08:00
hetao 085dff8d29 refactor: refine skills 2026-01-21 21:22:56 +08:00
hetao 5a45b9c131 feat: add SSE and HTTP transport support for MCP servers
- Add type, url, and headers fields to MCP server config
- Update MCP client to handle stdio, sse, and http transports
- Add todos field to ThreadState
- Add Deerflow branding requirement to frontend-design skill
- Update extensions_config.example.json with SSE/HTTP examples

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-21 16:14:00 +08:00
hetao 87752cafac feat: add SSE and HTTP transport support for MCP servers
- Add type, url, and headers fields to MCP server config
- Update MCP client to handle stdio, sse, and http transports
- Add todos field to ThreadState
- Add Deerflow branding requirement to frontend-design skill
- Update extensions_config.example.json with SSE/HTTP examples

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-21 16:14:00 +08:00
Henry Li fbe4d27ddd feat: use resolvedTheme instead of systemTheme 2026-01-21 10:48:29 +08:00
Henry Li 68b8083826 feat: use resolvedTheme instead of systemTheme 2026-01-21 10:48:29 +08:00
Henry Li 11c562eb98 refactor: move 2026-01-21 10:46:43 +08:00
Henry Li c7a40d6a23 refactor: move 2026-01-21 10:46:43 +08:00
Henry Li 54d29e254f docs: rewording 2026-01-21 10:46:18 +08:00
Henry Li f907f8ac16 docs: rewording 2026-01-21 10:46:18 +08:00
Henry Li e3d5b4960f feat: adjust colors 2026-01-21 10:35:50 +08:00
Henry Li ce4aa1e154 feat: adjust colors 2026-01-21 10:35:50 +08:00
Henry Li 26587ee970 feat: bring back the deer 2026-01-21 10:31:54 +08:00
Henry Li 1372dbefb2 feat: bring back the deer 2026-01-21 10:31:54 +08:00
Henry Li 220fc1c489 feat: auto open artifact 2026-01-21 09:45:55 +08:00
Henry Li 4467b1860f feat: auto open artifact 2026-01-21 09:45:55 +08:00
Henry Li 48742d1b59 feat: add code editor 2026-01-21 09:33:33 +08:00
Henry Li 6e024d6c8f feat: add code editor 2026-01-21 09:33:33 +08:00
Xun 6ec170cde5 fix: handle false values correctly in (#823)
Fixes a critical bug in the from_runnable_config() method where falsy values (like False, 0, and empty strings) were being incorrectly filtered out, causing configuration fields to revert to their default values. The fix changes the filter condition from if v to if v is not None, ensuring only None values are skipped.
2026-01-21 09:33:20 +08:00
Henry Li 7c6eb4cc8b feat: enlarge shadow 2026-01-21 08:52:30 +08:00
Henry Li 4b7ee2bee2 feat: enlarge shadow 2026-01-21 08:52:30 +08:00
Henry Li d77b9922a6 feat: make artifact "floating" 2026-01-21 08:50:15 +08:00
Henry Li 28d724d55a feat: make artifact "floating" 2026-01-21 08:50:15 +08:00
Henry Li a2ca682b0c feat: change color themes 2026-01-21 08:37:30 +08:00
Henry Li adfce3c79c feat: change color themes 2026-01-21 08:37:30 +08:00
Henry Li 10d253f461 feat: support settings 2026-01-20 23:43:21 +08:00
Henry Li 1b70e00642 feat: support settings 2026-01-20 23:43:21 +08:00
hetaoBackend 3191a3845f feat: integrate todo middleware 2026-01-20 22:38:04 +08:00
hetaoBackend 7ead7c93f8 feat: integrate todo middleware 2026-01-20 22:38:04 +08:00
hetaoBackend adbb03fc26 fix: fix sandbox cp issue 2026-01-20 22:08:36 +08:00
hetaoBackend c5a2771636 fix: fix sandbox cp issue 2026-01-20 22:08:36 +08:00
hetaoBackend 5888a5ba16 fix: fix skill md path 2026-01-20 21:10:05 +08:00
hetaoBackend e58e5f1904 fix: fix skill md path 2026-01-20 21:10:05 +08:00
hetaoBackend abc6c21b11 feat: enable public skills by default 2026-01-20 20:37:51 +08:00
hetaoBackend 2d931105d5 feat: enable public skills by default 2026-01-20 20:37:51 +08:00
Xun 0e64c52975 refactor: Refactors the retriever function to use async/await (#821)
* refactor: Refactors the retriever function to use async/await
2026-01-20 19:56:26 +08:00
Henry Li faba2784e1 feat: save locale in cookies 2026-01-20 16:00:39 +08:00
Henry Li dc8c1f4ed6 feat: save locale in cookies 2026-01-20 16:00:39 +08:00
Henry Li 32a45eb043 feat: implement i18n 2026-01-20 14:06:47 +08:00
Henry Li ac9ef30780 feat: implement i18n 2026-01-20 14:06:47 +08:00
hetaoBackend 6ec023de8b fix: fix config 2026-01-20 13:58:28 +08:00
hetaoBackend 33e6197f65 fix: fix config 2026-01-20 13:58:28 +08:00
hetaoBackend 50810c8212 feat: add skills api 2026-01-20 13:57:36 +08:00
hetaoBackend 66df9b5927 feat: add skills api 2026-01-20 13:57:36 +08:00
hetaoBackend 8434cf4c60 feat: add MCP API endpoint and enhance API documentation
Add new MCP configuration management endpoint and enhance API documentation
with detailed descriptions, examples, and OpenAPI support for better
developer experience.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-20 13:20:50 +08:00
hetaoBackend 411d9d57c3 feat: add MCP API endpoint and enhance API documentation
Add new MCP configuration management endpoint and enhance API documentation
with detailed descriptions, examples, and OpenAPI support for better
developer experience.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-20 13:20:50 +08:00
hetaoBackend d11763dcc8 fix: fix backend 2026-01-20 09:58:27 +08:00
hetaoBackend 5c1bb675ba fix: fix backend 2026-01-20 09:58:27 +08:00
Henry Li a18f37779e docs: rewording 2026-01-20 09:26:29 +08:00
Henry Li b791b28afa docs: rewording 2026-01-20 09:26:29 +08:00
DanielWalnut 513332b746 feat: add nginx reversed proxy (#15)
* docs: add nginx reverse proxy documentation

Add comprehensive nginx configuration documentation to README including:
- Production deployment instructions with step-by-step setup
- Architecture diagram showing traffic routing between services
- Nginx features: unified entry point, CORS handling, SSE support
- Updated project structure with nginx.conf and service ports

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* feat: implement nginx

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-19 23:23:38 +08:00
DanielWalnut 7978e05dc1 feat: add nginx reversed proxy (#15)
* docs: add nginx reverse proxy documentation

Add comprehensive nginx configuration documentation to README including:
- Production deployment instructions with step-by-step setup
- Architecture diagram showing traffic routing between services
- Nginx features: unified entry point, CORS handling, SSE support
- Updated project structure with nginx.conf and service ports

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* feat: implement nginx

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-19 23:23:38 +08:00
Henry Li b8f9678d07 feat: use code block to display bash commands 2026-01-19 21:59:23 +08:00
Henry Li 5d6162d006 feat: use code block to display bash commands 2026-01-19 21:59:23 +08:00
Henry Li fb265f2b1f feat: support NEXT_PUBLIC_LANGGRAPH_BASE_URL 2026-01-19 21:51:40 +08:00
Henry Li 58b5c2fcd5 feat: support NEXT_PUBLIC_LANGGRAPH_BASE_URL 2026-01-19 21:51:40 +08:00
hetaoBackend d6c1e5868d fix: fix proxy 2026-01-19 21:36:35 +08:00
hetaoBackend a6fcdbf50a fix: fix proxy 2026-01-19 21:36:35 +08:00
Henry Li d7dfffad90 feat: add ToggleGroup 2026-01-19 19:41:46 +08:00
Henry Li 24ca87d650 feat: add ToggleGroup 2026-01-19 19:41:46 +08:00
hetaoBackend 1171598b2f feat: add MCP (Model Context Protocol) support
Add comprehensive MCP integration using langchain-mcp-adapters to enable
pluggable external tools from MCP servers.

Features:
- MCP server configuration via mcp_config.json
- Automatic lazy initialization for seamless use in both FastAPI and LangGraph Studio
- Support for multiple MCP servers (filesystem, postgres, github, brave-search, etc.)
- Environment variable resolution in configuration
- Tool caching mechanism for optimal performance
- Complete documentation and setup guide

Implementation:
- Add src/mcp module with client, tools, and cache components
- Integrate MCP config loading in AppConfig
- Update tool system to include MCP tools automatically
- Add eager initialization in FastAPI lifespan handler
- Add lazy initialization fallback for LangGraph Studio

Dependencies:
- Add langchain-mcp-adapters>=0.1.0

Documentation:
- Add MCP_SETUP.md with comprehensive setup guide
- Update CLAUDE.md with MCP system architecture
- Update config.example.yaml with MCP configuration notes
- Update README.md with MCP setup instructions

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-19 18:58:13 +08:00
hetaoBackend 74d4a16492 feat: add MCP (Model Context Protocol) support
Add comprehensive MCP integration using langchain-mcp-adapters to enable
pluggable external tools from MCP servers.

Features:
- MCP server configuration via mcp_config.json
- Automatic lazy initialization for seamless use in both FastAPI and LangGraph Studio
- Support for multiple MCP servers (filesystem, postgres, github, brave-search, etc.)
- Environment variable resolution in configuration
- Tool caching mechanism for optimal performance
- Complete documentation and setup guide

Implementation:
- Add src/mcp module with client, tools, and cache components
- Integrate MCP config loading in AppConfig
- Update tool system to include MCP tools automatically
- Add eager initialization in FastAPI lifespan handler
- Add lazy initialization fallback for LangGraph Studio

Dependencies:
- Add langchain-mcp-adapters>=0.1.0

Documentation:
- Add MCP_SETUP.md with comprehensive setup guide
- Update CLAUDE.md with MCP system architecture
- Update config.example.yaml with MCP configuration notes
- Update README.md with MCP setup instructions

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-19 18:58:13 +08:00
Henry Li 541586dc66 feat: support dynamic loading models 2026-01-19 18:54:04 +08:00
Henry Li 21f35b1d3c feat: support dynamic loading models 2026-01-19 18:54:04 +08:00
hetaoBackend 1a7c853811 fix: use shared httpx client to prevent premature closure in SSE streaming
The proxy was creating a temporary httpx.AsyncClient within an async context manager.
When returning StreamingResponse for SSE endpoints, the client was being closed before
the streaming generator could use it, causing "client has been closed" errors.

This change introduces a shared httpx.AsyncClient that persists for the application
lifecycle, properly cleaned up during shutdown. This also improves performance by
reusing TCP connections across requests.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-19 16:52:30 +08:00
hetaoBackend ffb9ed3198 fix: use shared httpx client to prevent premature closure in SSE streaming
The proxy was creating a temporary httpx.AsyncClient within an async context manager.
When returning StreamingResponse for SSE endpoints, the client was being closed before
the streaming generator could use it, causing "client has been closed" errors.

This change introduces a shared httpx.AsyncClient that persists for the application
lifecycle, properly cleaned up during shutdown. This also improves performance by
reusing TCP connections across requests.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-19 16:52:30 +08:00
hetaoBackend 8ea530e221 fix: stop tracking .claude/settings.local.json
Remove .claude/settings.local.json from git tracking and add it to .gitignore to prevent future accidental commits of local settings.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-19 16:28:03 +08:00
hetaoBackend 3a4149c437 fix: stop tracking .claude/settings.local.json
Remove .claude/settings.local.json from git tracking and add it to .gitignore to prevent future accidental commits of local settings.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-19 16:28:03 +08:00
DanielWalnut 9a3eaea54e feat: implement summarization (#14) 2026-01-19 16:17:31 +08:00
DanielWalnut f0a2381bd5 feat: implement summarization (#14) 2026-01-19 16:17:31 +08:00
Henry Li 1ef04c94ee fix: fix getBackendBaseURL() 2026-01-19 15:42:19 +08:00
Henry Li 1352b0e0ba fix: fix getBackendBaseURL() 2026-01-19 15:42:19 +08:00
Henry Li f3f66ee924 feat: add NEXT_PUBLIC_BACKEND_BASE_URL 2026-01-19 11:23:40 +08:00
Henry Li 9d18e4e12d feat: add NEXT_PUBLIC_BACKEND_BASE_URL 2026-01-19 11:23:40 +08:00
Henry Li d8391ca3ea feat: make new chat always on top 2026-01-19 00:00:35 +08:00
Henry Li b431567666 feat: make new chat always on top 2026-01-19 00:00:35 +08:00
Henry Li 63fa500716 fix: decode URL 2026-01-18 20:26:01 +08:00
Henry Li c321c9293a fix: decode URL 2026-01-18 20:26:01 +08:00
Henry Li dc04042b53 feat: support clarification tool 2026-01-18 20:17:32 +08:00
Henry Li 5624b0cd38 feat: support clarification tool 2026-01-18 20:17:32 +08:00
Henry Li 69b225082b feat: re-implement message group 2026-01-18 19:56:07 +08:00
Henry Li aa44566fef feat: re-implement message group 2026-01-18 19:56:07 +08:00
DanielWalnut 645923c3bc feat: add clarification feature (#13) 2026-01-18 19:55:36 +08:00
DanielWalnut e1a8d544b6 feat: add clarification feature (#13) 2026-01-18 19:55:36 +08:00
Henry Li dd80348b76 feat: support SSE write_file(0 2026-01-18 17:13:15 +08:00
Henry Li ec1964c829 feat: support SSE write_file(0 2026-01-18 17:13:15 +08:00
DanielWalnut c50540e3fc fix: Long thinking but with empty content (#12) 2026-01-18 14:21:40 +08:00
DanielWalnut 6f97dde5d1 fix: Long thinking but with empty content (#12) 2026-01-18 14:21:40 +08:00
DanielWalnut 1397f30f24 feat: implement lazy sandbox and thread data initialization (#11)
Defer sandbox acquisition and thread directory creation until first use to improve performance and reduce resource usage.

Changes:
- Add lazy_init parameter to SandboxMiddleware (default: true)
- Add ensure_sandbox_initialized() helper for lazy sandbox acquisition
- Update all sandbox tools to use lazy initialization
- Add lazy_init parameter to ThreadDataMiddleware (default: true)
- Create thread directories on-demand in AioSandboxProvider
- LocalSandbox already creates directories on write (no changes needed)

Benefits:
- Saves 1-2s Docker container startup for conversations without tools
- Reduces unnecessary directory creation and file system operations
- Backward compatible with lazy_init=false option

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-18 13:38:34 +08:00
DanielWalnut 5f4c58aa82 feat: implement lazy sandbox and thread data initialization (#11)
Defer sandbox acquisition and thread directory creation until first use to improve performance and reduce resource usage.

Changes:
- Add lazy_init parameter to SandboxMiddleware (default: true)
- Add ensure_sandbox_initialized() helper for lazy sandbox acquisition
- Update all sandbox tools to use lazy initialization
- Add lazy_init parameter to ThreadDataMiddleware (default: true)
- Create thread directories on-demand in AioSandboxProvider
- LocalSandbox already creates directories on write (no changes needed)

Benefits:
- Saves 1-2s Docker container startup for conversations without tools
- Reduces unnecessary directory creation and file system operations
- Backward compatible with lazy_init=false option

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-18 13:38:34 +08:00
Henry Li 8f0bd828d5 feat: add recursion_limit 2026-01-18 13:16:27 +08:00
Henry Li 41a22fde91 feat: add recursion_limit 2026-01-18 13:16:27 +08:00
Henry Li 6bf187c1c2 fix: fix message grouping issues 2026-01-18 13:07:56 +08:00
Henry Li 71eadc942f fix: fix message grouping issues 2026-01-18 13:07:56 +08:00
DanielWalnut bfe8a24350 fix: fix backend python execution (#10) 2026-01-18 12:41:48 +08:00
DanielWalnut 5a0912d0fd fix: fix backend python execution (#10) 2026-01-18 12:41:48 +08:00
Henry Li 23dc64fab1 feat: enhance message display 2026-01-18 11:25:46 +08:00
Henry Li 9605cec6d3 feat: enhance message display 2026-01-18 11:25:46 +08:00
Henry Li 59683fc12e feat: dim the placeholder 2026-01-18 09:57:30 +08:00
Henry Li f9242727c7 feat: dim the placeholder 2026-01-18 09:57:30 +08:00
Henry Li 54f58fd7eb Merge remote-tracking branch 'refs/remotes/origin/experimental' into experimental 2026-01-18 09:57:23 +08:00
Henry Li bffe802130 Merge remote-tracking branch 'refs/remotes/origin/experimental' into experimental 2026-01-18 09:57:23 +08:00
Henry Li 92fc19a3aa feat: remove model icon 2026-01-18 09:55:17 +08:00
Henry Li 449f04fc44 feat: remove model icon 2026-01-18 09:55:17 +08:00
DanielWalnut aa030410fc feat: fix todos (#9) 2026-01-17 23:23:12 +08:00
DanielWalnut 3261273ee3 feat: fix todos (#9) 2026-01-17 23:23:12 +08:00
Henry Li 3f1f6af30c feat: change back to 60px height 2026-01-17 22:01:17 +08:00
Henry Li fa07e9e903 feat: change back to 60px height 2026-01-17 22:01:17 +08:00
Henry Li 7ea7a7864e feat: use default sidebar width 2026-01-17 22:01:08 +08:00
Henry Li d0988b3cf0 feat: use default sidebar width 2026-01-17 22:01:08 +08:00
Loganaden Velvindron 2ed0eeb107 fix(docker): nodejs CVE-2025-59466 (#818)
https://nodejs.org/en/blog/vulnerability/january-2026-dos-mitigation-async-hooks
2026-01-17 21:41:21 +08:00
Henry Li caf761be59 fix: fix z index 2026-01-17 21:34:32 +08:00
Henry Li 88eb341115 fix: fix z index 2026-01-17 21:34:32 +08:00
Henry Li 5cda2b90fc feat: refine theme 2026-01-17 21:24:49 +08:00
Henry Li 36b7ac0ce4 feat: refine theme 2026-01-17 21:24:49 +08:00
Henry Li 70cd664d3f feat: adjust dark theme 2026-01-17 21:08:05 +08:00
Henry Li 00fc70536e feat: adjust dark theme 2026-01-17 21:08:05 +08:00
Henry Li 32a77cce84 feat: the DeerFlow theme is back 2026-01-17 20:59:42 +08:00
Henry Li 6c9b0f275b feat: the DeerFlow theme is back 2026-01-17 20:59:42 +08:00
Henry Li 094553ea42 feat: change light theme 2026-01-17 20:32:27 +08:00
Henry Li 79d87de523 feat: change light theme 2026-01-17 20:32:27 +08:00
Henry Li df65010e5f fix: remove unused imports 2026-01-17 19:47:51 +08:00
Henry Li e418eb6110 fix: remove unused imports 2026-01-17 19:47:51 +08:00
Henry Li 2bc5f30c4d feat: welcome, again 2026-01-17 19:46:02 +08:00
Henry Li a6e5ebe898 feat: welcome, again 2026-01-17 19:46:02 +08:00
Henry Li 06068dd07b feat: add reasoning check to message list item rendering 2026-01-17 18:02:19 +08:00
Henry Li 0ea448a220 feat: add reasoning check to message list item rendering 2026-01-17 18:02:19 +08:00
Henry Li b705a44f3c feat: pull up the input box when creating new thread 2026-01-17 18:02:01 +08:00
Henry Li cb54b5dffa feat: pull up the input box when creating new thread 2026-01-17 18:02:01 +08:00
Henry Li 85d9baf2b1 feat:enhance focus status 2026-01-17 17:52:15 +08:00
Henry Li 9bfa49ae07 feat:enhance focus status 2026-01-17 17:52:15 +08:00
Henry Li a64b0d226b feat: redesign step counter 2026-01-17 17:45:13 +08:00
Henry Li 62921ec96a feat: redesign step counter 2026-01-17 17:45:13 +08:00
Henry Li 97dbcc4bd6 fix: remove unused imports 2026-01-17 17:37:36 +08:00
Henry Li 63f3c9e2bb fix: remove unused imports 2026-01-17 17:37:36 +08:00
Henry Li d8f0f91238 feat: extract ThreadTitle component 2026-01-17 17:37:12 +08:00
Henry Li 7b33214a05 feat: extract ThreadTitle component 2026-01-17 17:37:12 +08:00
Henry Li f1c6991194 feat: integrated with artifacts in states 2026-01-17 17:21:37 +08:00
Henry Li 9a3f72869c feat: integrated with artifacts in states 2026-01-17 17:21:37 +08:00
Henry Li 384353d613 feat: remove ring 2026-01-17 17:21:05 +08:00
Henry Li ab65ab3af2 feat: remove ring 2026-01-17 17:21:05 +08:00
Henry Li a66d515214 chore: add TODO for checking duplicate files in state.artifacts 2026-01-17 16:25:51 +08:00
Henry Li d603771291 chore: add TODO for checking duplicate files in state.artifacts 2026-01-17 16:25:51 +08:00
Henry Li 1c74e9996f dos: update backend TODOs 2026-01-17 16:17:59 +08:00
Henry Li 1d3b3e8eb8 dos: update backend TODOs 2026-01-17 16:17:59 +08:00
Henry Li a663bcc37b feat: merge the last thinking with the previous group 2026-01-17 16:10:58 +08:00
Henry Li 1a3b70ac43 feat: merge the last thinking with the previous group 2026-01-17 16:10:58 +08:00
Henry Li 584eed0166 fix: do not display 'Untitled' 2026-01-17 15:48:43 +08:00
Henry Li be1e016ed4 fix: do not display 'Untitled' 2026-01-17 15:48:43 +08:00
Henry Li 56da1c990a feat: implement '/chats' 2026-01-17 15:44:49 +08:00
Henry Li e2d0246827 feat: implement '/chats' 2026-01-17 15:44:49 +08:00
Henry Li 228ec49f70 feat: add date time util 2026-01-17 15:44:38 +08:00
Henry Li c38dfdf0e0 feat: add date time util 2026-01-17 15:44:38 +08:00
Henry Li 0e8fdf6234 feat: shrink card size 2026-01-17 15:22:00 +08:00
Henry Li 31510879f2 feat: shrink card size 2026-01-17 15:22:00 +08:00
Henry Li 5dc40a9ade feat: add open in new window 2026-01-17 15:19:53 +08:00
Henry Li aa2677e9fd feat: add open in new window 2026-01-17 15:19:53 +08:00
Henry Li 962d8f04ec feat: support artifact preview 2026-01-17 15:09:44 +08:00
Henry Li 0c6f8353bf feat: support artifact preview 2026-01-17 15:09:44 +08:00
Henry Li ec5bbf6b51 feat: set artifacts layout 2026-01-17 11:02:33 +08:00
Henry Li 80c928fcf5 feat: set artifacts layout 2026-01-17 11:02:33 +08:00
Henry Li 9d1cf89532 chore: remove unused components 2026-01-17 10:09:43 +08:00
Henry Li 5d23dd0763 chore: remove unused components 2026-01-17 10:09:43 +08:00
Henry Li a973c82a1f chore: downgrade shiki since breaking changes 2026-01-17 10:05:55 +08:00
Henry Li 58ff7f8876 chore: downgrade shiki since breaking changes 2026-01-17 10:05:55 +08:00
Henry Li 664ccb922f style: format 2026-01-17 00:13:24 +08:00
Henry Li dfe8f58c16 style: format 2026-01-17 00:13:24 +08:00
Henry Li 4e7256a9d8 feat: make BETTER_AUTH_* optional 2026-01-17 00:13:17 +08:00
Henry Li 9a4cb616c9 feat: make BETTER_AUTH_* optional 2026-01-17 00:13:17 +08:00
Henry Li bb92dec8d5 feat: ignore components from 3rd parties 2026-01-17 00:12:57 +08:00
Henry Li 1a99ae9c36 feat: ignore components from 3rd parties 2026-01-17 00:12:57 +08:00
Henry Li 4613d6e16e refactor: rename 2026-01-17 00:05:19 +08:00
Henry Li c216093360 refactor: rename 2026-01-17 00:05:19 +08:00
Henry Li 9d64c7e076 feat: integrated with artifacts 2026-01-17 00:02:03 +08:00
Henry Li e5050c6c1e feat: integrated with artifacts 2026-01-17 00:02:03 +08:00
Henry Li 34ca58ed1b fix: fix broken when SSE 2026-01-16 23:15:53 +08:00
Henry Li 16a5ed9a73 fix: fix broken when SSE 2026-01-16 23:15:53 +08:00
DanielWalnut facde645d7 feat: add artifacts logic (#8) 2026-01-16 23:04:38 +08:00
DanielWalnut d5b3052cda feat: add artifacts logic (#8) 2026-01-16 23:04:38 +08:00
Henry Li 6464a67230 feat: remember sidebar state 2026-01-16 23:03:39 +08:00
Henry Li 0d11b21c84 feat: remember sidebar state 2026-01-16 23:03:39 +08:00
Henry Li f9853f037c feat: support basic file presenting 2026-01-16 22:35:20 +08:00
Henry Li 83f367b98a feat: support basic file presenting 2026-01-16 22:35:20 +08:00
DanielWalnut 4b69aed47b feat: add thread-safety and graceful shutdown to AioSandboxProvider (#7)
Add thread-safe port allocation and proper cleanup on process exit to
prevent port conflicts in concurrent environments and ensure containers
are stopped when the application terminates.

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-16 22:28:19 +08:00
DanielWalnut 50a1e407cf feat: add thread-safety and graceful shutdown to AioSandboxProvider (#7)
Add thread-safe port allocation and proper cleanup on process exit to
prevent port conflicts in concurrent environments and ensure containers
are stopped when the application terminates.

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-16 22:28:19 +08:00
Henry Li c0e63c5308 docs: rewording 2026-01-16 22:10:08 +08:00
Henry Li d2845f658f docs: rewording 2026-01-16 22:10:08 +08:00
Henry Li 93a231cfb1 feat: integrated with artifact resizable 2026-01-16 21:55:31 +08:00
Henry Li ca70e2dcf7 feat: integrated with artifact resizable 2026-01-16 21:55:31 +08:00
Henry Li 68fbf53fb2 chore: add resizable 2026-01-16 21:54:54 +08:00
Henry Li e1ddb1ee42 chore: add resizable 2026-01-16 21:54:54 +08:00
Henry Li 1517e8675d feat: add present_file tool 2026-01-16 21:48:00 +08:00
Henry Li 56b26c060e feat: add present_file tool 2026-01-16 21:48:00 +08:00
Henry Li 91eff99f01 feat: add flip display effect 2026-01-16 20:40:09 +08:00
Henry Li e37be40773 feat: add flip display effect 2026-01-16 20:40:09 +08:00
Henry Li c265734c6e feat: adjust layout 2026-01-16 20:06:30 +08:00
Henry Li 6e5dab76cc feat: adjust layout 2026-01-16 20:06:30 +08:00
Henry Li 03f0e3f0c7 refactor: move biz logic to core 2026-01-16 19:51:39 +08:00
Henry Li ce70b67459 refactor: move biz logic to core 2026-01-16 19:51:39 +08:00
Henry Li 7066a3b691 feat: adjust layout and style of tooltip 2026-01-16 19:51:14 +08:00
Henry Li f6c20dbcfe feat: adjust layout and style of tooltip 2026-01-16 19:51:14 +08:00
Henry Li b72eb61302 refactor: simplify parameter 2026-01-16 19:50:41 +08:00
Henry Li a8ba2de579 refactor: simplify parameter 2026-01-16 19:50:41 +08:00
Henry Li df396fc246 feat: add copy button 2026-01-16 19:50:23 +08:00
Henry Li 574b7e59ce feat: add copy button 2026-01-16 19:50:23 +08:00
Henry Li 6bd49ab411 chore: remove 2026-01-16 19:50:17 +08:00
Henry Li b5dcb2b1d3 chore: remove 2026-01-16 19:50:17 +08:00
DanielWalnut 9f755ecc30 feat: add skills system for specialized agent workflows (#6)
Implement a skills framework that enables specialized workflows for
specific tasks (e.g., PDF processing, web page generation). Skills are
discovered from the skills/ directory and automatically mounted in
sandboxes with path mapping support.

- Add SkillsConfig for configuring skills path and container mount point
- Implement dynamic skill loading from SKILL.md files with YAML frontmatter
- Add path mapping in LocalSandbox to translate container paths to local paths
- Mount skills directory in AIO Docker sandbox containers
- Update lead agent prompt to dynamically inject available skills
- Add setup documentation and expand config.example.yaml

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-16 14:44:51 +08:00
DanielWalnut cfa97f7a96 feat: add skills system for specialized agent workflows (#6)
Implement a skills framework that enables specialized workflows for
specific tasks (e.g., PDF processing, web page generation). Skills are
discovered from the skills/ directory and automatically mounted in
sandboxes with path mapping support.

- Add SkillsConfig for configuring skills path and container mount point
- Implement dynamic skill loading from SKILL.md files with YAML frontmatter
- Add path mapping in LocalSandbox to translate container paths to local paths
- Mount skills directory in AIO Docker sandbox containers
- Update lead agent prompt to dynamically inject available skills
- Add setup documentation and expand config.example.yaml

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-16 14:44:51 +08:00
Henry Li f19e3ae8ac fix: lastStep could be empty 2026-01-16 14:38:49 +08:00
Henry Li 5ef3cb57ee fix: lastStep could be empty 2026-01-16 14:38:49 +08:00
Henry Li 52b9d0cffc feat: remove scroll button 2026-01-16 14:38:33 +08:00
Henry Li a589fb3dae feat: remove scroll button 2026-01-16 14:38:33 +08:00
Henry Li 2105170d39 Merge remote-tracking branch 'refs/remotes/origin/experimental' into experimental 2026-01-16 14:04:21 +08:00
Henry Li d8b7d06e99 Merge remote-tracking branch 'refs/remotes/origin/experimental' into experimental 2026-01-16 14:04:21 +08:00
Henry Li faf80bb429 feat: rename 'model' to 'model_name' 2026-01-16 14:03:34 +08:00
Henry Li ac075477a0 feat: rename 'model' to 'model_name' 2026-01-16 14:03:34 +08:00
DanielWalnut 7284eb15f1 feat: add gateway module with FastAPI server (#5)
* chore: add .claude/ to .gitignore

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add gateway module with FastAPI server

- Add new gateway module with FastAPI app for API routing
- Add gateway and serve commands to Makefile
- Add fastapi, httpx, uvicorn, sse-starlette dependencies
- Fix model config retrieval in lead_agent (support both model_name and model)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-16 13:22:26 +08:00
DanielWalnut fb92a472e2 feat: add gateway module with FastAPI server (#5)
* chore: add .claude/ to .gitignore

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add gateway module with FastAPI server

- Add new gateway module with FastAPI app for API routing
- Add gateway and serve commands to Makefile
- Add fastapi, httpx, uvicorn, sse-starlette dependencies
- Fix model config retrieval in lead_agent (support both model_name and model)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-16 13:22:26 +08:00
Henry Li 1f03fb3749 fix: navigate to home only in open-mode 2026-01-16 09:58:16 +08:00
Henry Li 5c94e6d222 fix: navigate to home only in open-mode 2026-01-16 09:58:16 +08:00
Henry Li 7c6189668c feat: link to home page 2026-01-16 09:56:30 +08:00
Henry Li 5fa98bf6cd feat: link to home page 2026-01-16 09:56:30 +08:00
Henry Li 028f402ff5 feat: store the local settings 2026-01-16 09:55:02 +08:00
Henry Li 3a62deb3fd feat: store the local settings 2026-01-16 09:55:02 +08:00
Henry Li 3f2bfded41 feat: enable edit context options 2026-01-16 09:37:04 +08:00
Henry Li cad12068ef feat: enable edit context options 2026-01-16 09:37:04 +08:00
Henry Li 956f8f0cfb refactor: rename 2026-01-16 09:15:04 +08:00
Henry Li c5b4c90b83 refactor: rename 2026-01-16 09:15:04 +08:00
Henry Li 08e0a1da16 chore: remove 2026-01-16 09:14:25 +08:00
Henry Li 608b703d26 chore: remove 2026-01-16 09:14:25 +08:00
Henry Li e9846c1dda refactor: refine folder structure and rename 2026-01-16 09:13:02 +08:00
Henry Li fe7504daed refactor: refine folder structure and rename 2026-01-16 09:13:02 +08:00
Henry Li 61499624a0 feat: adjust message group layout 2026-01-15 23:56:42 +08:00
Henry Li 7680a5adba feat: adjust message group layout 2026-01-15 23:56:42 +08:00
Henry Li 00ad4206c4 feat: enhance label 2026-01-15 23:47:36 +08:00
Henry Li f353831ac9 feat: enhance label 2026-01-15 23:47:36 +08:00
Henry Li c3cb4c348d feat: remove max-w- 2026-01-15 23:47:28 +08:00
Henry Li d45f48adde feat: remove max-w- 2026-01-15 23:47:28 +08:00
Henry Li 9f2b94ed52 feat: implement basic web app 2026-01-15 23:40:21 +08:00
Henry Li cecc684de1 feat: implement basic web app 2026-01-15 23:40:21 +08:00
DanielWalnut b44144dd2c feat: support function factory (#4) 2026-01-15 22:05:54 +08:00
DanielWalnut c7d68c6d3f feat: support function factory (#4) 2026-01-15 22:05:54 +08:00
DanielWalnut a39f799a7e fix: fix local path for local sandbox (#3) 2026-01-15 14:37:00 +08:00
DanielWalnut 3b879e277e fix: fix local path for local sandbox (#3) 2026-01-15 14:37:00 +08:00
DanielWalnut c92eedc572 feat: add thread data middleware (#2) 2026-01-15 13:22:30 +08:00
DanielWalnut 41442ccc2f feat: add thread data middleware (#2) 2026-01-15 13:22:30 +08:00
DanielWalnut ab427731dc feat: add AIO sandbox provider and auto title generation (#1)
- Add AioSandboxProvider for Docker-based sandbox execution with
  configurable container lifecycle, volume mounts, and port management
- Add TitleMiddleware to auto-generate thread titles after first
  user-assistant exchange using LLM
- Add Claude Code documentation (CLAUDE.md, AGENTS.md)
- Extend SandboxConfig with Docker-specific options (image, port, mounts)
- Fix hardcoded mount path to use expanduser
- Add agent-sandbox and dotenv dependencies

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 23:29:18 +08:00
DanielWalnut b2abfecf67 feat: add AIO sandbox provider and auto title generation (#1)
- Add AioSandboxProvider for Docker-based sandbox execution with
  configurable container lifecycle, volume mounts, and port management
- Add TitleMiddleware to auto-generate thread titles after first
  user-assistant exchange using LLM
- Add Claude Code documentation (CLAUDE.md, AGENTS.md)
- Extend SandboxConfig with Docker-specific options (image, port, mounts)
- Fix hardcoded mount path to use expanduser
- Add agent-sandbox and dotenv dependencies

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 23:29:18 +08:00
Henry Li de2d18561a feat: integrated with sandbox 2026-01-14 12:32:34 +08:00
Henry Li c1e9340062 chore: use .env 2026-01-14 12:32:15 +08:00
Henry Li 57dfc89ca1 chore: specify project name 2026-01-14 09:58:53 +08:00
Henry Li ce0b6f7754 chore: specify project title 2026-01-14 09:57:52 +08:00
Henry Li 5d6a7442d6 chore: remove tests 2026-01-14 09:52:34 +08:00
Henry Li c628c7f8db chore: create frontend project from boilerplate 2026-01-14 09:50:26 +08:00
Henry Li 3ff7613dd9 chore: mark backend folder as a Python project 2026-01-14 09:24:33 +08:00
Henry Li 2aeaf7c965 style: format 2026-01-14 09:21:19 +08:00
Henry Li 421488a991 chore: add lint and format 2026-01-14 09:20:05 +08:00
Henry Li 2e3a50d847 chore: update workspace structure 2026-01-14 09:19:54 +08:00
Henry Li e5c69cb7ee docs: update tool docs 2026-01-14 09:12:03 +08:00
Henry Li cb611f9270 chore: use ruff to lint and auto-format 2026-01-14 09:08:20 +08:00
Henry Li 7dc063ba25 feat: add agents 2026-01-14 07:20:00 +08:00
Henry Li cbbbac0c2b feat: add tools 2026-01-14 07:19:43 +08:00
Henry Li 57a02acb59 feat: add sandbox and local impl 2026-01-14 07:19:34 +08:00
Henry Li 4b5f529903 feat: integrated with Tavily and Jina AI 2026-01-14 07:17:22 +08:00
Henry Li 83bd7e4309 feat: add model modules 2026-01-14 07:16:45 +08:00
Henry Li 721b26a32f chore: add an empty __init__.py 2026-01-14 07:16:27 +08:00
Henry Li 86524a65f6 feat: add reflection modules 2026-01-14 07:16:07 +08:00
Henry Li 88ed3841c7 feat: add config modules 2026-01-14 07:15:58 +08:00
Henry Li c2a62a2266 chore: add Python and LangGraph stuff 2026-01-14 07:15:02 +08:00
Henry Li 81bd4dafa8 chore: add .gitignore for Python project 2026-01-14 07:14:00 +08:00
Henry Li dd545cfb97 chore: init 2026-01-14 07:09:20 +08:00
Willem Jiang 6b73a53999 fix(config): Add support for MCP server configuration parameters (#812)
* fix(config): Add support for MCP server configuration parameters

* refact: rename the sse_readtimeout to sse_read_timeout

* update the code with review comments

* update the MCP document for the latest change
2026-01-10 15:59:49 +08:00
Willem Jiang e52e69bdd4 fix(frontend):eliminating the empty divider issue on the frontend (#811)
* fix(frontend):eliminating the empty divider issue on the frontend

* Update the store.test.ts for the new changes
2026-01-09 23:34:07 +08:00
Willem Jiang 336040310c fix(frontend): passing the MCP header and env setting to backend (#810)
This pull request adds support for custom HTTP headers to the MCP server configuration and ensures that these headers are properly validated and included when adding new MCP servers. The changes are primarily focused on extending the schema and data handling for MCP server metadata.
2026-01-09 22:52:49 +08:00
MirzaSamadAhmedBaig 8c59f63d1b Fix message validation JSON import (#809)
* Fix message validation JSON import

* Update src/utils/context_manager.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-09 22:38:19 +08:00
Willem Jiang a376b0cb4e fix: Add runtime parameter to compress_messages method(#803)
* fix: Add runtime parameter to compress_messages method(#803)

    The compress_messages method was being called by PreModelHookMiddleware
    with both state and runtime parameters, but only accepted state parameter.
    This caused a TypeError when the middleware executed the pre_model_hook.

    Added optional runtime parameter to compress_messages signature to match
    the expected interface while maintaining backward compatibility.

* Update the code with the review comments
2026-01-07 20:36:15 +08:00
Willem Jiang d4ab77de5c fix: migrate from deprecated create_react_agent to langchain.agents.create_agent (#802)
* fix: migrate from deprecated create_react_agent to langchain.agents.create_agent

Fixes #799

- Replace deprecated langgraph.prebuilt.create_react_agent with
  langchain.agents.create_agent (LangGraph 1.0 migration)
- Add DynamicPromptMiddleware to handle dynamic prompt templates
  (replaces the 'prompt' callable parameter)
- Add PreModelHookMiddleware to handle pre-model hooks
  (replaces the 'pre_model_hook' parameter)
- Update AgentState import from langchain.agents in template.py
- Update tests to use the new API

* fix:update the code with review comments
2026-01-07 09:06:16 +08:00
Willem Jiang 1ced90b055 fix(frontend):added the display of the 'analyst' message #800 (#801) 2026-01-06 20:43:04 +08:00
dependabot[bot] 7e10b105ca build(deps): bump jspdf from 3.0.4 to 4.0.0 in /web (#798)
Bumps [jspdf](https://github.com/parallax/jsPDF) from 3.0.4 to 4.0.0.
- [Release notes](https://github.com/parallax/jsPDF/releases)
- [Changelog](https://github.com/parallax/jsPDF/blob/master/RELEASE.md)
- [Commits](https://github.com/parallax/jsPDF/compare/v3.0.4...v4.0.0)

---
updated-dependencies:
- dependency-name: jspdf
  dependency-version: 4.0.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-06 09:07:18 +08:00
Willem Jiang 7ebbb53b57 fix(frontend): render all tool calls in the frontend #796 (#797) 2026-01-05 22:24:52 +08:00
Willem Jiang 275aab9d42 fix(log): Enable the logging level when enabling the DEBUG environment variable (#793) 2026-01-01 09:32:42 +08:00
Willem Jiang a71b6bc41f fix(main): Passing the local parameter from the main interactive mode (#791) 2025-12-30 10:41:29 +08:00
YMG001 893ff82a7f fix(workflow): resolve locale hardcoding in src/workflow.py for interactive mode (#789) 2025-12-30 09:47:39 +08:00
Willem Jiang 5087d5012f fix(deps): update langchain-core to 1.2.5 to resolve CVE-2025-68664 (#787) 2025-12-27 21:36:17 +08:00
Willem Jiang bab60e6e3d fix(podcast): add fallback for models without json_object support (#747) (#785)
* fix(podcast): add fallback for models without json_object support (#747)

Models like Kimi K2 don't support response_format.type: json_object.
Add try-except to fall back to regular prompting with JSON parsing
when BadRequestError mentions json_object not supported.

- Add fallback to prompting + repair_json_output parsing
- Re-raise other BadRequestError types
- Add unit tests for script_writer_node with 100% coverage

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fixes: the unit test error of test_script_writer_node.py

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-26 23:04:20 +08:00
Willem Jiang 5a79f896c4 fix(metrics): update the polynomial regular expression used on uncontrolled data (#784) 2025-12-26 10:10:12 +08:00
Willem Jiang cd5c4877f3 fix(web): enable runtime API URL detection for cross-machine access (#777) (#783)
When NEXT_PUBLIC_API_URL is not explicitly configured, the frontend now
automatically detects the API URL based on the current page's hostname.
This allows accessing DeerFlow from other machines without rebuilding
the frontend.

- Add getBaseURL() helper with runtime window.location detection
- Use same protocol and hostname with default port 8000
- Preserve explicit NEXT_PUBLIC_API_URL configuration when set
- Fallback to localhost:8000 for SSR scenarios
2025-12-25 22:34:26 +08:00
Willem Jiang 8d9d767051 feat(eval): add report quality evaluation module and UI integration (#776)
* feat(eval): add report quality evaluation module

Addresses issue #773 - How to evaluate generated report quality objectively.

This module provides two evaluation approaches:
1. Automated metrics (no LLM required):
   - Citation count and source diversity
   - Word count compliance per report style
   - Section structure validation
   - Image inclusion tracking

2. LLM-as-Judge evaluation:
   - Factual accuracy scoring
   - Completeness assessment
   - Coherence evaluation
   - Relevance and citation quality checks

The combined evaluator provides a final score (1-10) and letter grade (A+ to F).

Files added:
- src/eval/__init__.py
- src/eval/metrics.py
- src/eval/llm_judge.py
- src/eval/evaluator.py
- tests/unit/eval/test_metrics.py
- tests/unit/eval/test_evaluator.py

* feat(eval): integrate report evaluation with web UI

This commit adds the web UI integration for the evaluation module:

Backend:
- Add EvaluateReportRequest/Response models in src/server/eval_request.py
- Add /api/report/evaluate endpoint to src/server/app.py

Frontend:
- Add evaluateReport API function in web/src/core/api/evaluate.ts
- Create EvaluationDialog component with grade badge, metrics display,
  and optional LLM deep evaluation
- Add evaluation button (graduation cap icon) to research-block.tsx toolbar
- Add i18n translations for English and Chinese

The evaluation UI allows users to:
1. View quick metrics-only evaluation (instant)
2. Optionally run deep LLM-based evaluation for detailed analysis
3. See grade (A+ to F), score (1-10), and metric breakdown

* feat(eval): improve evaluation reliability and add LLM judge tests

- Extract MAX_REPORT_LENGTH constant in llm_judge.py for maintainability
- Add comprehensive unit tests for LLMJudge class (parse_response,
  calculate_weighted_score, evaluate with mocked LLM)
- Pass reportStyle prop to EvaluationDialog for accurate evaluation criteria
- Add researchQueries store map to reliably associate queries with research
- Add getResearchQuery helper to retrieve query by researchId
- Remove unused imports in test_metrics.py

* fix(eval): use resolveServiceURL for evaluate API endpoint

The evaluateReport function was using a relative URL '/api/report/evaluate'
which sent requests to the Next.js server instead of the FastAPI backend.
Changed to use resolveServiceURL() consistent with other API functions.

* fix: improve type accuracy and React hooks in evaluation components

- Fix get_word_count_target return type from Optional[Dict] to Dict since it always returns a value via default fallback
- Fix useEffect dependency issue in EvaluationDialog using useRef to prevent unwanted re-evaluations
- Add aria-label to GradeBadge for screen reader accessibility
2025-12-25 21:55:48 +08:00
geniusroad 84a7f7815c refactor(graph): Refactor tool loading logic within nodes (#782)
* refactor(graph): Optimize tool loading logic within nodes

- Pre-copy the default tool list during initialization
- Merge MCP server configuration with default tool handling
- Simplify conditional branches and unify agent creation logic
- Remove duplicated agent creation code blocks

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-25 21:10:04 +08:00
Willem Jiang fb319aaa44 test: add unit tests for global connection pool (Issue #778) (#780)
* test: add unit tests for global connection pool (Issue #778)

- Add TestLifespanFunction class with 9 tests for lifespan management:
  - PostgreSQL/MongoDB pool initialization success/failure
  - Cleanup on shutdown
  - Skip initialization when not configured

- Add TestGlobalConnectionPoolUsage class with 4 tests:
  - Using global pools when available
  - Fallback to per-request connections

- Fix missing dict_row import in app.py (bug from PR #757)

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-23 23:06:39 +08:00
YikB 83e9d7c9e5 feat:Database connections use connection pools (#757)
* feat: Implement DeerFlow API server with chat streaming, Langgraph orchestration, and various content generation capabilities.

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-23 20:35:08 +08:00
Loganaden Velvindron 1f403a9f79 Fix typo in vulnerability reporting instructions (#772) 2025-12-21 17:07:13 +08:00
Willem Jiang 4dde77986a Added the security policy 2025-12-21 09:12:02 +08:00
Willem Jiang 04296cdf5a feat: add resource upload support for RAG (#768)
* feat: add resource upload support for RAG

- Backend: Added ingest_file method to Retriever and MilvusRetriever
- Backend: Added /api/rag/upload endpoint
- Frontend: Added RAGTab in settings for uploading resources
- Frontend: Updated translations and settings registration

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply suggestions from code review

* Apply suggestions from code review of src/rag/milvus.py

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-19 09:55:34 +08:00
Jiahe Wu 3e8f2ce3ad feat(web): add enable_web_search frontend UI (#681) (#766)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-12-17 23:36:32 +08:00
Jiahe Wu b85130b849 fix: display direct_response message in frontend (#763) (#764)
Extract message content from direct_response tool call args
and display it as the message content when tool call completes.

Note: This is a workaround. The message is not streamed because
direct_response uses tool calling mechanism where args are JSON,
not natural language text that can be streamed directly.
2025-12-17 21:04:37 +08:00
Jiahe Wu a4f64abd1f feat(web): add multi-format report export (Markdown, HTML, PDF, Word,… (#756)
* feat(web): add multi-format report export (Markdown, HTML, PDF, Word, Image)

* fix: correct import order for docx (lint error)

* fix(web): address Copilot review comments for multi-format export

- Add i18n support for dropdown menu items (en/zh)

- Add DOMPurify for HTML sanitization (XSS protection)

- Fix async handling for canvas.toBlob with Promise wrapper

- Add toast notifications for export errors

- Fix Tooltip + DropdownMenuTrigger nesting (accessibility)

- Ensure container cleanup in finally block

* fix(web): enhance markdown parsing for PDF and Word export

- Add list support (bullet and numbered) for PDF export
- Add parseInlineMarkdown helper for Word export to handle bold, italic, code, links
- Add list support for Word export (bullet and numbered)
- Address Copilot review comments from PR #756

* fix(web): address PR review feedback for multi-format export

- Extract PDF formatting magic numbers into PDF_CONSTANTS

- Add Tooltip wrapper for download dropdown button

- Reduce triggerDownload cleanup timeout from 1000ms to 100ms

- Use marked.Lexer.lexInline for robust markdown parsing

- Add console.warn for image export cleanup errors

- Add numbering config for Word document ordered lists

- Fix CSS class typo: px-5pb-20 -> px-5 pb-20

- Remove unreachable dead code in parseInlineMarkdown

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-12-16 09:06:24 +08:00
Willem Jiang 2a97170b6c feat: add Serper search engine support (#762)
* feat: add Serper search engine support

* docs: update configuration guide and env example for Serper

* test: add test case for Serper with missing API key
2025-12-15 23:04:26 +08:00
Jiahe Wu 93d81d450d feat: add enable_web_search config to disable web search (#681) (#760)
* feat: add enable_web_search config to disable web search (#681)

* fix: skip enforce_researcher_search validation when web search is disabled

- Return json.dumps([]) instead of empty string for consistency in background_investigation_node

- Add enable_web_search check to skip validation warning when user intentionally disabled web search

- Add warning log when researcher has no tools available

- Update tests to include new enable_web_search parameter

* fix: address Copilot review feedback

- Coordinate enforce_web_search with enable_web_search in validate_and_fix_plan

- Fix misleading comment in background_investigation_node

* docs: add warning about local RAG setup when disabling web search

* docs: add web search toggle section to configuration guide
2025-12-15 19:17:24 +08:00
Jiahe Wu c686ab7016 fix: handle greetings without triggering research workflow (#755)
* fix: handle greetings without triggering research workflow (#733)

* test: update tests for direct_response tool behavior

* fix: address Copilot review comments for coordinator_node - Extract locale from direct_response tool_args - Fix import sorting (ruff I001)

* fix: remove locale extraction from tool_args in direct_response

Use locale from state instead of tool_args to avoid potential side effects. The locale is already properly passed from frontend via state.

* fix: only fallback to planner when clarification is enabled

In legacy mode (BRANCH 1), no tool calls should end the workflow gracefully instead of falling back to planner. This fixes the test_coordinator_node_no_tool_calls integration test.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-12-13 20:25:46 +08:00
dependabot[bot] a6d8deee8b build(deps): bump next from 15.4.8 to 15.4.10 in /web (#758)
Bumps [next](https://github.com/vercel/next.js) from 15.4.8 to 15.4.10.
- [Release notes](https://github.com/vercel/next.js/releases)
- [Changelog](https://github.com/vercel/next.js/blob/canary/release.js)
- [Commits](https://github.com/vercel/next.js/compare/v15.4.8...v15.4.10)

---
updated-dependencies:
- dependency-name: next
  dependency-version: 15.4.10
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-12-12 10:36:47 +08:00
blueberrycongee 4c2592ac85 docs: add more MCP integration examples (#441) (#754) 2025-12-11 21:21:37 +08:00
Willem Jiang ec99338c9a fix(agents): patch _run in ToolInterceptor to ensure interrupt triggering (#753)
Fixes #752

* fix(agents): patch _run in ToolInterceptor to ensure interrupt triggering

* Update the code with review comments
2025-12-10 22:15:08 +08:00
Willem Jiang 84c449cf79 fix(checkpoint): clear in-memory store after successful persistence (#751)
* fix(checkpoint): clear in-memory store after successful persistence

* test(checkpoint): add unit test for memory leak check

* Update tests/unit/checkpoint/test_memory_leak.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-09 23:32:13 +08:00
infoquest-byteplus fde7a69562 Add the InfoQuest banner to the README (#748)
* add readme iq pic

* add readme iq pic

* add readme iq pic

* add readme iq pic

* add readme iq pic

* add readme iq pic
2025-12-08 17:42:04 +08:00
Willem Jiang 3bf4e1defb fix: setup WindowsSelectorEventLoopPolicy in the first place #741 (#742)
* fix: setup WindowsSelectorEventLoopPolicy in the first place #741

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-12-06 22:10:13 +08:00
Willem Jiang 3191e81939 fix: passing the locale to create_react_agent (#745) 2025-12-06 11:15:11 +08:00
dependabot[bot] 21da2630f3 build(deps): bump next from 15.4.7 to 15.4.8 in /web (#744)
Bumps [next](https://github.com/vercel/next.js) from 15.4.7 to 15.4.8.
- [Release notes](https://github.com/vercel/next.js/releases)
- [Changelog](https://github.com/vercel/next.js/blob/canary/release.js)
- [Commits](https://github.com/vercel/next.js/compare/v15.4.7...v15.4.8)

---
updated-dependencies:
- dependency-name: next
  dependency-version: 15.4.8
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-12-05 14:30:00 +08:00
zhangga bd6c50de33 feat:Strip code blocks in plan data. (#738)
* feat:Strip code blocks in plan data.

* feat: add repair_json_output for current_plan_content

---------

Co-authored-by: ryan-gz <ryzhangga1991@gmail.com>
2025-12-04 23:04:29 +08:00
Willem Jiang c36ab393f1 fix: update Interrupt object attribute access for LangGraph 1.0+ (#730) (#731)
* Update uv.lock to sync with pyproject.toml

* fix: update Interrupt object attribute access for LangGraph 1.0+ (#730)

The Interrupt class in LangGraph 1.0 no longer has the 'ns' attribute.
This change updates _create_interrupt_event() to use the new 'id'
attribute instead, with a fallback to thread_id for compatibility.

Changes:
- Replace event_data["__interrupt__"][0].ns[0] with interrupt.id
- Use getattr() with fallback for backward compatibility
- Update debug log message from 'ns=' to 'id='
- Add unit tests for _create_interrupt_event function

* fix the unit test error and address review comment

---------

Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-12-02 11:16:00 +08:00
Willem Jiang e1772d52a9 Clear-text logging of sensitive information (#732) 2025-12-02 09:58:28 +08:00
infoquest-byteplus 7ec9e45702 feat: support infoquest (#708)
* support infoquest

* support html checker

* support html checker

* change line break format

* change line break format

* change line break format

* change line break format

* change line break format

* change line break format

* change line break format

* change line break format

* Fix several critical issues in the codebase
- Resolve crawler panic by improving error handling
- Fix plan validation to prevent invalid configurations
- Correct InfoQuest crawler JSON conversion logic

* add test for infoquest

* add test for infoquest

* Add InfoQuest introduction to the README

* add test for infoquest

* fix readme for infoquest

* fix readme for infoquest

* resolve the conflict

* resolve the conflict

* resolve the conflict

* Fix formatting of INFOQUEST in SearchEngine enum

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-02 08:16:35 +08:00
Willem Jiang e179fb1632 fix(web): handle incomplete JSON in MCP tool call arguments (#528) (#727)
* fix(web): handle incomplete JSON in MCP tool call arguments (#528)

When using stream_mode=["messages", "updates"] with MCP tools, tool call
arguments arrive in chunks that may be incomplete JSON (missing closing
braces). This caused JSON.parse() to throw errors in the frontend.

Changes:
- Add safeParseToolArgs() function using best-effort-json-parser to
  gracefully handle incomplete JSON from streaming
- Replace direct JSON.parse() with safe parser in mergeMessage()
- Add comprehensive tests for tool call argument parsing scenarios

* Address the code review comments
2025-11-29 16:38:29 +08:00
Willem Jiang 4a78cfe12a fix(llm): filter unexpected config keys to prevent LangChain warnings (#411) (#726)
* fix(llm): filter unexpected config keys to prevent LangChain warnings (#411)

Add allowlist validation for LLM configuration keys to prevent unexpected
parameters like SEARCH_ENGINE from being passed to LLM constructors.

Changes:
- Add ALLOWED_LLM_CONFIG_KEYS set with valid LLM configuration parameters
- Filter out unexpected keys before creating LLM instances
- Log clear warning messages when unexpected keys are removed
- Add unit test for configuration key filtering

This fixes the confusing LangChain warning "WARNING! SEARCH_ENGINE is not
default parameter. SEARCH_ENGINE was transferred to model_kwargs" that
occurred when users accidentally placed configuration keys in wrong sections
of conf.yaml.

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-11-29 16:13:05 +08:00
Willem Jiang 2e010a4619 feat: add analysis step type for non-code reasoning tasks (#677) (#723)
Add a new "analysis" step type to handle reasoning and synthesis tasks
that don't require code execution, addressing the concern that routing
all non-search tasks to the coder agent was inappropriate.

Changes:
- Add ANALYSIS enum value to StepType in planner_model.py
- Create analyst_node for pure LLM reasoning without tools
- Update graph routing to route analysis steps to analyst agent
- Add analyst agent to AGENT_LLM_MAP configuration
- Create analyst prompts (English and Chinese)
- Update planner prompts with guidance on choosing between
  analysis (reasoning/synthesis) and processing (code execution)
- Change default step_type inference from "processing" to "analysis"
  when need_search=false

Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-11-29 09:46:55 +08:00
Willem Jiang 2cc19d6309 Update uv.lock to sync with pyproject.toml (#724) 2025-11-28 22:33:49 +08:00
Willem Jiang 170c4eb33c Upgrade langchain version to 1.x (#720)
* fix: revert the part of patch of issue-710 to extract the content from the plan

* Upgrade the ddgs for the new compatible version

* Upgraded langchain to 1.1.0
updated langchain related package to the new compatable version

* Update pyproject.toml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-11-28 22:09:13 +08:00
Willem Jiang b24f4d3f38 fix: apply context compression to prevent token overflow (Issue #721) (#722)
* fix: apply context compression to prevent token overflow (Issue #721)

- Add token_limit configuration to conf.yaml.example for BASIC_MODEL and REASONING_MODEL
- Implement context compression in _execute_agent_step() before agent invocation
- Preserve first 3 messages (system prompt + context) during compression
- Enhance ContextManager logging with better token count reporting
- Prevent 400 Input tokens exceeded errors by automatically compressing message history

* feat: add model-based token limit inference for Issue #721

- Add smart default token limits based on common LLM models
- Support model name inference when token_limit not explicitly configured
- Models include: OpenAI (GPT-4o, GPT-4, etc.), Claude, Gemini, Doubao, DeepSeek, etc.
- Conservative defaults prevent token overflow even without explicit configuration
- Priority: explicit config > model inference > safe default (100,000 tokens)
- Ensures Issue #721 protection for all users, not just those with token_limit set
2025-11-28 18:52:42 +08:00
Willem Jiang 223ec57fe4 fix: the frontend error when cancle the research plan (#719)
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-11-28 08:35:42 +08:00
Willem Jiang 4559197505 fix: revert the part of patch of issue-710 to extract the content from the plan (#718) 2025-11-27 23:59:31 +08:00
Willem Jiang ca4ada5aa7 fix: multiple web_search ToolMessages only showing last result (#717)
* fix: Missing Required Fields in Plan Validation

* fix: the exception of plan validation

* Fixed the test errors

* Addressed the comments of the PR reviews

* fix: multiple web_search ToolMessages only showing last result
2025-11-27 21:47:08 +08:00
Willem Jiang 667916959b fix: the exception of plan validation (#714)
* fix: Missing Required Fields in Plan Validation

* fix: the exception of plan validation

* Fixed the test errors

* Addressed the comments of the PR reviews
2025-11-27 19:39:25 +08:00
Willem Jiang bec97f02ae fix: the crawling error when encountering PDF URLs (#707)
* fix: the crawling error when encountering PDF URLs

* Added the unit test for the new feature of crawl tool

* fix: address the code review problems

* fix: address the code review problems
2025-11-25 09:24:52 +08:00
Willem Jiang da514337da fix: the validation Error with qwen-max-latest Model (#706)
* fix: the validation Error with qwen-max-latest Model

    - Added comprehensive unit tests in tests/unit/graph/test_nodes.py for the new extract_plan_content function
    - Tests cover various input types: string, AIMessage, dictionary, other types
    - Includes a specific test case for issue #703 with the qwen-max-latest model
    - All tests pass successfully, confirming the function handles different input types correctly

* feat: address the code review concerns
2025-11-24 21:13:15 +08:00
Willem Jiang 478291df07 fix: ensure researcher agent uses web search tool instead of generating URLs (#702) (#704)
* fix: ensure researcher agent uses web search tool instead of generating URLs (#702)

- Add enforce_researcher_search configuration option (default: True) to control web search requirement
- Strengthen researcher prompts in both English and Chinese with explicit instructions to use web_search tool
- Implement validate_web_search_usage function to detect if web search tool was used during research
- Add validation logic that warns when researcher doesn't use web search tool
- Enhance logging for web search tools with special markers for easy tracking
- Skip validation during unit tests to avoid test failures
- Update _execute_agent_step to accept config parameter for proper configuration access

This addresses issue #702 where the researcher agent was generating URLs on its own instead of using the web search tool.

* fix: addressed the code review comment

* fix the unit test error and update the code
2025-11-24 20:07:28 +08:00
Copilot cc9414f978 Add unit tests for PPT composer locale handling (#696)
* Initial plan

* Add unit tests for PPT composer localization

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

* Add ppt_content_*.md to .gitignore and remove temp files

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-11-22 17:03:06 +08:00
pizzahan 1bfcf9f429 feat: enable ppt_composer.zh_CN.md with request.locale (#694)
* feat: enable ppt_composer.zh_CN.md with request.locale

* fix: GeneratePPTRequest miss locale field

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-11-22 16:56:18 +08:00
Zts0hg 2d1a0997eb feat: be compatible with case: json_object is not supported by used model (#673)
* feat: 兼容使用的模型不支持json结构化输出的情况

* fix: add explicit validation that the response content is valid JSON before proceeding to parse it

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-11-21 09:41:34 +08:00
agoudbg 164ef5b8be refactor: Welcome layout and conditional rendering (#690)
* refactor: Welcome layout and conditional rendering

Improves flex layout and spacing in ConversationStarter, and updates MessagesBlock to conditionally render ConversationStarter or MessageListView based on chat state. This streamlines the UI and removes redundant rendering logic.

* fix: replay mode

* fix: Remove unnecessary inset-0

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-11-21 09:27:14 +08:00
Copilot b7a4b0f446 Add GitHub Copilot instructions for repository context (#698)
* Initial plan

* Add comprehensive GitHub Copilot instructions

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
2025-11-17 09:57:03 +08:00
Anush aa027faf95 feat: Qdrant Vector Search Support (#684)
* feat: Qdrant vector search support

Signed-off-by: Anush008 <anushshetty90@gmail.com>

* chore: Review updates

Signed-off-by: Anush008 <anushshetty90@gmail.com>

---------

Signed-off-by: Anush008 <anushshetty90@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-11-11 19:35:00 +08:00
js0205 70dbd21bdf docs: add comprehensive debugging guide and improve troubleshooting documentation (#688)
This commit addresses issue #682 by providing clear documentation on how to view complete model output and debug DeerFlow workflows.

Changes:
- Add new DEBUGGING.md guide with detailed instructions for:
  - Viewing complete model output
  - Enabling debug logging
  - Configuring LangChain verbose logging
  - Setting up LangSmith tracing
  - Docker Compose debugging tips
  - Common troubleshooting scenarios

- Update .env.example with:
  - Clearer comments for DEBUG setting
  - Documentation for LANGCHAIN_VERBOSE and LANGCHAIN_DEBUG options
  - Improved LangSmith configuration guidance

- Enhance docs/FAQ.md with:
  - How to view complete model output
  - How to enable debug logging
  - How to troubleshoot common issues
  - Links to the new debugging guide

These documentation improvements make it easier for users to:
- Debug workflow issues
- View LLM prompts and responses
- Troubleshoot deployment problems
- Monitor performance with LangSmith

Fixes #682
2025-11-10 09:34:16 +08:00
Qiyuan Jiao a38c8584d7 feat: add edit and refresh functionality for MCP servers in settings tab (#680)
* feat: add edit and refresh functionality for MCP servers in settings tab

* feat: fix lint error and enhance MCP server dialog with validation and error handling

* fix: add missing newline at the end of en.json file

* feat: only refreshing specific servers

* feat: add validation messages for MCP server configuration and improve server update logic
2025-11-06 10:38:45 +08:00
Willem Jiang fea585ae3d fix: prevent DOM error when removing temporary download link (#675) (#676)
Add defensive checks before removeChild to prevent 'Failed to execute removeChild' error when the element has already been removed from DOM. Wrap URL.revokeObjectURL in finally block to ensure proper resource cleanup.
2025-10-31 22:30:34 +08:00
Willem Jiang 6ae4bc588a fix: remove the unnessary conditional edge. (#671) 2025-10-29 10:12:32 +08:00
Willem Jiang 0415f622da fix: presever the local setting between frontend and backend (#670)
* fix: presever the local setting between frontend and backend

* Added unit test for the state preservation

* fix: passing the locale to the agent call

* fix: apply the fix after code review
2025-10-28 21:45:29 +08:00
Willem Jiang eb4c3b8ef6 fix: pass the locale through the frontend chat (#668)
* fix: pass the locale through the frontend chat

* update the code as the review suggestion

* update the code to fix the lint error
2025-10-28 08:41:54 +08:00
Willem Jiang b4c09aa4b1 security: add log injection attack prevention with input sanitization (#667)
* security: add log injection attack prevention with input sanitization

- Created src/utils/log_sanitizer.py to sanitize user-controlled input before logging
- Prevents log injection attacks using newlines, tabs, carriage returns, etc.
- Escapes dangerous characters: \n, \r, \t, \0, \x1b
- Provides specialized functions for different input types:
  - sanitize_log_input: general purpose sanitization
  - sanitize_thread_id: for user-provided thread IDs
  - sanitize_user_content: for user messages (more aggressive truncation)
  - sanitize_agent_name: for agent identifiers
  - sanitize_tool_name: for tool names
  - sanitize_feedback: for user interrupt feedback
  - create_safe_log_message: template-based safe message creation

- Updated src/server/app.py to sanitize all user input in logging:
  - Thread IDs from request parameter
  - Message content from user
  - Agent names and node information
  - Tool names and feedback

- Updated src/agents/tool_interceptor.py to sanitize:
  - Tool names during execution
  - User feedback during interrupt handling
  - Tool input data

- Added 29 comprehensive unit tests covering:
  - Classic newline injection attacks
  - Carriage return injection
  - Tab and null character injection
  - HTML/ANSI escape sequence injection
  - Combined multi-character attacks
  - Truncation and length limits

Fixes potential log forgery vulnerability where malicious users could inject
fake log entries via unsanitized input containing control characters.
2025-10-27 20:57:23 +08:00
Willem Jiang ccd7535072 fix: make SSE buffer size configurable to prevent overflow during multi-round searches (#664) (#665)
* fix: make SSE buffer size configurable to prevent overflow during multi-round searches (Issue #664)

- Add NEXT_PUBLIC_MAX_STREAM_BUFFER_SIZE environment variable for frontend SSE stream buffer
- Default to 1MB for backward compatibility, users can increase to 5-10MB for large searches
- Enhance error message with actual buffer sizes and guidance on configuration
- Add validation schema in env.js with positive integer requirement
- Document configuration in .env.example with clear examples and use cases

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-27 17:03:38 +08:00
Willem Jiang 83f1334db0 feat: add comprehensive debug logging for issue #477 hanging/freezing diagnosis (#662)
* feat: add comprehensive debug logging for issue #477 hanging/freezing diagnosis
- Add debug logging to src/server/app.py for event streaming and message chunk processing
- Track graph event flow with thread IDs for correlation
- Add detailed logging in interrupt event processing
- Add debug logging to src/agents/tool_interceptor.py for tool execution and interrupt handling
- Log interrupt decision flow and user feedback processing
- Add debug logging to src/graph/nodes.py for agent node execution
- Track step execution progress and agent coordination in research_team_node
- Add debug logging to src/agents/agents.py for agent creation and tool wrapping
- Update server.py to enable debug logging when --log-level debug is specified
- Add thread ID correlation throughout for better diagnostics
- Helps diagnose hanging/freezing issues during workflow execution

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-27 08:21:30 +08:00
Willem Jiang e9f0a02f1f docs: add tool-specific interrupts configuration to conf.yaml.example (#661)
This allows users to configure which tools should trigger interrupts
before execution, useful for reviewing sensitive operations.
2025-10-27 07:21:41 +08:00
Willem Jiang 6ded818f62 fix: handle escaped curly braces in LaTeX formulas (#608) (#660)
- Add unescape for \{ and \} characters in unescapeMarkdownSpecialChars()
- These are commonly used in LaTeX commands like \mathcal{F}
- Add test cases for Fourier transform notation and mixed escape scenarios
- All 118 tests pass including 4 new edge case tests for issue #608
2025-10-26 10:15:35 +08:00
Willem Jiang bcc403ecd3 feat: implement tool-specific interrupts for create_react_agent (#572) (#659)
* feat: implement tool-specific interrupts for create_react_agent (#572)

          Add selective tool interrupt capability allowing interrupts before specific tools
          rather than all tools. Users can now configure which tools trigger interrupts via
          the interrupt_before_tools parameter.

          Changes:
          - Create ToolInterceptor class to handle tool-specific interrupt logic
          - Add interrupt_before_tools parameter to create_agent() function
          - Extend Configuration with interrupt_before_tools field
          - Add interrupt_before_tools to ChatRequest API
          - Update nodes.py to pass interrupt configuration to agents
          - Update app.py workflow to support tool interrupt configuration
          - Add comprehensive unit tests for tool interceptor

          Features:
          - Selective tool interrupts: interrupt only specific tools by name
          - Approval keywords: recognize user approval (approved, proceed, accept, etc.)
          - Backward compatible: optional parameter, existing code unaffected
          - Flexible: works with default tools and MCP-powered tools
          - Works with existing resume mechanism for seamless workflow

          Example usage:
            request = ChatRequest(
              messages=[...],
              interrupt_before_tools=['db_tool', 'sensitive_api']
            )

* test: add comprehensive integration tests for tool-specific interrupts (#572)

Add 24 integration tests covering all aspects of the tool interceptor feature:

Test Coverage:
- Agent creation with tool interrupts
- Configuration support (with/without interrupts)
- ChatRequest API integration
- Multiple tools with selective interrupts
- User approval/rejection flows
- Tool wrapping and functionality preservation
- Error handling and edge cases
- Approval keyword recognition
- Complex tool inputs
- Logging and monitoring

All tests pass with 100% coverage of tool interceptor functionality.

Tests verify:
✓ Selective tool interrupts work correctly
✓ Only specified tools trigger interrupts
✓ Non-matching tools execute normally
✓ User feedback is properly parsed
✓ Tool functionality is preserved after wrapping
✓ Error handling works as expected
✓ Configuration options are properly respected
✓ Logging provides useful debugging info

* fix: mock get_llm_by_type in agent creation test

Fix test_agent_creation_with_tool_interrupts which was failing because
get_llm_by_type() was being called before create_react_agent was mocked.

Changes:
- Add mock for get_llm_by_type in test
- Use context manager composition for multiple patches
- Test now passes and validates tool wrapping correctly

All 24 integration tests now pass successfully.

* refactor: use mock assertion methods for consistent and clearer error messages

Update integration tests to use mock assertion methods instead of direct
attribute checking for consistency and clearer error messages:

Changes:
- Replace 'assert mock_interrupt.called' with 'mock_interrupt.assert_called()'
- Replace 'assert not mock_interrupt.called' with 'mock_interrupt.assert_not_called()'

Benefits:
- Consistent with pytest-mock and unittest.mock best practices
- Clearer error messages when assertions fail
- Better IDE autocompletion support
- More professional test code

All 42 tests pass with improved assertion patterns.

* refactor: use default_factory for interrupt_before_tools consistency

Improve consistency between ChatRequest and Configuration implementations:

Changes:
- ChatRequest.interrupt_before_tools: Use Field(default_factory=list) instead of Optional[None]
- Remove unnecessary 'or []' conversion in app.py line 505
- Aligns with Configuration.interrupt_before_tools implementation pattern
- No functional changes - all tests still pass

Benefits:
- Consistent field definition across codebase
- Simpler and cleaner code
- Reduced chance of None/empty list bugs
- Better alignment with Pydantic best practices

All 42 tests passing.

* refactor: improve tool input formatting in interrupt messages

Enhance tool input representation for better readability in interrupt messages:

Changes:
- Add json import for better formatting
- Create _format_tool_input() static method with JSON serialization
- Use JSON formatting for dicts, lists, tuples with indent=2
- Fall back to str() for non-serializable types
- Handle None input specially (returns 'No input')
- Improve interrupt message formatting with better spacing

Benefits:
- Complex tool inputs now display as readable JSON
- Nested structures are properly indented and visible
- Better user experience when reviewing tool inputs before approval
- Handles edge cases gracefully with fallbacks
- Improved logging output for debugging

Example improvements:
Before: {'query': 'SELECT...', 'limit': 10, 'nested': {'key': 'value'}}
After:
{
  "query": "SELECT...",
  "limit": 10,
  "nested": {
    "key": "value"
  }
}

All 42 tests still passing.

* test: add comprehensive unit tests for tool input formatting
2025-10-26 09:47:03 +08:00
Willem Jiang 0441038672 fix: improve config loading resilience for non-localhost access (#510) (#658)
* fix: improve config loading resilience for non-localhost access (#510)

- Add DEFAULT_CONFIG fallback to always return valid config even if fetch fails
- Implement retry logic with exponential backoff (max 2 retries) to handle transient failures
- Add 5-second fetch timeout to prevent hanging on unreachable backends
- Improve error logging with clear messages about config fetch status
- Always return DeerFlowConfig (never null) to prevent UI rendering issues
- Add safety checks in input-box component to verify reasoning models before access
- Improve type safety: verify array length before accessing array indices
- Add comprehensive documentation in .env.example with examples for different deployment scenarios
- Document NEXT_PUBLIC_API_URL variable behavior and fallback mechanism

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: add nullish coalescing to prevent TypeScript error in input-box

- Add ?? operator to handle potential undefined value when accessing reasoning[0]
- Fixes TS2322 error: Type 'string | undefined' is not assignable to type 'string | number | Date'

---------

Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-26 07:34:12 +08:00
Willem Jiang c7a82b82b4 fix: parsed json with extra tokens issue (#656)
Fixes #598 

* fix: parsed json with extra tokens issue

* Added unit test for json.ts

* fix the json unit test running issue

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update the code with code review suggestion

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-10-26 07:24:25 +08:00
Willem Jiang fd5a9aeae4 fix: handle [ACCEPTED] feedback gracefully without TypeError in plan review (#657)
* fix: handle [ACCEPTED] feedback gracefully without TypeError in plan review (#607)

- Add explicit None/empty feedback check to prevent processing None values
- Normalize feedback string once using strip().upper() instead of repeated calls
- Replace TypeError exception with graceful fallback to planner node
- Handle invalid feedback formats by logging warning and returning to planner
- Maintain backward compatibility for '[ACCEPTED]' and '[EDIT_PLAN]' formats
- Add test cases for None feedback, empty string feedback, and invalid formats
- Update existing test to verify graceful handling instead of exception raising

* Update src/graph/nodes.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-25 22:06:19 +08:00
Willem Jiang 1d71f8910e fix: react key warnings from duplicate message IDs + establish jest testing framework (#655)
* fix: resolve issue #588 - react key warnings from duplicate message IDs + establish jest testing framework

* Update the makefile and workflow with the js test

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-25 20:46:43 +08:00
Willem Jiang f2be4d6af1 fix: prevent tool name concatenation in consecutive tool calls to fix #523 (#654)
- Implement index-based grouping of tool call chunks in _process_tool_call_chunks()
- Add _validate_tool_call_chunks() for debug logging and validation
- Enhance _process_message_chunk() with tool call ID validation and boundary detection
- Add comprehensive unit tests (17 tests) for tool call chunk processing
- Fix issue where tool names were incorrectly concatenated (e.g., 'web_searchweb_search')
- Ensure chunks from different tool calls (different indices) remain properly separated
- Add detailed logging for debugging tool call streaming issues

* update the code with suggestions of reviewing
2025-10-24 22:26:25 +08:00
Willem Jiang 36bf5c9ccd fix: repair missing step_type fields in Plan validation (#653)
* fix: resolve issue #650 - repair missing step_type fields in Plan validation

- Add step_type repair logic to validate_and_fix_plan() to auto-infer missing step_type
- Infer as 'research' when need_search=true, 'processing' when need_search=false
- Add explicit CRITICAL REQUIREMENT section to planner.md emphasizing step_type mandatory for every step
- Include validation checklist and examples showing both research and processing steps
- Add 23 comprehensive unit tests for validate_and_fix_plan() covering all scenarios
- Add 4 integration tests specifically for Issue #650 with actual Plan validation
- Prevents Pydantic ValidationError: 'Field required' for missing step_type

* Update tests/unit/graph/test_plan_validation.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/unit/graph/test_plan_validation.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update the planner.zh_CN.md with recent changes of planner.md

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-24 21:26:48 +08:00
Willem Jiang 975b344ca7 fix: resolve issue #651 - crawl error with None content handling (#652)
* fix: resolve issue #651 - crawl error with None content handling
Fixed issue #651 by adding comprehensive null-safety checks and error handling to the crawl system.
The fix prevents the ‘TypeError: Incoming markup is of an invalid type: None’ crash by:
1. Validating HTTP responses from Jina API
2. Handling None/empty content at extraction stage
3. Adding fallback handling in Article markdown/message conversion
4. Improving error diagnostics with detailed logging
5. Adding 16 new tests with 100% coverage for critical paths

* Update src/crawler/readability_extractor.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/crawler/article.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-24 17:06:54 +08:00
jimmyuconn1982 2001a7c223 Fix: clarification bugs - max rounds, locale passing, and over-clarification (#647)
Fixes: Max rounds bug, locale passing bug, over-clarification issue

* reslove Copilot spelling comments

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-10-24 16:43:39 +08:00
Willem Jiang 5eada04f50 feat: Add comprehensive Chinese localization support for issue #412 (#649)
* feat: Add comprehensive Chinese localization support for issue #412

          - Add locale parameter to ChatRequest model to capture user's language preference
          - Implement language-aware template loading in template.py with fallback to English
          - Update all apply_prompt_template calls to pass locale through the workflow
          - Create Chinese translations for 14 core prompt files:
            * Main agents: coordinator, planner, researcher, reporter, coder
            * Subprocess agents: podcast_script_writer, ppt_composer, prompt_enhancer
            * Writing assistant: all 6 prose prompts
          - Update app.py to extract and propagate locale through workflow state
          - Support both zh-CN and en-US locales with automatic fallback
          - Ensure locale flows through all agent nodes and template rendering

* address the review suggestions
2025-10-24 16:31:19 +08:00
Willem Jiang 052490b116 fix: resolve issue #467 - message content validation and Tavily search error handling (#645)
* fix: resolve issue #467 - message content validation and Tavily search error handling

This commit implements a comprehensive fix for issue #467 where the application
crashed with 'Field required: input.messages.3.content' error when generating reports.

## Root Cause Analysis
The issue had multiple interconnected causes:
1. Tavily tool returned mixed types (lists/error strings) instead of consistent JSON
2. background_investigation_node didn't handle error cases properly, returning None
3. Missing message content validation before LLM calls
4. Insufficient error diagnostics for content-related errors

## Changes Made

### Part 1: Fix Tavily Search Tool (tavily_search_results_with_images.py)
- Modified _run() and _arun() methods to return JSON strings instead of mixed types
- Error responses now return JSON: {"error": repr(e)}
- Successful responses return JSON string: json.dumps(cleaned_results)
- Ensures tool results always have valid string content for ToolMessages

### Part 2: Fix background_investigation_node Error Handling (graph/nodes.py)
- Initialize background_investigation_results to empty list instead of None
- Added proper JSON parsing for string responses from Tavily tool
- Handle error responses with explicit error logging
- Always return valid JSON (empty list if error) instead of None

### Part 3: Add Message Content Validation (utils/context_manager.py)
- New validate_message_content() function validates all messages before LLM calls
- Ensures all messages have content attribute and valid string content
- Converts complex types (lists, dicts) to JSON strings
- Provides graceful fallback for messages with issues

### Part 4: Enhanced Error Diagnostics (_execute_agent_step in graph/nodes.py)
- Call message validation before agent invocation
- Add detailed logging for content-related errors
- Log message types, content types, and lengths when validation fails
- Helps with future debugging of similar issues

## Testing
- All unit tests pass (395 tests)
- Python syntax verified for all modified files
- No breaking changes to existing functionality

* test: update tests for issue #467 fixes

Update test expectations to match the new implementation:
- Tavily search tool now returns JSON strings instead of mixed types
- background_investigation_node returns empty list [] for errors instead of None
- All tests updated to verify the new behavior
- All 391 tests pass successfully

* Update src/graph/nodes.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-23 22:08:14 +08:00
Willem Jiang c15c480fe6 docs: provide comprehensive API documentation for the backend server (#646)
fixes #538
2025-10-23 19:40:22 +08:00
Qiyuan Jiao 829cb39b25 fix: Optimize the performance of stream data processing and add anti-… (#642)
* fix: Optimize the performance of stream data processing and add anti-shake and batch update mechanisms

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 修复消息批量更新重复问题

- 将 pendingUpdates 从数组改为 Map,使用 message.id 作为键
- 避免在16ms窗口内多次更新同一消息导致的重复处理
- 优化了批量更新性能,减少冗余的映射操作

* fix lint error

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-10-22 23:08:18 +08:00
Willem Jiang 9ece3fd9c3 fix: support additional Tavily search parameters via configuration to fix #548 (#643)
* fix: support additional Tavily search parameters via configuration to fix #548

- Add include_answer, search_depth, include_raw_content, include_images, include_image_descriptions to SEARCH_ENGINE config
- Update get_web_search_tool() to load these parameters from configuration with sensible defaults
- Parameters are now properly passed to TavilySearchWithImages during initialization
- This fixes 'got an unexpected keyword argument' errors when using web_search tool
- Update tests to verify new parameters are correctly set

* test: add comprehensive unit tests for web search configuration loading

- Add test for custom configuration values (include_answer, search_depth, etc.)
- Add test for empty configuration (all defaults)
- Add test for image_descriptions logic when include_images is false
- Add test for partial configuration
- Add test for missing config file
- Add test for multiple domains in include/exclude lists

All 7 new tests pass and provide comprehensive coverage of configuration loading
and parameter handling for Tavily search tool initialization.

* test: verify all Tavily configuration parameters are optional

Add 8 comprehensive tests to verify that all Tavily engine configuration
parameters are truly optional:

- test_tavily_with_no_search_engine_section: SEARCH_ENGINE section missing
- test_tavily_with_completely_empty_config: Entire config missing
- test_tavily_with_only_include_answer_param: Single param, rest default
- test_tavily_with_only_search_depth_param: Single param, rest default
- test_tavily_with_only_include_domains_param: Domain param, rest default
- test_tavily_with_explicit_false_boolean_values: False values work correctly
- test_tavily_with_empty_domain_lists: Empty lists handled correctly
- test_tavily_all_parameters_optional_mix: Multiple missing params work

These tests verify:
- Tool creation never fails regardless of missing configuration
- All parameters have sensible defaults
- Boolean parameters can be explicitly set to False
- Any combination of optional parameters works
- Domain lists can be empty or omitted

All 15 Tavily configuration tests pass successfully.
2025-10-22 22:56:02 +08:00
jimmyuconn1982 003f081a7b fix: Refine clarification workflow state handling (#641)
* fix: support local models by making thought field optional in Plan model

- Make thought field optional in Plan model to fix Pydantic validation errors with local models
- Add Ollama configuration example to conf.yaml.example
- Update documentation to include local model support
- Improve planner prompt with better JSON format requirements

Fixes local model integration issues where models like qwen3:14b would fail
due to missing thought field in JSON output.

* feat: Add intelligent clarification feature for research queries

- Add multi-turn clarification process to refine vague research questions
- Implement three-dimension clarification standard (Tech/App, Focus, Scope)
- Add clarification state management in coordinator node
- Update coordinator prompt with detailed clarification guidelines
- Add UI settings to enable/disable clarification feature (disabled by default)
- Update workflow to handle clarification rounds recursively
- Add comprehensive test coverage for clarification functionality
- Update documentation with clarification feature usage guide

Key components:
- src/graph/nodes.py: Core clarification logic and state management
- src/prompts/coordinator.md: Detailed clarification guidelines
- src/workflow.py: Recursive clarification handling
- web/: UI settings integration
- tests/: Comprehensive test coverage
- docs/: Updated configuration guide

* fix: Improve clarification conversation continuity

- Add comprehensive conversation history to clarification context
- Include previous exchanges summary in system messages
- Add explicit guidelines for continuing rounds in coordinator prompt
- Prevent LLM from starting new topics during clarification
- Ensure topic continuity across clarification rounds

Fixes issue where LLM would restart clarification instead of building upon previous exchanges.

* fix: Add conversation history to clarification context

* fix: resolve clarification feature message to planer, prompt, test issues

- Optimize coordinator.md prompt template for better clarification flow
- Simplify final message sent to planner after clarification
- Fix API key assertion issues in test_search.py

* fix: Add configurable max_clarification_rounds and comprehensive tests

- Add max_clarification_rounds parameter for external configuration
- Add comprehensive test cases for clarification feature in test_app.py
- Fixes issues found during interactive mode testing where:
  - Recursive call failed due to missing initial_state parameter
  - Clarification exited prematurely at max rounds
  - Incorrect logging of max rounds reached

* Move clarification tests to test_nodes.py and add max_clarification_rounds to zh.json

* fix: add max_clarification_rounds parameter passing from frontend to backend

- Add max_clarification_rounds parameter in store.ts sendMessage function
- Add max_clarification_rounds type definition in chat.ts
- Ensure frontend settings page clarification rounds are correctly passed to backend

* fix: refine clarification workflow state handling and coverage

- Add clarification history reconstruction
- Fix clarified topic accumulation
- Add clarified_research_topic state field
- Preserve clarification state in recursive calls
- Add comprehensive test coverage

* refactor: optimize coordinator logic and type annotations

- Simplify handoff topic logic in coordinator_node
- Update type annotations from Tuple to tuple
- Improve code readability and maintainability

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-10-22 22:49:07 +08:00
Willem Jiang 9371ad23ee Polish the Makefile comment (#644) 2025-10-22 17:06:41 +08:00
Willem Jiang add0a701f4 fix: ensure web search is performed for research plans to fix #535 (#640)
* fix: ensure web search is performed for research plans to fix #535

          When using certain models (DeepSeek-V3, Qwen3, or local deployments), the
          agent framework failed to trigger web search tools, resulting in hallucinated
          data. This fix implements multiple safeguards:

          1. Add enforce_web_search configuration flag:
             - New config option to mandate web search in research plans
             - Defaults to False for backward compatibility

          2. Add plan validation function validate_and_fix_plan():
             - Validates that plans include at least one research step with web search
             - Enforces web search requirement when enabled
             - Adds default research step if plan has no steps

          3. Enhance coordinator_node fallback logic:
             - When model fails to call tools, fallback to planner instead of __end__
             - Ensures workflow continues even when tool calling fails
             - Logs detailed diagnostic info for debugging

          4. Update prompts for stricter requirements:
             - planner.md: Add MANDATORY web search requirement and clear warnings
             - coordinator.md: Add CRITICAL tool calling requirement
             - Emphasize consequences of missing web search (hallucinated data)

          5. Update tests to reflect new behavior:
             - test_coordinator_node_no_tool_calls: Expect planner instead of __end__
             - test_coordinator_empty_llm_response_corner_case: Same expectation

          Fixes #535 by ensuring:
          - Web search is always performed for research tasks
          - Workflow doesn't terminate on tool calling failures
          - Models with poor tool calling support can still proceed
          - No hallucinated data without real information gathering

* Update src/graph/nodes.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/graph/nodes.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* accept the review suggestion of getting configuration

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-22 08:27:06 +08:00
Willem Jiang 2ff7d9adf8 Added the Agent.md file to work with AI agent (#639) 2025-10-21 15:25:13 +08:00
Willem Jiang 1a16677d1a fix: unescape markdown-escaped characters in math formulas to fix #608 (#637)
When editing reports, tiptap-markdown escapes special characters (*, _, [, ])
which corrupts LaTeX formulas. This fix:

1. Adds unescapeLatexInMath() function to reverse markdown escaping within
   math delimiters ($...$ and 94410...94410)
2. Applies the unescape function in the editor's onChange callback to clean
   the markdown before storing it
3. Adds comprehensive tests covering edge cases and round-trip scenarios

The fix ensures formulas like $(f * g)[n]$ remain unescaped when editing,
preventing display errors after save/reload.
2025-10-21 10:06:31 +08:00
Willem Jiang cb5c477371 Using tsx to run the ts unit tests (#638) 2025-10-21 10:05:50 +08:00
Willem Jiang d30c4d00d3 fix: convert crawl_tool dict return to JSON string for type consistency (#636)
Keep fixing #631
This pull request updates the crawl_tool function to return its results as a JSON string instead of a dictionary, and adjusts the unit tests accordingly to handle the new return type. The changes ensure consistent serialization of output and proper validation in tests.
2025-10-21 10:00:33 +08:00
Willem Jiang e2ff765460 fix: correct image result format for OpenAI compatibility to fix #632 (#634)
- Change image result type from 'image' to 'image_url' to match OpenAI API expectations
- Wrap image URL in dict structure: {"url": "..."} instead of plain string
- Update SearchResultPostProcessor to handle dict-based image_url during duplicate removal
- Update tests to validate new image format

This fixes the 400 error: Invalid value: 'image'. Supported values are: 'text', 'image_url'...

Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-10-20 23:14:09 +08:00
Willem Jiang 3689bc0e69 fix: handle non-string tool results to fix #631 (#633)
- Backend: Convert non-string content (lists, dicts) to JSON strings in _create_event_stream_message to ensure frontend always receives string content
- Frontend: Add type guard before calling startsWith() on toolCall.result for defensive programming

This fixes the TypeError: toolCall.result.startsWith is not a function when tools return complex objects.
2025-10-20 23:10:58 +08:00
Willem Jiang 984aa69acf fix: optimize animations to prevent browser freeze with many research steps (#630)
Fixes #570 where browser freezes when research plan has 8+ steps.

Performance optimizations:
- Add animation throttling: only animate first 10 activity items
- Reduce animation durations (0.4s → 0.3s for activities, 0.2s → 0.15s for results)
- Remove scale animations (GPU-intensive) from search results
- Limit displayed results (20 pages, 10 images max)
- Add conditional animations based on item index
- Cap animation delays to prevent excessive staggering
- Add React.memo to ActivityMessage and ActivityListItem components

These changes significantly improve performance when rendering multiple
research steps while maintaining visual appeal for smaller lists.
2025-10-19 19:24:57 +08:00
Willem Jiang 5af036f19f fix: add missing RunnableConfig parameter to human_feedback_node (#629)
* fix: add missing RunnableConfig parameter to human_feedback_node

This fixes issue #569 where interrupt() was being called outside of a runnable context.
The human_feedback_node was missing the config: RunnableConfig parameter that all other
node functions have, which caused RuntimeError when interrupt() tried to access the config.

- Add config: RunnableConfig parameter to function signature
- Add State type annotation to state parameter for consistency
- Maintains LangGraph execution context required by interrupt()

* test: update human_feedback_node tests to pass RunnableConfig parameter

Update all test functions that call human_feedback_node to include the new
required config parameter. These tests were failing because they were not
providing the RunnableConfig argument after the fix to add proper LangGraph
execution context.

Tests updated:
- test_human_feedback_node_auto_accepted
- test_human_feedback_node_edit_plan
- test_human_feedback_node_accepted
- test_human_feedback_node_invalid_interrupt
- test_human_feedback_node_json_decode_error_first_iteration
- test_human_feedback_node_json_decode_error_second_iteration
- test_human_feedback_node_not_enough_context

All tests now pass the mock_config fixture to human_feedback_node.
2025-10-19 17:35:06 +08:00
Willem Jiang 57c9c2dcd5 fix: improve error handling in researcher and coder nodes (#596)
- Wrap agent.ainvoke() calls in try-except blocks
- Log full exception tracebacks for better debugging
- Return detailed error messages to users instead of generic 'internal error'
- Include step title and agent name in error context
- Allow workflow to continue gracefully when agent execution fails
- Store error details in observations for audit trail
2025-10-19 16:33:14 +08:00
Willem Jiang 497a2a39cf fix:the formual display error after report editing (#627) 2025-10-17 15:34:43 +08:00
Willem Jiang c6348e70c6 fix: prevent repeated content animation during thinking streaming (#614) (#623)
* fix: prevent repeated content animation during thinking streaming (#614)

- Implement chunked rendering using reasoningContentChunks
- Static content (previous chunks) renders without animation
- Only current streaming chunk animates
- Disable animation on plan content (title, thought, steps) during streaming
- Animation applies after content finishes streaming (when complete)
- Prevents visual duplication of repeated sentences in thinking process
2025-10-16 19:48:05 +08:00
Willem Jiang d9f829b608 Add frontend tests step to frontend lint workflow 2025-10-16 19:19:07 +08:00
Willem Jiang 025ea6b94e fix: add unique key prop to conversation starter list items (#619)
- Changed key from question text to combination of index and question text
- Ensures unique keys even if translation has duplicate questions
- Resolves React warning: 'Each child in a list should have a unique key prop'
2025-10-16 18:24:36 +08:00
Willem Jiang 120fcfb316 fix: configure Windows event loop policy for PostgreSQL async compatibility (#618)
- Set asyncio.WindowsSelectorEventLoopPolicy() on Windows at app module level
- Ensures psycopg can run in async mode on Windows regardless of entry point
- Fixes 'ProactorEventLoop' error when using PostgreSQL checkpointer
- Works with all entry points: server.py, uvicorn, langgraph dev, etc.
2025-10-16 17:59:06 +08:00
Willem Jiang 9b127c55f2 chore: add frontend unit tests to lint-frontend make target
Added 'node --test tests/*.test.ts' to the lint-frontend target to ensure
frontend unit tests are run as part of the CI/quality checks workflow.

This ensures:
- Math formula normalization tests run before build
- Tests are validated alongside linting and type checking
- All 19 frontend tests pass before deployment

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2025-10-15 08:52:50 +08:00
Willem Jiang 779de40f10 fix: exclude test files from TypeScript type checking
Test files use .ts extensions in imports for Node's native test runner
compatibility, which conflicts with TypeScript's default behavior.
Excluding test files from tsconfig allows tests to run with Node while
maintaining strict type checking for the main codebase.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2025-10-15 08:52:50 +08:00
Willem Jiang 58c1743ed5 fix: resolve math formula display abnormal after editing report
This fix addresses the issue where math formulas become corrupted or
incorrectly displayed after editing the generated report in the editor.

**Root Cause:**
The issue occurred due to incompatibility between markdown processing
in the display component and the Tiptap editor:
1. Display component used \[\] and \(\) LaTeX delimiters
2. Tiptap Mathematics extension expects $ and 70868 delimiters
3. tiptap-markdown didn't have built-in math node serialization
4. Math syntax was lost/corrupted during editor save operations

**Solution Implemented:**
1. Created MathematicsWithMarkdown extension that adds markdown
   serialization support to Tiptap's Mathematics nodes
2. Added math delimiter normalization functions:
   - normalizeMathForEditor(): Converts LaTeX delimiters to $/70868
   - normalizeMathForDisplay(): Standardizes all delimiters to 70868
3. Updated Markdown component to use new normalization
4. Updated ReportEditor to normalize content before loading

**Changes:**
- web/src/components/editor/math-serializer.ts (new)
- web/src/components/editor/extensions.tsx
- web/src/components/editor/index.tsx
- web/src/components/deer-flow/markdown.tsx
- web/src/core/utils/markdown.ts
- web/tests/markdown-math-editor.test.ts (new)
- web/tests/markdown-katex.test.ts

**Testing:**
- Added 15 comprehensive tests for math normalization round-trip
- All tests passing (math editor + existing katex tests)
- Verified TypeScript compilation and linting

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2025-10-15 08:52:50 +08:00
jimmyuconn1982 24e2d86f7b fix: add max_clarification_rounds parameter passing from frontend to backend (#616)
Bug Fix
This PR fixes the issue where max_clarification_rounds parameter was not being passed from the frontend to the backend, causing a TypeError: '<' not supported between instances of 'int' and 'NoneType' error.

Technical Details
The issue was that the frontend was not passing the max_clarification_rounds parameter to the backend API, causing the backend to receive None values and fail during comparison operations. This fix ensures the parameter is properly typed and passed through the entire request chain.
2025-10-14 17:56:20 +08:00
jimmyuconn1982 2510cc61de feat: Add intelligent clarification feature in coordinate step for research queries (#613)
* fix: support local models by making thought field optional in Plan model

- Make thought field optional in Plan model to fix Pydantic validation errors with local models
- Add Ollama configuration example to conf.yaml.example
- Update documentation to include local model support
- Improve planner prompt with better JSON format requirements

Fixes local model integration issues where models like qwen3:14b would fail
due to missing thought field in JSON output.

* feat: Add intelligent clarification feature for research queries

- Add multi-turn clarification process to refine vague research questions
- Implement three-dimension clarification standard (Tech/App, Focus, Scope)
- Add clarification state management in coordinator node
- Update coordinator prompt with detailed clarification guidelines
- Add UI settings to enable/disable clarification feature (disabled by default)
- Update workflow to handle clarification rounds recursively
- Add comprehensive test coverage for clarification functionality
- Update documentation with clarification feature usage guide

Key components:
- src/graph/nodes.py: Core clarification logic and state management
- src/prompts/coordinator.md: Detailed clarification guidelines
- src/workflow.py: Recursive clarification handling
- web/: UI settings integration
- tests/: Comprehensive test coverage
- docs/: Updated configuration guide

* fix: Improve clarification conversation continuity

- Add comprehensive conversation history to clarification context
- Include previous exchanges summary in system messages
- Add explicit guidelines for continuing rounds in coordinator prompt
- Prevent LLM from starting new topics during clarification
- Ensure topic continuity across clarification rounds

Fixes issue where LLM would restart clarification instead of building upon previous exchanges.

* fix: Add conversation history to clarification context

* fix: resolve clarification feature message to planer, prompt, test issues

- Optimize coordinator.md prompt template for better clarification flow
- Simplify final message sent to planner after clarification
- Fix API key assertion issues in test_search.py

* fix: Add configurable max_clarification_rounds and comprehensive tests

- Add max_clarification_rounds parameter for external configuration
- Add comprehensive test cases for clarification feature in test_app.py
- Fixes issues found during interactive mode testing where:
  - Recursive call failed due to missing initial_state parameter
  - Clarification exited prematurely at max rounds
  - Incorrect logging of max rounds reached

* Move clarification tests to test_nodes.py and add max_clarification_rounds to zh.json
2025-10-14 13:35:57 +08:00
Willem Jiang 81c91dda43 feature: clean up the temp file which are generated when running the unit test of milvus (#612)
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-10-12 22:10:15 +08:00
Willem Jiang 2a6455c436 feature: add formula rander in the markdown (#611)
* feature: add formula rander in the markdown

* fixed the lint errors
2025-10-11 23:05:09 +08:00
jovial f80af8e132 chore: fix incorrect filename in conf.yaml.example comments (#609) 2025-10-11 10:00:22 +08:00
Willem Jiang 79b9cdb59a feature:Add the debug setting on vscode (#606) 2025-10-05 22:07:23 +08:00
jimmyuconn1982 24f6905c18 fix: support local models by making thought field optional in Plan model (#601)
- Make thought field optional in Plan model to fix Pydantic validation errors with local models
- Add Ollama configuration example to conf.yaml.example
- Update documentation to include local model support
- Improve planner prompt with better JSON format requirements

Fixes local model integration issues where models like qwen3:14b would fail
due to missing thought field in JSON output.

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-09-28 08:48:39 +08:00
Fancy-hjyp 5f4eb38fdb feat: add context compress (#590)
* feat:Add context compress

* feat: Add unit test

* feat: add unit test for context manager

* feat: add postprocessor param && code format

* feat: add configuration guide

* fix: fix the configuration_guide

* fix: fix the unit test

* fix: fix the default value

* feat: add test and log for context_manager
2025-09-27 21:42:22 +08:00
HagonChan c214999606 feat: add strategic_investment report style (#595)
* add strategic_investment mode

* make format

* make lint

* fix: repair
lint-frontend
2025-09-24 09:50:36 +08:00
Gordon 1c27e0f2ae feat: add support for searx/searxng (#253)
* add searx/searxng support

* nit

* Fix indentation in search.py for readability

* Clean up imports in search.py

Removed unused imports from search.py

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-09-22 18:54:30 +08:00
Fancy-hjyp 6bb0b95579 feat:support config tavily search results (#591)
* feat:support config tavily search results

* feat: support config tavily search results

* feat: update the default value of include_images

* fix: fix the test

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-09-22 18:26:50 +08:00
irrcombat 150a730f98 Debug deerflow server, web with vscode (#592)
* Debug deerflow server, web with vscode

Signed-off-by: shjy <asdf_0403@qq.com>

* removed the duplicated setting of .vscode

---------

Signed-off-by: shjy <asdf_0403@qq.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-09-22 13:59:34 +08:00
Chayton Bai 7694bb5d72 feat: support dify in rag module (#550)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-09-16 20:30:45 +08:00
lele3436 5085bf8ee9 feat: support for moi in RAG module (#571)
* feat: add support for moi

* small adjust

* small adjust

* according 2 comments

* add more intro

* add more intro
2025-09-16 20:25:59 +08:00
Willem Jiang ea0fe62971 fix: don't expose internal application error to client (#585) 2025-09-16 10:01:24 +08:00
Willem Jiang 79ab7365c0 fix: log the exception of graph execution (#577)
* fix: log the exeption of graph execution

* Update src/server/app.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-09-14 21:20:25 +08:00
lele3436 26a587c24e fix: frontend supports chinese for listing datasets in RAG (#582)
* fix-web-rag

* Update resource-suggestion.tsx
2025-09-14 20:19:56 +08:00
HagonChan bbc49a04a6 feat: add Google AI Studio API support with platform-based detection (#502)
* feat: add Google AI Studio API support with platform-based detection

* chore: update configuration_guide.md

* fix the uv.lock formate error

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-09-13 08:49:05 +08:00
CHANGXUBO dd9af1eb50 feat: Implement Milvus retriver for RAG (#516)
* feat: Implement MilvusRetriever with embedding model and resource management

* chore: Update configuration and loader files for consistency

* chore: Clean up test_milvus.py for improved readability and organization

* feat: Add tests for DashscopeEmbeddings query and document embedding methods

* feat: Add tests for embedding model initialization and example file loading in MilvusProvider

* chore: Remove unused imports and clean up test_milvus.py for better readability

* chore: Clean up test_milvus.py for improved readability and organization

* chore: Clean up test_milvus.py for improved readability and organization

* fix: replace print statements with logging in recursion limit function

* Implement feature X to enhance user experience and optimize performance

* refactor: clean up unused imports and comments in AboutTab component

* Implement feature X to enhance user experience and fix bug Y in module Z

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-09-12 22:20:55 +08:00
jimma eec8e4dd60 refactor(logging): add explicit error log message (#576)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-09-12 22:09:08 +08:00
Qiyuan Jiao 6d1d7f2d9e fix: Remove duplicate assignment operations for the tool_call_chunks field (#575) 2025-09-12 21:56:53 +08:00
Willem Jiang c2186c1ef8 doc: updated the README of the bootstrap (#568) 2025-09-10 21:00:16 +08:00
voroq 005712679c docs: add deployment note for Linux servers (#565)
* docs: add deployment note for Linux servers

- allow external connections by changing the host to 0.0.0.0.
- security warning to remind users to secure their environment before exposing the service.

* docs: add deployment note for Linux servers

- allow external connections by changing the host to 0.0.0.0.
- security warning to remind users to secure their environment before exposing the service.
2025-09-09 23:05:04 +08:00
Willem Jiang 317acdffad fix: the stdio and sse mcp server loading issue (#566) 2025-09-09 23:02:15 +08:00
Willem Jiang 4c17d88029 feat: creating mogodb and postgres mock instance in checkpoint test (#561)
* fix: using mongomock for the checkpoint test

* Add postgres mock setting to the unit test

* Added utils file of postgres_mock_utils

* fixed the runtime loading error of deerflow server
2025-09-09 22:49:11 +08:00
CHANGXUBO 7138ba36bc Add psycopg dependencies instruction for checkpointing (#564)
* Add psycopg dependencies instruction for checkpointing

* fix: update Dockerfile to improve dependency installation process
2025-09-09 17:28:18 +08:00
voroq 38ff2f7276 fix: correct typo in MongoDB connection string within .env.example (#560)
* fix: correct typo in MongoDB connection string within .env.example

- Changes 'ongodb' to 'mongodb' in LANGGRAPH_CHECKPOINT_DB_URL example.
- This ensures the example configuration is valid and can be used directly.

* Update LANGGRAPH_CHECKPOINT_DB_URL in .env.example

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-09-08 06:54:21 +08:00
dependabot[bot] a8a2e29e2c build(deps): bump next from 15.3.0 to 15.4.7 in /web (#556)
Bumps [next](https://github.com/vercel/next.js) from 15.3.0 to 15.4.7.
- [Release notes](https://github.com/vercel/next.js/releases)
- [Changelog](https://github.com/vercel/next.js/blob/canary/release.js)
- [Commits](https://github.com/vercel/next.js/compare/v15.3.0...v15.4.7)

---
updated-dependencies:
- dependency-name: next
  dependency-version: 15.4.7
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-09-04 22:26:33 +08:00
Willem Jiang a41ced1345 fix: the search content return tuple issue (#555) 2025-09-04 15:45:30 +08:00
Willem Jiang 8f127df948 Fixed the deepseek v3 planning issue #545 (#554) 2025-09-04 10:09:49 +08:00
12november 5f1981ac9b fix deer-flow/src/prompts/prose/prose_zap.md (#553) 2025-09-03 19:32:09 +08:00
Willem Jiang 72f9c59195 feat: add lint check of front-end (#534)
* feat: add lint check of front-end

* add pnpm installation

* add pnpm installation
2025-08-22 21:08:53 +08:00
Willem Jiang 0a02843666 Fix: build of font end of #466 (#530) 2025-08-21 23:25:52 +08:00
道心坚定韩道友 f17e5bd6c8 FIX/Adapt message box to handle long text in frontend (#466)
* fix:ui

* fix:ui bug

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-08-21 10:31:54 +08:00
CHANGXUBO db6c1bf7cb fix: update TavilySearchWithImages to inherit from TavilySearchResults (#522) 2025-08-21 09:52:12 +08:00
Anoyer-lzh 270d8c3712 fix: env parameters exception when configuring SSE or HTTP MCP server (#513)
* fix: _create_streamable_http_session() got an unexpected keyword argument 'env'

fix unit error

* update md

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-08-20 17:23:57 +08:00
Willem Jiang b08e9ad3ac fix: GitHub workflow action version warning (#520)
* fix: using commit hash as the action version

* fix: using commit hash as the action version

---------

Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-08-20 14:39:02 +08:00
Willem Jiang c6d152a074 fix: using commit hash as the action version (#519)
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-08-20 13:52:00 +08:00
Willem Jiang 44d328f696 fix: polish the makefile to provide help command (#518) 2025-08-20 09:51:42 +08:00
zgjja 3b4e993531 feat: 1. replace black with ruff for fomatting and sort import (#489)
2. use tavily from`langchain-tavily` rather than the older one from `langchain-community`

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-08-17 22:57:23 +08:00
CHANGXUBO 1bfec3ad05 feat: Enhance chat streaming and tool call processing (#498)
* feat: Enhance chat streaming and tool call processing

- Added support for MongoDB checkpointer in the chat streaming workflow.
- Introduced functions to process tool call chunks and sanitize arguments.
- Improved event message creation with additional metadata.
- Enhanced error handling for JSON serialization in event messages.
- Updated the frontend to convert escaped characters in tool call arguments.
- Refactored the workflow input preparation and initial message processing.
- Added new dependencies for MongoDB integration and tool argument sanitization.

* fix: Update MongoDB checkpointer configuration to use LANGGRAPH_CHECKPOINT_DB_URL

* feat: Add support for Postgres checkpointing and update README with database recommendations

* feat: Implement checkpoint saver functionality and update MongoDB connection handling

* refactor: Improve code formatting and readability in app.py and json_utils.py

* refactor: Clean up commented code and improve formatting in server.py

* refactor: Remove unused imports and improve code organization in app.py

* refactor: Improve code organization and remove unnecessary comments in app.py

* chore: use langgraph-checkpoint-postgres==2.0.21 to avoid the JSON convert issue in the latest version, implement chat stream persistant with Postgres

* feat: add MongoDB and PostgreSQL support for LangGraph checkpointing, enhance environment variable handling

* fix: update comments for clarity on Windows event loop policy

* chore: remove empty code changes in MongoDB and PostgreSQL checkpoint tests

* chore: clean up unused imports and code in checkpoint-related files

* chore: remove empty code changes in test_checkpoint.py

* chore: remove empty code changes in test_checkpoint.py

* chore: remove empty code changes in test_checkpoint.py

* test: update status code assertions in MCP endpoint tests to allow for 403 responses

* test: update MCP endpoint tests to assert specific status codes and enable MCP server configuration

* chore: remove unnecessary environment variables from unittest workflow

* fix: invert condition for MCP server configuration check to raise 403 when disabled

* chore: remove pymongo from test dependencies in uv.lock

* chore:  optimize the _get_agent_name method

* test: enhance ChatStreamManager tests for PostgreSQL and MongoDB initialization

* test: add persistence tests for ChatStreamManager with PostgreSQL and MongoDB

* test: add unit tests for ChatStreamManager initialization with PostgreSQL and MongoDB

* test: enhance persistence tests for ChatStreamManager with PostgreSQL and MongoDB to verify message aggregation

* test: add unit tests for ChatStreamManager with PostgreSQL and MongoDB

* test: add unit tests for ChatStreamManager initialization with PostgreSQL and MongoDB

* test: add unit tests for ChatStreamManager initialization with PostgreSQL and MongoDB

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-08-16 21:03:12 +08:00
CHANGXUBO d65b8f8fcc feat: Add llms to support the latest Open Source SOTA models (#497)
* fix: update README and configuration guide for new model support and reasoning capabilities

* fix: format code for consistency in agent and node files

* fix: update test cases for environment variable handling in llm configuration

* fix: refactor message chunk conversion functions for improved clarity and maintainability

* refactor: remove enable_thinking parameter from LLM configuration functions

* chore: update agent-LLM mapping for consistency

* chore: update LLM configuration handling for improved clarity

* test: add unit tests for Dashscope message chunk conversion and LLM configuration

* test: add unit tests for message chunk conversion in Dashscope

* test: add unit tests for message chunk conversion in Dashscope

* chore: remove unused imports from test_dashscope.py

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-08-13 22:29:22 +08:00
Willem Jiang ea17e82514 fix: backend server docker instance only listen to localhost (#508) 2025-08-11 19:28:21 +08:00
HagonChan a4d6171c17 fix: tool name mismatch issue (#506) 2025-08-08 19:31:22 +08:00
orifake e1f0595c3c refactor: refine coordinator prompt for Human In The Loop (#505) 2025-08-07 09:00:58 +08:00
Willem Jiang 9e691ecf20 fix: added configuration of python_repl (#503)
* fix: added configuration of python_repl

* fix the lint and unit test errors

* fix the lint and unit test errors

* fix:the lint check errors
2025-08-06 14:27:03 +08:00
Willem Jiang 4218cddab5 fix: langchain-mcp-adapters version conflict (#500)
* fix: langchain-mcp-adapters version conflict

* fix the lint error
2025-08-04 10:36:31 +08:00
Willem Jiang ba7509d9ae fix: build of the web (#492) 2025-07-31 11:54:37 +08:00
Willem Jiang aca9dcf643 fix:try to fix the docker build of front-end (#487) 2025-07-30 09:52:53 +08:00
Willem Jiang 98ef913b88 fix: docker build with uv.lock updated (#486) 2025-07-29 22:42:27 +08:00
suntp e178483971 fix: Add streamable MCP server support (#468)
* fix: Add streamable MCP server support(#349)

* “Revert-timeout”

* fix lint and test check

* modify streamable error notify

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-29 14:04:04 +08:00
HansleCho bedf7d4af2 Feat: Add Wikipedia search engine (#478)
* feat: add Wikipedia search engine

* wikipedia

* make format
2025-07-29 13:58:08 +08:00
alexezio 89c1b689dc fix: dotenv flags error (#472)
Co-authored-by: BENLIAN <benchi.lian@thoughtworks.com>
2025-07-24 16:53:40 +08:00
Zhonghao Liu f92bf0ca22 Feat: Cross-Language Search for RAGFlow (#469)
* cross-language search

* test passed
2025-07-24 16:39:02 +08:00
殷逸维 660395485c remove volengine package (#464) 2025-07-23 06:06:57 +08:00
道心坚定韩道友 32d8e514e1 fix:env AGENT_RECURSION_LIMIT not work (#453)
* fix:env AGENT_RECURSION_LIMIT not work

* fix:add test

* black tests/unit/config/test_configuration.py

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-22 15:23:21 +08:00
Affan Shaikhsurab b197b0f4cb Fix empty tuple agent (#458)
* feat: add support for 'unknown' message agent in MessageListItem and Message type

* fix: update default agent name from 'unknown' to 'planner' in workflow generator

* fix: remove handling for 'unknown' agent in MessageListItem

* fix: remove 'unknown' agent from Message interface

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-22 15:20:12 +08:00
DanielWalnut 6d8853b7c7 refine the research prompt (#460) 2025-07-22 14:49:04 +08:00
DanielWalnut c7edaf3e84 refine the research prompt (#459) 2025-07-22 14:13:10 +08:00
orifake e6ba1fcd82 fix: JSON parse error in link.tsx (#448)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-20 14:14:18 +08:00
Willem Jiang 4d65d20f01 fix: keep applying quick fix for #446 (#450)
* fix: the Backend returns 400 error

* fix: keep applying quick fix

* fix the lint error

* fixed .env.example settings
2025-07-20 14:10:46 +08:00
Willem Jiang ff67366c5c fix: the Backend returns 400 error (#449) 2025-07-20 11:38:18 +08:00
Willem Jiang d34f48819d feat: polish the mcp-server configure feature (#447)
* feat: disable the MCP server configuation by default

* Fixed the lint and test errors

* fix the lint error

* feat:update the mcp config documents and tests

* fixed the lint errors
2025-07-19 09:33:32 +08:00
Willem Jiang 75ad3e0dc6 feat: disable the MCP server configuation by default (#444)
* feat: disable the MCP server configuation by default

* Fixed the lint and test errors

* fix the lint error
2025-07-19 08:39:42 +08:00
DanielWalnut dbb24d7d14 fix: fix the bug introduced by coordinator messages update (#445) 2025-07-18 21:36:13 +08:00
Willem Jiang 933f3bb83a feat: add CORS setting for the backend application (#443)
* feat: add CORS setting for the backend application

* fix the formate issue
2025-07-18 18:04:03 +08:00
道心坚定韩道友 f17b06f206 fix:planner AttributeError 'list' object has no attribute 'get' (#436) 2025-07-18 09:27:15 +08:00
xiaofeng c14c548e0c fix:The console UI directly throws an error when user input is empty (#438) 2025-07-17 18:15:13 +08:00
Kuro Akuta c89b35805d fix: fix the coordinator's forgetting of its own messages. (#433) 2025-07-17 08:36:31 +08:00
DanielWalnut 774473cc18 fix: fix unit test cases for prompt enhancer (#431) 2025-07-16 11:41:43 +08:00
Affan Shaikhsurab b04225b7c8 fix: handle empty agent tuple in streaming workflow (#427)
Prevents IndexError when agent[0] is accessed on empty tuple,
resolving display issues with Gemini 2.0 Flash model.

Fixes #425
2025-07-16 08:59:11 +08:00
DanielWalnut b155e1eca6 refactor: refine the prompt enhancer pipeline (#426) 2025-07-15 19:21:59 +08:00
DanielWalnut 448001f532 refactor: human feedback doesn't need to check enough context (#423) 2025-07-15 18:51:41 +08:00
Willem Jiang 0f118fda92 fix: clean up the builder code (#417)
* fix: clean up the builder code

* fix:reformat the code
2025-07-15 17:22:50 +08:00
Affan Shaikhsurab ae30517f48 Update configuration_guide.md (#416)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-14 19:05:07 +08:00
orifake 8bdc6bfa2d fix: missing i18n message (#410)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-14 18:56:17 +08:00
HagonChan afbcdd68d8 fix: add missing translation for chat.page (#409)
* fix: Error: MISSING_MESSAGE: Could not resolve chat.page` in messages for locale 'en'

Fixed a `MISSING_MESSAGE` error that was occurring on the chat page due to missing translation keys for `chat.page` in the internationalization messages.

* Update en.json
2025-07-14 18:54:01 +08:00
Willem Jiang bf3bcee8e3 fix: main build fix for the merge #237 (#407) 2025-07-13 09:44:28 +08:00
Shiwen Cheng 0c46f8361b feat: support AzureChatOpenAI under configuring azure_endpoint or AZURE_OPENAI_ENDPOINT (#237)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-13 09:27:57 +08:00
Willem Jiang 86a89acac3 fix: update the reasoning model url in conf.yaml.example (#406) 2025-07-13 08:22:37 +08:00
Willem Jiang 2121510f63 fix:catch toolCalls doesn't return validate json (#405)
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-07-12 23:31:43 +08:00
cmq2525 0dc6c16c42 fix: repair_json_output cannot process msgs that do not starts with {, [ or ``` (#384)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-12 23:29:22 +08:00
vvky 5abf8c1f5e fix: correctly remove outermost code block markers in model responses (fix markdown rendering issue) (#386)
* fix: correctly remove outermost code block markers in frontend

* fix: correctly remove outermost quote block markers in 'dropMarkdownQuote'

* fix: correctly remove outermost quote block markers in 'dropMarkdownQuote'

* fix: correctly remove outermost quote block markers in 'dropMarkdownQuote'

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-12 22:19:30 +08:00
Willem Jiang 70b86d8464 feat: add the Chinese i8n support on the setting table (#404)
* feat: Added i8n to the mcp table

* feat: Added i8n to the about table
2025-07-12 21:28:08 +08:00
johnny0120 e1187d7d02 feat: add i18n support and add Chinese (#372)
* feat: add i18n support and add Chinese

* fix: resolve conflicts

* Update en.json with cancle settings

* Update zh.json with settngs cancle

---------

Co-authored-by: johnny0120 <15564476+johnny0120@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-07-12 15:18:28 +08:00
Willem Jiang 136f7eaa4e fix:upgrade uv version to avoid the big change of uv.lock (#402)
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-07-12 14:46:17 +08:00
Willem Jiang 3c46201ff0 fix: fix the lint check errors of the main branch (#403) 2025-07-12 14:43:25 +08:00
yihong 2363b21447 fix: some lint fix using tools (#98)
* fix: some lint fix using tools

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: md lint

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: some lint fix using tools

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: tests

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-12 13:59:02 +08:00
Willem Jiang 0d3255cdae feat: add the vscode unit test debug settings (#346) 2025-07-12 13:27:47 +08:00
Kirk Lin 9f8f060506 feat(llm): Add retry mechanism for LLM API calls (#400)
* feat(llm): Add retry mechanism for LLM API calls

* feat: configure max_retries for LLM calls via conf.yaml

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-12 10:12:07 +08:00
HagonChan dfd4712d9f feat: add Domain Control Features for Tavily Search Engine (#401)
* feat: add Domain Control Features for Tavily Search Engine

* fixed

* chore: update config.md
2025-07-12 08:53:51 +08:00
MaojiaSheng 859c6e3c5d doc: add knowledgebase rag examples in readme (#383)
* doc: add private knowledgebase examples in readme

* doc: add private knowledgebase examples in readme

* doc: add private knowledgebase examples in readme

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-07 13:05:01 +08:00
Willem Jiang d8016809b2 fix: the typo of setup-uv action (#393)
* fix: spine the github hash on the third party actions

* fix: the typo of action

* fix: try to fix the build by specify the action version
2025-07-07 08:43:11 +08:00
Willem Jiang 6c254c0783 fix: spine the github hash on the third party actions (#392) 2025-07-07 08:18:17 +08:00
殷逸维 d4fbc86b28 fix: docker build (#385)
* fix: docker build

* modify base docker image
2025-07-05 11:07:52 +08:00
Abeautifulsnow 7ad11bf86c refactor: simplify style mapping by using upper case only (#378)
* improve: add abort btn to abort the mcp add request.

* refactor: simplify style mapping by using upper case only

* format: execute uv run black --preview . to format python files.
2025-07-04 08:27:20 +08:00
殷逸维 be893eae2b feat: integrate VikingDB Knowledge Base into rag retrieving tool (#381)
Co-authored-by: Henry Li <henry1943@163.com>
2025-07-03 10:06:42 +08:00
Johannes Maron 5977b4a03e Publish containers to GitHub (#375)
This workflow creates two offical container images:

* `ghcr.io/codingjoe/deer-flow:main`
* `ghcr.io/codingjoe/deer-flow-web:main`
2025-06-29 20:55:51 +08:00
JeffJiang 52dfdd83ae fix: next server fetch error (#374) 2025-06-27 14:23:04 +08:00
Willem Jiang f27c96e692 fix: the lint error of llm.py (#369) 2025-06-26 10:36:26 +08:00
Tony M b7373fbe70 Add support for self-signed certs from model providers (#276)
* Add support for self-signed certs from model providers

* cleanup

---------

Co-authored-by: tonydoesathing <tmastromarino@cpacket.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-06-26 10:17:26 +08:00
Abeautifulsnow 9c2d4724e3 improve: add abort btn to abort the mcp add request. (#284) 2025-06-26 08:51:46 +08:00
johnny0120 aa06cd6fb6 fix: replace json before js fence (#344) 2025-06-26 08:40:32 +08:00
Young 82e1b65792 fix: settings tab display name (#250) 2025-06-19 14:33:00 +08:00
Willem Jiang dcdd7288ed test: add unit tests of the app (#305)
* test: add unit tests in server

* test: add unit tests of app.py in server

* test: reformat the codes

* test: add more tests to cover the exception part

* test: add more tests on the server app part

* fix: don't show the detail exception to the client

* test: try to fix the CI test

* fix: keep the TTS API call without exposure information

* Fixed the unit test errors

* Fixed the lint error
2025-06-18 14:13:05 +08:00
Willem Jiang 89f3d731c9 Fix: the test errors of test_nodes (#345) 2025-06-18 11:59:33 +08:00
Willem Jiang c0b04aaba2 test: add unit tests for graph (#296)
* test: added unit test of builder

* test: Add unit tests for nodes.py

* test: add more unit tests in test_nodes

* test: try to fix the unit test error on GitHub

* test: reformate the code of test_nodes.py

* Fix the test error of reset the local argument

* Fixed the test error by setup args

* reformat the code
2025-06-18 10:05:02 +08:00
Willem Jiang 4048ca67dd test: add test of json_utils (#309)
* test: add test of json_utils

* reformat the code
2025-06-18 10:04:46 +08:00
lelili2021 30a189cf26 fix: update several links related to volcengine in Readme (#333) 2025-06-17 08:49:29 +08:00
Ryan Guo e03b12b97f doc: provide a workable guideline update for ollama user (#323) 2025-06-17 08:47:54 +08:00
Luludle 8823ffdb6a fix: add line breaks to mcp edit dialog (#313) 2025-06-17 08:31:35 +08:00
3Spiders 4fe43153b1 fix(web): priority displayName for settings name error (#336) 2025-06-17 08:26:13 +08:00
Willem Jiang 4fb053b6d2 Revert "fix: solves the malformed json output and pydantic validation error p…" (#325)
This reverts commit a7315b46df.
2025-06-14 22:04:03 +08:00
DanielWalnut 19fa1e97c3 feat: add deep think feature (#311)
* feat: implement backend logic

* feat: implement api/config endpoint

* rename the symbol

* feat: re-implement configuration at client-side

* feat: add client-side of deep thinking

* fix backend bug

* feat: add reasoning block

* docs: update readme

* fix: translate into English

* fix: change icon to lightbulb

* feat: ignore more bad cases

* feat: adjust thinking layout, and implement auto scrolling

* docs: add comments

---------

Co-authored-by: Henry Li <henry1943@163.com>
2025-06-14 13:12:43 +08:00
Tax a7315b46df fix: solves the malformed json output and pydantic validation error produced by the 'planner' node by forcing the llm response to strictly comply with the pydantic 'Plan' model (#322) 2025-06-14 10:13:30 +08:00
JeffJiang 03e6a1a6e7 fix: mcp config styles (#320) 2025-06-13 18:01:19 +08:00
Lan 7d38e5f900 feat: append try catch (#280) 2025-06-12 20:43:50 +08:00
Willem Jiang 4c2fe2e7f5 test: add more unit tests of tools (#315)
* test: add more test on test_tts.py

* test: add unit test of search and retriever in tools

* test: remove the main code of search.py

* test: add the travily_search unit test

* reformate the codes

* test: add unit tests of tools

* Added the pytest-asyncio dependency

* added the license header of test_tavily_search_api_wrapper.py
2025-06-12 20:43:32 +08:00
Henry Li bb7dc6e98c docs: add VolcEngine introduction. (#314) 2025-06-12 13:36:29 +08:00
1325 changed files with 211824 additions and 92287 deletions
+181
View File
@@ -0,0 +1,181 @@
---
name: smoke-test
description: End-to-end smoke test skill for DeerFlow. Guides through: 1) Pulling latest code, 2) Docker OR Local installation and deployment (user preference, default to Local if Docker network issues), 3) Service availability verification, 4) Health check, 5) Final test report. Use when the user says "run smoke test", "smoke test deployment", "verify installation", "test service availability", "end-to-end test", or similar.
---
# DeerFlow Smoke Test Skill
This skill guides the Agent through DeerFlow's full end-to-end smoke test workflow, including code updates, deployment (supporting both Docker and local installation modes), service availability verification, and health checks.
## Deployment Mode Selection
This skill supports two deployment modes:
- **Local installation mode** (recommended, especially when network issues occur) - Run all services directly on the local machine
- **Docker mode** - Run all services inside Docker containers
**Selection strategy**:
- If the user explicitly asks for Docker mode, use Docker
- If network issues occur (such as slow image pulls), automatically switch to local mode
- Default to local mode whenever possible
## Structure
```
smoke-test/
├── SKILL.md ← You are here - core workflow and logic
├── scripts/
│ ├── check_docker.sh ← Check the Docker environment
│ ├── check_local_env.sh ← Check local environment dependencies
│ ├── frontend_check.sh ← Frontend page smoke check
│ ├── pull_code.sh ← Pull the latest code
│ ├── deploy_docker.sh ← Docker deployment
│ ├── deploy_local.sh ← Local deployment
│ └── health_check.sh ← Service health check
├── references/
│ ├── SOP.md ← Standard operating procedure
│ └── troubleshooting.md ← Troubleshooting guide
└── templates/
├── report.local.template.md ← Local mode smoke test report template
└── report.docker.template.md ← Docker mode smoke test report template
```
## Standard Operating Procedure (SOP)
### Phase 1: Code Update Check
1. **Confirm current directory** - Verify that the current working directory is the DeerFlow project root
2. **Check Git status** - See whether there are uncommitted changes
3. **Pull the latest code** - Use `git pull origin main` to get the latest updates
4. **Confirm code update** - Verify that the latest code was pulled successfully
### Phase 2: Deployment Mode Selection and Environment Check
**Choose deployment mode**:
- Ask for user preference, or choose automatically based on network conditions
- Default to local installation mode
**Local mode environment check**:
1. **Check Node.js version** - Requires 22+
2. **Check pnpm** - Package manager
3. **Check uv** - Python package manager
4. **Check nginx** - Reverse proxy
5. **Check required ports** - Confirm that ports 2026, 3000, 8001, and 2024 are not occupied
**Docker mode environment check** (if Docker is selected):
1. **Check whether Docker is installed** - Run `docker --version`
2. **Check Docker daemon status** - Run `docker info`
3. **Check Docker Compose availability** - Run `docker compose version`
4. **Check required ports** - Confirm that port 2026 is not occupied
### Phase 3: Configuration Preparation
1. **Check whether config.yaml exists**
- If it does not exist, run `make config` to generate it
- If it already exists, check whether it needs an upgrade with `make config-upgrade`
2. **Check the .env file**
- Verify that required environment variables are configured
- Especially model API keys such as `OPENAI_API_KEY`
### Phase 4: Deployment Execution
**Local mode deployment**:
1. **Check dependencies** - Run `make check`
2. **Install dependencies** - Run `make install`
3. **(Optional) Pre-pull the sandbox image** - If needed, run `make setup-sandbox`
4. **Start services** - Run `make dev-daemon` (background mode, recommended) or `make dev` (foreground mode)
5. **Wait for startup** - Give all services enough time to start completely (90-120 seconds recommended)
**Docker mode deployment** (if Docker is selected):
1. **Initialize Docker environment** - Run `make docker-init`
2. **Start Docker services** - Run `make docker-start`
3. **Wait for startup** - Give all containers enough time to start completely (60 seconds recommended)
### Phase 5: Service Health Check
**Local mode health check**:
1. **Check process status** - Confirm that LangGraph, Gateway, Frontend, and Nginx processes are all running
2. **Check frontend service** - Visit `http://localhost:2026` and verify that the page loads
3. **Check API Gateway** - Verify the `http://localhost:2026/health` endpoint
4. **Check LangGraph service** - Verify the availability of relevant endpoints
5. **Frontend route smoke check** - Run `bash .agent/skills/smoke-test/scripts/frontend_check.sh` to verify key routes under `/workspace`
**Docker mode health check** (when using Docker):
1. **Check container status** - Run `docker ps` and confirm that all containers are running
2. **Check frontend service** - Visit `http://localhost:2026` and verify that the page loads
3. **Check API Gateway** - Verify the `http://localhost:2026/health` endpoint
4. **Check LangGraph service** - Verify the availability of relevant endpoints
5. **Frontend route smoke check** - Run `bash .agent/skills/smoke-test/scripts/frontend_check.sh` to verify key routes under `/workspace`
### Optional Functional Verification
1. **List available models** - Verify that model configuration loads correctly
2. **List available skills** - Verify that the skill directory is mounted correctly
3. **Simple chat test** - Send a simple message to verify the end-to-end flow
### Phase 6: Generate Test Report
1. **Collect all test results** - Summarize execution status for each phase
2. **Record encountered issues** - If anything fails, record the error details
3. **Generate the final report** - Use the template that matches the selected deployment mode to create the complete test report, including overall conclusion, detailed key test cases, and explicit frontend page / route results
4. **Provide follow-up recommendations** - Offer suggestions based on the test results
## Execution Rules
- **Follow the sequence** - Execute strictly in the order described above
- **Idempotency** - Every step should be safe to repeat
- **Error handling** - If a step fails, stop and report the issue, then provide troubleshooting suggestions
- **Detailed logging** - Record the execution result and status of each step
- **User confirmation** - Ask for confirmation before potentially risky operations such as overwriting config
- **Mode preference** - Prefer local mode to avoid network-related issues
- **Template requirement** - The final report must use the matching template under `templates/`; do not output a free-form summary instead of the template-based report
- **Report clarity** - The execution summary must include the overall pass/fail conclusion plus per-case result explanations, and frontend smoke check results must be listed explicitly in the report
- **Optional phase handling** - If functional verification is not executed, do not present it as a separate skipped phase in the final report
## Known Acceptable Warnings
The following warnings can appear during smoke testing and do not block a successful result:
- Feishu/Lark SSL errors in Gateway logs (certificate verification failure) can be ignored if that channel is not enabled
- Warnings in LangGraph logs about missing methods in the custom checkpointer, such as `adelete_for_runs` or `aprune`, do not affect the core functionality
## Key Tools
Use the following tools during execution:
1. **bash** - Run shell commands
2. **present_file** - Show generated reports and important files
3. **task_tool** - Organize complex steps with subtasks when needed
## Success Criteria
Smoke test pass criteria (local mode):
- [x] Latest code is pulled successfully
- [x] Local environment check passes (Node.js 22+, pnpm, uv, nginx)
- [x] Configuration files are set up correctly
- [x] `make check` passes
- [x] `make install` completes successfully
- [x] `make dev` starts successfully
- [x] All service processes run normally
- [x] Frontend page is accessible
- [x] Frontend route smoke check passes (`/workspace` key routes)
- [x] API Gateway health check passes
- [x] Test report is generated completely
Smoke test pass criteria (Docker mode):
- [x] Latest code is pulled successfully
- [x] Docker environment check passes
- [x] Configuration files are set up correctly
- [x] `make docker-init` completes successfully
- [x] `make docker-start` completes successfully
- [x] All Docker containers run normally
- [x] Frontend page is accessible
- [x] Frontend route smoke check passes (`/workspace` key routes)
- [x] API Gateway health check passes
- [x] Test report is generated completely
## Read Reference Files
Before starting execution, read the following reference files:
1. `references/SOP.md` - Detailed step-by-step operating instructions
2. `references/troubleshooting.md` - Common issues and solutions
3. `templates/report.local.template.md` - Local mode test report template
4. `templates/report.docker.template.md` - Docker mode test report template
+452
View File
@@ -0,0 +1,452 @@
# DeerFlow Smoke Test Standard Operating Procedure (SOP)
This document describes the detailed operating steps for each phase of the DeerFlow smoke test.
## Phase 1: Code Update Check
### 1.1 Confirm Current Directory
**Objective**: Verify that the current working directory is the DeerFlow project root.
**Steps**:
1. Run `pwd` to view the current working directory
2. Check whether the directory contains the following files/directories:
- `Makefile`
- `backend/`
- `frontend/`
- `config.example.yaml`
**Success Criteria**: The current directory contains all of the files/directories listed above.
---
### 1.2 Check Git Status
**Objective**: Check whether there are uncommitted changes.
**Steps**:
1. Run `git status`
2. Check whether the output includes "Changes not staged for commit" or "Untracked files"
**Notes**:
- If there are uncommitted changes, recommend that the user commit or stash them first to avoid conflicts while pulling
- If the user confirms that they want to continue, this step can be skipped
---
### 1.3 Pull the Latest Code
**Objective**: Fetch the latest code updates.
**Steps**:
1. Run `git fetch origin main`
2. Run `git pull origin main`
**Success Criteria**:
- The commands succeed without errors
- The output shows "Already up to date" or indicates that new commits were pulled successfully
---
### 1.4 Confirm Code Update
**Objective**: Verify that the latest code was pulled successfully.
**Steps**:
1. Run `git log -1 --oneline` to view the latest commit
2. Record the commit hash and message
---
## Phase 2: Deployment Mode Selection and Environment Check
### 2.1 Choose Deployment Mode
**Objective**: Decide whether to use local mode or Docker mode.
**Decision Flow**:
1. Prefer local mode first to avoid network-related issues
2. If the user explicitly requests Docker, use Docker
3. If Docker network issues occur, switch to local mode automatically
---
### 2.2 Local Mode Environment Check
**Objective**: Verify that local development environment dependencies are satisfied.
#### 2.2.1 Check Node.js Version
**Steps**:
1. If nvm is used, run `nvm use 22` to switch to Node 22+
2. Run `node --version`
**Success Criteria**: Version >= 22.x
**Failure Handling**:
- If the version is too low, ask the user to install/switch Node.js with nvm:
```bash
nvm install 22
nvm use 22
```
- Or install it from the official website: https://nodejs.org/
---
#### 2.2.2 Check pnpm
**Steps**:
1. Run `pnpm --version`
**Success Criteria**: The command returns pnpm version information.
**Failure Handling**:
- If pnpm is not installed, ask the user to install it with `npm install -g pnpm`
---
#### 2.2.3 Check uv
**Steps**:
1. Run `uv --version`
**Success Criteria**: The command returns uv version information.
**Failure Handling**:
- If uv is not installed, ask the user to install uv
---
#### 2.2.4 Check nginx
**Steps**:
1. Run `nginx -v`
**Success Criteria**: The command returns nginx version information.
**Failure Handling**:
- macOS: install with Homebrew using `brew install nginx`
- Linux: install using the system package manager
---
#### 2.2.5 Check Required Ports
**Steps**:
1. Run the following commands to check ports:
```bash
lsof -i :2026 # Main port
lsof -i :3000 # Frontend
lsof -i :8001 # Gateway
lsof -i :2024 # LangGraph
```
**Success Criteria**: All ports are free, or they are occupied only by DeerFlow-related processes.
**Failure Handling**:
- If a port is occupied, ask the user to stop the related process
---
### 2.3 Docker Mode Environment Check (If Docker Is Selected)
#### 2.3.1 Check Whether Docker Is Installed
**Steps**:
1. Run `docker --version`
**Success Criteria**: The command returns Docker version information, such as "Docker version 24.x.x".
---
#### 2.3.2 Check Docker Daemon Status
**Steps**:
1. Run `docker info`
**Success Criteria**: The command runs successfully and shows Docker system information.
**Failure Handling**:
- If it fails, ask the user to start Docker Desktop or the Docker service
---
#### 2.3.3 Check Docker Compose Availability
**Steps**:
1. Run `docker compose version`
**Success Criteria**: The command returns Docker Compose version information.
---
#### 2.3.4 Check Required Ports
**Steps**:
1. Run `lsof -i :2026` (macOS/Linux) or `netstat -ano | findstr :2026` (Windows)
**Success Criteria**: Port 2026 is free, or it is occupied only by a DeerFlow-related process.
**Failure Handling**:
- If the port is occupied by another process, ask the user to stop that process or change the configuration
---
## Phase 3: Configuration Preparation
### 3.1 Check config.yaml
**Steps**:
1. Check whether `config.yaml` exists
2. If it does not exist, run `make config`
3. If it already exists, consider running `make config-upgrade` to merge new fields
**Validation**:
- Check whether at least one model is configured in config.yaml
- Check whether the model configuration references the correct environment variables
---
### 3.2 Check the .env File
**Steps**:
1. Check whether the `.env` file exists
2. If it does not exist, copy it from `.env.example`
3. Check whether the following environment variables are configured:
- `OPENAI_API_KEY` (or other model API keys)
- Other required settings
---
## Phase 4: Deployment Execution
### 4.1 Local Mode Deployment
#### 4.1.1 Check Dependencies
**Steps**:
1. Run `make check`
**Description**: This command validates all required tools (Node.js 22+, pnpm, uv, nginx).
---
#### 4.1.2 Install Dependencies
**Steps**:
1. Run `make install`
**Description**: This command installs both backend and frontend dependencies.
**Notes**:
- This step may take some time
- If network issues cause failures, try using a closer or mirrored package registry
---
#### 4.1.3 (Optional) Pre-pull the Sandbox Image
**Steps**:
1. If Docker / Container sandbox is used, run `make setup-sandbox`
**Description**: This step is optional and not needed for local sandbox mode.
---
#### 4.1.4 Start Services
**Steps**:
1. Run `make dev-daemon` (background mode)
**Description**: This command starts all services (LangGraph, Gateway, Frontend, Nginx).
**Notes**:
- `make dev` runs in the foreground and stops with Ctrl+C
- `make dev-daemon` runs in the background
- Use `make stop` to stop services
---
#### 4.1.5 Wait for Services to Start
**Steps**:
1. Wait 90-120 seconds for all services to start completely
2. You can monitor startup progress by checking these log files:
- `logs/langgraph.log`
- `logs/gateway.log`
- `logs/frontend.log`
- `logs/nginx.log`
---
### 4.2 Docker Mode Deployment (If Docker Is Selected)
#### 4.2.1 Initialize the Docker Environment
**Steps**:
1. Run `make docker-init`
**Description**: This command pulls the sandbox image if needed.
---
#### 4.2.2 Start Docker Services
**Steps**:
1. Run `make docker-start`
**Description**: This command builds and starts all required Docker containers.
---
#### 4.2.3 Wait for Services to Start
**Steps**:
1. Wait 60-90 seconds for all services to start completely
2. You can run `make docker-logs` to monitor startup progress
---
## Phase 5: Service Health Check
### 5.1 Local Mode Health Check
#### 5.1.1 Check Process Status
**Steps**:
1. Run the following command to check processes:
```bash
ps aux | grep -E "(langgraph|uvicorn|next|nginx)" | grep -v grep
```
**Success Criteria**: Confirm that the following processes are running:
- LangGraph (`langgraph dev`)
- Gateway (`uvicorn app.gateway.app:app`)
- Frontend (`next dev` or `next start`)
- Nginx (`nginx`)
---
#### 5.1.2 Check Frontend Service
**Steps**:
1. Use curl or a browser to visit `http://localhost:2026`
2. Verify that the page loads normally
**Example curl command**:
```bash
curl -I http://localhost:2026
```
**Success Criteria**: Returns an HTTP 200 status code.
---
#### 5.1.3 Check API Gateway
**Steps**:
1. Visit `http://localhost:2026/health`
**Example curl command**:
```bash
curl http://localhost:2026/health
```
**Success Criteria**: Returns health status JSON.
---
#### 5.1.4 Check LangGraph Service
**Steps**:
1. Visit relevant LangGraph endpoints to verify availability
---
### 5.2 Docker Mode Health Check (When Using Docker)
#### 5.2.1 Check Container Status
**Steps**:
1. Run `docker ps`
2. Confirm that the following containers are running:
- `deer-flow-nginx`
- `deer-flow-frontend`
- `deer-flow-gateway`
- `deer-flow-langgraph` (if not in gateway mode)
---
#### 5.2.2 Check Frontend Service
**Steps**:
1. Use curl or a browser to visit `http://localhost:2026`
2. Verify that the page loads normally
**Example curl command**:
```bash
curl -I http://localhost:2026
```
**Success Criteria**: Returns an HTTP 200 status code.
---
#### 5.2.3 Check API Gateway
**Steps**:
1. Visit `http://localhost:2026/health`
**Example curl command**:
```bash
curl http://localhost:2026/health
```
**Success Criteria**: Returns health status JSON.
---
#### 5.2.4 Check LangGraph Service
**Steps**:
1. Visit relevant LangGraph endpoints to verify availability
---
## Optional Functional Verification
### 6.1 List Available Models
**Steps**: Verify the model list through the API or UI.
---
### 6.2 List Available Skills
**Steps**: Verify the skill list through the API or UI.
---
### 6.3 Simple Chat Test
**Steps**: Send a simple message to test the complete workflow.
---
## Phase 6: Generate the Test Report
### 6.1 Collect Test Results
Summarize the execution status of each phase and record successful and failed items.
### 6.2 Record Issues
If anything fails, record detailed error information.
### 6.3 Generate the Report
Use the template to create a complete test report.
### 6.4 Provide Recommendations
Provide follow-up recommendations based on the test results.
@@ -0,0 +1,612 @@
# Troubleshooting Guide
This document lists common issues encountered during DeerFlow smoke testing and how to resolve them.
## Code Update Issues
### Issue: `git pull` Fails with a Merge Conflict Warning
**Symptoms**:
```
error: Your local changes to the following files would be overwritten by merge
```
**Solutions**:
1. Option A: Commit local changes first
```bash
git add .
git commit -m "Save local changes"
git pull origin main
```
2. Option B: Stash local changes
```bash
git stash
git pull origin main
git stash pop # Restore changes later if needed
```
3. Option C: Discard local changes (use with caution)
```bash
git reset --hard HEAD
git pull origin main
```
---
## Local Mode Environment Issues
### Issue: Node.js Version Is Too Old
**Symptoms**:
```
Node.js version is too old. Requires 22+, got x.x.x
```
**Solutions**:
1. Install or upgrade Node.js with nvm:
```bash
nvm install 22
nvm use 22
```
2. Or download and install it from the official website: https://nodejs.org/
3. Verify the version:
```bash
node --version
```
---
### Issue: pnpm Is Not Installed
**Symptoms**:
```
command not found: pnpm
```
**Solutions**:
1. Install pnpm with npm:
```bash
npm install -g pnpm
```
2. Or use the official installation script:
```bash
curl -fsSL https://get.pnpm.io/install.sh | sh -
```
3. Verify the installation:
```bash
pnpm --version
```
---
### Issue: uv Is Not Installed
**Symptoms**:
```
command not found: uv
```
**Solutions**:
1. Use the official installation script:
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```
2. macOS users can also install it with Homebrew:
```bash
brew install uv
```
3. Verify the installation:
```bash
uv --version
```
---
### Issue: nginx Is Not Installed
**Symptoms**:
```
command not found: nginx
```
**Solutions**:
1. macOS (Homebrew):
```bash
brew install nginx
```
2. Ubuntu/Debian:
```bash
sudo apt update
sudo apt install nginx
```
3. CentOS/RHEL:
```bash
sudo yum install nginx
```
4. Verify the installation:
```bash
nginx -v
```
---
### Issue: Port Is Already in Use
**Symptoms**:
```
Error: listen EADDRINUSE: address already in use :::2026
```
**Solutions**:
1. Find the process using the port:
```bash
lsof -i :2026 # macOS/Linux
netstat -ano | findstr :2026 # Windows
```
2. Stop that process:
```bash
kill -9 <PID> # macOS/Linux
taskkill /PID <PID> /F # Windows
```
3. Or stop DeerFlow services first:
```bash
make stop
```
---
## Local Mode Dependency Installation Issues
### Issue: `make install` Fails Due to Network Timeout
**Symptoms**:
Network timeouts or connection failures occur during dependency installation.
**Solutions**:
1. Configure pnpm to use a mirror registry:
```bash
pnpm config set registry https://registry.npmmirror.com
```
2. Configure uv to use a mirror registry:
```bash
uv pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
```
3. Retry the installation:
```bash
make install
```
---
### Issue: Python Dependency Installation Fails
**Symptoms**:
Errors occur during `uv sync`.
**Solutions**:
1. Clean the uv cache:
```bash
cd backend
uv cache clean
```
2. Resync dependencies:
```bash
cd backend
uv sync
```
3. View detailed error logs:
```bash
cd backend
uv sync --verbose
```
---
### Issue: Frontend Dependency Installation Fails
**Symptoms**:
Errors occur during `pnpm install`.
**Solutions**:
1. Clean the pnpm cache:
```bash
cd frontend
pnpm store prune
```
2. Remove node_modules and the lock file:
```bash
cd frontend
rm -rf node_modules pnpm-lock.yaml
```
3. Reinstall:
```bash
cd frontend
pnpm install
```
---
## Local Mode Service Startup Issues
### Issue: Services Exit Immediately After Startup
**Symptoms**:
Processes exit quickly after running `make dev-daemon`.
**Solutions**:
1. Check log files:
```bash
tail -f logs/langgraph.log
tail -f logs/gateway.log
tail -f logs/frontend.log
tail -f logs/nginx.log
```
2. Check whether config.yaml is configured correctly
3. Check environment variables in the .env file
4. Confirm that required ports are not occupied
5. Stop all services and restart:
```bash
make stop
make dev-daemon
```
---
### Issue: Nginx Fails to Start Because Temp Directories Do Not Exist
**Symptoms**:
```
nginx: [emerg] mkdir() "/opt/homebrew/var/run/nginx/client_body_temp" failed (2: No such file or directory)
```
**Solutions**:
Add local temp directory configuration to `docker/nginx/nginx.local.conf` so nginx uses the repository's temp directory.
Add the following at the beginning of the `http` block:
```nginx
client_body_temp_path temp/client_body_temp;
proxy_temp_path temp/proxy_temp;
fastcgi_temp_path temp/fastcgi_temp;
uwsgi_temp_path temp/uwsgi_temp;
scgi_temp_path temp/scgi_temp;
```
Note: The `temp/` directory under the repository root is created automatically by `make dev` or `make dev-daemon`.
---
### Issue: Nginx Fails to Start (General)
**Symptoms**:
The nginx process fails to start or reports an error.
**Solutions**:
1. Check the nginx configuration:
```bash
nginx -t -c docker/nginx/nginx.local.conf -p .
```
2. Check nginx logs:
```bash
tail -f logs/nginx.log
```
3. Ensure no other nginx process is running:
```bash
ps aux | grep nginx
```
4. If needed, stop existing nginx processes:
```bash
pkill -9 nginx
```
---
### Issue: Frontend Compilation Fails
**Symptoms**:
Compilation errors appear in `frontend.log`.
**Solutions**:
1. Check frontend logs:
```bash
tail -f logs/frontend.log
```
2. Check whether Node.js version is 22+
3. Reinstall frontend dependencies:
```bash
cd frontend
rm -rf node_modules .next
pnpm install
```
4. Restart services:
```bash
make stop
make dev-daemon
```
---
### Issue: Gateway Fails to Start
**Symptoms**:
Errors appear in `gateway.log`.
**Solutions**:
1. Check gateway logs:
```bash
tail -f logs/gateway.log
```
2. Check whether config.yaml exists and has valid formatting
3. Check whether Python dependencies are complete:
```bash
cd backend
uv sync
```
4. Confirm that the LangGraph service is running normally (if not in gateway mode)
---
### Issue: LangGraph Fails to Start
**Symptoms**:
Errors appear in `langgraph.log`.
**Solutions**:
1. Check LangGraph logs:
```bash
tail -f logs/langgraph.log
```
2. Check config.yaml
3. Check whether Python dependencies are complete
4. Confirm that port 2024 is not occupied
---
## Docker-Related Issues
### Issue: Docker Commands Cannot Run
**Symptoms**:
```
Cannot connect to the Docker daemon
```
**Solutions**:
1. Confirm that Docker Desktop is running
2. macOS: check whether the Docker icon appears in the top menu bar
3. Linux: run `sudo systemctl start docker`
4. Run `docker info` again to verify
---
### Issue: `make docker-init` Fails to Pull the Image
**Symptoms**:
```
Error pulling image: connection refused
```
**Solutions**:
1. Check network connectivity
2. Configure a Docker image mirror if needed
3. Check whether a proxy is required
4. Switch to local installation mode if necessary (recommended)
---
## Configuration File Issues
### Issue: config.yaml Is Missing or Invalid
**Symptoms**:
```
Error: could not read config.yaml
```
**Solutions**:
1. Regenerate the configuration file:
```bash
make config
```
2. Check YAML syntax:
- Make sure indentation is correct (use 2 spaces)
- Make sure there are no tab characters
- Check that there is a space after each colon
3. Use a YAML validation tool to check the format
---
### Issue: Model API Key Is Not Configured
**Symptoms**:
After services start, API requests fail with authentication errors.
**Solutions**:
1. Edit the .env file and add the API key:
```bash
OPENAI_API_KEY=your-actual-api-key-here
```
2. Restart services (local mode):
```bash
make stop
make dev-daemon
```
3. Restart services (Docker mode):
```bash
make docker-stop
make docker-start
```
4. Confirm that the model configuration in config.yaml references the environment variable correctly
---
## Service Health Check Issues
### Issue: Frontend Page Is Not Accessible
**Symptoms**:
The browser shows a connection failure when visiting http://localhost:2026.
**Solutions** (local mode):
1. Confirm that the nginx process is running:
```bash
ps aux | grep nginx
```
2. Check nginx logs:
```bash
tail -f logs/nginx.log
```
3. Check firewall settings
**Solutions** (Docker mode):
1. Confirm that the nginx container is running:
```bash
docker ps | grep nginx
```
2. Check nginx logs:
```bash
cd docker && docker compose -p deer-flow-dev -f docker-compose-dev.yaml logs nginx
```
3. Check firewall settings
---
### Issue: API Gateway Health Check Fails
**Symptoms**:
Accessing `/health` returns an error or times out.
**Solutions** (local mode):
1. Check gateway logs:
```bash
tail -f logs/gateway.log
```
2. Confirm that config.yaml exists and has valid formatting
3. Check whether Python dependencies are complete
4. Confirm that the LangGraph service is running normally
**Solutions** (Docker mode):
1. Check gateway container logs:
```bash
make docker-logs-gateway
```
2. Confirm that config.yaml is mounted correctly
3. Check whether Python dependencies are complete
4. Confirm that the LangGraph service is running normally
---
## Common Diagnostic Commands
### Local Mode Diagnostics
#### View All Service Processes
```bash
ps aux | grep -E "(langgraph|uvicorn|next|nginx)" | grep -v grep
```
#### View Service Logs
```bash
# View all logs
tail -f logs/*.log
# View specific service logs
tail -f logs/langgraph.log
tail -f logs/gateway.log
tail -f logs/frontend.log
tail -f logs/nginx.log
```
#### Stop All Services
```bash
make stop
```
#### Fully Reset the Local Environment
```bash
make stop
make clean
make config
make install
make dev-daemon
```
---
### Docker Mode Diagnostics
#### View All Container Status
```bash
docker ps -a
```
#### View Container Resource Usage
```bash
docker stats
```
#### Enter a Container for Debugging
```bash
docker exec -it deer-flow-gateway sh
```
#### Clean Up All DeerFlow-Related Containers and Images
```bash
make docker-stop
cd docker && docker compose -p deer-flow-dev -f docker-compose-dev.yaml down -v
```
#### Fully Reset the Docker Environment
```bash
make docker-stop
make clean
make config
make docker-init
make docker-start
```
---
## Get More Help
If the solutions above do not resolve the issue:
1. Check the GitHub issues for the project: https://github.com/bytedance/deer-flow/issues
2. Review the project documentation: README.md and the `backend/docs/` directory
3. Open a new issue and include detailed error logs
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env bash
set -e
echo "=========================================="
echo " Checking Docker Environment"
echo "=========================================="
echo ""
# Check whether Docker is installed
if command -v docker >/dev/null 2>&1; then
echo "✓ Docker is installed"
docker --version
else
echo "✗ Docker is not installed"
exit 1
fi
echo ""
# Check the Docker daemon
if docker info >/dev/null 2>&1; then
echo "✓ Docker daemon is running normally"
else
echo "✗ Docker daemon is not running"
echo " Please start Docker Desktop or the Docker service"
exit 1
fi
echo ""
# Check Docker Compose
if docker compose version >/dev/null 2>&1; then
echo "✓ Docker Compose is available"
docker compose version
else
echo "✗ Docker Compose is not available"
exit 1
fi
echo ""
# Check port 2026
if ! command -v lsof >/dev/null 2>&1; then
echo "✗ lsof is required to check whether port 2026 is available"
exit 1
fi
port_2026_usage="$(lsof -nP -iTCP:2026 -sTCP:LISTEN 2>/dev/null || true)"
if [ -n "$port_2026_usage" ]; then
echo "⚠ Port 2026 is already in use"
echo " Occupying process:"
echo "$port_2026_usage"
deerflow_process_found=0
while IFS= read -r pid; do
if [ -z "$pid" ]; then
continue
fi
process_command="$(ps -p "$pid" -o command= 2>/dev/null || true)"
case "$process_command" in
*[Dd]eer[Ff]low*|*[Dd]eerflow*|*[Nn]ginx*deerflow*|*deerflow/*[Nn]ginx*)
deerflow_process_found=1
;;
esac
done <<EOF
$(printf '%s\n' "$port_2026_usage" | awk 'NR > 1 {print $2}')
EOF
if [ "$deerflow_process_found" -eq 1 ]; then
echo "✓ Port 2026 is occupied by DeerFlow"
else
echo "✗ Port 2026 must be free before starting DeerFlow"
exit 1
fi
else
echo "✓ Port 2026 is available"
fi
echo ""
echo "=========================================="
echo " Docker Environment Check Complete"
echo "=========================================="
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env bash
set -e
echo "=========================================="
echo " Checking Local Development Environment"
echo "=========================================="
echo ""
all_passed=true
# Check Node.js
echo "1. Checking Node.js..."
if command -v node >/dev/null 2>&1; then
NODE_VERSION=$(node --version | sed 's/v//')
NODE_MAJOR=$(echo "$NODE_VERSION" | cut -d. -f1)
if [ "$NODE_MAJOR" -ge 22 ]; then
echo "✓ Node.js is installed (version: $NODE_VERSION)"
else
echo "✗ Node.js version is too old (current: $NODE_VERSION, required: 22+)"
all_passed=false
fi
else
echo "✗ Node.js is not installed"
all_passed=false
fi
echo ""
# Check pnpm
echo "2. Checking pnpm..."
if command -v pnpm >/dev/null 2>&1; then
echo "✓ pnpm is installed (version: $(pnpm --version))"
else
echo "✗ pnpm is not installed"
echo " Install command: npm install -g pnpm"
all_passed=false
fi
echo ""
# Check uv
echo "3. Checking uv..."
if command -v uv >/dev/null 2>&1; then
echo "✓ uv is installed (version: $(uv --version))"
else
echo "✗ uv is not installed"
all_passed=false
fi
echo ""
# Check nginx
echo "4. Checking nginx..."
if command -v nginx >/dev/null 2>&1; then
echo "✓ nginx is installed (version: $(nginx -v 2>&1))"
else
echo "✗ nginx is not installed"
echo " macOS: brew install nginx"
echo " Linux: install it with the system package manager"
all_passed=false
fi
echo ""
# Check ports
echo "5. Checking ports..."
if ! command -v lsof >/dev/null 2>&1; then
echo "✗ lsof is not installed, so port availability cannot be verified"
echo " Install lsof and rerun this check"
all_passed=false
else
for port in 2026 3000 8001 2024; do
if lsof -i :$port >/dev/null 2>&1; then
echo "⚠ Port $port is already in use:"
lsof -i :$port | head -2
all_passed=false
else
echo "✓ Port $port is available"
fi
done
fi
echo ""
# Summary
echo "=========================================="
echo " Environment Check Summary"
echo "=========================================="
echo ""
if [ "$all_passed" = true ]; then
echo "✅ All environment checks passed!"
echo ""
echo "Next step: run make install to install dependencies"
exit 0
else
echo "❌ Some checks failed. Please fix the issues above first"
exit 1
fi
+65
View File
@@ -0,0 +1,65 @@
#!/usr/bin/env bash
set -e
echo "=========================================="
echo " Docker Deployment"
echo "=========================================="
echo ""
# Check config.yaml
if [ ! -f "config.yaml" ]; then
echo "config.yaml does not exist. Generating it..."
make config
echo ""
echo "⚠ Please edit config.yaml to configure your models and API keys"
echo " Then run this script again"
exit 1
else
echo "✓ config.yaml exists"
fi
echo ""
# Check the .env file
if [ ! -f ".env" ]; then
echo ".env does not exist. Copying it from the example..."
if [ -f ".env.example" ]; then
cp .env.example .env
echo "✓ Created the .env file"
else
echo "⚠ .env.example does not exist. Please create the .env file manually"
fi
else
echo "✓ .env file exists"
fi
echo ""
# Check the frontend .env file
if [ ! -f "frontend/.env" ]; then
echo "frontend/.env does not exist. Copying it from the example..."
if [ -f "frontend/.env.example" ]; then
cp frontend/.env.example frontend/.env
echo "✓ Created the frontend/.env file"
else
echo "⚠ frontend/.env.example does not exist. Please create frontend/.env manually"
fi
else
echo "✓ frontend/.env file exists"
fi
echo ""
# Initialize the Docker environment
echo "Initializing the Docker environment..."
make docker-init
echo ""
# Start Docker services
echo "Starting Docker services..."
make docker-start
echo ""
echo "=========================================="
echo " Deployment Complete"
echo "=========================================="
echo ""
echo "🌐 Access URL: http://localhost:2026"
echo "📋 View logs: make docker-logs"
echo "🛑 Stop services: make docker-stop"
+63
View File
@@ -0,0 +1,63 @@
#!/usr/bin/env bash
set -e
echo "=========================================="
echo " Local Mode Deployment"
echo "=========================================="
echo ""
# Check config.yaml
if [ ! -f "config.yaml" ]; then
echo "config.yaml does not exist. Generating it..."
make config
echo ""
echo "⚠ Please edit config.yaml to configure your models and API keys"
echo " Then run this script again"
exit 1
else
echo "✓ config.yaml exists"
fi
echo ""
# Check the .env file
if [ ! -f ".env" ]; then
echo ".env does not exist. Copying it from the example..."
if [ -f ".env.example" ]; then
cp .env.example .env
echo "✓ Created the .env file"
else
echo "⚠ .env.example does not exist. Please create the .env file manually"
fi
else
echo "✓ .env file exists"
fi
echo ""
# Check dependencies
echo "Checking dependencies..."
make check
echo ""
# Install dependencies
echo "Installing dependencies..."
make install
echo ""
# Start services
echo "Starting services (background mode)..."
make dev-daemon
echo ""
echo "=========================================="
echo " Deployment Complete"
echo "=========================================="
echo ""
echo "🌐 Access URL: http://localhost:2026"
echo "📋 View logs:"
echo " - logs/langgraph.log"
echo " - logs/gateway.log"
echo " - logs/frontend.log"
echo " - logs/nginx.log"
echo "🛑 Stop services: make stop"
echo ""
echo "Please wait 90-120 seconds for all services to start completely, then run the health check"
@@ -0,0 +1,70 @@
#!/usr/bin/env bash
set +e
echo "=========================================="
echo " Frontend Page Smoke Check"
echo "=========================================="
echo ""
BASE_URL="${BASE_URL:-http://localhost:2026}"
DOC_PATH="${DOC_PATH:-/en/docs}"
all_passed=true
check_status() {
local name="$1"
local url="$2"
local expected_re="$3"
local status
status="$(curl -s -o /dev/null -w "%{http_code}" -L "$url")"
if echo "$status" | grep -Eq "$expected_re"; then
echo "$name ($url) -> $status"
else
echo "$name ($url) -> $status (expected: $expected_re)"
all_passed=false
fi
}
check_final_url() {
local name="$1"
local url="$2"
local expected_path_re="$3"
local effective
effective="$(curl -s -o /dev/null -w "%{url_effective}" -L "$url")"
if echo "$effective" | grep -Eq "$expected_path_re"; then
echo "$name redirect target -> $effective"
else
echo "$name redirect target -> $effective (expected path: $expected_path_re)"
all_passed=false
fi
}
echo "1. Checking entry pages..."
check_status "Landing page" "${BASE_URL}/" "200"
check_status "Workspace redirect" "${BASE_URL}/workspace" "200|301|302|307|308"
check_final_url "Workspace redirect" "${BASE_URL}/workspace" "/workspace/chats/"
echo ""
echo "2. Checking key workspace routes..."
check_status "New chat page" "${BASE_URL}/workspace/chats/new" "200"
check_status "Chats list page" "${BASE_URL}/workspace/chats" "200"
check_status "Agents gallery page" "${BASE_URL}/workspace/agents" "200"
echo ""
echo "3. Checking docs route (optional)..."
check_status "Docs page" "${BASE_URL}${DOC_PATH}" "200|404"
echo ""
echo "=========================================="
echo " Frontend Smoke Check Summary"
echo "=========================================="
echo ""
if [ "$all_passed" = true ]; then
echo "✅ Frontend smoke checks passed!"
exit 0
else
echo "❌ Frontend smoke checks failed"
exit 1
fi
+125
View File
@@ -0,0 +1,125 @@
#!/usr/bin/env bash
set +e
echo "=========================================="
echo " Service Health Check"
echo "=========================================="
echo ""
all_passed=true
mode="${SMOKE_TEST_MODE:-auto}"
summary_hint="make logs"
print_step() {
echo "$1"
}
check_http_status() {
local name="$1"
local url="$2"
local expected_re="$3"
local status
status="$(curl -s -o /dev/null -w "%{http_code}" "$url" 2>/dev/null)"
if echo "$status" | grep -Eq "$expected_re"; then
echo "$name is accessible ($url -> $status)"
else
echo "$name is not accessible ($url -> ${status:-000})"
all_passed=false
fi
}
check_listen_port() {
local name="$1"
local port="$2"
if lsof -nP -iTCP:"$port" -sTCP:LISTEN >/dev/null 2>&1; then
echo "$name is listening on port $port"
else
echo "$name is not listening on port $port"
all_passed=false
fi
}
docker_available() {
command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1
}
detect_mode() {
case "$mode" in
local|docker)
echo "$mode"
return
;;
esac
if docker_available && docker ps --format "{{.Names}}" | grep -q "deer-flow"; then
echo "docker"
else
echo "local"
fi
}
mode="$(detect_mode)"
echo "Deployment mode: $mode"
echo ""
if [ "$mode" = "docker" ]; then
summary_hint="make docker-logs"
print_step "1. Checking container status..."
if docker ps --format "{{.Names}}" | grep -q "deer-flow"; then
echo "✓ Containers are running:"
docker ps --format " - {{.Names}} ({{.Status}})"
else
echo "✗ No DeerFlow-related containers are running"
all_passed=false
fi
else
summary_hint="logs/{langgraph,gateway,frontend,nginx}.log"
print_step "1. Checking local service ports..."
check_listen_port "Nginx" 2026
check_listen_port "Frontend" 3000
check_listen_port "Gateway" 8001
check_listen_port "LangGraph" 2024
fi
echo ""
echo "2. Waiting for services to fully start (30 seconds)..."
sleep 30
echo ""
echo "3. Checking frontend service..."
check_http_status "Frontend service" "http://localhost:2026" "200|301|302|307|308"
echo ""
echo "4. Checking API Gateway..."
health_response=$(curl -s http://localhost:2026/health 2>/dev/null)
if [ $? -eq 0 ] && [ -n "$health_response" ]; then
echo "✓ API Gateway health check passed"
echo " Response: $health_response"
else
echo "✗ API Gateway health check failed"
all_passed=false
fi
echo ""
echo "5. Checking LangGraph service..."
check_http_status "LangGraph service" "http://localhost:2024/" "200|301|302|307|308|404"
echo ""
echo "=========================================="
echo " Health Check Summary"
echo "=========================================="
echo ""
if [ "$all_passed" = true ]; then
echo "✅ All checks passed!"
echo ""
echo "🌐 Application URL: http://localhost:2026"
exit 0
else
echo "❌ Some checks failed"
echo ""
echo "Please review: $summary_hint"
exit 1
fi
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env bash
set -e
echo "=========================================="
echo " Pulling the Latest Code"
echo "=========================================="
echo ""
# Check whether the current directory is a Git repository
if [ ! -d ".git" ]; then
echo "✗ The current directory is not a Git repository"
exit 1
fi
# Check Git status
echo "Checking Git status..."
if git status --porcelain | grep -q .; then
echo "⚠ Uncommitted changes detected:"
git status --short
echo ""
echo "Please commit or stash your changes before continuing"
echo "Options:"
echo " 1. git add . && git commit -m 'Save changes'"
echo " 2. git stash (stash changes and restore them later)"
echo " 3. git reset --hard HEAD (discard local changes - use with caution)"
exit 1
else
echo "✓ Working tree is clean"
fi
echo ""
# Fetch remote updates
echo "Fetching remote updates..."
git fetch origin main
echo ""
# Pull the latest code
echo "Pulling the latest code..."
git pull origin main
echo ""
# Show the latest commit
echo "Latest commit:"
git log -1 --oneline
echo ""
echo "=========================================="
echo " Code Update Complete"
echo "=========================================="
@@ -0,0 +1,180 @@
# DeerFlow Smoke Test Report
**Test Date**: {{test_date}}
**Test Environment**: {{test_environment}}
**Deployment Mode**: Docker
**Test Version**: {{git_commit}}
---
## Execution Summary
| Metric | Status |
|------|------|
| Total Test Phases | 6 |
| Passed Phases | {{passed_stages}} |
| Failed Phases | {{failed_stages}} |
| Overall Conclusion | **{{overall_status}}** |
### Key Test Cases
| Case | Result | Details |
|------|--------|---------|
| Code update check | {{case_code_update}} | {{case_code_update_details}} |
| Environment check | {{case_env_check}} | {{case_env_check_details}} |
| Configuration preparation | {{case_config_prep}} | {{case_config_prep_details}} |
| Deployment | {{case_deploy}} | {{case_deploy_details}} |
| Health check | {{case_health_check}} | {{case_health_check_details}} |
| Frontend routes | {{case_frontend_routes_overall}} | {{case_frontend_routes_details}} |
---
## Detailed Test Results
### Phase 1: Code Update Check
- [x] Confirm current directory - {{status_dir_check}}
- [x] Check Git status - {{status_git_status}}
- [x] Pull latest code - {{status_git_pull}}
- [x] Confirm code update - {{status_git_verify}}
**Phase Status**: {{stage1_status}}
---
### Phase 2: Docker Environment Check
- [x] Docker version - {{status_docker_version}}
- [x] Docker daemon - {{status_docker_daemon}}
- [x] Docker Compose - {{status_docker_compose}}
- [x] Port check - {{status_port_check}}
**Phase Status**: {{stage2_status}}
---
### Phase 3: Configuration Preparation
- [x] config.yaml - {{status_config_yaml}}
- [x] .env file - {{status_env_file}}
- [x] Model configuration - {{status_model_config}}
**Phase Status**: {{stage3_status}}
---
### Phase 4: Docker Deployment
- [x] docker-init - {{status_docker_init}}
- [x] docker-start - {{status_docker_start}}
- [x] Service startup wait - {{status_wait_startup}}
**Phase Status**: {{stage4_status}}
---
### Phase 5: Service Health Check
- [x] Container status - {{status_containers}}
- [x] Frontend service - {{status_frontend}}
- [x] API Gateway - {{status_api_gateway}}
- [x] LangGraph service - {{status_langgraph}}
**Phase Status**: {{stage5_status}}
---
### Frontend Routes Smoke Results
| Route | Status | Details |
|-------|--------|---------|
| Landing `/` | {{landing_status}} | {{landing_details}} |
| Workspace redirect `/workspace` | {{workspace_redirect_status}} | target {{workspace_redirect_target}} |
| New chat `/workspace/chats/new` | {{new_chat_status}} | {{new_chat_details}} |
| Chats list `/workspace/chats` | {{chats_list_status}} | {{chats_list_details}} |
| Agents gallery `/workspace/agents` | {{agents_gallery_status}} | {{agents_gallery_details}} |
| Docs `{{docs_path}}` | {{docs_status}} | {{docs_details}} |
**Summary**: {{frontend_routes_summary}}
---
### Phase 6: Test Report Generation
- [x] Result summary - {{status_summary}}
- [x] Issue log - {{status_issues}}
- [x] Report generation - {{status_report}}
**Phase Status**: {{stage6_status}}
---
## Issue Log
### Issue 1
**Description**: {{issue1_description}}
**Severity**: {{issue1_severity}}
**Solution**: {{issue1_solution}}
---
## Environment Information
### Docker Version
```text
{{docker_version_output}}
```
### Git Information
```text
Repository: {{git_repo}}
Branch: {{git_branch}}
Commit: {{git_commit}}
Commit Message: {{git_commit_message}}
```
### Configuration Summary
- config.yaml exists: {{config_exists}}
- .env file exists: {{env_exists}}
- Number of configured models: {{model_count}}
---
## Container Status
| Container Name | Status | Uptime |
|----------|------|----------|
| deer-flow-nginx | {{nginx_status}} | {{nginx_uptime}} |
| deer-flow-frontend | {{frontend_status}} | {{frontend_uptime}} |
| deer-flow-gateway | {{gateway_status}} | {{gateway_uptime}} |
| deer-flow-langgraph | {{langgraph_status}} | {{langgraph_uptime}} |
---
## Recommendations and Next Steps
### If the Test Passes
1. [ ] Visit http://localhost:2026 to start using DeerFlow
2. [ ] Configure your preferred model if it is not configured yet
3. [ ] Explore available skills
4. [ ] Refer to the documentation to learn more features
### If the Test Fails
1. [ ] Review references/troubleshooting.md for common solutions
2. [ ] Check Docker logs: `make docker-logs`
3. [ ] Verify configuration file format and content
4. [ ] If needed, fully reset the environment: `make clean && make config && make docker-init && make docker-start`
---
## Appendix
### Full Logs
{{full_logs}}
### Tester
{{tester_name}}
---
*Report generated at: {{report_time}}*
@@ -0,0 +1,185 @@
# DeerFlow Smoke Test Report
**Test Date**: {{test_date}}
**Test Environment**: {{test_environment}}
**Deployment Mode**: Local
**Test Version**: {{git_commit}}
---
## Execution Summary
| Metric | Status |
|------|------|
| Total Test Phases | 6 |
| Passed Phases | {{passed_stages}} |
| Failed Phases | {{failed_stages}} |
| Overall Conclusion | **{{overall_status}}** |
### Key Test Cases
| Case | Result | Details |
|------|--------|---------|
| Code update check | {{case_code_update}} | {{case_code_update_details}} |
| Environment check | {{case_env_check}} | {{case_env_check_details}} |
| Configuration preparation | {{case_config_prep}} | {{case_config_prep_details}} |
| Deployment | {{case_deploy}} | {{case_deploy_details}} |
| Health check | {{case_health_check}} | {{case_health_check_details}} |
| Frontend routes | {{case_frontend_routes_overall}} | {{case_frontend_routes_details}} |
---
## Detailed Test Results
### Phase 1: Code Update Check
- [x] Confirm current directory - {{status_dir_check}}
- [x] Check Git status - {{status_git_status}}
- [x] Pull latest code - {{status_git_pull}}
- [x] Confirm code update - {{status_git_verify}}
**Phase Status**: {{stage1_status}}
---
### Phase 2: Local Environment Check
- [x] Node.js version - {{status_node_version}}
- [x] pnpm - {{status_pnpm}}
- [x] uv - {{status_uv}}
- [x] nginx - {{status_nginx}}
- [x] Port check - {{status_port_check}}
**Phase Status**: {{stage2_status}}
---
### Phase 3: Configuration Preparation
- [x] config.yaml - {{status_config_yaml}}
- [x] .env file - {{status_env_file}}
- [x] Model configuration - {{status_model_config}}
**Phase Status**: {{stage3_status}}
---
### Phase 4: Local Deployment
- [x] make check - {{status_make_check}}
- [x] make install - {{status_make_install}}
- [x] make dev-daemon / make dev - {{status_local_start}}
- [x] Service startup wait - {{status_wait_startup}}
**Phase Status**: {{stage4_status}}
---
### Phase 5: Service Health Check
- [x] Process status - {{status_processes}}
- [x] Frontend service - {{status_frontend}}
- [x] API Gateway - {{status_api_gateway}}
- [x] LangGraph service - {{status_langgraph}}
**Phase Status**: {{stage5_status}}
---
### Frontend Routes Smoke Results
| Route | Status | Details |
|-------|--------|---------|
| Landing `/` | {{landing_status}} | {{landing_details}} |
| Workspace redirect `/workspace` | {{workspace_redirect_status}} | target {{workspace_redirect_target}} |
| New chat `/workspace/chats/new` | {{new_chat_status}} | {{new_chat_details}} |
| Chats list `/workspace/chats` | {{chats_list_status}} | {{chats_list_details}} |
| Agents gallery `/workspace/agents` | {{agents_gallery_status}} | {{agents_gallery_details}} |
| Docs `{{docs_path}}` | {{docs_status}} | {{docs_details}} |
**Summary**: {{frontend_routes_summary}}
---
### Phase 6: Test Report Generation
- [x] Result summary - {{status_summary}}
- [x] Issue log - {{status_issues}}
- [x] Report generation - {{status_report}}
**Phase Status**: {{stage6_status}}
---
## Issue Log
### Issue 1
**Description**: {{issue1_description}}
**Severity**: {{issue1_severity}}
**Solution**: {{issue1_solution}}
---
## Environment Information
### Local Dependency Versions
```text
Node.js: {{node_version_output}}
pnpm: {{pnpm_version_output}}
uv: {{uv_version_output}}
nginx: {{nginx_version_output}}
```
### Git Information
```text
Repository: {{git_repo}}
Branch: {{git_branch}}
Commit: {{git_commit}}
Commit Message: {{git_commit_message}}
```
### Configuration Summary
- config.yaml exists: {{config_exists}}
- .env file exists: {{env_exists}}
- Number of configured models: {{model_count}}
---
## Local Service Status
| Service | Status | Endpoint |
|---------|--------|----------|
| Nginx | {{nginx_status}} | {{nginx_endpoint}} |
| Frontend | {{frontend_status}} | {{frontend_endpoint}} |
| Gateway | {{gateway_status}} | {{gateway_endpoint}} |
| LangGraph | {{langgraph_status}} | {{langgraph_endpoint}} |
---
## Recommendations and Next Steps
### If the Test Passes
1. [ ] Visit http://localhost:2026 to start using DeerFlow
2. [ ] Configure your preferred model if it is not configured yet
3. [ ] Explore available skills
4. [ ] Refer to the documentation to learn more features
### If the Test Fails
1. [ ] Review references/troubleshooting.md for common solutions
2. [ ] Check local logs: `logs/{langgraph,gateway,frontend,nginx}.log`
3. [ ] Verify configuration file format and content
4. [ ] If needed, fully reset the environment: `make stop && make clean && make install && make dev-daemon`
---
## Appendix
### Full Logs
{{full_logs}}
### Tester
{{tester_name}}
---
*Report generated at: {{report_time}}*
+18
View File
@@ -3,6 +3,7 @@ Dockerfile
.dockerignore
.git
.gitignore
docker/
# Python
__pycache__/
@@ -51,3 +52,20 @@ examples/
assets/
tests/
*.log
# Exclude directories not needed in Docker context
# Frontend build only needs frontend/
# Backend build only needs backend/
scripts/
logs/
docker/
skills/
frontend/.next
frontend/node_modules
backend/.venv
backend/htmlcov
backend/.coverage
*.md
!README.md
!frontend/README.md
!backend/README.md
+42 -27
View File
@@ -1,35 +1,50 @@
# Application Settings
DEBUG=True
APP_ENV=development
# Serper API Key (Google Search) - https://serper.dev
SERPER_API_KEY=your-serper-api-key
# docker build args
NEXT_PUBLIC_API_URL="http://localhost:8000/api"
# TAVILY API Key
TAVILY_API_KEY=your-tavily-api-key
AGENT_RECURSION_LIMIT=30
# Jina API Key
JINA_API_KEY=your-jina-api-key
# Search Engine, Supported values: tavily (recommended), duckduckgo, brave_search, arxiv
SEARCH_API=tavily
TAVILY_API_KEY=tvly-xxx
# BRAVE_SEARCH_API_KEY=xxx # Required only if SEARCH_API is brave_search
# JINA_API_KEY=jina_xxx # Optional, default is None
# InfoQuest API Key
INFOQUEST_API_KEY=your-infoquest-api-key
# CORS Origins (comma-separated) - e.g., http://localhost:3000,http://localhost:3001
# CORS_ORIGINS=http://localhost:3000
# Optional, RAG provider
# RAG_PROVIDER=ragflow
# RAGFLOW_API_URL="http://localhost:9388"
# RAGFLOW_API_KEY="ragflow-xxx"
# RAGFLOW_RETRIEVAL_SIZE=10
# Optional:
# FIRECRAWL_API_KEY=your-firecrawl-api-key
# VOLCENGINE_API_KEY=your-volcengine-api-key
# OPENAI_API_KEY=your-openai-api-key
# GEMINI_API_KEY=your-gemini-api-key
# DEEPSEEK_API_KEY=your-deepseek-api-key
# NOVITA_API_KEY=your-novita-api-key # OpenAI-compatible, see https://novita.ai
# MINIMAX_API_KEY=your-minimax-api-key # OpenAI-compatible, see https://platform.minimax.io
# VLLM_API_KEY=your-vllm-api-key # OpenAI-compatible
# FEISHU_APP_ID=your-feishu-app-id
# FEISHU_APP_SECRET=your-feishu-app-secret
# Optional, volcengine TTS for generating podcast
VOLCENGINE_TTS_APPID=xxx
VOLCENGINE_TTS_ACCESS_TOKEN=xxx
# VOLCENGINE_TTS_CLUSTER=volcano_tts # Optional, default is volcano_tts
# VOLCENGINE_TTS_VOICE_TYPE=BV700_V2_streaming # Optional, default is BV700_V2_streaming
# SLACK_BOT_TOKEN=your-slack-bot-token
# SLACK_APP_TOKEN=your-slack-app-token
# TELEGRAM_BOT_TOKEN=your-telegram-bot-token
# DISCORD_BOT_TOKEN=your-discord-bot-token
# Option, for langsmith tracing and monitoring
# Enable LangSmith to monitor and debug your LLM calls, agent runs, and tool executions.
# LANGSMITH_TRACING=true
# LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
# LANGSMITH_API_KEY="xxx"
# LANGSMITH_PROJECT="xxx"
# LANGSMITH_ENDPOINT=https://api.smith.langchain.com
# LANGSMITH_API_KEY=your-langsmith-api-key
# LANGSMITH_PROJECT=your-langsmith-project
# [!NOTE]
# For model settings and other configurations, please refer to `docs/configuration_guide.md`
# GitHub API Token
# GITHUB_TOKEN=your-github-token
# Database (only needed when config.yaml has database.backend: postgres)
# DATABASE_URL=postgresql://deerflow:password@localhost:5432/deerflow
#
# WECOM_BOT_ID=your-wecom-bot-id
# WECOM_BOT_SECRET=your-wecom-bot-secret
# DINGTALK_CLIENT_ID=your-dingtalk-client-id
# DINGTALK_CLIENT_SECRET=your-dingtalk-client-secret
# Set to "false" to disable Swagger UI, ReDoc, and OpenAPI schema in production
# GATEWAY_ENABLE_DOCS=false
+43
View File
@@ -0,0 +1,43 @@
# Normalize line endings to LF for all text files
* text=auto eol=lf
# Shell scripts and makefiles must always use LF
*.sh text eol=lf
Makefile text eol=lf
**/Makefile text eol=lf
# Common config/source files
*.yml text eol=lf
*.yaml text eol=lf
*.toml text eol=lf
*.json text eol=lf
*.md text eol=lf
*.py text eol=lf
*.ts text eol=lf
*.tsx text eol=lf
*.js text eol=lf
*.jsx text eol=lf
*.css text eol=lf
*.scss text eol=lf
*.html text eol=lf
*.env text eol=lf
# Windows scripts
*.bat text eol=crlf
*.cmd text eol=crlf
# Binary assets
*.png binary
*.jpg binary
*.jpeg binary
*.gif binary
*.webp binary
*.ico binary
*.pdf binary
*.zip binary
*.tar binary
*.gz binary
*.mp4 binary
*.mov binary
*.woff binary
*.woff2 binary
@@ -0,0 +1,128 @@
name: Runtime Information
description: Report runtime/environment details to help reproduce an issue.
title: "[runtime] "
labels:
- needs-triage
body:
- type: markdown
attributes:
value: |
Thanks for sharing runtime details.
Complete this form so maintainers can quickly reproduce and diagnose the problem.
- type: input
id: summary
attributes:
label: Problem summary
description: Short summary of the issue.
placeholder: e.g. make dev fails to start gateway service
validations:
required: true
- type: textarea
id: expected
attributes:
label: Expected behavior
placeholder: What did you expect to happen?
validations:
required: true
- type: textarea
id: actual
attributes:
label: Actual behavior
placeholder: What happened instead? Include key error lines.
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- macOS
- Linux
- Windows
- Other
validations:
required: true
- type: input
id: platform_details
attributes:
label: Platform details
description: Add architecture and shell if relevant.
placeholder: e.g. arm64, zsh
- type: input
id: python_version
attributes:
label: Python version
placeholder: e.g. Python 3.12.9
- type: input
id: node_version
attributes:
label: Node.js version
placeholder: e.g. v23.11.0
- type: input
id: pnpm_version
attributes:
label: pnpm version
placeholder: e.g. 10.26.2
- type: input
id: uv_version
attributes:
label: uv version
placeholder: e.g. 0.7.20
- type: dropdown
id: run_mode
attributes:
label: How are you running DeerFlow?
options:
- Local (make dev)
- Docker (make docker-dev)
- CI
- Other
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: Reproduction steps
description: Provide exact commands and sequence.
placeholder: |
1. make check
2. make install
3. make dev
4. ...
validations:
required: true
- type: textarea
id: logs
attributes:
label: Relevant logs
description: Paste key lines from logs (for example logs/gateway.log, logs/frontend.log).
render: shell
validations:
required: true
- type: textarea
id: git_info
attributes:
label: Git state
description: Share output of git branch and latest commit SHA.
placeholder: |
branch: feature/my-branch
commit: abcdef1
- type: textarea
id: additional
attributes:
label: Additional context
description: Add anything else that might help triage.
+213
View File
@@ -0,0 +1,213 @@
# Copilot Onboarding Instructions for DeerFlow
Use this file as the default operating guide for this repository. Follow it first, and only search the codebase when this file is incomplete or incorrect.
## 1) Repository Summary
DeerFlow is a full-stack "super agent harness".
- Backend: Python 3.12, LangGraph + FastAPI gateway, sandbox/tool system, memory, MCP integration.
- Frontend: Next.js 16 + React 19 + TypeScript + pnpm.
- Local dev entrypoint: root `Makefile` starts backend + frontend + nginx on `http://localhost:2026`.
- Docker dev entrypoint: `make docker-*` (mode-aware provisioner startup from `config.yaml`).
Current repo footprint is medium-large (backend service, frontend app, docker stack, skills library, docs).
## 2) Runtime and Toolchain Requirements
Validated in this repo on macOS:
- Node.js `>=22` (validated with Node `23.11.0`)
- pnpm (repo expects lockfile generated by pnpm 10; validated with pnpm `10.26.2` and `10.15.0`)
- Python `>=3.12` (CI uses `3.12`)
- `uv` (validated with `0.7.20`)
- `nginx` (required for `make dev` unified local endpoint)
Always run from repo root unless a command explicitly says otherwise.
## 3) Build/Test/Lint/Run - Verified Command Sequences
These were executed and validated in this repository.
### A. Bootstrap and install
1. Check prerequisites:
```bash
make check
```
Observed: passes when required tools are installed.
2. Install dependencies (recommended order: backend then frontend, as implemented by `make install`):
```bash
make install
```
### B. Backend CI-equivalent validation
Run from `backend/`:
```bash
make lint
make test
```
Validated results:
- `make lint`: pass (`ruff check .`)
- `make test`: pass (`277 passed, 15 warnings in ~76.6s`)
CI parity:
- `.github/workflows/backend-unit-tests.yml` runs on pull requests.
- CI executes `uv sync --group dev`, then `make lint`, then `make test` in `backend/`.
### C. Frontend validation
Run from `frontend/`.
Recommended reliable sequence:
```bash
pnpm lint
pnpm typecheck
BETTER_AUTH_SECRET=local-dev-secret pnpm build
```
Observed failure modes and workarounds:
- `pnpm build` fails without `BETTER_AUTH_SECRET` in production-mode env validation.
- Workaround: set `BETTER_AUTH_SECRET` (best) or set `SKIP_ENV_VALIDATION=1`.
- Even with `SKIP_ENV_VALIDATION=1`, Better Auth can still warn/error in logs about default secret; prefer setting a real non-default secret.
- `pnpm check` currently fails (`next lint` invocation is incompatible here and resolves to an invalid directory). Do not rely on `pnpm check`; run `pnpm lint` and `pnpm typecheck` explicitly.
### D. Run locally (all services)
From root:
```bash
make dev
```
Behavior:
- Stops existing local services first.
- Starts LangGraph (`2024`), Gateway (`8001`), Frontend (`3000`), nginx (`2026`).
- Unified app endpoint: `http://localhost:2026`.
- Logs: `logs/langgraph.log`, `logs/gateway.log`, `logs/frontend.log`, `logs/nginx.log`.
Stop services:
```bash
make stop
```
If tool sessions/timeouts interrupt `make dev`, run `make stop` again to ensure cleanup.
### E. Config bootstrap
From root:
```bash
make config
```
Important behavior:
- This intentionally aborts if `config.yaml` (or `config.yml`/`configure.yml`) already exists.
- Use `make config` only for first-time setup in a clean clone.
## 4) Command Order That Minimizes Failures
Use this exact order for local code changes:
1. `make check`
2. `make install` (if frontend fails with proxy errors, rerun frontend install with proxy vars unset)
3. Backend checks: `cd backend && make lint && make test`
4. Frontend checks: `cd frontend && pnpm lint && pnpm typecheck`
5. Frontend build (if UI changes or release-sensitive changes): `BETTER_AUTH_SECRET=... pnpm build`
Always run backend lint/tests before opening PRs because that is what CI enforces.
## 5) Project Layout and Architecture (High-Value Paths)
Root-level orchestration and config:
- `Makefile` - main local/dev/docker command entrypoints
- `config.example.yaml` - primary app config template
- `config.yaml` - local active config (gitignored)
- `docker/docker-compose-dev.yaml` - Docker dev topology
- `.github/workflows/backend-unit-tests.yml` - PR validation workflow
Backend core:
- `backend/packages/harness/deerflow/agents/` - lead agent, middleware chain, memory
- `backend/app/gateway/` - FastAPI gateway API
- `backend/packages/harness/deerflow/sandbox/` - sandbox provider + tool wrappers
- `backend/packages/harness/deerflow/subagents/` - subagent registry/execution
- `backend/packages/harness/deerflow/mcp/` - MCP integration
- `backend/langgraph.json` - graph entrypoint (`deerflow.agents:make_lead_agent`)
- `backend/pyproject.toml` - Python deps and `requires-python`
- `backend/ruff.toml` - lint/format policy
- `backend/tests/` - backend unit and integration-like tests
Frontend core:
- `frontend/src/app/` - Next.js routes/pages
- `frontend/src/components/` - UI components
- `frontend/src/core/` - app logic (threads, tools, API, models)
- `frontend/src/env.js` - env schema/validation (critical for build behavior)
- `frontend/package.json` - scripts/deps
- `frontend/eslint.config.js` - lint rules
- `frontend/tsconfig.json` - TS config
Skills and assets:
- `skills/public/` - built-in skill packs loaded by agent runtime
## 6) Pre-Checkin / Validation Expectations
Before submitting changes, run at minimum:
- Backend: `cd backend && make lint && make test`
- Frontend (if touched): `cd frontend && pnpm lint && pnpm typecheck`
- Frontend build when changing env/auth/routing/build-sensitive files: `BETTER_AUTH_SECRET=... pnpm build`
If touching orchestration/config (`Makefile`, `docker/*`, `config*.yaml`), also run `make dev` and verify the four services start.
## 7) Non-Obvious Dependencies and Gotchas
- Proxy env vars can silently break frontend network operations (`pnpm install`/registry access).
- `BETTER_AUTH_SECRET` is effectively required for reliable frontend production build validation.
- Next.js may warn about multiple lockfiles and workspace root inference; this is currently a warning, not a build blocker.
- `make config` is non-idempotent by design when config already exists.
- `make dev` includes process cleanup and can emit shutdown logs/noise if interrupted; this is expected.
## 8) Root Inventory (quick reference)
Important root entries:
- `.github/`
- `backend/`
- `frontend/`
- `docker/`
- `skills/`
- `scripts/`
- `docs/`
- `README.md`
- `CONTRIBUTING.md`
- `Makefile`
- `config.example.yaml`
- `extensions_config.example.json`
## 9) Instruction Priority
Trust this onboarding guide first.
Only do broad repo searches (`grep/find/code search`) when:
- you need file-level implementation details not listed here,
- a command here fails and you need updated replacement behavior,
- or CI/workflow definitions have changed since this file was written.
+40
View File
@@ -0,0 +1,40 @@
name: Unit Tests
on:
push:
branches: [ 'main' ]
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
concurrency:
group: unit-tests-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
backend-unit-tests:
if: github.event.pull_request.draft == false
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: '3.12'
- name: Install uv
uses: astral-sh/setup-uv@v7
- name: Install backend dependencies
working-directory: backend
run: uv sync --group dev
- name: Run unit tests of backend
working-directory: backend
run: make test
+63
View File
@@ -0,0 +1,63 @@
name: E2E Tests
on:
push:
branches: [ 'main' ]
paths:
- 'frontend/**'
- '.github/workflows/e2e-tests.yml'
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
paths:
- 'frontend/**'
- '.github/workflows/e2e-tests.yml'
concurrency:
group: e2e-tests-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
e2e-tests:
if: ${{ github.event_name != 'pull_request' || github.event.pull_request.draft == false }}
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '22'
- name: Enable Corepack
run: corepack enable
- name: Use pinned pnpm version
run: corepack prepare pnpm@10.26.2 --activate
- name: Install frontend dependencies
working-directory: frontend
run: pnpm install --frozen-lockfile
- name: Install Playwright Chromium
working-directory: frontend
run: npx playwright install chromium --with-deps
- name: Run E2E tests
working-directory: frontend
run: pnpm exec playwright test
env:
SKIP_ENV_VALIDATION: '1'
- name: Upload Playwright report
uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: playwright-report
path: frontend/playwright-report/
retention-days: 7
+43
View File
@@ -0,0 +1,43 @@
name: Frontend Unit Tests
on:
push:
branches: [ 'main' ]
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
concurrency:
group: frontend-unit-tests-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
frontend-unit-tests:
if: github.event.pull_request.draft == false
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '22'
- name: Enable Corepack
run: corepack enable
- name: Use pinned pnpm version
run: corepack prepare pnpm@10.26.2 --activate
- name: Install frontend dependencies
working-directory: frontend
run: pnpm install --frozen-lockfile
- name: Run unit tests of frontend
working-directory: frontend
run: make test
+74
View File
@@ -0,0 +1,74 @@
name: Lint Check
on:
push:
branches: [ 'main' ]
pull_request:
branches: [ '*' ]
permissions:
contents: read
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: '3.12'
- name: Install uv
uses: astral-sh/setup-uv@v7
- name: Install dependencies
working-directory: backend
run: |
uv sync --group dev
- name: Lint backend
working-directory: backend
run: make lint
lint-frontend:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '22'
- name: Enable Corepack
run: corepack enable
- name: Use pinned pnpm version
run: corepack prepare pnpm@10.26.2 --activate
- name: Install frontend dependencies
run: |
cd frontend
pnpm install --frozen-lockfile
- name: Check frontend formatting
run: |
cd frontend
pnpm format
- name: Run frontend linting
run: |
cd frontend
pnpm lint
- name: Check TypeScript types
run: |
cd frontend
pnpm typecheck
- name: Build frontend
run: |
cd frontend
BETTER_AUTH_SECRET=local-dev-secret pnpm build
-31
View File
@@ -1,31 +0,0 @@
name: Lint Check
on:
push:
branches: [ 'main' ]
pull_request:
branches: [ '*' ]
permissions:
contents: read
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install the latest version of uv
uses: astral-sh/setup-uv@v5
with:
version: "latest"
- name: Install dependencies
run: |
uv venv --python 3.12
uv pip install -e ".[dev]"
- name: Run linters
run: |
source .venv/bin/activate
make lint
-48
View File
@@ -1,48 +0,0 @@
name: Test Cases Check
on:
push:
branches: [ 'main' ]
pull_request:
branches: [ '*' ]
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install the latest version of uv
uses: astral-sh/setup-uv@v5
with:
version: "latest"
- name: Install dependencies
run: |
uv venv --python 3.12
uv pip install -e ".[dev]"
uv pip install -e ".[test]"
- name: Run test cases with coverage
run: |
source .venv/bin/activate
TAVILY_API_KEY=mock-key make coverage
- name: Generate HTML Coverage Report
run: |
source .venv/bin/activate
python -m coverage html -d coverage_html
- name: Upload Coverage Report
uses: actions/upload-artifact@v4
with:
name: coverage-report
path: coverage_html/
- name: Display Coverage Summary
run: |
source .venv/bin/activate
python -m coverage report
+48 -13
View File
@@ -1,13 +1,20 @@
# Python-generated files
# DeerFlow docker image cache
docker/.cache/
# oh-my-claudecode state
.omc/
# OS generated files
.DS_Store
*.local
._*
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db
# Python cache
__pycache__/
*.py[oc]
build/
dist/
wheels/
*.egg-info
.coverage
agent_history.gif
static/browser_history/*.gif
*.pyc
*.pyo
# Virtual environments
.venv
@@ -16,12 +23,40 @@ venv/
# Environment variables
.env
# user conf
conf.yaml
# Configuration files
config.yaml
mcp_config.json
extensions_config.json
# IDE
.idea/
.langgraph_api/
.vscode/
# coverage report
# Coverage report
coverage.xml
coverage/
.deer-flow/
.claude/
skills/custom/*
logs/
log/
debug.log
# Local git hooks (keep only on this machine, do not push)
.githooks/
# pnpm
.pnpm-store
sandbox_image_cache.tar
# ignore the legacy `web` folder
web/
# Deployment artifacts
backend/Dockerfile.langgraph
config.yaml.bak
.playwright-mcp
/frontend/test-results/
/frontend/playwright-report/
.gstack/
.worktrees
+33
View File
@@ -0,0 +1,33 @@
repos:
# Backend: ruff lint + format via uv (uses the same ruff version as backend deps)
- repo: local
hooks:
- id: ruff
name: ruff lint
entry: bash -c 'cd backend && uv run ruff check --fix "${@/#backend\//}"' --
language: system
types_or: [python]
files: ^backend/
- id: ruff-format
name: ruff format
entry: bash -c 'cd backend && uv run ruff format "${@/#backend\//}"' --
language: system
types_or: [python]
files: ^backend/
# Frontend: eslint + prettier (must run from frontend/ for node_modules resolution)
- repo: local
hooks:
- id: frontend-eslint
name: eslint (frontend)
entry: bash -c 'cd frontend && npx eslint --fix "${@/#frontend\//}"' --
language: system
types_or: [javascript, tsx, ts]
files: ^frontend/
- id: frontend-prettier
name: prettier (frontend)
entry: bash -c 'cd frontend && npx prettier --write "${@/#frontend\//}"' --
language: system
files: ^frontend/
types_or: [javascript, tsx, ts, json, css]
-60
View File
@@ -1,60 +0,0 @@
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: 当前文件",
"type": "debugpy",
"request": "launch",
"program": "${file}",
"console": "integratedTerminal",
"justMyCode": true
},
{
"name": "Python: main.py",
"type": "debugpy",
"request": "launch",
"program": "${workspaceFolder}/main.py",
"console": "integratedTerminal",
"justMyCode": false,
"env": {
"PYTHONPATH": "${workspaceFolder}"
},
"args": [
"--debug", "--max_plan_iterations", "1", "--max_step_num", "1"
]
},
{
"name": "Python: llm.py",
"type": "debugpy",
"request": "launch",
"program": "${workspaceFolder}/src/llms/llm.py",
"console": "integratedTerminal",
"justMyCode": true,
"env": {
"PYTHONPATH": "${workspaceFolder}"
}
},
{
"name": "Python: server.py",
"type": "debugpy",
"request": "launch",
"program": "${workspaceFolder}/server.py",
"console": "integratedTerminal",
"justMyCode": false,
"env": {
"PYTHONPATH": "${workspaceFolder}"
}
},
{
"name": "Python: graph.py",
"type": "debugpy",
"request": "launch",
"program": "${workspaceFolder}/src/ppt/graph/builder.py",
"console": "integratedTerminal",
"justMyCode": false,
"env": {
"PYTHONPATH": "${workspaceFolder}"
}
},
]
}
+128
View File
@@ -0,0 +1,128 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
willem.jiang@gmail.com.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.
-133
View File
@@ -1,133 +0,0 @@
# Contributing to DeerFlow
Thank you for your interest in contributing to DeerFlow! We welcome contributions of all kinds from the community.
## Ways to Contribute
There are many ways you can contribute to DeerFlow:
- **Code Contributions**: Add new features, fix bugs, or improve performance
- **Documentation**: Improve README, add code comments, or create examples
- **Bug Reports**: Submit detailed bug reports through issues
- **Feature Requests**: Suggest new features or improvements
- **Code Reviews**: Review pull requests from other contributors
- **Community Support**: Help others in discussions and issues
## Development Setup
1. Fork the repository
2. Clone your fork:
```bash
git clone https://github.com/bytedance/deer-flow.git
cd deer-flow
```
3. Set up your development environment:
```bash
# Install dependencies, uv will take care of the python interpreter and venv creation
uv sync
# For development, install additional dependencies
uv pip install -e ".[dev]"
uv pip install -e ".[test]"
```
4. Configure pre-commit hooks:
```bash
chmod +x pre-commit
ln -s ../../pre-commit .git/hooks/pre-commit
```
## Development Process
1. Create a new branch:
```bash
git checkout -b feature/amazing-feature
```
2. Make your changes following our coding standards:
- Write clear, documented code
- Follow PEP 8 style guidelines
- Add tests for new features
- Update documentation as needed
3. Run tests and checks:
```bash
make test # Run tests
make lint # Run linting
make format # Format code
make coverage # Check test coverage
```
4. Commit your changes:
```bash
git commit -m 'Add some amazing feature'
```
5. Push to your fork:
```bash
git push origin feature/amazing-feature
```
6. Open a Pull Request
## Pull Request Guidelines
- Fill in the pull request template completely
- Include tests for new features
- Update documentation as needed
- Ensure all tests pass and there are no linting errors
- Keep pull requests focused on a single feature or fix
- Reference any related issues
## Code Style
- Follow PEP 8 guidelines
- Use type hints where possible
- Write descriptive docstrings
- Keep functions and methods focused and single-purpose
- Comment complex logic
- Python version requirement: >= 3.12
## Testing
Run the test suite:
```bash
# Run all tests
make test
# Run specific test file
pytest tests/integration/test_workflow.py
# Run with coverage
make coverage
```
## Code Quality
```bash
# Run linting
make lint
# Format code
make format
```
## Community Guidelines
- Be respectful and inclusive
- Follow our code of conduct
- Help others learn and grow
- Give constructive feedback
- Stay focused on improving the project
## Need Help?
If you need help with anything:
- Check existing issues and discussions
- Join our community channels
- Ask questions in discussions
## License
By contributing to DeerFlow, you agree that your contributions will be licensed under the MIT License.
We appreciate your contributions to making DeerFlow better!
+340
View File
@@ -0,0 +1,340 @@
# Contributing to DeerFlow
Thank you for your interest in contributing to DeerFlow! This guide will help you set up your development environment and understand our development workflow.
## Development Environment Setup
We offer two development environments. **Docker is recommended** for the most consistent and hassle-free experience.
### Option 1: Docker Development (Recommended)
Docker provides a consistent, isolated environment with all dependencies pre-configured. No need to install Node.js, Python, or nginx on your local machine.
#### Prerequisites
- Docker Desktop or Docker Engine
- pnpm (for caching optimization)
#### Setup Steps
1. **Configure the application**:
```bash
# Copy example configuration
cp config.example.yaml config.yaml
# Set your API keys
export OPENAI_API_KEY="your-key-here"
# or edit config.yaml directly
```
2. **Initialize Docker environment** (first time only):
```bash
make docker-init
```
This will:
- Build Docker images
- Install frontend dependencies (pnpm)
- Install backend dependencies (uv)
- Share pnpm cache with host for faster builds
3. **Start development services**:
```bash
make docker-start
```
`make docker-start` reads `config.yaml` and starts `provisioner` only for provisioner/Kubernetes sandbox mode.
All services will start with hot-reload enabled:
- Frontend changes are automatically reloaded
- Backend changes trigger automatic restart
- LangGraph server supports hot-reload
4. **Access the application**:
- Web Interface: http://localhost:2026
- API Gateway: http://localhost:2026/api/*
- LangGraph: http://localhost:2026/api/langgraph/*
#### Docker Commands
```bash
# Build the custom k3s image (with pre-cached sandbox image)
make docker-init
# Start Docker services (mode-aware, localhost:2026)
make docker-start
# Stop Docker development services
make docker-stop
# View Docker development logs
make docker-logs
# View Docker frontend logs
make docker-logs-frontend
# View Docker gateway logs
make docker-logs-gateway
```
If Docker builds are slow in your network, you can override the default package registries before running `make docker-init` or `make docker-start`:
```bash
export UV_INDEX_URL=https://pypi.org/simple
export NPM_REGISTRY=https://registry.npmjs.org
```
#### Recommended host resources
Use these as practical starting points for development and review environments:
| Scenario | Starting point | Recommended | Notes |
|---------|-----------|------------|-------|
| `make dev` on one machine | 4 vCPU, 8 GB RAM | 8 vCPU, 16 GB RAM | Best when DeerFlow uses hosted model APIs. |
| `make docker-start` review environment | 4 vCPU, 8 GB RAM | 8 vCPU, 16 GB RAM | Docker image builds and sandbox containers need extra headroom. |
| Shared Linux test server | 8 vCPU, 16 GB RAM | 16 vCPU, 32 GB RAM | Prefer this for heavier multi-agent runs or multiple reviewers. |
`2 vCPU / 4 GB` environments often fail to start reliably or become unresponsive under normal DeerFlow workloads.
#### Linux: Docker daemon permission denied
If `make docker-init`, `make docker-start`, or `make docker-stop` fails on Linux with an error like below, your current user likely does not have permission to access the Docker daemon socket:
```text
unable to get image 'deer-flow-dev-langgraph': permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
```
Recommended fix: add your current user to the `docker` group so Docker commands work without `sudo`.
1. Confirm the `docker` group exists:
```bash
getent group docker
```
2. Add your current user to the `docker` group:
```bash
sudo usermod -aG docker $USER
```
3. Apply the new group membership. The most reliable option is to log out completely and then log back in. If you want to refresh the current shell session instead, run:
```bash
newgrp docker
```
4. Verify Docker access:
```bash
docker ps
```
5. Retry the DeerFlow command:
```bash
make docker-stop
make docker-start
```
If `docker ps` still reports a permission error after `usermod`, fully log out and log back in before retrying.
#### Docker Architecture
```
Host Machine
Docker Compose (deer-flow-dev)
├→ nginx (port 2026) ← Reverse proxy
├→ web (port 3000) ← Frontend with hot-reload
├→ api (port 8001) ← Gateway API with hot-reload
├→ langgraph (port 2024) ← LangGraph server with hot-reload
└→ provisioner (optional, port 8002) ← Started only in provisioner/K8s sandbox mode
```
**Benefits of Docker Development**:
- ✅ Consistent environment across different machines
- ✅ No need to install Node.js, Python, or nginx locally
- ✅ Isolated dependencies and services
- ✅ Easy cleanup and reset
- ✅ Hot-reload for all services
- ✅ Production-like environment
### Option 2: Local Development
If you prefer to run services directly on your machine:
#### Prerequisites
Check that you have all required tools installed:
```bash
make check
```
Required tools:
- Node.js 22+
- pnpm
- uv (Python package manager)
- nginx
#### Setup Steps
1. **Configure the application** (same as Docker setup above)
2. **Install dependencies** (this also sets up pre-commit hooks):
```bash
make install
```
3. **Run development server** (starts all services with nginx):
```bash
make dev
```
4. **Access the application**:
- Web Interface: http://localhost:2026
- All API requests are automatically proxied through nginx
#### Manual Service Control
If you need to start services individually:
1. **Start backend services**:
```bash
# Terminal 1: Start LangGraph Server (port 2024)
cd backend
make dev
# Terminal 2: Start Gateway API (port 8001)
cd backend
make gateway
# Terminal 3: Start Frontend (port 3000)
cd frontend
pnpm dev
```
2. **Start nginx**:
```bash
make nginx
# or directly: nginx -c $(pwd)/docker/nginx/nginx.local.conf -g 'daemon off;'
```
3. **Access the application**:
- Web Interface: http://localhost:2026
#### Nginx Configuration
The nginx configuration provides:
- Unified entry point on port 2026
- Routes `/api/langgraph/*` to LangGraph Server (2024)
- Routes other `/api/*` endpoints to Gateway API (8001)
- Routes non-API requests to Frontend (3000)
- Centralized CORS handling
- SSE/streaming support for real-time agent responses
- Optimized timeouts for long-running operations
## Project Structure
```
deer-flow/
├── config.example.yaml # Configuration template
├── extensions_config.example.json # MCP and Skills configuration template
├── Makefile # Build and development commands
├── scripts/
│ └── docker.sh # Docker management script
├── docker/
│ ├── docker-compose-dev.yaml # Docker Compose configuration
│ └── nginx/
│ ├── nginx.conf # Nginx config for Docker
│ └── nginx.local.conf # Nginx config for local dev
├── backend/ # Backend application
│ ├── src/
│ │ ├── gateway/ # Gateway API (port 8001)
│ │ ├── agents/ # LangGraph agents (port 2024)
│ │ ├── mcp/ # Model Context Protocol integration
│ │ ├── skills/ # Skills system
│ │ └── sandbox/ # Sandbox execution
│ ├── docs/ # Backend documentation
│ └── Makefile # Backend commands
├── frontend/ # Frontend application
│ └── Makefile # Frontend commands
└── skills/ # Agent skills
├── public/ # Public skills
└── custom/ # Custom skills
```
## Architecture
```
Browser
Nginx (port 2026) ← Unified entry point
├→ Frontend (port 3000) ← / (non-API requests)
├→ Gateway API (port 8001) ← /api/models, /api/mcp, /api/skills, /api/threads/*/artifacts
└→ LangGraph Server (port 2024) ← /api/langgraph/* (agent interactions)
```
## Development Workflow
1. **Create a feature branch**:
```bash
git checkout -b feature/your-feature-name
```
2. **Make your changes** with hot-reload enabled
3. **Format and lint your code** (CI will reject unformatted code):
```bash
# Backend
cd backend
make format # ruff check --fix + ruff format
# Frontend
cd frontend
pnpm format:write # Prettier
```
4. **Test your changes** thoroughly
5. **Commit your changes**:
```bash
git add .
git commit -m "feat: description of your changes"
```
6. **Push and create a Pull Request**:
```bash
git push origin feature/your-feature-name
```
## Testing
```bash
# Backend tests
cd backend
make test
# Frontend unit tests
cd frontend
make test
# Frontend E2E tests (requires Chromium; builds and auto-starts the Next.js production server)
cd frontend
make test-e2e
```
### PR Regression Checks
Every pull request triggers the following CI workflows:
- **Backend unit tests** — [.github/workflows/backend-unit-tests.yml](.github/workflows/backend-unit-tests.yml)
- **Frontend unit tests** — [.github/workflows/frontend-unit-tests.yml](.github/workflows/frontend-unit-tests.yml)
- **Frontend E2E tests** — [.github/workflows/e2e-tests.yml](.github/workflows/e2e-tests.yml) (triggered only when `frontend/` files change)
## Code Style
- **Backend (Python)**: We use `ruff` for linting and formatting. Run `make format` before committing.
- **Frontend (TypeScript)**: We use ESLint and Prettier. Run `pnpm format:write` before committing.
- CI enforces formatting — PRs with unformatted code will fail the lint check.
## Documentation
- [Configuration Guide](backend/docs/CONFIGURATION.md) - Setup and configuration
- [Architecture Overview](backend/CLAUDE.md) - Technical architecture
- [MCP Setup Guide](backend/docs/MCP_SERVER.md) - Model Context Protocol configuration
## Need Help?
- Check existing [Issues](https://github.com/bytedance/deer-flow/issues)
- Read the [Documentation](backend/docs/)
- Ask questions in [Discussions](https://github.com/bytedance/deer-flow/discussions)
## License
By contributing to DeerFlow, you agree that your contributions will be licensed under the [MIT License](./LICENSE).
-24
View File
@@ -1,24 +0,0 @@
FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim
# Install uv.
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
WORKDIR /app
# Pre-cache the application dependencies.
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync --locked --no-install-project
# Copy the application into the container.
COPY . /app
# Install the application dependencies.
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --locked
EXPOSE 8000
# Run the application.
CMD ["uv", "run", "python", "server.py", "--host", "0.0.0.0", "--port", "8000"]
+87
View File
@@ -0,0 +1,87 @@
# DeerFlow Install
This file is for coding agents. If the DeerFlow repository is not already cloned and open, clone `https://github.com/bytedance/deer-flow.git` first, then continue from the repository root.
## Goal
Bootstrap a DeerFlow local development workspace on the user's machine with the least risky path available.
Default preference:
1. Docker development environment
2. Local development environment
Do not assume API keys or model credentials exist. Set up everything that can be prepared safely, then stop with a concise summary of what the user still needs to provide.
## Operating Rules
- Be idempotent. Re-running this document should not damage an existing setup.
- Prefer existing repo commands over ad hoc shell commands.
- Do not use `sudo` or install system packages without explicit user approval.
- Do not overwrite existing user config values unless the user asks.
- If a step fails, stop, explain the blocker, and provide the smallest next action.
- If multiple setup paths are possible, prefer Docker when Docker is already available.
## Success Criteria
Consider the setup successful when all of the following are true:
- The DeerFlow repository is cloned and the current working directory is the repo root.
- `config.yaml` exists.
- For Docker setup, `make docker-init` completed successfully and Docker prerequisites are prepared, but services are not assumed to be running yet.
- For local setup, `make check` passed or reported no missing prerequisites, and `make install` completed successfully.
- The user receives the exact next command to launch DeerFlow.
- The user also receives any missing model configuration or referenced environment variable names from `config.yaml`, without inspecting secret-bearing files for actual values.
## Steps
- If the current directory is not the DeerFlow repository root, clone `https://github.com/bytedance/deer-flow.git` if needed, then change into the repository root.
- Confirm the current directory is the DeerFlow repository root by checking that `Makefile`, `backend/`, `frontend/`, and `config.example.yaml` exist.
- Detect whether `config.yaml` already exists.
- If `config.yaml` does not exist, run `make config`.
- Detect whether Docker is available and the daemon is reachable with `docker info`.
- If Docker is available:
- Run `make docker-init`.
- Treat this as Docker prerequisite preparation only. Do not claim that app services, compose validation, or image builds have already succeeded.
- Do not start long-running services unless the user explicitly asks or this setup request clearly includes launch verification.
- Tell the user the recommended next command is `make docker-start`.
- If Docker is not available:
- Run `make check`.
- If `make check` reports missing system dependencies such as `node`, `pnpm`, `uv`, or `nginx`, stop and report the missing tools instead of attempting privileged installs.
- If prerequisites are satisfied, run `make install`.
- Tell the user the recommended next command is `make dev`.
- Inspect `config.yaml` only for missing model entries or referenced environment variable placeholders. Do not read `.env`, `frontend/.env`, or other secret-bearing files.
- If no model is configured, tell the user they must add at least one entry under `models` in `config.yaml`.
- If `config.yaml` references variables such as `$OPENAI_API_KEY`, tell the user which variable names still need real values, but do not verify them by opening secret-bearing files.
- If the repository already appears configured, avoid repeating expensive work unless it is necessary to verify the environment.
## Verification
Use the lightest verification that matches the chosen setup path.
For Docker setup:
- Confirm `make docker-init` completed successfully.
- Confirm `config.yaml` exists.
- State explicitly that Docker services were not started and `make docker-start` is still the first real launch step.
- Do not leave background services running unless the user asked for that.
For local setup:
- Confirm `make install` completed successfully.
- Confirm `config.yaml` exists.
- Do not leave background services running unless the user asked for that.
## Final Response Format
Return a short status report with:
1. Setup path used: Docker or local
2. Setup level reached: Docker prerequisites prepared or local dependencies installed
3. Files created or detected: for example `config.yaml`
4. Remaining user action: model config, env var values, auth files, or nothing
5. Exact next command to start DeerFlow
## EXECUTE NOW
Complete the steps above. When finished, stop after the setup boundary and report status instead of continuing into unrelated project work.
+2 -1
View File
@@ -1,6 +1,7 @@
MIT License
Copyright (c) 2025 Bytedance Ltd. and/or its affiliates
Copyright (c) 2025-2026 DeerFlow Authors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
@@ -18,4 +19,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
SOFTWARE.
+176 -15
View File
@@ -1,22 +1,183 @@
.PHONY: lint format install-dev serve test coverage
# DeerFlow - Unified Development Environment
install-dev:
uv pip install -e ".[dev]" && uv pip install -e ".[test]"
.PHONY: help config config-upgrade check install setup doctor dev dev-daemon start start-daemon stop up down clean docker-init docker-start docker-stop docker-logs docker-logs-frontend docker-logs-gateway
format:
uv run black --preview .
BASH ?= bash
BACKEND_UV_RUN = cd backend && uv run
lint:
uv run black --check .
# Detect OS for Windows compatibility
ifeq ($(OS),Windows_NT)
SHELL := cmd.exe
PYTHON ?= python
# Run repo shell scripts through Git Bash when Make is launched from cmd.exe / PowerShell.
RUN_WITH_GIT_BASH = call scripts\run-with-git-bash.cmd
else
PYTHON ?= python3
RUN_WITH_GIT_BASH =
endif
serve:
uv run server.py --reload
help:
@echo "DeerFlow Development Commands:"
@echo " make setup - Interactive setup wizard (recommended for new users)"
@echo " make doctor - Check configuration and system requirements"
@echo " make config - Generate local config files (aborts if config already exists)"
@echo " make config-upgrade - Merge new fields from config.example.yaml into config.yaml"
@echo " make check - Check if all required tools are installed"
@echo " make install - Install all dependencies (frontend + backend + pre-commit hooks)"
@echo " make setup-sandbox - Pre-pull sandbox container image (recommended)"
@echo " make dev - Start all services in development mode (with hot-reloading)"
@echo " make dev-daemon - Start dev services in background (daemon mode)"
@echo " make start - Start all services in production mode (optimized, no hot-reloading)"
@echo " make start-daemon - Start prod services in background (daemon mode)"
@echo " make stop - Stop all running services"
@echo " make clean - Clean up processes and temporary files"
@echo ""
@echo "Docker Production Commands:"
@echo " make up - Build and start production Docker services (localhost:2026)"
@echo " make down - Stop and remove production Docker containers"
@echo ""
@echo "Docker Development Commands:"
@echo " make docker-init - Pull the sandbox image"
@echo " make docker-start - Start Docker services (mode-aware from config.yaml, localhost:2026)"
@echo " make docker-stop - Stop Docker development services"
@echo " make docker-logs - View Docker development logs"
@echo " make docker-logs-frontend - View Docker frontend logs"
@echo " make docker-logs-gateway - View Docker gateway logs"
test:
uv run pytest tests/
## Setup & Diagnosis
setup:
@$(BACKEND_UV_RUN) python ../scripts/setup_wizard.py
langgraph-dev:
uvx --refresh --from "langgraph-cli[inmem]" --with-editable . --python 3.12 langgraph dev --allow-blocking
doctor:
@$(BACKEND_UV_RUN) python ../scripts/doctor.py
coverage:
uv run pytest --cov=src tests/ --cov-report=term-missing --cov-report=xml
config:
@$(PYTHON) ./scripts/configure.py
config-upgrade:
@$(RUN_WITH_GIT_BASH) ./scripts/config-upgrade.sh
# Check required tools
check:
@$(PYTHON) ./scripts/check.py
# Install all dependencies
install:
@echo "Installing backend dependencies..."
@cd backend && uv sync
@echo "Installing frontend dependencies..."
@cd frontend && pnpm install
@echo "Installing pre-commit hooks..."
@$(BACKEND_UV_RUN) --with pre-commit pre-commit install
@echo "✓ All dependencies installed"
@echo ""
@echo "=========================================="
@echo " Optional: Pre-pull Sandbox Image"
@echo "=========================================="
@echo ""
@echo "If you plan to use Docker/Container-based sandbox, you can pre-pull the image:"
@echo " make setup-sandbox"
@echo ""
# Pre-pull sandbox Docker image (optional but recommended)
setup-sandbox:
@echo "=========================================="
@echo " Pre-pulling Sandbox Container Image"
@echo "=========================================="
@echo ""
@IMAGE=$$(grep -A 20 "# sandbox:" config.yaml 2>/dev/null | grep "image:" | awk '{print $$2}' | head -1); \
if [ -z "$$IMAGE" ]; then \
IMAGE="enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest"; \
echo "Using default image: $$IMAGE"; \
else \
echo "Using configured image: $$IMAGE"; \
fi; \
echo ""; \
if command -v container >/dev/null 2>&1 && [ "$$(uname)" = "Darwin" ]; then \
echo "Detected Apple Container on macOS, pulling image..."; \
container image pull "$$IMAGE" || echo "⚠ Apple Container pull failed, will try Docker"; \
fi; \
if command -v docker >/dev/null 2>&1; then \
echo "Pulling image using Docker..."; \
if docker pull "$$IMAGE"; then \
echo ""; \
echo "✓ Sandbox image pulled successfully"; \
else \
echo ""; \
echo "⚠ Failed to pull sandbox image (this is OK for local sandbox mode)"; \
fi; \
else \
echo "✗ Neither Docker nor Apple Container is available"; \
echo " Please install Docker: https://docs.docker.com/get-docker/"; \
exit 1; \
fi
# Start all services in development mode (with hot-reloading)
dev:
@$(PYTHON) ./scripts/check.py
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev
# Start all services in production mode (with optimizations)
start:
@$(PYTHON) ./scripts/check.py
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod
# Start all services in daemon mode (background)
dev-daemon:
@$(PYTHON) ./scripts/check.py
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev --daemon
# Start prod services in daemon mode (background)
start-daemon:
@$(PYTHON) ./scripts/check.py
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod --daemon
# Stop all services
stop:
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --stop
# Clean up
clean: stop
@echo "Cleaning up..."
@-rm -rf backend/.deer-flow 2>/dev/null || true
@-rm -rf backend/.langgraph_api 2>/dev/null || true
@-rm -rf logs/*.log 2>/dev/null || true
@echo "✓ Cleanup complete"
# ==========================================
# Docker Development Commands
# ==========================================
# Initialize Docker containers and install dependencies
docker-init:
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh init
# Start Docker development environment
docker-start:
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh start
# Stop Docker development environment
docker-stop:
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh stop
# View Docker development logs
docker-logs:
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs
# View Docker development logs
docker-logs-frontend:
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs --frontend
docker-logs-gateway:
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs --gateway
# ==========================================
# Production Docker Commands
# ==========================================
# Build and start production services
up:
@$(RUN_WITH_GIT_BASH) ./scripts/deploy.sh
# Stop and remove production containers
down:
@$(RUN_WITH_GIT_BASH) ./scripts/deploy.sh down
+688 -498
View File
File diff suppressed because it is too large Load Diff
-495
View File
@@ -1,495 +0,0 @@
# 🦌 DeerFlow
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![DeepWiki](https://img.shields.io/badge/DeepWiki-bytedance%2Fdeer--flow-blue.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACwAAAAyCAYAAAAnWDnqAAAAAXNSR0IArs4c6QAAA05JREFUaEPtmUtyEzEQhtWTQyQLHNak2AB7ZnyXZMEjXMGeK/AIi+QuHrMnbChYY7MIh8g01fJoopFb0uhhEqqcbWTp06/uv1saEDv4O3n3dV60RfP947Mm9/SQc0ICFQgzfc4CYZoTPAswgSJCCUJUnAAoRHOAUOcATwbmVLWdGoH//PB8mnKqScAhsD0kYP3j/Yt5LPQe2KvcXmGvRHcDnpxfL2zOYJ1mFwrryWTz0advv1Ut4CJgf5uhDuDj5eUcAUoahrdY/56ebRWeraTjMt/00Sh3UDtjgHtQNHwcRGOC98BJEAEymycmYcWwOprTgcB6VZ5JK5TAJ+fXGLBm3FDAmn6oPPjR4rKCAoJCal2eAiQp2x0vxTPB3ALO2CRkwmDy5WohzBDwSEFKRwPbknEggCPB/imwrycgxX2NzoMCHhPkDwqYMr9tRcP5qNrMZHkVnOjRMWwLCcr8ohBVb1OMjxLwGCvjTikrsBOiA6fNyCrm8V1rP93iVPpwaE+gO0SsWmPiXB+jikdf6SizrT5qKasx5j8ABbHpFTx+vFXp9EnYQmLx02h1QTTrl6eDqxLnGjporxl3NL3agEvXdT0WmEost648sQOYAeJS9Q7bfUVoMGnjo4AZdUMQku50McCcMWcBPvr0SzbTAFDfvJqwLzgxwATnCgnp4wDl6Aa+Ax283gghmj+vj7feE2KBBRMW3FzOpLOADl0Isb5587h/U4gGvkt5v60Z1VLG8BhYjbzRwyQZemwAd6cCR5/XFWLYZRIMpX39AR0tjaGGiGzLVyhse5C9RKC6ai42ppWPKiBagOvaYk8lO7DajerabOZP46Lby5wKjw1HCRx7p9sVMOWGzb/vA1hwiWc6jm3MvQDTogQkiqIhJV0nBQBTU+3okKCFDy9WwferkHjtxib7t3xIUQtHxnIwtx4mpg26/HfwVNVDb4oI9RHmx5WGelRVlrtiw43zboCLaxv46AZeB3IlTkwouebTr1y2NjSpHz68WNFjHvupy3q8TFn3Hos2IAk4Ju5dCo8B3wP7VPr/FGaKiG+T+v+TQqIrOqMTL1VdWV1DdmcbO8KXBz6esmYWYKPwDL5b5FA1a0hwapHiom0r/cKaoqr+27/XcrS5UwSMbQAAAABJRU5ErkJggg==)](https://deepwiki.com/bytedance/deer-flow)
<!-- DeepWiki badge generated by https://deepwiki.ryoppippi.com/ -->
[English](./README.md) | [简体中文](./README_zh.md) | [日本語](./README_ja.md) | [Deutsch](./README_de.md) | [Español](./README_es.md) | [Русский](./README_ru.md) | [Portuguese](./README_pt.md)
> Aus Open Source entstanden, an Open Source zurückgeben.
**DeerFlow** (**D**eep **E**xploration and **E**fficient **R**esearch **Flow**) ist ein Community-getriebenes Framework für tiefgehende Recherche, das auf der großartigen Arbeit der Open-Source-Community aufbaut. Unser Ziel ist es, Sprachmodelle mit spezialisierten Werkzeugen für Aufgaben wie Websuche, Crawling und Python-Code-Ausführung zu kombinieren und gleichzeitig der Community, die dies möglich gemacht hat, etwas zurückzugeben.
Besuchen Sie [unsere offizielle Website](https://deerflow.tech/) für weitere Details.
## Demo
### Video
https://github.com/user-attachments/assets/f3786598-1f2a-4d07-919e-8b99dfa1de3e
In dieser Demo zeigen wir, wie man DeerFlow nutzt, um:
- Nahtlos mit MCP-Diensten zu integrieren
- Den Prozess der tiefgehenden Recherche durchzuführen und einen umfassenden Bericht mit Bildern zu erstellen
- Podcast-Audio basierend auf dem generierten Bericht zu erstellen
### Wiedergaben
- [Wie hoch ist der Eiffelturm im Vergleich zum höchsten Gebäude?](https://deerflow.tech/chat?replay=eiffel-tower-vs-tallest-building)
- [Was sind die angesagtesten Repositories auf GitHub?](https://deerflow.tech/chat?replay=github-top-trending-repo)
- [Einen Artikel über traditionelle Gerichte aus Nanjing schreiben](https://deerflow.tech/chat?replay=nanjing-traditional-dishes)
- [Wie dekoriert man eine Mietwohnung?](https://deerflow.tech/chat?replay=rental-apartment-decoration)
- [Besuchen Sie unsere offizielle Website, um weitere Wiedergaben zu entdecken.](https://deerflow.tech/#case-studies)
---
## 📑 Inhaltsverzeichnis
- [🚀 Schnellstart](#schnellstart)
- [🌟 Funktionen](#funktionen)
- [🏗️ Architektur](#architektur)
- [🛠️ Entwicklung](#entwicklung)
- [🗣️ Text-zu-Sprache-Integration](#text-zu-sprache-integration)
- [📚 Beispiele](#beispiele)
- [❓ FAQ](#faq)
- [📜 Lizenz](#lizenz)
- [💖 Danksagungen](#danksagungen)
- [⭐ Star-Verlauf](#star-verlauf)
## Schnellstart
DeerFlow ist in Python entwickelt und kommt mit einer in Node.js geschriebenen Web-UI. Um einen reibungslosen Einrichtungsprozess zu gewährleisten, empfehlen wir die Verwendung der folgenden Tools:
### Empfohlene Tools
- **[`uv`](https://docs.astral.sh/uv/getting-started/installation/):**
Vereinfacht die Verwaltung von Python-Umgebungen und Abhängigkeiten. `uv` erstellt automatisch eine virtuelle Umgebung im Stammverzeichnis und installiert alle erforderlichen Pakete für Sie—keine manuelle Installation von Python-Umgebungen notwendig.
- **[`nvm`](https://github.com/nvm-sh/nvm):**
Verwalten Sie mühelos mehrere Versionen der Node.js-Laufzeit.
- **[`pnpm`](https://pnpm.io/installation):**
Installieren und verwalten Sie Abhängigkeiten des Node.js-Projekts.
### Umgebungsanforderungen
Stellen Sie sicher, dass Ihr System die folgenden Mindestanforderungen erfüllt:
- **[Python](https://www.python.org/downloads/):** Version `3.12+`
- **[Node.js](https://nodejs.org/en/download/):** Version `22+`
### Installation
```bash
# Repository klonen
git clone https://github.com/bytedance/deer-flow.git
cd deer-flow
# Abhängigkeiten installieren, uv kümmert sich um den Python-Interpreter und die Erstellung der venv sowie die Installation der erforderlichen Pakete
uv sync
# Konfigurieren Sie .env mit Ihren API-Schlüsseln
# Tavily: https://app.tavily.com/home
# Brave_SEARCH: https://brave.com/search/api/
# volcengine TTS: Fügen Sie Ihre TTS-Anmeldedaten hinzu, falls vorhanden
cp .env.example .env
# Siehe die Abschnitte 'Unterstützte Suchmaschinen' und 'Text-zu-Sprache-Integration' unten für alle verfügbaren Optionen
# Konfigurieren Sie conf.yaml für Ihr LLM-Modell und API-Schlüssel
# Weitere Details finden Sie unter 'docs/configuration_guide.md'
cp conf.yaml.example conf.yaml
# Installieren Sie marp für PPT-Generierung
# https://github.com/marp-team/marp-cli?tab=readme-ov-file#use-package-manager
brew install marp-cli
```
Optional können Sie Web-UI-Abhängigkeiten über [pnpm](https://pnpm.io/installation) installieren:
```bash
cd deer-flow/web
pnpm install
```
### Konfigurationen
Weitere Informationen finden Sie im [Konfigurationsleitfaden](docs/configuration_guide.md).
> [!HINWEIS]
> Lesen Sie den Leitfaden sorgfältig, bevor Sie das Projekt starten, und aktualisieren Sie die Konfigurationen entsprechend Ihren spezifischen Einstellungen und Anforderungen.
### Konsolen-UI
Der schnellste Weg, um das Projekt auszuführen, ist die Verwendung der Konsolen-UI.
```bash
# Führen Sie das Projekt in einer bash-ähnlichen Shell aus
uv run main.py
```
### Web-UI
Dieses Projekt enthält auch eine Web-UI, die ein dynamischeres und ansprechenderes interaktives Erlebnis bietet.
> [!HINWEIS]
> Sie müssen zuerst die Abhängigkeiten der Web-UI installieren.
```bash
# Führen Sie sowohl den Backend- als auch den Frontend-Server im Entwicklungsmodus aus
# Unter macOS/Linux
./bootstrap.sh -d
# Unter Windows
bootstrap.bat -d
```
Öffnen Sie Ihren Browser und besuchen Sie [`http://localhost:3000`](http://localhost:3000), um die Web-UI zu erkunden.
Weitere Details finden Sie im Verzeichnis [`web`](./web/).
## Unterstützte Suchmaschinen
DeerFlow unterstützt mehrere Suchmaschinen, die in Ihrer `.env`-Datei über die Variable `SEARCH_API` konfiguriert werden können:
- **Tavily** (Standard): Eine spezialisierte Such-API für KI-Anwendungen
- Erfordert `TAVILY_API_KEY` in Ihrer `.env`-Datei
- Registrieren Sie sich unter: https://app.tavily.com/home
- **DuckDuckGo**: Datenschutzorientierte Suchmaschine
- Kein API-Schlüssel erforderlich
- **Brave Search**: Datenschutzorientierte Suchmaschine mit erweiterten Funktionen
- Erfordert `BRAVE_SEARCH_API_KEY` in Ihrer `.env`-Datei
- Registrieren Sie sich unter: https://brave.com/search/api/
- **Arxiv**: Wissenschaftliche Papiersuche für akademische Forschung
- Kein API-Schlüssel erforderlich
- Spezialisiert auf wissenschaftliche und akademische Papiere
Um Ihre bevorzugte Suchmaschine zu konfigurieren, setzen Sie die Variable `SEARCH_API` in Ihrer `.env`-Datei:
```bash
# Wählen Sie eine: tavily, duckduckgo, brave_search, arxiv
SEARCH_API=tavily
```
## Funktionen
### Kernfähigkeiten
- 🤖 **LLM-Integration**
- Unterstützt die Integration der meisten Modelle über [litellm](https://docs.litellm.ai/docs/providers).
- Unterstützung für Open-Source-Modelle wie Qwen
- OpenAI-kompatible API-Schnittstelle
- Mehrstufiges LLM-System für unterschiedliche Aufgabenkomplexitäten
### Tools und MCP-Integrationen
- 🔍 **Suche und Abruf**
- Websuche über Tavily, Brave Search und mehr
- Crawling mit Jina
- Fortgeschrittene Inhaltsextraktion
- 🔗 **MCP Nahtlose Integration**
- Erweiterte Fähigkeiten für privaten Domänenzugriff, Wissensgraphen, Webbrowsing und mehr
- Erleichtert die Integration verschiedener Forschungswerkzeuge und -methoden
### Menschliche Zusammenarbeit
- 🧠 **Mensch-in-der-Schleife**
- Unterstützt interaktive Modifikation von Forschungsplänen mit natürlicher Sprache
- Unterstützt automatische Akzeptanz von Forschungsplänen
- 📝 **Bericht-Nachbearbeitung**
- Unterstützt Notion-ähnliche Blockbearbeitung
- Ermöglicht KI-Verfeinerungen, einschließlich KI-unterstützter Polierung, Satzkürzung und -erweiterung
- Angetrieben von [tiptap](https://tiptap.dev/)
### Inhaltserstellung
- 🎙️ **Podcast- und Präsentationserstellung**
- KI-gestützte Podcast-Skripterstellung und Audiosynthese
- Automatisierte Erstellung einfacher PowerPoint-Präsentationen
- Anpassbare Vorlagen für maßgeschneiderte Inhalte
## Architektur
DeerFlow implementiert eine modulare Multi-Agenten-Systemarchitektur, die für automatisierte Forschung und Codeanalyse konzipiert ist. Das System basiert auf LangGraph und ermöglicht einen flexiblen zustandsbasierten Workflow, bei dem Komponenten über ein klar definiertes Nachrichtenübermittlungssystem kommunizieren.
![Architekturdiagramm](./assets/architecture.png)
> Sehen Sie es live auf [deerflow.tech](https://deerflow.tech/#multi-agent-architecture)
Das System verwendet einen optimierten Workflow mit den folgenden Komponenten:
1. **Koordinator**: Der Einstiegspunkt, der den Workflow-Lebenszyklus verwaltet
- Initiiert den Forschungsprozess basierend auf Benutzereingaben
- Delegiert Aufgaben bei Bedarf an den Planer
- Fungiert als primäre Schnittstelle zwischen dem Benutzer und dem System
2. **Planer**: Strategische Komponente für Aufgabenzerlegung und -planung
- Analysiert Forschungsziele und erstellt strukturierte Ausführungspläne
- Bestimmt, ob ausreichend Kontext verfügbar ist oder ob weitere Forschung benötigt wird
- Verwaltet den Forschungsablauf und entscheidet, wann der endgültige Bericht erstellt wird
3. **Forschungsteam**: Eine Sammlung spezialisierter Agenten, die den Plan ausführen:
- **Forscher**: Führt Websuchen und Informationssammlung mit Tools wie Websuchmaschinen, Crawling und sogar MCP-Diensten durch.
- **Codierer**: Behandelt Codeanalyse, -ausführung und technische Aufgaben mit dem Python REPL Tool.
Jeder Agent hat Zugriff auf spezifische Tools, die für seine Rolle optimiert sind, und operiert innerhalb des LangGraph-Frameworks
4. **Reporter**: Endphasenprozessor für Forschungsergebnisse
- Aggregiert Erkenntnisse vom Forschungsteam
- Verarbeitet und strukturiert die gesammelten Informationen
- Erstellt umfassende Forschungsberichte
## Text-zu-Sprache-Integration
DeerFlow enthält jetzt eine Text-zu-Sprache (TTS)-Funktion, mit der Sie Forschungsberichte in Sprache umwandeln können. Diese Funktion verwendet die volcengine TTS API, um hochwertige Audios aus Text zu generieren. Funktionen wie Geschwindigkeit, Lautstärke und Tonhöhe können ebenfalls angepasst werden.
### Verwendung der TTS API
Sie können auf die TTS-Funktionalität über den Endpunkt `/api/tts` zugreifen:
```bash
# Beispiel API-Aufruf mit curl
curl --location 'http://localhost:8000/api/tts' \
--header 'Content-Type: application/json' \
--data '{
"text": "Dies ist ein Test der Text-zu-Sprache-Funktionalität.",
"speed_ratio": 1.0,
"volume_ratio": 1.0,
"pitch_ratio": 1.0
}' \
--output speech.mp3
```
## Entwicklung
### Testen
Führen Sie die Testsuite aus:
```bash
# Alle Tests ausführen
make test
# Spezifische Testdatei ausführen
pytest tests/integration/test_workflow.py
# Mit Abdeckung ausführen
make coverage
```
### Codequalität
```bash
# Lint ausführen
make lint
# Code formatieren
make format
```
### Debugging mit LangGraph Studio
DeerFlow verwendet LangGraph für seine Workflow-Architektur. Sie können LangGraph Studio verwenden, um den Workflow in Echtzeit zu debuggen und zu visualisieren.
#### LangGraph Studio lokal ausführen
DeerFlow enthält eine `langgraph.json`-Konfigurationsdatei, die die Graphstruktur und Abhängigkeiten für das LangGraph Studio definiert. Diese Datei verweist auf die im Projekt definierten Workflow-Graphen und lädt automatisch Umgebungsvariablen aus der `.env`-Datei.
##### Mac
```bash
# Installieren Sie den uv-Paketmanager, wenn Sie ihn noch nicht haben
curl -LsSf https://astral.sh/uv/install.sh | sh
# Installieren Sie Abhängigkeiten und starten Sie den LangGraph-Server
uvx --refresh --from "langgraph-cli[inmem]" --with-editable . --python 3.12 langgraph dev --allow-blocking
```
##### Windows / Linux
```bash
# Abhängigkeiten installieren
pip install -e .
pip install -U "langgraph-cli[inmem]"
# LangGraph-Server starten
langgraph dev
```
Nach dem Start des LangGraph-Servers sehen Sie mehrere URLs im Terminal:
- API: http://127.0.0.1:2024
- Studio UI: https://smith.langchain.com/studio/?baseUrl=http://127.0.0.1:2024
- API-Dokumentation: http://127.0.0.1:2024/docs
Öffnen Sie den Studio UI-Link in Ihrem Browser, um auf die Debugging-Schnittstelle zuzugreifen.
#### Verwendung von LangGraph Studio
In der Studio UI können Sie:
1. Den Workflow-Graphen visualisieren und sehen, wie Komponenten verbunden sind
2. Die Ausführung in Echtzeit verfolgen, um zu sehen, wie Daten durch das System fließen
3. Den Zustand in jedem Schritt des Workflows inspizieren
4. Probleme durch Untersuchung von Ein- und Ausgaben jeder Komponente debuggen
5. Feedback während der Planungsphase geben, um Forschungspläne zu verfeinern
Wenn Sie ein Forschungsthema in der Studio UI einreichen, können Sie die gesamte Workflow-Ausführung sehen, einschließlich:
- Die Planungsphase, in der der Forschungsplan erstellt wird
- Die Feedback-Schleife, in der Sie den Plan ändern können
- Die Forschungs- und Schreibphasen für jeden Abschnitt
- Die Erstellung des endgültigen Berichts
### Aktivieren von LangSmith-Tracing
DeerFlow unterstützt LangSmith-Tracing, um Ihnen beim Debuggen und Überwachen Ihrer Workflows zu helfen. Um LangSmith-Tracing zu aktivieren:
1. Stellen Sie sicher, dass Ihre `.env`-Datei die folgenden Konfigurationen enthält (siehe `.env.example`):
```bash
LANGSMITH_TRACING=true
LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
LANGSMITH_API_KEY="xxx"
LANGSMITH_PROJECT="xxx"
```
2. Starten Sie das Tracing mit LangSmith lokal, indem Sie folgenden Befehl ausführen:
```bash
langgraph dev
```
Dies aktiviert die Trace-Visualisierung in LangGraph Studio und sendet Ihre Traces zur Überwachung und Analyse an LangSmith.
## Beispiele
Die folgenden Beispiele demonstrieren die Fähigkeiten von DeerFlow:
### Forschungsberichte
1. **OpenAI Sora Bericht** - Analyse von OpenAIs Sora KI-Tool
- Diskutiert Funktionen, Zugang, Prompt-Engineering, Einschränkungen und ethische Überlegungen
- [Vollständigen Bericht ansehen](examples/openai_sora_report.md)
2. **Googles Agent-to-Agent-Protokoll Bericht** - Überblick über Googles Agent-to-Agent (A2A)-Protokoll
- Diskutiert seine Rolle in der KI-Agentenkommunikation und seine Beziehung zum Model Context Protocol (MCP) von Anthropic
- [Vollständigen Bericht ansehen](examples/what_is_agent_to_agent_protocol.md)
3. **Was ist MCP?** - Eine umfassende Analyse des Begriffs "MCP" in mehreren Kontexten
- Untersucht Model Context Protocol in KI, Monocalciumphosphat in der Chemie und Micro-channel Plate in der Elektronik
- [Vollständigen Bericht ansehen](examples/what_is_mcp.md)
4. **Bitcoin-Preisschwankungen** - Analyse der jüngsten Bitcoin-Preisbewegungen
- Untersucht Markttrends, regulatorische Einflüsse und technische Indikatoren
- Bietet Empfehlungen basierend auf historischen Daten
- [Vollständigen Bericht ansehen](examples/bitcoin_price_fluctuation.md)
5. **Was ist LLM?** - Eine eingehende Erforschung großer Sprachmodelle
- Diskutiert Architektur, Training, Anwendungen und ethische Überlegungen
- [Vollständigen Bericht ansehen](examples/what_is_llm.md)
6. **Wie nutzt man Claude für tiefgehende Recherche?** - Best Practices und Workflows für die Verwendung von Claude in der tiefgehenden Forschung
- Behandelt Prompt-Engineering, Datenanalyse und Integration mit anderen Tools
- [Vollständigen Bericht ansehen](examples/how_to_use_claude_deep_research.md)
7. **KI-Adoption im Gesundheitswesen: Einflussfaktoren** - Analyse der Faktoren, die die KI-Adoption im Gesundheitswesen vorantreiben
- Diskutiert KI-Technologien, Datenqualität, ethische Überlegungen, wirtschaftliche Bewertungen, organisatorische Bereitschaft und digitale Infrastruktur
- [Vollständigen Bericht ansehen](examples/AI_adoption_in_healthcare.md)
8. **Auswirkungen des Quantencomputing auf die Kryptographie** - Analyse der Auswirkungen des Quantencomputing auf die Kryptographie
- Diskutiert Schwachstellen der klassischen Kryptographie, Post-Quanten-Kryptographie und quantenresistente kryptographische Lösungen
- [Vollständigen Bericht ansehen](examples/Quantum_Computing_Impact_on_Cryptography.md)
9. **Cristiano Ronaldos Leistungshöhepunkte** - Analyse der Leistungshöhepunkte von Cristiano Ronaldo
- Diskutiert seine Karriereerfolge, internationalen Tore und Leistungen in verschiedenen Spielen
- [Vollständigen Bericht ansehen](examples/Cristiano_Ronaldo's_Performance_Highlights.md)
Um diese Beispiele auszuführen oder Ihre eigenen Forschungsberichte zu erstellen, können Sie die folgenden Befehle verwenden:
```bash
# Mit einer spezifischen Anfrage ausführen
uv run main.py "Welche Faktoren beeinflussen die KI-Adoption im Gesundheitswesen?"
# Mit benutzerdefinierten Planungsparametern ausführen
uv run main.py --max_plan_iterations 3 "Wie wirkt sich Quantencomputing auf die Kryptographie aus?"
# Im interaktiven Modus mit eingebauten Fragen ausführen
uv run main.py --interactive
# Oder mit grundlegendem interaktiven Prompt ausführen
uv run main.py
# Alle verfügbaren Optionen anzeigen
uv run main.py --help
```
### Interaktiver Modus
Die Anwendung unterstützt jetzt einen interaktiven Modus mit eingebauten Fragen in Englisch und Chinesisch:
1. Starten Sie den interaktiven Modus:
```bash
uv run main.py --interactive
```
2. Wählen Sie Ihre bevorzugte Sprache (English oder 中文)
3. Wählen Sie aus einer Liste von eingebauten Fragen oder wählen Sie die Option, Ihre eigene Frage zu stellen
4. Das System wird Ihre Frage verarbeiten und einen umfassenden Forschungsbericht generieren
### Mensch-in-der-Schleife
DeerFlow enthält einen Mensch-in-der-Schleife-Mechanismus, der es Ihnen ermöglicht, Forschungspläne vor ihrer Ausführung zu überprüfen, zu bearbeiten und zu genehmigen:
1. **Planüberprüfung**: Wenn Mensch-in-der-Schleife aktiviert ist, präsentiert das System den generierten Forschungsplan zur Überprüfung vor der Ausführung
2. **Feedback geben**: Sie können:
- Den Plan akzeptieren, indem Sie mit `[ACCEPTED]` antworten
- Den Plan bearbeiten, indem Sie Feedback geben (z.B., `[EDIT PLAN] Fügen Sie mehr Schritte zur technischen Implementierung hinzu`)
- Das System wird Ihr Feedback einarbeiten und einen überarbeiteten Plan generieren
3. **Automatische Akzeptanz**: Sie können die automatische Akzeptanz aktivieren, um den Überprüfungsprozess zu überspringen:
- Über API: Setzen Sie `auto_accepted_plan: true` in Ihrer Anfrage
4. **API-Integration**: Bei Verwendung der API können Sie Feedback über den Parameter `feedback` geben:
```json
{
"messages": [{"role": "user", "content": "Was ist Quantencomputing?"}],
"thread_id": "my_thread_id",
"auto_accepted_plan": false,
"feedback": "[EDIT PLAN] Mehr über Quantenalgorithmen aufnehmen"
}
```
### Kommandozeilenargumente
Die Anwendung unterstützt mehrere Kommandozeilenargumente, um ihr Verhalten anzupassen:
- **query**: Die zu verarbeitende Forschungsanfrage (kann mehrere Wörter umfassen)
- **--interactive**: Im interaktiven Modus mit eingebauten Fragen ausführen
- **--max_plan_iterations**: Maximale Anzahl von Planungszyklen (Standard: 1)
- **--max_step_num**: Maximale Anzahl von Schritten in einem Forschungsplan (Standard: 3)
- **--debug**: Detaillierte Debug-Protokollierung aktivieren
## FAQ
Weitere Informationen finden Sie in der [FAQ.md](docs/FAQ.md).
## Lizenz
Dieses Projekt ist Open Source und unter der [MIT-Lizenz](./LICENSE) verfügbar.
## Danksagungen
DeerFlow baut auf der unglaublichen Arbeit der Open-Source-Community auf. Wir sind allen Projekten und Mitwirkenden zutiefst dankbar, deren Bemühungen DeerFlow möglich gemacht haben. Wahrhaftig stehen wir auf den Schultern von Riesen.
Wir möchten unsere aufrichtige Wertschätzung den folgenden Projekten für ihre unschätzbaren Beiträge aussprechen:
- **[LangChain](https://github.com/langchain-ai/langchain)**: Ihr außergewöhnliches Framework unterstützt unsere LLM-Interaktionen und -Ketten und ermöglicht nahtlose Integration und Funktionalität.
- **[LangGraph](https://github.com/langchain-ai/langgraph)**: Ihr innovativer Ansatz zur Multi-Agenten-Orchestrierung war maßgeblich für die Ermöglichung der ausgeklügelten Workflows von DeerFlow.
Diese Projekte veranschaulichen die transformative Kraft der Open-Source-Zusammenarbeit, und wir sind stolz darauf, auf ihren Grundlagen aufzubauen.
### Hauptmitwirkende
Ein herzliches Dankeschön geht an die Hauptautoren von `DeerFlow`, deren Vision, Leidenschaft und Engagement dieses Projekt zum Leben erweckt haben:
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
- **[Henry Li](https://github.com/magiccube/)**
Ihr unerschütterliches Engagement und Fachwissen waren die treibende Kraft hinter dem Erfolg von DeerFlow. Wir fühlen uns geehrt, Sie an der Spitze dieser Reise zu haben.
## Star-Verlauf
[![Star History Chart](https://api.star-history.com/svg?repos=bytedance/deer-flow&type=Date)](https://star-history.com/#bytedance/deer-flow&Date)
-554
View File
@@ -1,554 +0,0 @@
# 🦌 DeerFlow
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![DeepWiki](https://img.shields.io/badge/DeepWiki-bytedance%2Fdeer--flow-blue.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACwAAAAyCAYAAAAnWDnqAAAAAXNSR0IArs4c6QAAA05JREFUaEPtmUtyEzEQhtWTQyQLHNak2AB7ZnyXZMEjXMGeK/AIi+QuHrMnbChYY7MIh8g01fJoopFb0uhhEqqcbWTp06/uv1saEDv4O3n3dV60RfP947Mm9/SQc0ICFQgzfc4CYZoTPAswgSJCCUJUnAAoRHOAUOcATwbmVLWdGoH//PB8mnKqScAhsD0kYP3j/Yt5LPQe2KvcXmGvRHcDnpxfL2zOYJ1mFwrryWTz0advv1Ut4CJgf5uhDuDj5eUcAUoahrdY/56ebRWeraTjMt/00Sh3UDtjgHtQNHwcRGOC98BJEAEymycmYcWwOprTgcB6VZ5JK5TAJ+fXGLBm3FDAmn6oPPjR4rKCAoJCal2eAiQp2x0vxTPB3ALO2CRkwmDy5WohzBDwSEFKRwPbknEggCPB/imwrycgxX2NzoMCHhPkDwqYMr9tRcP5qNrMZHkVnOjRMWwLCcr8ohBVb1OMjxLwGCvjTikrsBOiA6fNyCrm8V1rP93iVPpwaE+gO0SsWmPiXB+jikdf6SizrT5qKasx5j8ABbHpFTx+vFXp9EnYQmLx02h1QTTrl6eDqxLnGjporxl3NL3agEvXdT0WmEost648sQOYAeJS9Q7bfUVoMGnjo4AZdUMQku50McCcMWcBPvr0SzbTAFDfvJqwLzgxwATnCgnp4wDl6Aa+Ax283gghmj+vj7feE2KBBRMW3FzOpLOADl0Isb5587h/U4gGvkt5v60Z1VLG8BhYjbzRwyQZemwAd6cCR5/XFWLYZRIMpX39AR0tjaGGiGzLVyhse5C9RKC6ai42ppWPKiBagOvaYk8lO7DajerabOZP46Lby5wKjw1HCRx7p9sVMOWGzb/vA1hwiWc6jm3MvQDTogQkiqIhJV0nBQBTU+3okKCFDy9WwferkHjtxib7t3xIUQtHxnIwtx4mpg26/HfwVNVDb4oI9RHmx5WGelRVlrtiw43zboCLaxv46AZeB3IlTkwouebTr1y2NjSpHz68WNFjHvupy3q8TFn3Hos2IAk4Ju5dCo8B3wP7VPr/FGaKiG+T+v+TQqIrOqMTL1VdWV1DdmcbO8KXBz6esmYWYKPwDL5b5FA1a0hwapHiom0r/cKaoqr+27/XcrS5UwSMbQAAAABJRU5ErkJggg==)](https://deepwiki.com/bytedance/deer-flow)
<!-- DeepWiki badge generated by https://deepwiki.ryoppippi.com/ -->
[English](./README.md) | [简体中文](./README_zh.md) | [日本語](./README_ja.md) | [Deutsch](./README_de.md) | [Español](./README_es.md) | [Русский](./README_ru.md) | [Portuguese](./README_pt.md)
> Originado del código abierto, retribuido al código abierto.
**DeerFlow** (**D**eep **E**xploration and **E**fficient **R**esearch **Flow**) es un marco de Investigación Profunda impulsado por la comunidad que se basa en el increíble trabajo de la comunidad de código abierto. Nuestro objetivo es combinar modelos de lenguaje con herramientas especializadas para tareas como búsqueda web, rastreo y ejecución de código Python, mientras devolvemos a la comunidad que hizo esto posible.
Por favor, visita [nuestra página web oficial](https://deerflow.tech/) para más detalles.
## Demostración
### Video
https://github.com/user-attachments/assets/f3786598-1f2a-4d07-919e-8b99dfa1de3e
En esta demostración, mostramos cómo usar DeerFlow para:
- Integrar perfectamente con servicios MCP
- Realizar el proceso de Investigación Profunda y producir un informe completo con imágenes
- Crear audio de podcast basado en el informe generado
### Repeticiones
- [¿Qué altura tiene la Torre Eiffel comparada con el edificio más alto?](https://deerflow.tech/chat?replay=eiffel-tower-vs-tallest-building)
- [¿Cuáles son los repositorios más populares en GitHub?](https://deerflow.tech/chat?replay=github-top-trending-repo)
- [Escribir un artículo sobre los platos tradicionales de Nanjing](https://deerflow.tech/chat?replay=nanjing-traditional-dishes)
- [¿Cómo decorar un apartamento de alquiler?](https://deerflow.tech/chat?replay=rental-apartment-decoration)
- [Visita nuestra página web oficial para explorar más repeticiones.](https://deerflow.tech/#case-studies)
---
## 📑 Tabla de Contenidos
- [🚀 Inicio Rápido](#inicio-rápido)
- [🌟 Características](#características)
- [🏗️ Arquitectura](#arquitectura)
- [🛠️ Desarrollo](#desarrollo)
- [🐳 Docker](#docker)
- [🗣️ Integración de Texto a Voz](#integración-de-texto-a-voz)
- [📚 Ejemplos](#ejemplos)
- [❓ Preguntas Frecuentes](#preguntas-frecuentes)
- [📜 Licencia](#licencia)
- [💖 Agradecimientos](#agradecimientos)
- [⭐ Historial de Estrellas](#historial-de-estrellas)
## Inicio Rápido
DeerFlow está desarrollado en Python y viene con una interfaz web escrita en Node.js. Para garantizar un proceso de configuración sin problemas, recomendamos utilizar las siguientes herramientas:
### Herramientas Recomendadas
- **[`uv`](https://docs.astral.sh/uv/getting-started/installation/):**
Simplifica la gestión del entorno Python y las dependencias. `uv` crea automáticamente un entorno virtual en el directorio raíz e instala todos los paquetes necesarios por ti—sin necesidad de instalar entornos Python manualmente.
- **[`nvm`](https://github.com/nvm-sh/nvm):**
Gestiona múltiples versiones del entorno de ejecución Node.js sin esfuerzo.
- **[`pnpm`](https://pnpm.io/installation):**
Instala y gestiona dependencias del proyecto Node.js.
### Requisitos del Entorno
Asegúrate de que tu sistema cumple con los siguientes requisitos mínimos:
- **[Python](https://www.python.org/downloads/):** Versión `3.12+`
- **[Node.js](https://nodejs.org/en/download/):** Versión `22+`
### Instalación
```bash
# Clonar el repositorio
git clone https://github.com/bytedance/deer-flow.git
cd deer-flow
# Instalar dependencias, uv se encargará del intérprete de python, la creación del entorno virtual y la instalación de los paquetes necesarios
uv sync
# Configurar .env con tus claves API
# Tavily: https://app.tavily.com/home
# Brave_SEARCH: https://brave.com/search/api/
# volcengine TTS: Añade tus credenciales TTS si las tienes
cp .env.example .env
# Ver las secciones 'Motores de Búsqueda Compatibles' e 'Integración de Texto a Voz' a continuación para todas las opciones disponibles
# Configurar conf.yaml para tu modelo LLM y claves API
# Por favor, consulta 'docs/configuration_guide.md' para más detalles
cp conf.yaml.example conf.yaml
# Instalar marp para la generación de presentaciones
# https://github.com/marp-team/marp-cli?tab=readme-ov-file#use-package-manager
brew install marp-cli
```
Opcionalmente, instala las dependencias de la interfaz web vía [pnpm](https://pnpm.io/installation):
```bash
cd deer-flow/web
pnpm install
```
### Configuraciones
Por favor, consulta la [Guía de Configuración](docs/configuration_guide.md) para más detalles.
> [!NOTA]
> Antes de iniciar el proyecto, lee la guía cuidadosamente y actualiza las configuraciones para que coincidan con tus ajustes y requisitos específicos.
### Interfaz de Consola
La forma más rápida de ejecutar el proyecto es utilizar la interfaz de consola.
```bash
# Ejecutar el proyecto en un shell tipo bash
uv run main.py
```
### Interfaz Web
Este proyecto también incluye una Interfaz Web, que ofrece una experiencia interactiva más dinámica y atractiva.
> [!NOTA]
> Necesitas instalar primero las dependencias de la interfaz web.
```bash
# Ejecutar tanto el servidor backend como el frontend en modo desarrollo
# En macOS/Linux
./bootstrap.sh -d
# En Windows
bootstrap.bat -d
```
Abre tu navegador y visita [`http://localhost:3000`](http://localhost:3000) para explorar la interfaz web.
Explora más detalles en el directorio [`web`](./web/).
## Motores de Búsqueda Compatibles
DeerFlow soporta múltiples motores de búsqueda que pueden configurarse en tu archivo `.env` usando la variable `SEARCH_API`:
- **Tavily** (predeterminado): Una API de búsqueda especializada para aplicaciones de IA
- Requiere `TAVILY_API_KEY` en tu archivo `.env`
- Regístrate en: https://app.tavily.com/home
- **DuckDuckGo**: Motor de búsqueda centrado en la privacidad
- No requiere clave API
- **Brave Search**: Motor de búsqueda centrado en la privacidad con características avanzadas
- Requiere `BRAVE_SEARCH_API_KEY` en tu archivo `.env`
- Regístrate en: https://brave.com/search/api/
- **Arxiv**: Búsqueda de artículos científicos para investigación académica
- No requiere clave API
- Especializado en artículos científicos y académicos
Para configurar tu motor de búsqueda preferido, establece la variable `SEARCH_API` en tu archivo `.env`:
```bash
# Elige uno: tavily, duckduckgo, brave_search, arxiv
SEARCH_API=tavily
```
## Características
### Capacidades Principales
- 🤖 **Integración de LLM**
- Soporta la integración de la mayoría de los modelos a través de [litellm](https://docs.litellm.ai/docs/providers).
- Soporte para modelos de código abierto como Qwen
- Interfaz API compatible con OpenAI
- Sistema LLM de múltiples niveles para diferentes complejidades de tareas
### Herramientas e Integraciones MCP
- 🔍 **Búsqueda y Recuperación**
- Búsqueda web a través de Tavily, Brave Search y más
- Rastreo con Jina
- Extracción avanzada de contenido
- 🔗 **Integración Perfecta con MCP**
- Amplía capacidades para acceso a dominio privado, gráfico de conocimiento, navegación web y más
- Facilita la integración de diversas herramientas y metodologías de investigación
### Colaboración Humana
- 🧠 **Humano en el Bucle**
- Soporta modificación interactiva de planes de investigación usando lenguaje natural
- Soporta aceptación automática de planes de investigación
- 📝 **Post-Edición de Informes**
- Soporta edición de bloques tipo Notion
- Permite refinamientos por IA, incluyendo pulido asistido por IA, acortamiento y expansión de oraciones
- Impulsado por [tiptap](https://tiptap.dev/)
### Creación de Contenido
- 🎙️ **Generación de Podcasts y Presentaciones**
- Generación de guiones de podcast y síntesis de audio impulsadas por IA
- Creación automatizada de presentaciones PowerPoint simples
- Plantillas personalizables para contenido a medida
## Arquitectura
DeerFlow implementa una arquitectura modular de sistema multi-agente diseñada para investigación automatizada y análisis de código. El sistema está construido sobre LangGraph, permitiendo un flujo de trabajo flexible basado en estados donde los componentes se comunican a través de un sistema de paso de mensajes bien definido.
![Diagrama de Arquitectura](./assets/architecture.png)
> Vélo en vivo en [deerflow.tech](https://deerflow.tech/#multi-agent-architecture)
El sistema emplea un flujo de trabajo racionalizado con los siguientes componentes:
1. **Coordinador**: El punto de entrada que gestiona el ciclo de vida del flujo de trabajo
- Inicia el proceso de investigación basado en la entrada del usuario
- Delega tareas al planificador cuando corresponde
- Actúa como la interfaz principal entre el usuario y el sistema
2. **Planificador**: Componente estratégico para descomposición y planificación de tareas
- Analiza objetivos de investigación y crea planes de ejecución estructurados
- Determina si hay suficiente contexto disponible o si se necesita más investigación
- Gestiona el flujo de investigación y decide cuándo generar el informe final
3. **Equipo de Investigación**: Una colección de agentes especializados que ejecutan el plan:
- **Investigador**: Realiza búsquedas web y recopilación de información utilizando herramientas como motores de búsqueda web, rastreo e incluso servicios MCP.
- **Programador**: Maneja análisis de código, ejecución y tareas técnicas utilizando la herramienta Python REPL.
Cada agente tiene acceso a herramientas específicas optimizadas para su rol y opera dentro del marco LangGraph
4. **Reportero**: Procesador de etapa final para los resultados de la investigación
- Agrega hallazgos del equipo de investigación
- Procesa y estructura la información recopilada
- Genera informes de investigación completos
## Integración de Texto a Voz
DeerFlow ahora incluye una función de Texto a Voz (TTS) que te permite convertir informes de investigación a voz. Esta función utiliza la API TTS de volcengine para generar audio de alta calidad a partir de texto. Características como velocidad, volumen y tono también son personalizables.
### Usando la API TTS
Puedes acceder a la funcionalidad TTS a través del punto final `/api/tts`:
```bash
# Ejemplo de llamada API usando curl
curl --location 'http://localhost:8000/api/tts' \
--header 'Content-Type: application/json' \
--data '{
"text": "Esto es una prueba de la funcionalidad de texto a voz.",
"speed_ratio": 1.0,
"volume_ratio": 1.0,
"pitch_ratio": 1.0
}' \
--output speech.mp3
```
## Desarrollo
### Pruebas
Ejecuta el conjunto de pruebas:
```bash
# Ejecutar todas las pruebas
make test
# Ejecutar archivo de prueba específico
pytest tests/integration/test_workflow.py
# Ejecutar con cobertura
make coverage
```
### Calidad del Código
```bash
# Ejecutar linting
make lint
# Formatear código
make format
```
### Depuración con LangGraph Studio
DeerFlow utiliza LangGraph para su arquitectura de flujo de trabajo. Puedes usar LangGraph Studio para depurar y visualizar el flujo de trabajo en tiempo real.
#### Ejecutando LangGraph Studio Localmente
DeerFlow incluye un archivo de configuración `langgraph.json` que define la estructura del grafo y las dependencias para LangGraph Studio. Este archivo apunta a los grafos de flujo de trabajo definidos en el proyecto y carga automáticamente variables de entorno desde el archivo `.env`.
##### Mac
```bash
# Instala el gestor de paquetes uv si no lo tienes
curl -LsSf https://astral.sh/uv/install.sh | sh
# Instala dependencias e inicia el servidor LangGraph
uvx --refresh --from "langgraph-cli[inmem]" --with-editable . --python 3.12 langgraph dev --allow-blocking
```
##### Windows / Linux
```bash
# Instalar dependencias
pip install -e .
pip install -U "langgraph-cli[inmem]"
# Iniciar el servidor LangGraph
langgraph dev
```
Después de iniciar el servidor LangGraph, verás varias URLs en la terminal:
- API: http://127.0.0.1:2024
- UI de Studio: https://smith.langchain.com/studio/?baseUrl=http://127.0.0.1:2024
- Docs de API: http://127.0.0.1:2024/docs
Abre el enlace de UI de Studio en tu navegador para acceder a la interfaz de depuración.
#### Usando LangGraph Studio
En la UI de Studio, puedes:
1. Visualizar el grafo de flujo de trabajo y ver cómo se conectan los componentes
2. Rastrear la ejecución en tiempo real para ver cómo fluyen los datos a través del sistema
3. Inspeccionar el estado en cada paso del flujo de trabajo
4. Depurar problemas examinando entradas y salidas de cada componente
5. Proporcionar retroalimentación durante la fase de planificación para refinar planes de investigación
Cuando envías un tema de investigación en la UI de Studio, podrás ver toda la ejecución del flujo de trabajo, incluyendo:
- La fase de planificación donde se crea el plan de investigación
- El bucle de retroalimentación donde puedes modificar el plan
- Las fases de investigación y escritura para cada sección
- La generación del informe final
### Habilitando el Rastreo de LangSmith
DeerFlow soporta el rastreo de LangSmith para ayudarte a depurar y monitorear tus flujos de trabajo. Para habilitar el rastreo de LangSmith:
1. Asegúrate de que tu archivo `.env` tenga las siguientes configuraciones (ver `.env.example`):
```bash
LANGSMITH_TRACING=true
LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
LANGSMITH_API_KEY="xxx"
LANGSMITH_PROJECT="xxx"
```
2. Inicia el rastreo y visualiza el grafo localmente con LangSmith ejecutando:
```bash
langgraph dev
```
Esto habilitará la visualización de rastros en LangGraph Studio y enviará tus rastros a LangSmith para monitoreo y análisis.
## Docker
También puedes ejecutar este proyecto con Docker.
Primero, necesitas leer la [configuración](docs/configuration_guide.md) a continuación. Asegúrate de que los archivos `.env` y `.conf.yaml` estén listos.
Segundo, para construir una imagen Docker de tu propio servidor web:
```bash
docker build -t deer-flow-api .
```
Finalmente, inicia un contenedor Docker que ejecute el servidor web:
```bash
# Reemplaza deer-flow-api-app con tu nombre de contenedor preferido
docker run -d -t -p 8000:8000 --env-file .env --name deer-flow-api-app deer-flow-api
# detener el servidor
docker stop deer-flow-api-app
```
### Docker Compose (incluye tanto backend como frontend)
DeerFlow proporciona una configuración docker-compose para ejecutar fácilmente tanto el backend como el frontend juntos:
```bash
# construir imagen docker
docker compose build
# iniciar el servidor
docker compose up
```
## Ejemplos
Los siguientes ejemplos demuestran las capacidades de DeerFlow:
### Informes de Investigación
1. **Informe sobre OpenAI Sora** - Análisis de la herramienta IA Sora de OpenAI
- Discute características, acceso, ingeniería de prompts, limitaciones y consideraciones éticas
- [Ver informe completo](examples/openai_sora_report.md)
2. **Informe sobre el Protocolo Agent to Agent de Google** - Visión general del protocolo Agent to Agent (A2A) de Google
- Discute su papel en la comunicación de agentes IA y su relación con el Model Context Protocol (MCP) de Anthropic
- [Ver informe completo](examples/what_is_agent_to_agent_protocol.md)
3. **¿Qué es MCP?** - Un análisis completo del término "MCP" en múltiples contextos
- Explora Model Context Protocol en IA, Fosfato Monocálcico en química y Placa de Microcanales en electrónica
- [Ver informe completo](examples/what_is_mcp.md)
4. **Fluctuaciones del Precio de Bitcoin** - Análisis de los movimientos recientes del precio de Bitcoin
- Examina tendencias del mercado, influencias regulatorias e indicadores técnicos
- Proporciona recomendaciones basadas en datos históricos
- [Ver informe completo](examples/bitcoin_price_fluctuation.md)
5. **¿Qué es LLM?** - Una exploración en profundidad de los Modelos de Lenguaje Grandes
- Discute arquitectura, entrenamiento, aplicaciones y consideraciones éticas
- [Ver informe completo](examples/what_is_llm.md)
6. **¿Cómo usar Claude para Investigación Profunda?** - Mejores prácticas y flujos de trabajo para usar Claude en investigación profunda
- Cubre ingeniería de prompts, análisis de datos e integración con otras herramientas
- [Ver informe completo](examples/how_to_use_claude_deep_research.md)
7. **Adopción de IA en Salud: Factores de Influencia** - Análisis de factores que impulsan la adopción de IA en salud
- Discute tecnologías IA, calidad de datos, consideraciones éticas, evaluaciones económicas, preparación organizativa e infraestructura digital
- [Ver informe completo](examples/AI_adoption_in_healthcare.md)
8. **Impacto de la Computación Cuántica en la Criptografía** - Análisis del impacto de la computación cuántica en la criptografía
- Discute vulnerabilidades de la criptografía clásica, criptografía post-cuántica y soluciones criptográficas resistentes a la cuántica
- [Ver informe completo](examples/Quantum_Computing_Impact_on_Cryptography.md)
9. **Aspectos Destacados del Rendimiento de Cristiano Ronaldo** - Análisis de los aspectos destacados del rendimiento de Cristiano Ronaldo
- Discute sus logros profesionales, goles internacionales y rendimiento en varios partidos
- [Ver informe completo](examples/Cristiano_Ronaldo's_Performance_Highlights.md)
Para ejecutar estos ejemplos o crear tus propios informes de investigación, puedes usar los siguientes comandos:
```bash
# Ejecutar con una consulta específica
uv run main.py "¿Qué factores están influyendo en la adopción de IA en salud?"
# Ejecutar con parámetros de planificación personalizados
uv run main.py --max_plan_iterations 3 "¿Cómo impacta la computación cuántica en la criptografía?"
# Ejecutar en modo interactivo con preguntas integradas
uv run main.py --interactive
# O ejecutar con prompt interactivo básico
uv run main.py
# Ver todas las opciones disponibles
uv run main.py --help
```
### Modo Interactivo
La aplicación ahora soporta un modo interactivo con preguntas integradas tanto en inglés como en chino:
1. Lanza el modo interactivo:
```bash
uv run main.py --interactive
```
2. Selecciona tu idioma preferido (English o 中文)
3. Elige de una lista de preguntas integradas o selecciona la opción para hacer tu propia pregunta
4. El sistema procesará tu pregunta y generará un informe de investigación completo
### Humano en el Bucle
DeerFlow incluye un mecanismo de humano en el bucle que te permite revisar, editar y aprobar planes de investigación antes de que sean ejecutados:
1. **Revisión del Plan**: Cuando el humano en el bucle está habilitado, el sistema presentará el plan de investigación generado para tu revisión antes de la ejecución
2. **Proporcionando Retroalimentación**: Puedes:
- Aceptar el plan respondiendo con `[ACCEPTED]`
- Editar el plan proporcionando retroalimentación (p.ej., `[EDIT PLAN] Añadir más pasos sobre implementación técnica`)
- El sistema incorporará tu retroalimentación y generará un plan revisado
3. **Auto-aceptación**: Puedes habilitar la auto-aceptación para omitir el proceso de revisión:
- Vía API: Establece `auto_accepted_plan: true` en tu solicitud
4. **Integración API**: Cuando uses la API, puedes proporcionar retroalimentación a través del parámetro `feedback`:
```json
{
"messages": [{ "role": "user", "content": "¿Qué es la computación cuántica?" }],
"thread_id": "my_thread_id",
"auto_accepted_plan": false,
"feedback": "[EDIT PLAN] Incluir más sobre algoritmos cuánticos"
}
```
### Argumentos de Línea de Comandos
La aplicación soporta varios argumentos de línea de comandos para personalizar su comportamiento:
- **query**: La consulta de investigación a procesar (puede ser múltiples palabras)
- **--interactive**: Ejecutar en modo interactivo con preguntas integradas
- **--max_plan_iterations**: Número máximo de ciclos de planificación (predeterminado: 1)
- **--max_step_num**: Número máximo de pasos en un plan de investigación (predeterminado: 3)
- **--debug**: Habilitar registro detallado de depuración
## Preguntas Frecuentes
Por favor, consulta [FAQ.md](docs/FAQ.md) para más detalles.
## Licencia
Este proyecto es de código abierto y está disponible bajo la [Licencia MIT](./LICENSE).
## Agradecimientos
DeerFlow está construido sobre el increíble trabajo de la comunidad de código abierto. Estamos profundamente agradecidos a todos los proyectos y contribuyentes cuyos esfuerzos han hecho posible DeerFlow. Verdaderamente, nos apoyamos en hombros de gigantes.
Nos gustaría extender nuestro sincero agradecimiento a los siguientes proyectos por sus invaluables contribuciones:
- **[LangChain](https://github.com/langchain-ai/langchain)**: Su excepcional marco impulsa nuestras interacciones y cadenas LLM, permitiendo integración y funcionalidad sin problemas.
- **[LangGraph](https://github.com/langchain-ai/langgraph)**: Su enfoque innovador para la orquestación multi-agente ha sido instrumental en permitir los sofisticados flujos de trabajo de DeerFlow.
Estos proyectos ejemplifican el poder transformador de la colaboración de código abierto, y estamos orgullosos de construir sobre sus cimientos.
### Contribuyentes Clave
Un sentido agradecimiento va para los autores principales de `DeerFlow`, cuya visión, pasión y dedicación han dado vida a este proyecto:
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
- **[Henry Li](https://github.com/magiccube/)**
Su compromiso inquebrantable y experiencia han sido la fuerza impulsora detrás del éxito de DeerFlow. Nos sentimos honrados de tenerlos al timón de este viaje.
## Historial de Estrellas
[![Gráfico de Historial de Estrellas](https://api.star-history.com/svg?repos=bytedance/deer-flow&type=Date)](https://star-history.com/#bytedance/deer-flow&Date)
+629
View File
@@ -0,0 +1,629 @@
# 🦌 DeerFlow - 2.0
[English](./README.md) | [中文](./README_zh.md) | [日本語](./README_ja.md) | Français | [Русский](./README_ru.md)
[![Python](https://img.shields.io/badge/Python-3.12%2B-3776AB?logo=python&logoColor=white)](./backend/pyproject.toml)
[![Node.js](https://img.shields.io/badge/Node.js-22%2B-339933?logo=node.js&logoColor=white)](./Makefile)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
> Le 28 février 2026, DeerFlow a décroché la 🏆 1re place sur GitHub Trending suite au lancement de la version 2. Un immense merci à notre incroyable communauté — c'est grâce à vous ! 💪🔥
DeerFlow (**D**eep **E**xploration and **E**fficient **R**esearch **Flow**) est un **super agent harness** open source qui orchestre des **sub-agents**, de la **mémoire** et des **sandboxes** pour accomplir pratiquement n'importe quelle tâche — le tout propulsé par des **skills extensibles**.
https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
> [!NOTE]
> **DeerFlow 2.0 est une réécriture complète.** Il ne partage aucun code avec la v1. Si vous cherchez le framework Deep Research original, il est maintenu sur la [branche `1.x`](https://github.com/bytedance/deer-flow/tree/main-1.x) — les contributions y sont toujours les bienvenues. Le développement actif a migré vers la 2.0.
## Site officiel
[<img width="2880" height="1600" alt="image" src="https://github.com/user-attachments/assets/a598c49f-3b2f-41ea-a052-05e21349188a" />](https://deerflow.tech)
Découvrez-en plus et regardez des **démos réelles** sur notre [**site officiel**](https://deerflow.tech).
## Coding Plan de ByteDance Volcengine
<img width="4808" height="2400" alt="英文方舟" src="https://github.com/user-attachments/assets/2ecc7b9d-50be-4185-b1f7-5542d222fb2d" />
- Nous recommandons fortement d'utiliser Doubao-Seed-2.0-Code, DeepSeek v3.2 et Kimi 2.5 pour exécuter DeerFlow
- [En savoir plus](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
- [Développeurs en Chine continentale, cliquez ici](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
## InfoQuest
DeerFlow intègre désormais le toolkit de recherche et de crawling intelligent développé par BytePlus — [InfoQuest (essai gratuit en ligne)](https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest)
<a href="https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest" target="_blank">
<img
src="https://sf16-sg.tiktokcdn.com/obj/eden-sg/hubseh7bsbps/20251208-160108.png" alt="InfoQuest_banner"
/>
</a>
---
## Table des matières
- [🦌 DeerFlow - 2.0](#-deerflow---20)
- [Site officiel](#site-officiel)
- [InfoQuest](#infoquest)
- [Table des matières](#table-des-matières)
- [Installation en une phrase pour un coding agent](#installation-en-une-phrase-pour-un-coding-agent)
- [Démarrage rapide](#démarrage-rapide)
- [Configuration](#configuration)
- [Lancer l'application](#lancer-lapplication)
- [Option 1 : Docker (recommandé)](#option-1--docker-recommandé)
- [Option 2 : Développement local](#option-2--développement-local)
- [Avancé](#avancé)
- [Mode Sandbox](#mode-sandbox)
- [Serveur MCP](#serveur-mcp)
- [Canaux de messagerie](#canaux-de-messagerie)
- [Traçage LangSmith](#traçage-langsmith)
- [Du Deep Research au Super Agent Harness](#du-deep-research-au-super-agent-harness)
- [Fonctionnalités principales](#fonctionnalités-principales)
- [Skills et outils](#skills-et-outils)
- [Intégration Claude Code](#intégration-claude-code)
- [Sub-Agents](#sub-agents)
- [Sandbox et système de fichiers](#sandbox-et-système-de-fichiers)
- [Context Engineering](#context-engineering)
- [Mémoire à long terme](#mémoire-à-long-terme)
- [Modèles recommandés](#modèles-recommandés)
- [Client Python intégré](#client-python-intégré)
- [Documentation](#documentation)
- [⚠️ Avertissement de sécurité](#-avertissement-de-sécurité)
- [Contribuer](#contribuer)
- [Licence](#licence)
- [Remerciements](#remerciements)
- [Contributeurs principaux](#contributeurs-principaux)
- [Star History](#star-history)
## Installation en une phrase pour un coding agent
Si vous utilisez Claude Code, Codex, Cursor, Windsurf ou un autre coding agent, vous pouvez simplement lui envoyer cette phrase :
```text
Aide-moi à cloner DeerFlow si nécessaire, puis à initialiser son environnement de développement local en suivant https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md
```
Ce prompt est destiné aux coding agents. Il leur demande de cloner le dépôt si nécessaire, de privilégier Docker quand il est disponible, puis de s'arrêter avec la commande exacte pour lancer DeerFlow et la liste des configurations encore manquantes.
## Démarrage rapide
### Configuration
1. **Cloner le dépôt DeerFlow**
```bash
git clone https://github.com/bytedance/deer-flow.git
cd deer-flow
```
2. **Générer les fichiers de configuration locaux**
Depuis le répertoire racine du projet (`deer-flow/`), exécutez :
```bash
make config
```
Cette commande crée les fichiers de configuration locaux à partir des templates fournis.
3. **Configurer le(s) modèle(s) de votre choix**
Éditez `config.yaml` et définissez au moins un modèle :
```yaml
models:
- name: gpt-4 # Internal identifier
display_name: GPT-4 # Human-readable name
use: langchain_openai:ChatOpenAI # LangChain class path
model: gpt-4 # Model identifier for API
api_key: $OPENAI_API_KEY # API key (recommended: use env var)
max_tokens: 4096 # Maximum tokens per request
temperature: 0.7 # Sampling temperature
- name: openrouter-gemini-2.5-flash
display_name: Gemini 2.5 Flash (OpenRouter)
use: langchain_openai:ChatOpenAI
model: google/gemini-2.5-flash-preview
api_key: $OPENAI_API_KEY # OpenRouter still uses the OpenAI-compatible field name here
base_url: https://openrouter.ai/api/v1
- name: gpt-5-responses
display_name: GPT-5 (Responses API)
use: langchain_openai:ChatOpenAI
model: gpt-5
api_key: $OPENAI_API_KEY
use_responses_api: true
output_version: responses/v1
```
OpenRouter et les passerelles compatibles OpenAI similaires doivent être configurés avec `langchain_openai:ChatOpenAI` et `base_url`. Si vous préférez utiliser un nom de variable d'environnement propre au fournisseur, pointez `api_key` vers cette variable explicitement (par exemple `api_key: $OPENROUTER_API_KEY`).
Pour router les modèles OpenAI via `/v1/responses`, continuez d'utiliser `langchain_openai:ChatOpenAI` et définissez `use_responses_api: true` avec `output_version: responses/v1`.
Exemples de providers basés sur un CLI :
```yaml
models:
- name: gpt-5.4
display_name: GPT-5.4 (Codex CLI)
use: deerflow.models.openai_codex_provider:CodexChatModel
model: gpt-5.4
supports_thinking: true
supports_reasoning_effort: true
- name: claude-sonnet-4.6
display_name: Claude Sonnet 4.6 (Claude Code OAuth)
use: deerflow.models.claude_provider:ClaudeChatModel
model: claude-sonnet-4-6
max_tokens: 4096
supports_thinking: true
```
- Codex CLI lit `~/.codex/auth.json`
- L'endpoint Responses de Codex rejette actuellement `max_tokens` et `max_output_tokens`, donc `CodexChatModel` n'expose pas de limite de tokens par requête
- Claude Code accepte `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_AUTH_TOKEN`, `CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR`, `CLAUDE_CODE_CREDENTIALS_PATH`, ou en clair `~/.claude/.credentials.json`
- Sur macOS, DeerFlow ne sonde pas le Keychain automatiquement. Exportez l'auth Claude Code explicitement si nécessaire :
```bash
eval "$(python3 scripts/export_claude_code_oauth.py --print-export)"
```
4. **Définir les clés API pour le(s) modèle(s) configuré(s)**
Choisissez l'une des méthodes suivantes :
- Option A : Éditer le fichier `.env` à la racine du projet (recommandé)
```bash
TAVILY_API_KEY=your-tavily-api-key
OPENAI_API_KEY=your-openai-api-key
# OpenRouter also uses OPENAI_API_KEY when your config uses langchain_openai:ChatOpenAI + base_url.
# Add other provider keys as needed
INFOQUEST_API_KEY=your-infoquest-api-key
```
- Option B : Exporter les variables d'environnement dans votre shell
```bash
export OPENAI_API_KEY=your-openai-api-key
```
Pour les providers basés sur un CLI :
- Codex CLI : `~/.codex/auth.json`
- Claude Code OAuth : handoff explicite via env/fichier ou `~/.claude/.credentials.json`
- Option C : Éditer `config.yaml` directement (non recommandé en production)
```yaml
models:
- name: gpt-4
api_key: your-actual-api-key-here # Replace placeholder
```
### Lancer l'application
#### Option 1 : Docker (recommandé)
**Développement** (hot-reload, montage des sources) :
```bash
make docker-init # Pull sandbox image (only once or when image updates)
make docker-start # Start services (auto-detects sandbox mode from config.yaml)
```
`make docker-start` ne lance `provisioner` que si `config.yaml` utilise le mode provisioner (`sandbox.use: deerflow.community.aio_sandbox:AioSandboxProvider` avec `provisioner_url`).
Les processus backend récupèrent automatiquement les changements dans `config.yaml` au prochain accès à la configuration, donc les mises à jour de métadonnées des modèles ne nécessitent pas de redémarrage manuel en développement.
> [!TIP]
> Sous Linux, si les commandes Docker échouent avec `permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock`, ajoutez votre utilisateur au groupe `docker` et reconnectez-vous avant de réessayer. Voir [CONTRIBUTING.md](CONTRIBUTING.md#linux-docker-daemon-permission-denied) pour la solution complète.
**Production** (build des images en local, montage de la config et des données) :
```bash
make up # Build images and start all production services
make down # Stop and remove containers
```
> [!NOTE]
> Le serveur d'agents LangGraph fonctionne actuellement via `langgraph dev` (le serveur CLI open source).
Accès : http://localhost:2026
Voir [CONTRIBUTING.md](CONTRIBUTING.md) pour le guide complet de développement avec Docker.
#### Option 2 : Développement local
Si vous préférez lancer les services en local :
Prérequis : complétez d'abord les étapes de « Configuration » ci-dessus (`make config` et clés API des modèles). `make dev` nécessite un fichier de configuration valide (par défaut `config.yaml` à la racine du projet ; modifiable via `DEER_FLOW_CONFIG_PATH`).
1. **Vérifier les prérequis** :
```bash
make check # Verifies Node.js 22+, pnpm, uv, nginx
```
2. **Installer les dépendances** :
```bash
make install # Install backend + frontend dependencies
```
3. **(Optionnel) Pré-télécharger l'image sandbox** :
```bash
# Recommended if using Docker/Container-based sandbox
make setup-sandbox
```
4. **Démarrer les services** :
```bash
make dev
```
5. **Accès** : http://localhost:2026
### Avancé
#### Mode Sandbox
DeerFlow supporte plusieurs modes d'exécution sandbox :
- **Exécution locale** (exécute le code sandbox directement sur la machine hôte)
- **Exécution Docker** (exécute le code sandbox dans des conteneurs Docker isolés)
- **Exécution Docker avec Kubernetes** (exécute le code sandbox dans des pods Kubernetes via le service provisioner)
En développement Docker, le démarrage des services suit le mode sandbox défini dans `config.yaml`. En mode Local/Docker, `provisioner` n'est pas démarré.
Voir le [Guide de configuration Sandbox](backend/docs/CONFIGURATION.md#sandbox) pour configurer le mode de votre choix.
#### Serveur MCP
DeerFlow supporte des serveurs MCP et des skills configurables pour étendre ses capacités.
Pour les serveurs MCP HTTP/SSE, les flux de tokens OAuth sont supportés (`client_credentials`, `refresh_token`).
Voir le [Guide MCP Server](backend/docs/MCP_SERVER.md) pour les instructions détaillées.
#### Canaux de messagerie
DeerFlow peut recevoir des tâches depuis des applications de messagerie. Les canaux démarrent automatiquement une fois configurés — aucune IP publique n'est requise.
| Canal | Transport | Difficulté |
|---------|-----------|------------|
| Telegram | Bot API (long-polling) | Facile |
| Slack | Socket Mode | Modérée |
| Feishu / Lark | WebSocket | Modérée |
| DingTalk | Stream Push (WebSocket) | Modérée |
**Configuration dans `config.yaml` :**
```yaml
channels:
# LangGraph Server URL (default: http://localhost:2024)
langgraph_url: http://localhost:2024
# Gateway API URL (default: http://localhost:8001)
gateway_url: http://localhost:8001
# Optional: global session defaults for all mobile channels
session:
assistant_id: lead_agent
config:
recursion_limit: 100
context:
thinking_enabled: true
is_plan_mode: false
subagent_enabled: false
feishu:
enabled: true
app_id: $FEISHU_APP_ID
app_secret: $FEISHU_APP_SECRET
# domain: https://open.feishu.cn # China (default)
# domain: https://open.larksuite.com # International
slack:
enabled: true
bot_token: $SLACK_BOT_TOKEN # xoxb-...
app_token: $SLACK_APP_TOKEN # xapp-... (Socket Mode)
allowed_users: [] # empty = allow all
telegram:
enabled: true
bot_token: $TELEGRAM_BOT_TOKEN
allowed_users: [] # empty = allow all
# Optional: per-channel / per-user session settings
session:
assistant_id: mobile_agent
context:
thinking_enabled: false
users:
"123456789":
assistant_id: vip_agent
config:
recursion_limit: 150
context:
thinking_enabled: true
subagent_enabled: true
dingtalk:
enabled: true
client_id: $DINGTALK_CLIENT_ID # ClientId depuis DingTalk Open Platform
client_secret: $DINGTALK_CLIENT_SECRET # ClientSecret depuis DingTalk Open Platform
allowed_users: [] # vide = tout le monde autorisé
card_template_id: "" # Optionnel : ID de modèle AI Card pour l'effet machine à écrire en streaming
```
Définissez les clés API correspondantes dans votre fichier `.env` :
```bash
# Telegram
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrSTUvwxYZ
# Slack
SLACK_BOT_TOKEN=xoxb-...
SLACK_APP_TOKEN=xapp-...
# Feishu / Lark
FEISHU_APP_ID=cli_xxxx
FEISHU_APP_SECRET=your_app_secret
# DingTalk
DINGTALK_CLIENT_ID=your_client_id
DINGTALK_CLIENT_SECRET=your_client_secret
```
**Configuration Telegram**
1. Ouvrez une conversation avec [@BotFather](https://t.me/BotFather), envoyez `/newbot`, et copiez le token HTTP API.
2. Définissez `TELEGRAM_BOT_TOKEN` dans `.env` et activez le canal dans `config.yaml`.
**Configuration Slack**
1. Créez une Slack App sur [api.slack.com/apps](https://api.slack.com/apps) → Create New App → From scratch.
2. Dans **OAuth & Permissions**, ajoutez les Bot Token Scopes : `app_mentions:read`, `chat:write`, `im:history`, `im:read`, `im:write`, `files:write`.
3. Activez le **Socket Mode** → générez un App-Level Token (`xapp-…`) avec le scope `connections:write`.
4. Dans **Event Subscriptions**, abonnez-vous aux bot events : `app_mention`, `message.im`.
5. Définissez `SLACK_BOT_TOKEN` et `SLACK_APP_TOKEN` dans `.env` et activez le canal dans `config.yaml`.
**Configuration Feishu / Lark**
1. Créez une application sur [Feishu Open Platform](https://open.feishu.cn/) → activez la capacité **Bot**.
2. Ajoutez les permissions : `im:message`, `im:message.p2p_msg:readonly`, `im:resource`.
3. Dans **Events**, abonnez-vous à `im.message.receive_v1` et sélectionnez le mode **Long Connection**.
4. Copiez l'App ID et l'App Secret. Définissez `FEISHU_APP_ID` et `FEISHU_APP_SECRET` dans `.env` et activez le canal dans `config.yaml`.
**Configuration DingTalk**
1. Créez une application sur [DingTalk Open Platform](https://open.dingtalk.com/) et activez la capacité **Robot**.
2. Dans la page de configuration du robot, définissez le mode de réception des messages sur **Stream**.
3. Copiez le `Client ID` et le `Client Secret`. Définissez `DINGTALK_CLIENT_ID` et `DINGTALK_CLIENT_SECRET` dans `.env` et activez le canal dans `config.yaml`.
4. *(Optionnel)* Pour activer les réponses en streaming AI Card (effet machine à écrire), créez un modèle **AI Card** sur la [plateforme de cartes DingTalk](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card), puis définissez `card_template_id` dans `config.yaml` avec l'ID du modèle. Vous devez également demander les permissions `Card.Streaming.Write` et `Card.Instance.Write`.
**Commandes**
Une fois un canal connecté, vous pouvez interagir avec DeerFlow directement depuis le chat :
| Commande | Description |
|---------|-------------|
| `/new` | Démarrer une nouvelle conversation |
| `/status` | Afficher les infos du thread en cours |
| `/models` | Lister les modèles disponibles |
| `/memory` | Consulter la mémoire |
| `/help` | Afficher l'aide |
> Les messages sans préfixe de commande sont traités comme du chat classique — DeerFlow crée un thread et répond de manière conversationnelle.
#### Traçage LangSmith
DeerFlow intègre nativement [LangSmith](https://smith.langchain.com) pour l'observabilité. Une fois activé, tous les appels LLM, les exécutions d'agents et les exécutions d'outils sont tracés et visibles dans le tableau de bord LangSmith.
Ajoutez les lignes suivantes à votre fichier `.env` :
```bash
LANGSMITH_TRACING=true
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
LANGSMITH_PROJECT=xxx
```
Pour les déploiements Docker, le traçage est désactivé par défaut. Définissez `LANGSMITH_TRACING=true` et `LANGSMITH_API_KEY` dans votre `.env` pour l'activer.
## Du Deep Research au Super Agent Harness
DeerFlow a démarré comme un framework de Deep Research — et la communauté s'en est emparée. Depuis le lancement, les développeurs l'ont poussé bien au-delà de la recherche : construction de pipelines de données, génération de présentations, mise en place de dashboards, automatisation de workflows de contenu. Des usages qu'on n'avait jamais anticipés.
Ça nous a révélé quelque chose d'important : DeerFlow n'était pas qu'un simple outil de recherche. C'était un **harness** — un runtime qui donne aux agents l'infrastructure nécessaire pour vraiment accomplir du travail.
On l'a donc reconstruit de zéro.
DeerFlow 2.0 n'est plus un framework à assembler soi-même. C'est un super agent harness — clé en main et entièrement extensible. Construit sur LangGraph et LangChain, il embarque tout ce dont un agent a besoin out of the box : un système de fichiers, de la mémoire, des skills, une exécution sandboxée, et la capacité de planifier et de lancer des sub-agents pour les tâches complexes et multi-étapes.
Utilisez-le tel quel. Ou démontez-le et faites-en le vôtre.
## Fonctionnalités principales
### Skills et outils
Les skills sont ce qui permet à DeerFlow de faire *pratiquement n'importe quoi*.
Un Agent Skill standard est un module de capacité structuré — un fichier Markdown qui définit un workflow, des bonnes pratiques et des références vers des ressources associées. DeerFlow est livré avec des skills intégrés pour la recherche, la génération de rapports, la création de présentations, les pages web, la génération d'images et de vidéos, et bien plus. Mais la vraie force réside dans l'extensibilité : ajoutez vos propres skills, remplacez ceux fournis, ou combinez-les en workflows composites.
Les skills sont chargés progressivement — uniquement quand la tâche le nécessite, pas tous en même temps. Ça permet de garder la fenêtre de contexte légère et de bien fonctionner même avec des modèles sensibles au nombre de tokens.
Quand vous installez des archives `.skill` via le Gateway, DeerFlow accepte les métadonnées frontmatter optionnelles standard comme `version`, `author` et `compatibility`, plutôt que de rejeter des skills externes par ailleurs valides.
Les outils suivent la même philosophie. DeerFlow est livré avec un ensemble d'outils de base — recherche web, fetch de pages web, opérations sur les fichiers, exécution bash — et supporte les outils custom via des serveurs MCP et des fonctions Python. Remplacez n'importe quoi. Ajoutez n'importe quoi.
Les suggestions de suivi générées par le Gateway normalisent désormais aussi bien la sortie texte brut du modèle que le contenu riche au format bloc/liste avant de parser la réponse en tableau JSON, de sorte que les wrappers de contenu propres à chaque provider ne suppriment plus silencieusement les suggestions.
```
# Paths inside the sandbox container
/mnt/skills/public
├── research/SKILL.md
├── report-generation/SKILL.md
├── slide-creation/SKILL.md
├── web-page/SKILL.md
└── image-generation/SKILL.md
/mnt/skills/custom
└── your-custom-skill/SKILL.md ← yours
```
#### Intégration Claude Code
Le skill `claude-to-deerflow` vous permet d'interagir avec une instance DeerFlow en cours d'exécution directement depuis [Claude Code](https://docs.anthropic.com/en/docs/claude-code). Envoyez des tâches de recherche, vérifiez le statut, gérez les threads — le tout sans quitter le terminal.
**Installer le skill** :
```bash
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
```
Assurez-vous ensuite que DeerFlow tourne (par défaut sur `http://localhost:2026`) et utilisez la commande `/claude-to-deerflow` dans Claude Code.
**Ce que vous pouvez faire** :
- Envoyer des messages à DeerFlow et recevoir des réponses en streaming
- Choisir le mode d'exécution : flash (rapide), standard, pro (planification), ultra (sub-agents)
- Vérifier la santé de DeerFlow, lister les modèles/skills/agents
- Gérer les threads et l'historique des conversations
- Upload des fichiers pour analyse
**Variables d'environnement** (optionnel, pour des endpoints custom) :
```bash
DEERFLOW_URL=http://localhost:2026 # Unified proxy base URL
DEERFLOW_GATEWAY_URL=http://localhost:2026 # Gateway API
DEERFLOW_LANGGRAPH_URL=http://localhost:2026/api/langgraph # LangGraph API
```
Voir [`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md) pour la référence API complète.
### Sub-Agents
Les tâches complexes tiennent rarement en une seule passe. DeerFlow les décompose.
L'agent principal peut lancer des sub-agents à la volée — chacun avec son propre contexte délimité, ses outils et ses conditions d'arrêt. Les sub-agents s'exécutent en parallèle quand c'est possible, remontent des résultats structurés, et l'agent principal synthétise le tout en une sortie cohérente.
C'est comme ça que DeerFlow gère les tâches qui prennent de quelques minutes à plusieurs heures : une tâche de recherche peut se déployer en une dizaine de sub-agents, chacun explorant un angle différent, puis converger vers un seul rapport — ou un site web — ou un jeu de slides avec des visuels générés. Un seul harness, de nombreuses mains.
### Sandbox et système de fichiers
DeerFlow ne se contente pas de *parler* de faire les choses. Il dispose de son propre ordinateur.
Chaque tâche s'exécute dans un conteneur Docker isolé avec un système de fichiers complet — skills, workspace, uploads, outputs. L'agent lit, écrit et édite des fichiers. Il exécute des commandes bash et du code. Il visualise des images. Le tout sandboxé, le tout auditable, zéro contamination entre les sessions.
C'est la différence entre un chatbot avec accès à des outils et un agent doté d'un véritable environnement d'exécution.
```
# Paths inside the sandbox container
/mnt/user-data/
├── uploads/ ← your files
├── workspace/ ← agents' working directory
└── outputs/ ← final deliverables
```
### Context Engineering
**Contexte isolé des Sub-Agents** : chaque sub-agent s'exécute dans son propre contexte isolé. Il ne peut voir ni le contexte de l'agent principal, ni celui des autres sub-agents. L'objectif est de garantir que chaque sub-agent reste concentré sur sa tâche sans être parasité par des informations non pertinentes.
**Résumé** : au sein d'une session, DeerFlow gère le contexte de manière agressive — en résumant les sous-tâches terminées, en déchargeant les résultats intermédiaires vers le système de fichiers, en compressant ce qui n'est plus immédiatement pertinent. Ça lui permet de rester efficace sur des tâches longues et multi-étapes sans faire exploser la fenêtre de contexte.
### Mémoire à long terme
La plupart des agents oublient tout dès qu'une conversation se termine. DeerFlow, lui, se souvient.
D'une session à l'autre, DeerFlow construit une mémoire persistante de votre profil, de vos préférences et de vos connaissances accumulées. Plus vous l'utilisez, mieux il vous connaît — votre style d'écriture, votre stack technique, vos workflows récurrents. La mémoire est stockée localement et reste sous votre contrôle.
Les mises à jour de la mémoire ignorent désormais les entrées de faits en double au moment de l'application, de sorte que les préférences et le contexte répétés ne s'accumulent plus indéfiniment entre les sessions.
## Modèles recommandés
DeerFlow est agnostique en termes de modèle — il fonctionne avec n'importe quel LLM implémentant l'API compatible OpenAI. Cela dit, il offre de meilleures performances avec des modèles qui supportent :
- **De longues fenêtres de contexte** (100k+ tokens) pour la recherche approfondie et les tâches multi-étapes
- **Des capacités de raisonnement** pour la planification adaptative et la décomposition de tâches complexes
- **Des entrées multimodales** pour la compréhension d'images et de vidéos
- **Un usage fiable des outils (tool use)** pour des appels de fonctions et des sorties structurées fiables
## Client Python intégré
DeerFlow peut être utilisé comme bibliothèque Python intégrée sans lancer l'ensemble des services HTTP. Le `DeerFlowClient` fournit un accès direct in-process à toutes les capacités d'agent et de Gateway, en retournant les mêmes schémas de réponse que l'API HTTP Gateway. Le HTTP Gateway expose également `DELETE /api/threads/{thread_id}` pour supprimer les données de thread locales gérées par DeerFlow après la suppression du thread LangGraph :
```python
from deerflow.client import DeerFlowClient
client = DeerFlowClient()
# Chat
response = client.chat("Analyze this paper for me", thread_id="my-thread")
# Streaming (LangGraph SSE protocol: values, messages-tuple, end)
for event in client.stream("hello"):
if event.type == "messages-tuple" and event.data.get("type") == "ai":
print(event.data["content"])
# Configuration & management — returns Gateway-aligned dicts
models = client.list_models() # {"models": [...]}
skills = client.list_skills() # {"skills": [...]}
client.update_skill("web-search", enabled=True)
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
```
Toutes les méthodes retournant des dicts sont validées en CI contre les modèles de réponse Pydantic du Gateway (`TestGatewayConformance`), garantissant que le client intégré reste synchronisé avec les schémas de l'API HTTP. Voir `backend/packages/harness/deerflow/client.py` pour la documentation API complète.
## Documentation
- [Guide de contribution](CONTRIBUTING.md) - Mise en place de l'environnement de développement et workflow
- [Guide de configuration](backend/docs/CONFIGURATION.md) - Instructions d'installation et de configuration
- [Vue d'ensemble de l'architecture](backend/CLAUDE.md) - Détails de l'architecture technique
- [Architecture backend](backend/README.md) - Architecture backend et référence API
## ⚠️ Avertissement de sécurité
### Un déploiement inapproprié peut introduire des risques de sécurité
DeerFlow dispose de capacités clés à hauts privilèges, notamment **l'exécution de commandes système, les opérations sur les ressources et l'invocation de logique métier**. Il est conçu par défaut pour être **déployé dans un environnement local de confiance (accessible uniquement via l'interface de loopback 127.0.0.1)**. Si vous déployez l'agent dans des environnements non fiables — tels que des réseaux LAN, des serveurs cloud publics ou d'autres environnements accessibles depuis plusieurs terminaux — sans mesures de sécurité strictes, cela peut introduire des risques, notamment :
- **Invocation non autorisée** : les fonctionnalités de l'agent pourraient être découvertes par des tiers non autorisés ou des scanners malveillants, déclenchant des requêtes non autorisées en masse qui exécutent des opérations à haut risque (commandes système, lecture/écriture de fichiers), pouvant causer de graves conséquences.
- **Risques juridiques et de conformité** : si l'agent est utilisé illégalement pour mener des cyberattaques, du vol de données ou d'autres activités illicites, cela peut entraîner des responsabilités juridiques et des risques de conformité.
### Recommandations de sécurité
**Note : nous recommandons fortement de déployer DeerFlow dans un environnement réseau local de confiance.** Si vous avez besoin d'un déploiement multi-appareils ou multi-réseaux, vous devez mettre en place des mesures de sécurité strictes, par exemple :
- **Liste blanche d'IP** : utilisez `iptables`, ou déployez des pare-feux matériels / commutateurs avec ACL, pour **configurer des règles de liste blanche d'IP** et refuser l'accès à toutes les autres adresses IP.
- **Passerelle d'authentification** : configurez un proxy inverse (ex. nginx) et **activez une authentification forte en amont**, bloquant tout accès non authentifié.
- **Isolation réseau** : si possible, placez l'agent et les appareils de confiance dans le **même VLAN dédié**, isolé des autres équipements réseau.
- **Restez informé** : continuez à suivre les mises à jour de sécurité du projet DeerFlow.
## Contribuer
Les contributions sont les bienvenues ! Consultez [CONTRIBUTING.md](CONTRIBUTING.md) pour la mise en place de l'environnement de développement, le workflow et les conventions.
La couverture de tests de régression inclut la détection du mode sandbox Docker et les tests de gestion du kubeconfig-path du provisioner dans `backend/tests/`.
## Licence
Ce projet est open source et disponible sous la [Licence MIT](./LICENSE).
## Remerciements
DeerFlow est construit sur le travail remarquable de la communauté open source. Nous sommes profondément reconnaissants envers tous les projets et contributeurs dont les efforts ont rendu DeerFlow possible. Nous nous tenons véritablement sur les épaules de géants.
Nous tenons à exprimer notre sincère gratitude aux projets suivants pour leurs contributions inestimables :
- **[LangChain](https://github.com/langchain-ai/langchain)** : leur excellent framework propulse nos interactions LLM et nos chaînes, permettant une intégration et des fonctionnalités fluides.
- **[LangGraph](https://github.com/langchain-ai/langgraph)** : leur approche innovante de l'orchestration multi-agents a été déterminante pour les workflows sophistiqués de DeerFlow.
Ces projets illustrent le pouvoir transformateur de la collaboration open source, et nous sommes fiers de bâtir sur leurs fondations.
### Contributeurs principaux
Un grand merci aux auteurs principaux de `DeerFlow`, dont la vision, la passion et le dévouement ont donné vie à ce projet :
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
- **[Henry Li](https://github.com/magiccube/)**
Votre engagement sans faille et votre expertise sont le moteur du succès de DeerFlow. Nous sommes honorés de vous avoir à la barre de cette aventure.
## Star History
[![Star History Chart](https://api.star-history.com/svg?repos=bytedance/deer-flow&type=Date)](https://star-history.com/#bytedance/deer-flow&Date)
+531 -502
View File
File diff suppressed because it is too large Load Diff
-545
View File
@@ -1,545 +0,0 @@
# 🦌 DeerFlow
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![DeepWiki](https://img.shields.io/badge/DeepWiki-bytedance%2Fdeer--flow-blue.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACwAAAAyCAYAAAAnWDnqAAAAAXNSR0IArs4c6QAAA05JREFUaEPtmUtyEzEQhtWTQyQLHNak2AB7ZnyXZMEjXMGeK/AIi+QuHrMnbChYY7MIh8g01fJoopFb0uhhEqqcbWTp06/uv1saEDv4O3n3dV60RfP947Mm9/SQc0ICFQgzfc4CYZoTPAswgSJCCUJUnAAoRHOAUOcATwbmVLWdGoH//PB8mnKqScAhsD0kYP3j/Yt5LPQe2KvcXmGvRHcDnpxfL2zOYJ1mFwrryWTz0advv1Ut4CJgf5uhDuDj5eUcAUoahrdY/56ebRWeraTjMt/00Sh3UDtjgHtQNHwcRGOC98BJEAEymycmYcWwOprTgcB6VZ5JK5TAJ+fXGLBm3FDAmn6oPPjR4rKCAoJCal2eAiQp2x0vxTPB3ALO2CRkwmDy5WohzBDwSEFKRwPbknEggCPB/imwrycgxX2NzoMCHhPkDwqYMr9tRcP5qNrMZHkVnOjRMWwLCcr8ohBVb1OMjxLwGCvjTikrsBOiA6fNyCrm8V1rP93iVPpwaE+gO0SsWmPiXB+jikdf6SizrT5qKasx5j8ABbHpFTx+vFXp9EnYQmLx02h1QTTrl6eDqxLnGjporxl3NL3agEvXdT0WmEost648sQOYAeJS9Q7bfUVoMGnjo4AZdUMQku50McDcMWcBPvr0SzbTAFDfvJqwLzgxwATnCgnp4wDl6Aa+Ax283gghmj+vj7feE2KBBRMW3FzOpLOADl0Isb5587h/U4gGvkt5v60Z1VLG8BhYjbzRwyQZemwAd6cCR5/XFWLYZRIMpX39AR0tjaGGiGzLVyhse5C9RKC6ai42ppWPKiBagOvaYk8lO7DajerabOZP46Lby5wKjw1HCRx7p9sVMOWGzb/vA1hwiWc6jm3MvQDTogQkiqIhJV0nBQBTU+3okKCFDy9WwferkHjtxib7t3xIUQtHxnIwtx4mpg26/HfwVNVDb4oI9RHmx5WGelRVlrtiw43zboCLaxv46AZeB3IlTkwouebTr1y2NjSpHz68WNFjHvupy3q8TFn3Hos2IAk4Ju5dCo8B3wP7VPr/FGaKiG+T+v+TQqIrOqMTL1VdWV1DdmcbO8KXBz6esmYWYKPwDL5b5FA1a0hwapHiom0r/cKaoqr+27/XcrS5UwSMbQAAAABJRU5ErkJggg==)](https://deepwiki.com/bytedance/deer-flow)
<!-- DeepWiki badge generated by https://deepwiki.ryoppippi.com/ -->
[English](./README.md) | [简体中文](./README_zh.md) | [日本語](./README_ja.md) | [Deutsch](./README_de.md) | [Español](./README_es.md) | [Русский](./README_ru.md) | [Portuguese](./README_pt.md)
> Originado do Open Source, de volta ao Open Source
**DeerFlow** (**D**eep **E**xploration and **E**fficient **R**esearch **Flow**) é um framework de Pesquisa Profunda orientado-a-comunidade que baseia-se em um íncrivel trabalho da comunidade open source. Nosso objetivo é combinar modelos de linguagem com ferramentas especializadas para tarefas como busca na web, crawling, e execução de código Python, enquanto retribui com a comunidade que o tornou possível.
Por favor, visite [Nosso Site Oficial](https://deerflow.tech/) para maiores detalhes.
## Demo
### Video
https://github.com/user-attachments/assets/f3786598-1f2a-4d07-919e-8b99dfa1de3e
Nesse demo, nós demonstramos como usar o DeerFlow para:
In this demo, we showcase how to use DeerFlow to:
- Integração fácil com serviços MCP
- Conduzir o processo de Pesquisa Profunda e produzir um relatório abrangente com imagens
- Criar um áudio podcast baseado no relatório gerado
### Replays
- [Quão alta é a Torre Eiffel comparada ao prédio mais alto?](https://deerflow.tech/chat?replay=eiffel-tower-vs-tallest-building)
- [Quais são os top repositórios tendência no GitHub?](https://deerflow.tech/chat?replay=github-top-trending-repo)
- [Escreva um artigo sobre os pratos tradicionais de Nanjing's](https://deerflow.tech/chat?replay=nanjing-traditional-dishes)
- [Como decorar um apartamento alugado?](https://deerflow.tech/chat?replay=rental-apartment-decoration)
- [Visite nosso site oficial para explorar mais replays.](https://deerflow.tech/#case-studies)
---
## 📑 Tabela de Conteúdos
- [🚀 Início Rápido](#Início-Rápido)
- [🌟 Funcionalidades](#funcionalidades)
- [🏗️ Arquitetura](#arquitetura)
- [🛠️ Desenvolvimento](#desenvolvimento)
- [🐳 Docker](#docker)
- [🗣️ Texto-para-fala Integração](#texto-para-fala-integração)
- [📚 Exemplos](#exemplos)
- [❓ FAQ](#faq)
- [📜 Licença](#licença)
- [💖 Agradecimentos](#agradecimentos)
- [🏆 Contribuidores-Chave](#contribuidores-chave)
- [⭐ Histórico de Estrelas](#Histórico-Estrelas)
## Início-Rápido
DeerFlow é desenvolvido em Python, e vem com uma IU web escrita em Node.js. Para garantir um processo de configuração fácil, nós recomendamos o uso das seguintes ferramentas:
### Ferramentas Recomendadas
- **[`uv`](https://docs.astral.sh/uv/getting-started/installation/):**
Simplifica o gerenciamento de dependência de ambientes Python. `uv` automaticamente cria um ambiente virtual no diretório raiz e instala todos os pacotes necessários para não haver a necessidade de instalar ambientes Python manualmente
- **[`nvm`](https://github.com/nvm-sh/nvm):**
Gerencia múltiplas versões do ambiente de execução do Node.js sem esforço.
- **[`pnpm`](https://pnpm.io/installation):**
Instala e gerencia dependências do projeto Node.js.
### Requisitos de Ambiente
Certifique-se de que seu sistema atenda os seguintes requisitos mínimos:
- **[Python](https://www.python.org/downloads/):** Versão `3.12+`
- **[Node.js](https://nodejs.org/en/download/):** Versão `22+`
### Instalação
```bash
# Clone o repositório
git clone https://github.com/bytedance/deer-flow.git
cd deer-flow
# Instale as dependências, uv irá lidar com o interpretador do python e a criação do venv, e instalar os pacotes necessários
uv sync
# Configure .env com suas chaves de API
# Tavily: https://app.tavily.com/home
# Brave_SEARCH: https://brave.com/search/api/
# volcengine TTS: Adicione sua credencial TTS caso você a possua
cp .env.example .env
# Veja as seções abaixo 'Supported Search Engines' and 'Texto-para-Fala Integração' para todas as opções disponíveis
# Configure o conf.yaml para o seu modelo LLM e chaves API
# Por favor, consulte 'docs/configuration_guide.md' para maiores detalhes
cp conf.yaml.example conf.yaml
# Instale marp para geração de ppt
# https://github.com/marp-team/marp-cli?tab=readme-ov-file#use-package-manager
brew install marp-cli
```
Opcionalmente, instale as dependências IU web via [pnpm](https://pnpm.io/installation):
```bash
cd deer-flow/web
pnpm install
```
### Configurações
Por favor, consulte o [Guia de Configuração](docs/configuration_guide.md) para maiores detalhes.
> [!NOTA]
> Antes de iniciar o projeto, leia o guia detalhadamente, e atualize as configurações para baterem com os seus requisitos e configurações específicas.
### Console IU
A maneira mais rápida de rodar o projeto é usar o console IU.
```bash
# Execute o projeto em um shell tipo-bash
uv run main.py
```
### Web IU
Esse projeto também inclui uma IU Web, trazendo uma experiência mais interativa, dinâmica e engajadora.
> [!NOTA]
> Você precisa instalar as dependências do IU web primeiro.
```bash
# Execute ambos os servidores de backend e frontend em modo desenvolvimento
# No macOS/Linux
./bootstrap.sh -d
# No Windows
bootstrap.bat -d
```
Abra seu navegador e visite [`http://localhost:3000`](http://localhost:3000) para explorar a IU web.
Explore mais detalhes no diretório [`web`](./web/) .
## Mecanismos de Busca Suportados
DeerFlow suporta múltiplos mecanismos de busca que podem ser configurados no seu arquivo `.env` usando a variável `SEARCH_API`:
- **Tavily** (padrão): Uma API de busca especializada para aplicações de IA
- Requer `TAVILY_API_KEY` no seu arquivo `.env`
- Inscreva-se em: https://app.tavily.com/home
- **DuckDuckGo**: Mecanismo de busca focado em privacidade
- Não requer chave API
- **Brave Search**: Mecanismo de busca focado em privacidade com funcionalidades avançadas
- Requer `BRAVE_SEARCH_API_KEY` no seu arquivo `.env`
- Inscreva-se em: https://brave.com/search/api/
- **Arxiv**: Busca de artigos científicos para pesquisa acadêmica
- Não requer chave API
- Especializado em artigos científicos e acadêmicos
Para configurar o seu mecanismo preferido, defina a variável `SEARCH_API` no seu arquivo:
```bash
# Escolha uma: tavily, duckduckgo, brave_search, arxiv
SEARCH_API=tavily
```
## Funcionalidades
### Principais Funcionalidades
- 🤖 **Integração LLM**
- Suporta a integração da maioria dos modelos através de [litellm](https://docs.litellm.ai/docs/providers).
- Suporte a modelos open source como Qwen
- Interface API compatível com a OpenAI
- Sistema LLM multicamadas para diferentes complexidades de tarefa
### Ferramentas e Integrações MCP
- 🔍 **Busca e Recuperação**
- Busca web com Tavily, Brave Search e mais
- Crawling com Jina
- Extração de Conteúdo avançada
- 🔗 **Integração MCP perfeita**
- Expansão de capacidades de acesso para acesso a domínios privados, grafo de conhecimento, navegação web e mais
- Integração facilitdade de diversas ferramentas de pesquisa e metodologias
### Colaboração Humana
- 🧠 **Humano-no-processo**
- Suporta modificação interativa de planos de pesquisa usando linguagem natural
- Suporta auto-aceite de planos de pesquisa
- 📝 **Relatório Pós-Edição**
- Suporta edição de edição de blocos estilo Notion
- Permite refinamentos de IA, incluindo polimento de IA assistida, encurtamento de frase, e expansão
- Distribuído por [tiptap](https://tiptap.dev/)
### Criação de Conteúdo
- 🎙️ **Geração de Podcast e apresentação**
- Script de geração de podcast e síntese de áudio movido por IA
- Criação automatizada de apresentações PowerPoint simples
- Templates customizáveis para conteúdo personalizado
## Arquitetura
DeerFlow implementa uma arquitetura de sistema multi-agente modular designada para pesquisa e análise de código automatizada. O sistema é construído em LangGraph, possibilitando um fluxo de trabalho flexível baseado-em-estado onde os componentes se comunicam através de um sistema de transmissão de mensagens bem-definido.
![Diagrama de Arquitetura](./assets/architecture.png)
> Veja ao vivo em [deerflow.tech](https://deerflow.tech/#multi-agent-architecture)
O sistema emprega um fluxo de trabalho simplificado com os seguintes componentes:
1. **Coordenador**: O ponto de entrada que gerencia o ciclo de vida do fluxo de trabalho
- Inicia o processo de pesquisa baseado na entrada do usuário
- Delega tarefas so planejador quando apropriado
- Atua como a interface primária entre o usuário e o sistema
2. **Planejador**: Componente estratégico para a decomposição e planejamento
- Analisa objetivos de pesquisa e cria planos de execução estruturados
- Determina se há contexto suficiente disponível ou se mais pesquisa é necessária
- Gerencia o fluxo de pesquisa e decide quando gerar o relatório final
3. **Time de Pesquisa**: Uma coleção de agentes especializados que executam o plano:
- **Pesquisador**: Conduz buscas web e coleta informações utilizando ferramentas como mecanismos de busca web, crawling e mesmo serviços MCP.
- **Programador**: Lida com a análise de código, execução e tarefas técnicas como usar a ferramenta Python REPL.
Cada agente tem acesso à ferramentas específicas otimizadas para seu papel e opera dentro do fluxo de trabalho LangGraph.
4. **Repórter**: Estágio final do processador de estágio para saídas de pesquisa
- Resultados agregados do time de pesquisa
- Processa e estrutura as informações coletadas
- Gera relatórios abrangentes de pesquisas
## Texto-para-Fala Integração
DeerFlow agora inclui uma funcionalidade Texto-para-Fala (TTS) que permite que você converta relatórios de busca para voz. Essa funcionalidade usa o mecanismo de voz da API TTS para gerar áudio de alta qualidade a partir do texto. Funcionalidades como velocidade, volume e tom também são customizáveis.
### Usando a API TTS
Você pode acessar a funcionalidade TTS através do endpoint `/api/tts`:
```bash
# Exemplo de chamada da API usando curl
curl --location 'http://localhost:8000/api/tts' \
--header 'Content-Type: application/json' \
--data '{
"text": "This is a test of the text-to-speech functionality.",
"speed_ratio": 1.0,
"volume_ratio": 1.0,
"pitch_ratio": 1.0
}' \
--output speech.mp3
```
## Desenvolvimento
### Testando
Rode o conjunto de testes:
```bash
# Roda todos os testes
make test
# Roda um arquivo de teste específico
pytest tests/integration/test_workflow.py
# Roda com coverage
make coverage
```
### Qualidade de Código
```bash
# Roda o linting
make lint
# Formata de código
make format
```
### Debugando com o LangGraph Studio
DeerFlow usa LangGraph para sua arquitetura de fluxo de trabalho. Nós podemos usar o LangGraph Studio para debugar e visualizar o fluxo de trabalho em tempo real.
#### Rodando o LangGraph Studio Localmente
DeerFlow inclui um arquivo de configuração `langgraph.json` que define a estrutura do grafo e dependências para o LangGraph Studio. Esse arquivo aponta para o grafo do fluxo de trabalho definido no projeto e automaticamente carrega as variáveis de ambiente do arquivo `.env`.
##### Mac
```bash
# Instala o gerenciador de pacote uv caso você não o possua
curl -LsSf https://astral.sh/uv/install.sh | sh
# Instala as dependências e inicia o servidor LangGraph
uvx --refresh --from "langgraph-cli[inmem]" --with-editable . --python 3.12 langgraph dev --allow-blocking
```
##### Windows / Linux
```bash
# Instala as dependências
pip install -e .
pip install -U "langgraph-cli[inmem]"
# Inicia o servidor LangGraph
langgraph dev
```
Após iniciar o servidor LangGraph, você verá diversas URLs no seu terminal:
- API: http://127.0.0.1:2024
- Studio UI: https://smith.langchain.com/studio/?baseUrl=http://127.0.0.1:2024
- API Docs: http://127.0.0.1:2024/docs
Abra o link do Studio UI no seu navegador para acessar a interface de depuração.
#### Usando o LangGraph Studio
No Studio UI, você pode:
1. Visualizar o grafo do fluxo de trabalho e como seus componentes se conectam
2. Rastrear a execução em tempo-real e ver como os dados fluem através do sistema
3. Inspecionar o estado de cada passo do fluxo de trabalho
4. Depurar problemas ao examinar entradas e saídas de cada componente
5. Coletar feedback durante a fase de planejamento para refinar os planos de pesquisa
Quando você envia um tópico de pesquisa ao Studio UI, você será capaz de ver toda a execução do fluxo de trabalho, incluindo:
- A fase de planejamento onde o plano de pesquisa foi criado
- O processo de feedback onde você pode modificar o plano
- As fases de pesquisa e escrita de cada seção
- A geração do relatório final
## Docker
Você também pode executar esse projeto via Docker.
Primeiro, voce deve ler a [configuração](#configuration) below. Make sure `.env`, `.conf.yaml` files are ready.
Segundo, para fazer o build de sua imagem docker em seu próprio servidor:
```bash
docker build -t deer-flow-api .
```
E por fim, inicie um container docker rodando o servidor web:
```bash
# substitua deer-flow-api-app com seu nome de container preferido
docker run -d -t -p 8000:8000 --env-file .env --name deer-flow-api-app deer-flow-api
# pare o servidor
docker stop deer-flow-api-app
```
### Docker Compose (inclui ambos backend e frontend)
DeerFlow fornece uma estrutura docker-compose para facilmente executar ambos o backend e frontend juntos:
```bash
# building docker image
docker compose build
# start the server
docker compose up
```
## Exemplos:
Os seguintes exemplos demonstram as capacidades do DeerFlow:
### Relatórios de Pesquisa
1. **Relatório OpenAI Sora** - Análise da ferramenta Sora da OpenAI
- Discute funcionalidades, acesso, engenharia de prompt, limitações e considerações éticas
- [Veja o relatório completo](examples/openai_sora_report.md)
2. **Relatório Protocolo Agent-to-Agent do Google** - Visão geral do protocolo Agent-to-Agent (A2A) do Google
- Discute o seu papel na comunicação de Agente de IA e seu relacionamento com o Protocolo de Contexto de Modelo ( MCP ) da Anthropic
- [Veja o relatório completo](examples/what_is_agent_to_agent_protocol.md)
3. **O que é MCP?** - Uma análise abrangente to termo "MCP" através de múltiplos contextos
- Explora o Protocolo de Contexto de Modelo em IA, Fosfato Monocálcio em Química, e placa de microcanal em eletrônica
- [Veja o relatório completo](examples/what_is_mcp.md)
4. **Bitcoin Price Fluctuations** - Análise das recentes movimentações de preço do Bitcoin
- Examina tendências de mercado, influências regulatórias, e indicadores técnicos
- Fornece recomendações baseadas nos dados históricos
- [Veja o relatório completo](examples/bitcoin_price_fluctuation.md)
5. **O que é LLM?** - Uma exploração em profundidade de Large Language Models
- Discute arquitetura, treinamento, aplicações, e considerações éticas
- [Veja o relatório completo](examples/what_is_llm.md)
6. **Como usar Claude para Pesquisa Aprofundada?** - Melhores práticas e fluxos de trabalho para usar Claude em pesquisa aprofundada
- Cobre engenharia de prompt, análise de dados, e integração com outras ferramentas
- [Veja o relatório completo](examples/how_to_use_claude_deep_research.md)
7. **Adoção de IA na Área da Saúde: Fatores de Influência** - Análise dos fatores que levam à adoção de IA na área da saúde
- Discute tecnologias de IA, qualidade de dados, considerações éticas, avaliações econômicas, prontidão organizacional, e infraestrutura digital
- [Veja o relatório completo](examples/AI_adoption_in_healthcare.md)
8. **Impacto da Computação Quântica em Criptografia** - Análise dos impactos da computação quântica em criptografia
- Discture vulnerabilidades da criptografia clássica, criptografia pós-quântica, e soluções criptográficas de resistência-quântica
- [Veja o relatório completo](examples/Quantum_Computing_Impact_on_Cryptography.md)
9. **Destaques da Performance do Cristiano Ronaldo** - Análise dos destaques da performance do Cristiano Ronaldo
- Discute as suas conquistas de carreira, objetivos internacionais, e performance em diversas partidas
- [Veja o relatório completo](examples/Cristiano_Ronaldo's_Performance_Highlights.md)
Para executar esses exemplos ou criar seus próprios relatórios de pesquisa, você deve utilizar os seguintes comandos:
```bash
# Executa com uma consulta específica
uv run main.py "Quais fatores estão influenciando a adoção de IA na área da saúde?"
# Executa com parâmetros de planejamento customizados
uv run main.py --max_plan_iterations 3 "Como a computação quântica impacta na criptografia?"
# Executa em modo interativo com questões embutidas
uv run main.py --interactive
# Ou executa com um prompt interativo básico
uv run main.py
# Vê todas as opções disponíveis
uv run main.py --help
```
### Modo Interativo
A aplicação agora suporta um modo interativo com questões embutidas tanto em Inglês quanto Chinês:
1. Inicie o modo interativo:
```bash
uv run main.py --interactive
```
2. Selecione sua linguagem de preferência (English or 中文)
3. Escolha uma das questões embutidas da lista ou selecione a opção para perguntar sua própria questão
4. O sistema irá processar sua questão e gerar um relatório abrangente de pesquisa
### Humano no processo
DeerFlow inclue um mecanismo de humano no processo que permite a você revisar, editar e aprovar planos de pesquisa antes que estes sejam executados:
1. **Revisão de Plano**: Quando o humano no processo está habilitado, o sistema irá apresentar o plano de pesquisa gerado para sua revisão antes da execução
2. **Fornecimento de Feedback**: Você pode:
- Aceitar o plano respondendo com `[ACCEPTED]`
- Edite o plano fornecendo feedback (e.g., `[EDIT PLAN] Adicione mais passos sobre a implementação técnica`)
- O sistema irá incorporar seu feedback e gerar um plano revisado
3. **Auto-aceite**: Você pode habilitar o auto-aceite ou pular o processo de revisão:
- Via API: Defina `auto_accepted_plan: true` na sua requisição
4. **Integração de API**: Quanto usar a API, você pode fornecer um feedback através do parâmetro `feedback`:
```json
{
"messages": [{ "role": "user", "content": "O que é computação quântica?" }],
"thread_id": "my_thread_id",
"auto_accepted_plan": false,
"feedback": "[EDIT PLAN] Inclua mais sobre algoritmos quânticos"
}
```
### Argumentos via Linha de Comando
A aplicação suporta diversos argumentos via linha de comando para customizar o seu comportamento:
- **consulta**: A consulta de pesquisa a ser processada (podem ser múltiplas palavras)
- **--interativo**: Roda no modo interativo com questões embutidas
- **--max_plan_iterations**: Número máximo de ciclos de planejamento (padrão: 1)
- **--max_step_num**: Número máximo de passos em um plano de pesquisa (padrão: 3)
- **--debug**: Habilita Enable um log de depuração detalhado
## FAQ
Por favor consulte a [FAQ.md](docs/FAQ.md) para maiores detalhes.
## Licença
Esse projeto é open source e disponível sob a [MIT License](./LICENSE).
## Agradecimentos
DeerFlow é construído através do incrível trabalho da comunidade open-source. Nós somos profundamente gratos a todos os projetos e contribuidores cujos esforços tornaram o DeerFlow possível. Realmente, nós estamos apoiados nos ombros de gigantes.
Nós gostaríamos de extender nossos sinceros agradecimentos aos seguintes projetos por suas invaloráveis contribuições:
- **[LangChain](https://github.com/langchain-ai/langchain)**: O framework excepcional deles empodera nossas interações via LLM e correntes, permitindo uma integração perfeita e funcional.
- **[LangGraph](https://github.com/langchain-ai/langgraph)**: A abordagem inovativa para orquestração multi-agente deles tem sido foi fundamental em permitir o acesso dos fluxos de trabalho sofisticados do DeerFlow.
Esses projetos exemplificam o poder transformador da colaboração open-source, e nós temos orgulho de construir baseado em suas fundações.
### Contribuidores-Chave
Um sincero muito obrigado vai para os principais autores do `DeerFlow`, cuja visão, paixão, e dedicação trouxe esse projeto à vida:
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
- **[Henry Li](https://github.com/magiccube/)**
O seu compromisso inabalável e experiência tem sido a força por trás do sucesso do DeerFlow. Nós estamos honrados em tê-los no comando dessa trajetória.
## Histórico-Estrelas
[![Gráfico do Histórico de Estrelas](https://api.star-history.com/svg?repos=bytedance/deer-flow&type=Date)](https://star-history.com/#bytedance/deer-flow&Date)
+456 -505
View File
File diff suppressed because it is too large Load Diff
+553 -500
View File
File diff suppressed because it is too large Load Diff
+12
View File
@@ -0,0 +1,12 @@
# Security Policy
## Supported Versions
As deer-flow doesn't provide an official release yet, please use the latest version for the security updates.
Currently, we have two branches to maintain:
* main branch for deer-flow 2.x
* main-1.x branch for deer-flow 1.x
## Reporting a Vulnerability
Please go to https://github.com/bytedance/deer-flow/security to report the vulnerability you find.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 151 KiB

+28
View File
@@ -0,0 +1,28 @@
# Python-generated files
__pycache__/
*.py[oc]
build/
dist/
wheels/
*.egg-info
.coverage
.coverage.*
.ruff_cache
agent_history.gif
static/browser_history/*.gif
log/
log/*
# Virtual environments
.venv
venv/
# User config file
config.yaml
# Langgraph
.langgraph_api
# Claude Code settings
.claude/settings.local.json
+3
View File
@@ -0,0 +1,3 @@
{
"recommendations": ["charliermarsh.ruff"]
}
+11
View File
@@ -0,0 +1,11 @@
{
"window.title": "${activeEditorShort}${separator}${separator}deer-flow/backend",
"[python]": {
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.fixAll": "explicit",
"source.organizeImports": "explicit"
},
"editor.defaultFormatter": "charliermarsh.ruff"
}
}
+2
View File
@@ -0,0 +1,2 @@
For the backend architecture and design patterns:
@./CLAUDE.md
+566
View File
@@ -0,0 +1,566 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
DeerFlow is a LangGraph-based AI super agent system with a full-stack architecture. The backend provides a "super agent" with sandbox execution, persistent memory, subagent delegation, and extensible tool integration - all operating in per-thread isolated environments.
**Architecture**:
- **Gateway API** (port 8001): REST API plus embedded LangGraph-compatible agent runtime
- **Frontend** (port 3000): Next.js web interface
- **Nginx** (port 2026): Unified reverse proxy entry point
- **Provisioner** (port 8002, optional in Docker dev): Started only when sandbox is configured for provisioner/Kubernetes mode
**Runtime**:
- `make dev`, Docker dev, and production all run the agent runtime in Gateway via `RunManager` + `run_agent()` + `StreamBridge` (`packages/harness/deerflow/runtime/`). Nginx exposes that runtime at `/api/langgraph/*` and rewrites it to Gateway's native `/api/*` routers.
**Project Structure**:
```
deer-flow/
├── Makefile # Root commands (check, install, dev, stop)
├── config.yaml # Main application configuration
├── extensions_config.json # MCP servers and skills configuration
├── backend/ # Backend application (this directory)
│ ├── Makefile # Backend-only commands (dev, gateway, lint)
│ ├── langgraph.json # LangGraph Studio graph configuration
│ ├── packages/
│ │ └── harness/ # deerflow-harness package (import: deerflow.*)
│ │ ├── pyproject.toml
│ │ └── deerflow/
│ │ ├── agents/ # LangGraph agent system
│ │ │ ├── lead_agent/ # Main agent (factory + system prompt)
│ │ │ ├── middlewares/ # 10 middleware components
│ │ │ ├── memory/ # Memory extraction, queue, prompts
│ │ │ └── thread_state.py # ThreadState schema
│ │ ├── sandbox/ # Sandbox execution system
│ │ │ ├── local/ # Local filesystem provider
│ │ │ ├── sandbox.py # Abstract Sandbox interface
│ │ │ ├── tools.py # bash, ls, read/write/str_replace
│ │ │ └── middleware.py # Sandbox lifecycle management
│ │ ├── subagents/ # Subagent delegation system
│ │ │ ├── builtins/ # general-purpose, bash agents
│ │ │ ├── executor.py # Background execution engine
│ │ │ └── registry.py # Agent registry
│ │ ├── tools/builtins/ # Built-in tools (present_files, ask_clarification, view_image)
│ │ ├── mcp/ # MCP integration (tools, cache, client)
│ │ ├── models/ # Model factory with thinking/vision support
│ │ ├── skills/ # Skills discovery, loading, parsing
│ │ ├── config/ # Configuration system (app, model, sandbox, tool, etc.)
│ │ ├── community/ # Community tools (tavily, jina_ai, firecrawl, image_search, aio_sandbox)
│ │ ├── reflection/ # Dynamic module loading (resolve_variable, resolve_class)
│ │ ├── utils/ # Utilities (network, readability)
│ │ └── client.py # Embedded Python client (DeerFlowClient)
│ ├── app/ # Application layer (import: app.*)
│ │ ├── gateway/ # FastAPI Gateway API
│ │ │ ├── app.py # FastAPI application
│ │ │ └── routers/ # FastAPI route modules (models, mcp, memory, skills, uploads, threads, artifacts, agents, suggestions, channels)
│ │ └── channels/ # IM platform integrations
│ ├── tests/ # Test suite
│ └── docs/ # Documentation
├── frontend/ # Next.js frontend application
└── skills/ # Agent skills directory
├── public/ # Public skills (committed)
└── custom/ # Custom skills (gitignored)
```
## Important Development Guidelines
### Documentation Update Policy
**CRITICAL: Always update README.md and CLAUDE.md after every code change**
When making code changes, you MUST update the relevant documentation:
- Update `README.md` for user-facing changes (features, setup, usage instructions)
- Update `CLAUDE.md` for development changes (architecture, commands, workflows, internal systems)
- Keep documentation synchronized with the codebase at all times
- Ensure accuracy and timeliness of all documentation
## Commands
**Root directory** (for full application):
```bash
make check # Check system requirements
make install # Install all dependencies (frontend + backend)
make dev # Start all services (Gateway + Frontend + Nginx), with config.yaml preflight
make start # Start production services locally
make stop # Stop all services
```
**Backend directory** (for backend development only):
```bash
make install # Install backend dependencies
make dev # Run Gateway API with reload (port 8001)
make gateway # Run Gateway API only (port 8001)
make test # Run all backend tests
make lint # Lint with ruff
make format # Format code with ruff
```
Regression tests related to Docker/provisioner behavior:
- `tests/test_docker_sandbox_mode_detection.py` (mode detection from `config.yaml`)
- `tests/test_provisioner_kubeconfig.py` (kubeconfig file/directory handling)
Boundary check (harness → app import firewall):
- `tests/test_harness_boundary.py` — ensures `packages/harness/deerflow/` never imports from `app.*`
CI runs these regression tests for every pull request via [.github/workflows/backend-unit-tests.yml](../.github/workflows/backend-unit-tests.yml).
## Architecture
### Harness / App Split
The backend is split into two layers with a strict dependency direction:
- **Harness** (`packages/harness/deerflow/`): Publishable agent framework package (`deerflow-harness`). Import prefix: `deerflow.*`. Contains agent orchestration, tools, sandbox, models, MCP, skills, config — everything needed to build and run agents.
- **App** (`app/`): Unpublished application code. Import prefix: `app.*`. Contains the FastAPI Gateway API and IM channel integrations (Feishu, Slack, Telegram, DingTalk).
**Dependency rule**: App imports deerflow, but deerflow never imports app. This boundary is enforced by `tests/test_harness_boundary.py` which runs in CI.
**Import conventions**:
```python
# Harness internal
from deerflow.agents import make_lead_agent
from deerflow.models import create_chat_model
# App internal
from app.gateway.app import app
from app.channels.service import start_channel_service
# App → Harness (allowed)
from deerflow.config import get_app_config
# Harness → App (FORBIDDEN — enforced by test_harness_boundary.py)
# from app.gateway.routers.uploads import ... # ← will fail CI
```
### Agent System
**Lead Agent** (`packages/harness/deerflow/agents/lead_agent/agent.py`):
- Entry point: `make_lead_agent(config: RunnableConfig)` registered in `langgraph.json`
- Dynamic model selection via `create_chat_model()` with thinking/vision support
- Tools loaded via `get_available_tools()` - combines sandbox, built-in, MCP, community, and subagent tools
- System prompt generated by `apply_prompt_template()` with skills, memory, and subagent instructions
**ThreadState** (`packages/harness/deerflow/agents/thread_state.py`):
- Extends `AgentState` with: `sandbox`, `thread_data`, `title`, `artifacts`, `todos`, `uploaded_files`, `viewed_images`
- Uses custom reducers: `merge_artifacts` (deduplicate), `merge_viewed_images` (merge/clear)
**Runtime Configuration** (via `config.configurable`):
- `thinking_enabled` - Enable model's extended thinking
- `model_name` - Select specific LLM model
- `is_plan_mode` - Enable TodoList middleware
- `subagent_enabled` - Enable task delegation tool
### Middleware Chain
Lead-agent middlewares are assembled in strict append order across `packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py` (`build_lead_runtime_middlewares`) and `packages/harness/deerflow/agents/lead_agent/agent.py` (`_build_middlewares`):
1. **ThreadDataMiddleware** - Creates per-thread directories under the user's isolation scope (`backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}`); resolves `user_id` via `get_effective_user_id()` (falls back to `"default"` in no-auth mode); Web UI thread deletion now follows LangGraph thread removal with Gateway cleanup of the local thread directory
2. **UploadsMiddleware** - Tracks and injects newly uploaded files into conversation
3. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state
4. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., due to user interruption), including raw provider tool-call payloads preserved only in `additional_kwargs["tool_calls"]`
5. **LLMErrorHandlingMiddleware** - Normalizes provider/model invocation failures into recoverable assistant-facing errors before later middleware/tool stages run
6. **GuardrailMiddleware** - Pre-tool-call authorization via pluggable `GuardrailProvider` protocol (optional, if `guardrails.enabled` in config). Evaluates each tool call and returns error ToolMessage on deny. Three provider options: built-in `AllowlistProvider` (zero deps), OAP policy providers (e.g. `aport-agent-guardrails`), or custom providers. See [docs/GUARDRAILS.md](docs/GUARDRAILS.md) for setup, usage, and how to implement a provider.
7. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations for security logging before tool execution continues
8. **ToolErrorHandlingMiddleware** - Converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting
9. **SummarizationMiddleware** - Context reduction when approaching token limits (optional, if enabled)
10. **TodoListMiddleware** - Task tracking with `write_todos` tool (optional, if plan_mode)
11. **TokenUsageMiddleware** - Records token usage metrics when token tracking is enabled (optional)
12. **TitleMiddleware** - Auto-generates thread title after first complete exchange and normalizes structured message content before prompting the title model
13. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses)
14. **ViewImageMiddleware** - Injects base64 image data before LLM call (conditional on vision support)
15. **DeferredToolFilterMiddleware** - Hides deferred tool schemas from the bound model until tool search is enabled (optional)
16. **SubagentLimitMiddleware** - Truncates excess `task` tool calls from model response to enforce `MAX_CONCURRENT_SUBAGENTS` limit (optional, if `subagent_enabled`)
17. **LoopDetectionMiddleware** - Detects repeated tool-call loops; hard-stop responses clear both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer
18. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
### Configuration System
**Main Configuration** (`config.yaml`):
Setup: Copy `config.example.yaml` to `config.yaml` in the **project root** directory.
**Config Versioning**: `config.example.yaml` has a `config_version` field. On startup, `AppConfig.from_file()` compares user version vs example version and emits a warning if outdated. Missing `config_version` = version 0. Run `make config-upgrade` to auto-merge missing fields. When changing the config schema, bump `config_version` in `config.example.yaml`.
**Config Caching**: `get_app_config()` caches the parsed config, but automatically reloads it when the resolved config path changes or the file's mtime increases. This keeps Gateway and LangGraph reads aligned with `config.yaml` edits without requiring a manual process restart.
Configuration priority:
1. Explicit `config_path` argument
2. `DEER_FLOW_CONFIG_PATH` environment variable
3. `config.yaml` in current directory (backend/)
4. `config.yaml` in parent directory (project root - **recommended location**)
Config values starting with `$` are resolved as environment variables (e.g., `$OPENAI_API_KEY`).
`ModelConfig` also declares `use_responses_api` and `output_version` so OpenAI `/v1/responses` can be enabled explicitly while still using `langchain_openai:ChatOpenAI`.
**Extensions Configuration** (`extensions_config.json`):
MCP servers and skills are configured together in `extensions_config.json` in project root:
Configuration priority:
1. Explicit `config_path` argument
2. `DEER_FLOW_EXTENSIONS_CONFIG_PATH` environment variable
3. `extensions_config.json` in current directory (backend/)
4. `extensions_config.json` in parent directory (project root - **recommended location**)
### Gateway API (`app/gateway/`)
FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWAY_ENABLE_DOCS=false` to disable `/docs`, `/redoc`, and `/openapi.json` in production (default: enabled).
**Routers**:
| Router | Endpoints |
|--------|-----------|
| **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details |
| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) |
| **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
| **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types |
| **Suggestions** (`/api/threads/{id}/suggestions`) | `POST /` - generate follow-up questions; rich list/block model content is normalized before JSON parsing |
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /../messages` - thread messages with feedback; `GET /../token-usage` - aggregate tokens |
| **Feedback** (`/api/threads/{id}/runs/{rid}/feedback`) | `PUT /` - upsert feedback; `DELETE /` - delete user feedback; `POST /` - create feedback; `GET /` - list feedback; `GET /stats` - aggregate stats; `DELETE /{fid}` - delete specific |
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` → Gateway.
### Sandbox System (`packages/harness/deerflow/sandbox/`)
**Interface**: Abstract `Sandbox` with `execute_command`, `read_file`, `write_file`, `list_dir`
**Provider Pattern**: `SandboxProvider` with `acquire`, `get`, `release` lifecycle
**Implementations**:
- `LocalSandboxProvider` - Singleton local filesystem execution with path mappings
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation
**Virtual Path System**:
- Agent sees: `/mnt/user-data/{workspace,uploads,outputs}`, `/mnt/skills`
- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`, `deer-flow/skills/`
- Translation: `replace_virtual_path()` / `replace_virtual_paths_in_command()`
- Detection: `is_local_sandbox()` checks `sandbox_id == "local"`
**Sandbox Tools** (in `packages/harness/deerflow/sandbox/tools.py`):
- `bash` - Execute commands with path translation and error handling
- `ls` - Directory listing (tree format, max 2 levels)
- `read_file` - Read file contents with optional line range
- `write_file` - Write/append to files, creates directories
- `str_replace` - Substring replacement (single or all occurrences); same-path serialization is scoped to `(sandbox.id, path)` so isolated sandboxes do not contend on identical virtual paths inside one process
### Subagent System (`packages/harness/deerflow/subagents/`)
**Built-in Agents**: `general-purpose` (all tools except `task`) and `bash` (command specialist)
**Execution**: Dual thread pool - `_scheduler_pool` (3 workers) + `_execution_pool` (3 workers)
**Concurrency**: `MAX_CONCURRENT_SUBAGENTS = 3` enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`), 15-minute timeout
**Flow**: `task()` tool → `SubagentExecutor` → background thread → poll 5s → SSE events → result
**Events**: `task_started`, `task_running`, `task_completed`/`task_failed`/`task_timed_out`
### Tool System (`packages/harness/deerflow/tools/`)
`get_available_tools(groups, include_mcp, model_name, subagent_enabled)` assembles:
1. **Config-defined tools** - Resolved from `config.yaml` via `resolve_variable()`
2. **MCP tools** - From enabled MCP servers (lazy initialized, cached with mtime invalidation)
3. **Built-in tools**:
- `present_files` - Make output files visible to user (only `/mnt/user-data/outputs`)
- `ask_clarification` - Request clarification (intercepted by ClarificationMiddleware → interrupts)
- `view_image` - Read image as base64 (added only if model supports vision)
4. **Subagent tool** (if enabled):
- `task` - Delegate to subagent (description, prompt, subagent_type, max_turns)
**Community tools** (`packages/harness/deerflow/community/`):
- `tavily/` - Web search (5 results default) and web fetch (4KB limit)
- `jina_ai/` - Web fetch via Jina reader API with readability extraction
- `firecrawl/` - Web scraping via Firecrawl API
**ACP agent tools**:
- `invoke_acp_agent` - Invokes external ACP-compatible agents from `config.yaml`
- ACP launchers must be real ACP adapters. The standard `codex` CLI is not ACP-compatible by itself; configure a wrapper such as `npx -y @zed-industries/codex-acp` or an installed `codex-acp` binary
- Missing ACP executables now return an actionable error message instead of a raw `[Errno 2]`
- Each ACP agent uses a per-thread workspace at `{base_dir}/users/{user_id}/threads/{thread_id}/acp-workspace/`. The workspace is accessible to the lead agent via the virtual path `/mnt/acp-workspace/` (read-only). In docker sandbox mode, the directory is volume-mounted into the container at `/mnt/acp-workspace` (read-only); in local sandbox mode, path translation is handled by `tools.py`
- `image_search/` - Image search via DuckDuckGo
### MCP System (`packages/harness/deerflow/mcp/`)
- Uses `langchain-mcp-adapters` `MultiServerMCPClient` for multi-server management
- **Lazy initialization**: Tools loaded on first use via `get_cached_mcp_tools()`
- **Cache invalidation**: Detects config file changes via mtime comparison
- **Transports**: stdio (command-based), SSE, HTTP
- **OAuth (HTTP/SSE)**: Supports token endpoint flows (`client_credentials`, `refresh_token`) with automatic token refresh + Authorization header injection
- **Runtime updates**: Gateway API saves to extensions_config.json; LangGraph detects via mtime
### Skills System (`packages/harness/deerflow/skills/`)
- **Location**: `deer-flow/skills/{public,custom}/`
- **Format**: Directory with `SKILL.md` (YAML frontmatter: name, description, license, allowed-tools)
- **Loading**: `load_skills()` recursively scans `skills/{public,custom}` for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json
- **Injection**: Enabled skills listed in agent system prompt with container paths
- **Installation**: `POST /api/skills/install` extracts .skill ZIP archive to custom/ directory
### Model Factory (`packages/harness/deerflow/models/factory.py`)
- `create_chat_model(name, thinking_enabled)` instantiates LLM from config via reflection
- Supports `thinking_enabled` flag with per-model `when_thinking_enabled` overrides
- Supports vLLM-style thinking toggles via `when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking` for Qwen reasoning models, while normalizing legacy `thinking` configs for backward compatibility
- Supports `supports_vision` flag for image understanding models
- Config values starting with `$` resolved as environment variables
- Missing provider modules surface actionable install hints from reflection resolvers (for example `uv add langchain-google-genai`)
### vLLM Provider (`packages/harness/deerflow/models/vllm_provider.py`)
- `VllmChatModel` subclasses `langchain_openai:ChatOpenAI` for vLLM 0.19.0 OpenAI-compatible endpoints
- Preserves vLLM's non-standard assistant `reasoning` field on full responses, streaming deltas, and follow-up tool-call turns
- Designed for configs that enable thinking through `extra_body.chat_template_kwargs.enable_thinking` on vLLM 0.19.0 Qwen reasoning models, while accepting the older `thinking` alias
### IM Channels System (`app/channels/`)
Bridges external messaging platforms (Feishu, Slack, Telegram, DingTalk) to the DeerFlow agent via the LangGraph Server.
**Architecture**: Channels communicate with Gateway through the `langgraph-sdk` HTTP client (same as the frontend), ensuring threads are created and managed server-side. The internal SDK client injects process-local internal auth plus a matching CSRF cookie/header pair so Gateway accepts state-changing thread/run requests from channel workers without relying on browser session cookies.
**Components**:
- `message_bus.py` - Async pub/sub hub (`InboundMessage` → queue → dispatcher; `OutboundMessage` → callbacks → channels)
- `store.py` - JSON-file persistence mapping `channel_name:chat_id[:topic_id]``thread_id` (keys are `channel:chat` for root conversations and `channel:chat:topic` for threaded conversations)
- `manager.py` - Core dispatcher: creates threads via `client.threads.create()`, routes commands, keeps Slack/Telegram on `client.runs.wait()`, and uses `client.runs.stream(["messages-tuple", "values"])` for Feishu incremental outbound updates
- `base.py` - Abstract `Channel` base class (start/stop/send lifecycle)
- `service.py` - Manages lifecycle of all configured channels from `config.yaml`
- `slack.py` / `feishu.py` / `telegram.py` / `dingtalk.py` - Platform-specific implementations (`feishu.py` tracks the running card `message_id` in memory and patches the same card in place; `dingtalk.py` optionally uses AI Card streaming for in-place updates when `card_template_id` is configured)
**Message Flow**:
1. External platform -> Channel impl -> `MessageBus.publish_inbound()`
2. `ChannelManager._dispatch_loop()` consumes from queue
3. For chat: look up/create thread through Gateway's LangGraph-compatible API
4. Feishu chat: `runs.stream()` → accumulate AI text → publish multiple outbound updates (`is_final=False`) → publish final outbound (`is_final=True`)
5. Slack/Telegram chat: `runs.wait()` → extract final response → publish outbound
6. Feishu channel sends one running reply card up front, then patches the same card for each outbound update (card JSON sets `config.update_multi=true` for Feishu's patch API requirement)
7. DingTalk AI Card mode (when `card_template_id` configured): `runs.stream()` → create card with initial text → stream updates via `PUT /v1.0/card/streaming` → finalize on `is_final=True`. Falls back to `sampleMarkdown` if card creation or streaming fails
8. For commands (`/new`, `/status`, `/models`, `/memory`, `/help`): handle locally or query Gateway API
9. Outbound → channel callbacks → platform reply
**Configuration** (`config.yaml` -> `channels`):
- `langgraph_url` - LangGraph-compatible Gateway API base URL (default: `http://localhost:8001/api`)
- `gateway_url` - Gateway API URL for auxiliary commands (default: `http://localhost:8001`)
- In Docker Compose, IM channels run inside the `gateway` container, so `localhost` points back to that container. Use `http://gateway:8001/api` for `langgraph_url` and `http://gateway:8001` for `gateway_url`, or set `DEER_FLOW_CHANNELS_LANGGRAPH_URL` / `DEER_FLOW_CHANNELS_GATEWAY_URL`.
- Per-channel configs: `feishu` (app_id, app_secret), `slack` (bot_token, app_token), `telegram` (bot_token), `dingtalk` (client_id, client_secret, optional `card_template_id` for AI Card streaming)
### Memory System (`packages/harness/deerflow/agents/memory/`)
**Components**:
- `updater.py` - LLM-based memory updates with fact extraction, whitespace-normalized fact deduplication (trims leading/trailing whitespace before comparing), and atomic file I/O
- `queue.py` - Debounced update queue (per-thread deduplication, configurable wait time); captures `user_id` at enqueue time so it survives the `threading.Timer` boundary
- `prompt.py` - Prompt templates for memory updates
- `storage.py` - File-based storage with per-user isolation; cache keyed by `(user_id, agent_name)` tuple
**Per-User Isolation**:
- Memory is stored per-user at `{base_dir}/users/{user_id}/memory.json`
- Per-agent per-user memory at `{base_dir}/users/{user_id}/agents/{agent_name}/memory.json`
- `user_id` is resolved via `get_effective_user_id()` from `deerflow.runtime.user_context`
- In no-auth mode, `user_id` defaults to `"default"` (constant `DEFAULT_USER_ID`)
- Absolute `storage_path` in config opts out of per-user isolation
- **Migration**: Run `PYTHONPATH=. python scripts/migrate_user_isolation.py` to move legacy `memory.json` and `threads/` into per-user layout; supports `--dry-run`
**Data Structure** (stored in `{base_dir}/users/{user_id}/memory.json`):
- **User Context**: `workContext`, `personalContext`, `topOfMind` (1-3 sentence summaries)
- **History**: `recentMonths`, `earlierContext`, `longTermBackground`
- **Facts**: Discrete facts with `id`, `content`, `category` (preference/knowledge/context/behavior/goal), `confidence` (0-1), `createdAt`, `source`
**Workflow**:
1. `MemoryMiddleware` filters messages (user inputs + final AI responses), captures `user_id` via `get_effective_user_id()`, and queues conversation with the captured `user_id`
2. Queue debounces (30s default), batches updates, deduplicates per-thread
3. Background thread invokes LLM to extract context updates and facts, using the stored `user_id` (not the contextvar, which is unavailable on timer threads)
4. Applies updates atomically (temp file + rename) with cache invalidation, skipping duplicate fact content before append
5. Next interaction injects top 15 facts + context into `<memory>` tags in system prompt
Focused regression coverage for the updater lives in `backend/tests/test_memory_updater.py`.
**Configuration** (`config.yaml``memory`):
- `enabled` / `injection_enabled` - Master switches
- `storage_path` - Path to memory.json (absolute path opts out of per-user isolation)
- `debounce_seconds` - Wait time before processing (default: 30)
- `model_name` - LLM for updates (null = default model)
- `max_facts` / `fact_confidence_threshold` - Fact storage limits (100 / 0.7)
- `max_injection_tokens` - Token limit for prompt injection (2000)
### Reflection System (`packages/harness/deerflow/reflection/`)
- `resolve_variable(path)` - Import module and return variable (e.g., `module.path:variable_name`)
- `resolve_class(path, base_class)` - Import and validate class against base class
### Config Schema
**`config.yaml`** key sections:
- `models[]` - LLM configs with `use` class path, `supports_thinking`, `supports_vision`, provider-specific fields
- vLLM reasoning models should use `deerflow.models.vllm_provider:VllmChatModel`; for Qwen-style parsers prefer `when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking`, and DeerFlow will also normalize the older `thinking` alias
- `tools[]` - Tool configs with `use` variable path and `group`
- `tool_groups[]` - Logical groupings for tools
- `sandbox.use` - Sandbox provider class path
- `skills.path` / `skills.container_path` - Host and container paths to skills directory
- `title` - Auto-title generation (enabled, max_words, max_chars, prompt_template)
- `summarization` - Context summarization (enabled, trigger conditions, keep policy)
- `subagents.enabled` - Master switch for subagent delegation
- `memory` - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens)
**`extensions_config.json`**:
- `mcpServers` - Map of server name → config (enabled, type, command, args, env, url, headers, oauth, description)
- `skills` - Map of skill name → state (enabled)
Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` methods.
### Embedded Client (`packages/harness/deerflow/client.py`)
`DeerFlowClient` provides direct in-process access to all DeerFlow capabilities without HTTP services. All return types align with the Gateway API response schemas, so consumer code works identically in HTTP and embedded modes.
**Architecture**: Imports the same `deerflow` modules that Gateway API uses. Shares the same config files and data directories. No FastAPI dependency.
**Agent Conversation**:
- `chat(message, thread_id)` — synchronous, accumulates streaming deltas per message-id and returns the final AI text
- `stream(message, thread_id)` — subscribes to LangGraph `stream_mode=["values", "messages", "custom"]` and yields `StreamEvent`:
- `"values"` — full state snapshot (title, messages, artifacts); AI text already delivered via `messages` mode is **not** re-synthesized here to avoid duplicate deliveries
- `"messages-tuple"` — per-chunk update: for AI text this is a **delta** (concat per `id` to rebuild the full message); tool calls and tool results are emitted once each
- `"custom"` — forwarded from `StreamWriter`
- `"end"` — stream finished (carries cumulative `usage` counted once per message id)
- Agent created lazily via `create_agent()` + `_build_middlewares()`, same as `make_lead_agent`
- Supports `checkpointer` parameter for state persistence across turns
- `reset_agent()` forces agent recreation (e.g. after memory or skill changes)
- See [docs/STREAMING.md](docs/STREAMING.md) for the full design: why Gateway and DeerFlowClient are parallel paths, LangGraph's `stream_mode` semantics, the per-id dedup invariants, and regression testing strategy
**Gateway Equivalent Methods** (replaces Gateway API):
| Category | Methods | Return format |
|----------|---------|---------------|
| Models | `list_models()`, `get_model(name)` | `{"models": [...]}`, `{name, display_name, ...}` |
| MCP | `get_mcp_config()`, `update_mcp_config(servers)` | `{"mcp_servers": {...}}` |
| Skills | `list_skills()`, `get_skill(name)`, `update_skill(name, enabled)`, `install_skill(path)` | `{"skills": [...]}` |
| Memory | `get_memory()`, `reload_memory()`, `get_memory_config()`, `get_memory_status()` | dict |
| Uploads | `upload_files(thread_id, files)`, `list_uploads(thread_id)`, `delete_upload(thread_id, filename)` | `{"success": true, "files": [...]}`, `{"files": [...], "count": N}` |
| Artifacts | `get_artifact(thread_id, path)``(bytes, mime_type)` | tuple |
**Key difference from Gateway**: Upload accepts local `Path` objects instead of HTTP `UploadFile`, rejects directory paths before copying, and reuses a single worker when document conversion must run inside an active event loop. Artifact returns `(bytes, mime_type)` instead of HTTP Response. The new Gateway-only thread cleanup route deletes `.deer-flow/threads/{thread_id}` after LangGraph thread deletion; there is no matching `DeerFlowClient` method yet. `update_mcp_config()` and `update_skill()` automatically invalidate the cached agent.
**Tests**: `tests/test_client.py` (77 unit tests including `TestGatewayConformance`), `tests/test_client_live.py` (live integration tests, requires config.yaml)
**Gateway Conformance Tests** (`TestGatewayConformance`): Validate that every dict-returning client method conforms to the corresponding Gateway Pydantic response model. Each test parses the client output through the Gateway model — if Gateway adds a required field that the client doesn't provide, Pydantic raises `ValidationError` and CI catches the drift. Covers: `ModelsListResponse`, `ModelResponse`, `SkillsListResponse`, `SkillResponse`, `SkillInstallResponse`, `McpConfigResponse`, `UploadResponse`, `MemoryConfigResponse`, `MemoryStatusResponse`.
## Development Workflow
### Test-Driven Development (TDD) — MANDATORY
**Every new feature or bug fix MUST be accompanied by unit tests. No exceptions.**
- Write tests in `backend/tests/` following the existing naming convention `test_<feature>.py`
- Run the full suite before and after your change: `make test`
- Tests must pass before a feature is considered complete
- For lightweight config/utility modules, prefer pure unit tests with no external dependencies
- If a module causes circular import issues in tests, add a `sys.modules` mock in `tests/conftest.py` (see existing example for `deerflow.subagents.executor`)
```bash
# Run all tests
make test
# Run a specific test file
PYTHONPATH=. uv run pytest tests/test_<feature>.py -v
```
### Running the Full Application
From the **project root** directory:
```bash
make dev
```
This starts all services and makes the application available at `http://localhost:2026`.
**All startup modes:**
| | **Local Foreground** | **Local Daemon** | **Docker Dev** | **Docker Prod** |
|---|---|---|---|---|
| **Dev** | `./scripts/serve.sh --dev`<br/>`make dev` | `./scripts/serve.sh --dev --daemon`<br/>`make dev-daemon` | `./scripts/docker.sh start`<br/>`make docker-start` | — |
| **Prod** | `./scripts/serve.sh --prod`<br/>`make start` | `./scripts/serve.sh --prod --daemon`<br/>`make start-daemon` | — | `./scripts/deploy.sh`<br/>`make up` |
| Action | Local | Docker Dev | Docker Prod |
|---|---|---|---|
| **Stop** | `./scripts/serve.sh --stop`<br/>`make stop` | `./scripts/docker.sh stop`<br/>`make docker-stop` | `./scripts/deploy.sh down`<br/>`make down` |
| **Restart** | `./scripts/serve.sh --restart [flags]` | `./scripts/docker.sh restart` | — |
**Nginx routing**:
- `/api/langgraph/*` → Gateway embedded runtime (8001), rewritten to `/api/*`
- `/api/*` (other) → Gateway API (8001)
- `/` (non-API) → Frontend (3000)
### Running Backend Services Separately
From the **backend** directory:
```bash
# Gateway API
make gateway
```
Direct access (without nginx):
- Gateway: `http://localhost:8001`
### Frontend Configuration
The frontend uses environment variables to connect to backend services:
- `NEXT_PUBLIC_LANGGRAPH_BASE_URL` - Defaults to `/api/langgraph` (through nginx)
- `NEXT_PUBLIC_BACKEND_BASE_URL` - Defaults to empty string (through nginx)
When using `make dev` from root, the frontend automatically connects through nginx.
## Key Features
### File Upload
Multi-file upload with automatic document conversion:
- Endpoint: `POST /api/threads/{thread_id}/uploads`
- Supports: PDF, PPT, Excel, Word documents (converted via `markitdown`)
- Rejects directory inputs before copying so uploads stay all-or-nothing
- Reuses one conversion worker per request when called from an active event loop
- Files stored in thread-isolated directories
- Agent receives uploaded file list via `UploadsMiddleware`
See [docs/FILE_UPLOAD.md](docs/FILE_UPLOAD.md) for details.
### Plan Mode
TodoList middleware for complex multi-step tasks:
- Controlled via runtime config: `config.configurable.is_plan_mode = True`
- Provides `write_todos` tool for task tracking
- One task in_progress at a time, real-time updates
See [docs/plan_mode_usage.md](docs/plan_mode_usage.md) for details.
### Context Summarization
Automatic conversation summarization when approaching token limits:
- Configured in `config.yaml` under `summarization` key
- Trigger types: tokens, messages, or fraction of max input
- Keeps recent messages while summarizing older ones
See [docs/summarization.md](docs/summarization.md) for details.
### Vision Support
For models with `supports_vision: true`:
- `ViewImageMiddleware` processes images in conversation
- `view_image_tool` added to agent's toolset
- Images automatically converted to base64 and injected into state
## Code Style
- Uses `ruff` for linting and formatting
- Line length: 240 characters
- Python 3.12+ with type hints
- Double quotes, space indentation
## Documentation
See `docs/` directory for detailed documentation:
- [CONFIGURATION.md](docs/CONFIGURATION.md) - Configuration options
- [ARCHITECTURE.md](docs/ARCHITECTURE.md) - Architecture details
- [API.md](docs/API.md) - API reference
- [SETUP.md](docs/SETUP.md) - Setup guide
- [FILE_UPLOAD.md](docs/FILE_UPLOAD.md) - File upload feature
- [PATH_EXAMPLES.md](docs/PATH_EXAMPLES.md) - Path types and usage
- [summarization.md](docs/summarization.md) - Context summarization
- [plan_mode_usage.md](docs/plan_mode_usage.md) - Plan mode with TodoList
+426
View File
@@ -0,0 +1,426 @@
# Contributing to DeerFlow Backend
Thank you for your interest in contributing to DeerFlow! This document provides guidelines and instructions for contributing to the backend codebase.
## Table of Contents
- [Getting Started](#getting-started)
- [Development Setup](#development-setup)
- [Project Structure](#project-structure)
- [Code Style](#code-style)
- [Making Changes](#making-changes)
- [Testing](#testing)
- [Pull Request Process](#pull-request-process)
- [Architecture Guidelines](#architecture-guidelines)
## Getting Started
### Prerequisites
- Python 3.12 or higher
- [uv](https://docs.astral.sh/uv/) package manager
- Git
- Docker (optional, for Docker sandbox testing)
### Fork and Clone
1. Fork the repository on GitHub
2. Clone your fork locally:
```bash
git clone https://github.com/YOUR_USERNAME/deer-flow.git
cd deer-flow
```
## Development Setup
### Install Dependencies
```bash
# From project root
cp config.example.yaml config.yaml
# Install backend dependencies
cd backend
make install
```
### Configure Environment
Set up your API keys for testing:
```bash
export OPENAI_API_KEY="your-api-key"
# Add other keys as needed
```
### Run the Development Server
```bash
# Terminal 1: LangGraph server
make dev
# Terminal 2: Gateway API
make gateway
```
## Project Structure
```
backend/src/
├── agents/ # Agent system
│ ├── lead_agent/ # Main agent implementation
│ │ └── agent.py # Agent factory and creation
│ ├── middlewares/ # Agent middlewares
│ │ ├── thread_data_middleware.py
│ │ ├── sandbox_middleware.py
│ │ ├── title_middleware.py
│ │ ├── uploads_middleware.py
│ │ ├── view_image_middleware.py
│ │ └── clarification_middleware.py
│ └── thread_state.py # Thread state definition
├── gateway/ # FastAPI Gateway
│ ├── app.py # FastAPI application
│ └── routers/ # Route handlers
│ ├── models.py # /api/models endpoints
│ ├── mcp.py # /api/mcp endpoints
│ ├── skills.py # /api/skills endpoints
│ ├── artifacts.py # /api/threads/.../artifacts
│ └── uploads.py # /api/threads/.../uploads
├── sandbox/ # Sandbox execution
│ ├── __init__.py # Sandbox interface
│ ├── local.py # Local sandbox provider
│ └── tools.py # Sandbox tools (bash, file ops)
├── tools/ # Agent tools
│ └── builtins/ # Built-in tools
│ ├── present_file_tool.py
│ ├── ask_clarification_tool.py
│ └── view_image_tool.py
├── mcp/ # MCP integration
│ └── manager.py # MCP server management
├── models/ # Model system
│ └── factory.py # Model factory
├── skills/ # Skills system
│ └── loader.py # Skills loader
├── config/ # Configuration
│ ├── app_config.py # Main app config
│ ├── extensions_config.py # Extensions config
│ └── summarization_config.py
├── community/ # Community tools
│ ├── tavily/ # Tavily web search
│ ├── jina/ # Jina web fetch
│ ├── firecrawl/ # Firecrawl scraping
│ └── aio_sandbox/ # Docker sandbox
├── reflection/ # Dynamic loading
│ └── __init__.py # Module resolution
└── utils/ # Utilities
└── __init__.py
```
## Code Style
### Linting and Formatting
We use `ruff` for both linting and formatting:
```bash
# Check for issues
make lint
# Auto-fix and format
make format
```
### Style Guidelines
- **Line length**: 240 characters maximum
- **Python version**: 3.12+ features allowed
- **Type hints**: Use type hints for function signatures
- **Quotes**: Double quotes for strings
- **Indentation**: 4 spaces (no tabs)
- **Imports**: Group by standard library, third-party, local
### Docstrings
Use docstrings for public functions and classes:
```python
def create_chat_model(name: str, thinking_enabled: bool = False) -> BaseChatModel:
"""Create a chat model instance from configuration.
Args:
name: The model name as defined in config.yaml
thinking_enabled: Whether to enable extended thinking
Returns:
A configured LangChain chat model instance
Raises:
ValueError: If the model name is not found in configuration
"""
...
```
## Making Changes
### Branch Naming
Use descriptive branch names:
- `feature/add-new-tool` - New features
- `fix/sandbox-timeout` - Bug fixes
- `docs/update-readme` - Documentation
- `refactor/config-system` - Code refactoring
### Commit Messages
Write clear, concise commit messages:
```
feat: add support for Claude 3.5 model
- Add model configuration in config.yaml
- Update model factory to handle Claude-specific settings
- Add tests for new model
```
Prefix types:
- `feat:` - New feature
- `fix:` - Bug fix
- `docs:` - Documentation
- `refactor:` - Code refactoring
- `test:` - Tests
- `chore:` - Build/config changes
## Testing
### Running Tests
```bash
uv run pytest
```
### Writing Tests
Place tests in the `tests/` directory mirroring the source structure:
```
tests/
├── test_models/
│ └── test_factory.py
├── test_sandbox/
│ └── test_local.py
└── test_gateway/
└── test_models_router.py
```
Example test:
```python
import pytest
from deerflow.models.factory import create_chat_model
def test_create_chat_model_with_valid_name():
"""Test that a valid model name creates a model instance."""
model = create_chat_model("gpt-4")
assert model is not None
def test_create_chat_model_with_invalid_name():
"""Test that an invalid model name raises ValueError."""
with pytest.raises(ValueError):
create_chat_model("nonexistent-model")
```
## Pull Request Process
### Before Submitting
1. **Ensure tests pass**: `uv run pytest`
2. **Run linter**: `make lint`
3. **Format code**: `make format`
4. **Update documentation** if needed
### PR Description
Include in your PR description:
- **What**: Brief description of changes
- **Why**: Motivation for the change
- **How**: Implementation approach
- **Testing**: How you tested the changes
### Review Process
1. Submit PR with clear description
2. Address review feedback
3. Ensure CI passes
4. Maintainer will merge when approved
## Architecture Guidelines
### Adding New Tools
1. Create tool in `packages/harness/deerflow/tools/builtins/` or `packages/harness/deerflow/community/`:
```python
# packages/harness/deerflow/tools/builtins/my_tool.py
from langchain_core.tools import tool
@tool
def my_tool(param: str) -> str:
"""Tool description for the agent.
Args:
param: Description of the parameter
Returns:
Description of return value
"""
return f"Result: {param}"
```
2. Register in `config.yaml`:
```yaml
tools:
- name: my_tool
group: my_group
use: deerflow.tools.builtins.my_tool:my_tool
```
### Adding New Middleware
1. Create middleware in `packages/harness/deerflow/agents/middlewares/`:
```python
# packages/harness/deerflow/agents/middlewares/my_middleware.py
from langchain.agents.middleware import BaseMiddleware
from langchain_core.runnables import RunnableConfig
class MyMiddleware(BaseMiddleware):
"""Middleware description."""
def transform_state(self, state: dict, config: RunnableConfig) -> dict:
"""Transform the state before agent execution."""
# Modify state as needed
return state
```
2. Register in `packages/harness/deerflow/agents/lead_agent/agent.py`:
```python
middlewares = [
ThreadDataMiddleware(),
SandboxMiddleware(),
MyMiddleware(), # Add your middleware
TitleMiddleware(),
ClarificationMiddleware(),
]
```
### Adding New API Endpoints
1. Create router in `app/gateway/routers/`:
```python
# app/gateway/routers/my_router.py
from fastapi import APIRouter
router = APIRouter(prefix="/my-endpoint", tags=["my-endpoint"])
@router.get("/")
async def get_items():
"""Get all items."""
return {"items": []}
@router.post("/")
async def create_item(data: dict):
"""Create a new item."""
return {"created": data}
```
2. Register in `app/gateway/app.py`:
```python
from app.gateway.routers import my_router
app.include_router(my_router.router)
```
### Configuration Changes
When adding new configuration options:
1. Update `packages/harness/deerflow/config/app_config.py` with new fields
2. Add default values in `config.example.yaml`
3. Document in `docs/CONFIGURATION.md`
### MCP Server Integration
To add support for a new MCP server:
1. Add configuration in `extensions_config.json`:
```json
{
"mcpServers": {
"my-server": {
"enabled": true,
"type": "stdio",
"command": "npx",
"args": ["-y", "@my-org/mcp-server"],
"description": "My MCP Server"
}
}
}
```
2. Update `extensions_config.example.json` with the new server
### Skills Development
To create a new skill:
1. Create directory in `skills/public/` or `skills/custom/`:
```
skills/public/my-skill/
└── SKILL.md
```
2. Write `SKILL.md` with YAML front matter:
```markdown
---
name: My Skill
description: What this skill does
license: MIT
allowed-tools:
- read_file
- write_file
- bash
---
# My Skill
Instructions for the agent when this skill is enabled...
```
## Questions?
If you have questions about contributing:
1. Check existing documentation in `docs/`
2. Look for similar issues or PRs on GitHub
3. Open a discussion or issue on GitHub
Thank you for contributing to DeerFlow!
+91
View File
@@ -0,0 +1,91 @@
# Backend Dockerfile — multi-stage build
# Stage 1 (builder): compiles native Python extensions with build-essential
# Stage 2 (dev): retains toolchain for dev containers (uv sync at startup)
# Stage 3 (runtime): clean image without compiler toolchain for production
# UV source image (override for restricted networks that cannot reach ghcr.io)
ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.7.20
FROM ${UV_IMAGE} AS uv-source
# ── Stage 1: Builder ──────────────────────────────────────────────────────────
FROM python:3.12-slim-bookworm AS builder
ARG NODE_MAJOR=22
ARG APT_MIRROR
ARG UV_INDEX_URL
# Optional extras to install (e.g. "postgres" for PostgreSQL support)
# Usage: docker build --build-arg UV_EXTRAS=postgres ...
ARG UV_EXTRAS
# Optionally override apt mirror for restricted networks (e.g. APT_MIRROR=mirrors.aliyun.com)
RUN if [ -n "${APT_MIRROR}" ]; then \
sed -i "s|deb.debian.org|${APT_MIRROR}|g" /etc/apt/sources.list.d/debian.sources 2>/dev/null || true; \
sed -i "s|deb.debian.org|${APT_MIRROR}|g" /etc/apt/sources.list 2>/dev/null || true; \
fi
# Install build tools + Node.js (build-essential needed for native Python extensions)
RUN apt-get update && apt-get install -y \
curl \
build-essential \
gnupg \
ca-certificates \
&& mkdir -p /etc/apt/keyrings \
&& curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg \
&& echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_${NODE_MAJOR}.x nodistro main" > /etc/apt/sources.list.d/nodesource.list \
&& apt-get update \
&& apt-get install -y nodejs \
&& rm -rf /var/lib/apt/lists/*
# Install uv (source image overridable via UV_IMAGE build arg)
COPY --from=uv-source /uv /uvx /usr/local/bin/
# Set working directory
WORKDIR /app
# Copy backend source code
COPY backend ./backend
# Install dependencies with cache mount
# When UV_EXTRAS is set (e.g. "postgres"), installs optional dependencies.
RUN --mount=type=cache,target=/root/.cache/uv \
sh -c "cd backend && UV_INDEX_URL=${UV_INDEX_URL:-https://pypi.org/simple} uv sync ${UV_EXTRAS:+--extra $UV_EXTRAS}"
# ── Stage 2: Dev ──────────────────────────────────────────────────────────────
# Retains compiler toolchain from builder so startup-time `uv sync` can build
# source distributions in development containers.
FROM builder AS dev
# Install Docker CLI (for DooD: allows starting sandbox containers via host Docker socket)
COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
EXPOSE 8001 2024
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
# ── Stage 3: Runtime ──────────────────────────────────────────────────────────
# Clean image without build-essential — reduces size (~200 MB) and attack surface.
FROM python:3.12-slim-bookworm
# Copy Node.js runtime from builder (provides npx for MCP servers)
COPY --from=builder /usr/bin/node /usr/bin/node
COPY --from=builder /usr/lib/node_modules /usr/lib/node_modules
RUN ln -s ../lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm \
&& ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/bin/npx
# Install Docker CLI (for DooD: allows starting sandbox containers via host Docker socket)
COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
# Install uv (source image overridable via UV_IMAGE build arg)
COPY --from=uv-source /uv /uvx /usr/local/bin/
# Set working directory
WORKDIR /app
# Copy backend with pre-built virtualenv from builder
COPY --from=builder /app/backend ./backend
# Expose ports (gateway: 8001, langgraph: 2024)
EXPOSE 8001 2024
# Default command (can be overridden in docker-compose)
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
+18
View File
@@ -0,0 +1,18 @@
install:
uv sync
dev:
PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --reload
gateway:
PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
test:
PYTHONPATH=. uv run pytest tests/ -v
lint:
uvx ruff check .
uvx ruff format --check .
format:
uvx ruff check . --fix && uvx ruff format .
+418
View File
@@ -0,0 +1,418 @@
# DeerFlow Backend
DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent memory, and extensible tool integration. The backend enables AI agents to execute code, browse the web, manage files, delegate tasks to subagents, and retain context across conversations - all in isolated, per-thread environments.
---
## Architecture
```
┌──────────────────────────────────────┐
│ Nginx (Port 2026) │
│ Unified reverse proxy │
└───────┬──────────────────┬───────────┘
│ │
/api/langgraph/* │ │ /api/* (other)
▼ ▼
┌────────────────────┐ ┌────────────────────────┐
│ LangGraph Server │ │ Gateway API (8001) │
│ (Port 2024) │ │ FastAPI REST │
│ │ │ │
│ ┌────────────────┐ │ │ Models, MCP, Skills, │
│ │ Lead Agent │ │ │ Memory, Uploads, │
│ │ ┌──────────┐ │ │ │ Artifacts │
│ │ │Middleware│ │ │ └────────────────────────┘
│ │ │ Chain │ │ │
│ │ └──────────┘ │ │
│ │ ┌──────────┐ │ │
│ │ │ Tools │ │ │
│ │ └──────────┘ │ │
│ │ ┌──────────┐ │ │
│ │ │Subagents │ │ │
│ │ └──────────┘ │ │
│ └────────────────┘ │
└────────────────────┘
```
**Request Routing** (via Nginx):
- `/api/langgraph/*` → LangGraph Server - agent interactions, threads, streaming
- `/api/*` (other) → Gateway API - models, MCP, skills, memory, artifacts, uploads, thread-local cleanup
- `/` (non-API) → Frontend - Next.js web interface
---
## Core Components
### Lead Agent
The single LangGraph agent (`lead_agent`) is the runtime entry point, created via `make_lead_agent(config)`. It combines:
- **Dynamic model selection** with thinking and vision support
- **Middleware chain** for cross-cutting concerns (9 middlewares)
- **Tool system** with sandbox, MCP, community, and built-in tools
- **Subagent delegation** for parallel task execution
- **System prompt** with skills injection, memory context, and working directory guidance
### Middleware Chain
Middlewares execute in strict order, each handling a specific concern:
| # | Middleware | Purpose |
|---|-----------|---------|
| 1 | **ThreadDataMiddleware** | Creates per-thread isolated directories (workspace, uploads, outputs) |
| 2 | **UploadsMiddleware** | Injects newly uploaded files into conversation context |
| 3 | **SandboxMiddleware** | Acquires sandbox environment for code execution |
| 4 | **SummarizationMiddleware** | Reduces context when approaching token limits (optional) |
| 5 | **TodoListMiddleware** | Tracks multi-step tasks in plan mode (optional) |
| 6 | **TitleMiddleware** | Auto-generates conversation titles after first exchange |
| 7 | **MemoryMiddleware** | Queues conversations for async memory extraction |
| 8 | **ViewImageMiddleware** | Injects image data for vision-capable models (conditional) |
| 9 | **ClarificationMiddleware** | Intercepts clarification requests and interrupts execution (must be last) |
### Sandbox System
Per-thread isolated execution with virtual path translation:
- **Abstract interface**: `execute_command`, `read_file`, `write_file`, `list_dir`
- **Providers**: `LocalSandboxProvider` (filesystem) and `AioSandboxProvider` (Docker, in community/)
- **Virtual paths**: `/mnt/user-data/{workspace,uploads,outputs}` → thread-specific physical directories
- **Skills path**: `/mnt/skills``deer-flow/skills/` directory
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
- **File-write safety**: `str_replace` serializes read-modify-write per `(sandbox.id, path)` so isolated sandboxes keep concurrency even when virtual paths match
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` (`bash` is disabled by default when using `LocalSandboxProvider`; use `AioSandboxProvider` for isolated shell access)
### Subagent System
Async task delegation with concurrent execution:
- **Built-in agents**: `general-purpose` (full toolset) and `bash` (command specialist, exposed only when shell access is available)
- **Concurrency**: Max 3 subagents per turn, 15-minute timeout
- **Execution**: Background thread pools with status tracking and SSE events
- **Flow**: Agent calls `task()` tool → executor runs subagent in background → polls for completion → returns result
### Memory System
LLM-powered persistent context retention across conversations:
- **Automatic extraction**: Analyzes conversations for user context, facts, and preferences
- **Structured storage**: User context (work, personal, top-of-mind), history, and confidence-scored facts
- **Debounced updates**: Batches updates to minimize LLM calls (configurable wait time)
- **System prompt injection**: Top facts + context injected into agent prompts
- **Storage**: JSON file with mtime-based cache invalidation
### Tool Ecosystem
| Category | Tools |
|----------|-------|
| **Sandbox** | `bash`, `ls`, `read_file`, `write_file`, `str_replace` |
| **Built-in** | `present_files`, `ask_clarification`, `view_image`, `task` (subagent) |
| **Community** | Tavily (web search), Jina AI (web fetch), Firecrawl (scraping), DuckDuckGo (image search) |
| **MCP** | Any Model Context Protocol server (stdio, SSE, HTTP transports) |
| **Skills** | Domain-specific workflows injected via system prompt |
### Gateway API
FastAPI application providing REST endpoints for frontend integration:
| Route | Purpose |
|-------|---------|
| `GET /api/models` | List available LLM models |
| `GET/PUT /api/mcp/config` | Manage MCP server configurations |
| `GET/PUT /api/skills` | List and manage skills |
| `POST /api/skills/install` | Install skill from `.skill` archive |
| `GET /api/memory` | Retrieve memory data |
| `POST /api/memory/reload` | Force memory reload |
| `GET /api/memory/config` | Memory configuration |
| `GET /api/memory/status` | Combined config + data |
| `POST /api/threads/{id}/uploads` | Upload files (auto-converts PDF/PPT/Excel/Word to Markdown, rejects directory paths) |
| `GET /api/threads/{id}/uploads/list` | List uploaded files |
| `DELETE /api/threads/{id}` | Delete DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
| `GET /api/threads/{id}/artifacts/{path}` | Serve generated artifacts |
### IM Channels
The IM bridge supports Feishu, Slack, and Telegram. Slack and Telegram still use the final `runs.wait()` response path, while Feishu now streams through `runs.stream(["messages-tuple", "values"])` and updates a single in-thread card in place.
For Feishu card updates, DeerFlow stores the running card's `message_id` per inbound message and patches that same card until the run finishes, preserving the existing `OK` / `DONE` reaction flow.
---
## Quick Start
### Prerequisites
- Python 3.12+
- [uv](https://docs.astral.sh/uv/) package manager
- API keys for your chosen LLM provider
### Installation
```bash
cd deer-flow
# Copy configuration files
cp config.example.yaml config.yaml
# Install backend dependencies
cd backend
make install
```
### Configuration
Edit `config.yaml` in the project root:
```yaml
models:
- name: gpt-4o
display_name: GPT-4o
use: langchain_openai:ChatOpenAI
model: gpt-4o
api_key: $OPENAI_API_KEY
supports_thinking: false
supports_vision: true
- name: gpt-5-responses
display_name: GPT-5 (Responses API)
use: langchain_openai:ChatOpenAI
model: gpt-5
api_key: $OPENAI_API_KEY
use_responses_api: true
output_version: responses/v1
supports_vision: true
```
Set your API keys:
```bash
export OPENAI_API_KEY="your-api-key-here"
```
### Running
**Full Application** (from project root):
```bash
make dev # Starts LangGraph + Gateway + Frontend + Nginx
```
Access at: http://localhost:2026
**Backend Only** (from backend directory):
```bash
# Terminal 1: LangGraph server
make dev
# Terminal 2: Gateway API
make gateway
```
Direct access: LangGraph at http://localhost:2024, Gateway at http://localhost:8001
---
## Project Structure
```
backend/
├── src/
│ ├── agents/ # Agent system
│ │ ├── lead_agent/ # Main agent (factory, prompts)
│ │ ├── middlewares/ # 9 middleware components
│ │ ├── memory/ # Memory extraction & storage
│ │ └── thread_state.py # ThreadState schema
│ ├── gateway/ # FastAPI Gateway API
│ │ ├── app.py # Application setup
│ │ └── routers/ # 6 route modules
│ ├── sandbox/ # Sandbox execution
│ │ ├── local/ # Local filesystem provider
│ │ ├── sandbox.py # Abstract interface
│ │ ├── tools.py # bash, ls, read/write/str_replace
│ │ └── middleware.py # Sandbox lifecycle
│ ├── subagents/ # Subagent delegation
│ │ ├── builtins/ # general-purpose, bash agents
│ │ ├── executor.py # Background execution engine
│ │ └── registry.py # Agent registry
│ ├── tools/builtins/ # Built-in tools
│ ├── mcp/ # MCP protocol integration
│ ├── models/ # Model factory
│ ├── skills/ # Skill discovery & loading
│ ├── config/ # Configuration system
│ ├── community/ # Community tools & providers
│ ├── reflection/ # Dynamic module loading
│ └── utils/ # Utilities
├── docs/ # Documentation
├── tests/ # Test suite
├── langgraph.json # LangGraph server configuration
├── pyproject.toml # Python dependencies
├── Makefile # Development commands
└── Dockerfile # Container build
```
---
## Configuration
### Main Configuration (`config.yaml`)
Place in project root. Config values starting with `$` resolve as environment variables.
Key sections:
- `models` - LLM configurations with class paths, API keys, thinking/vision flags
- `tools` - Tool definitions with module paths and groups
- `tool_groups` - Logical tool groupings
- `sandbox` - Execution environment provider
- `skills` - Skills directory paths
- `title` - Auto-title generation settings
- `summarization` - Context summarization settings
- `subagents` - Subagent system (enabled/disabled)
- `memory` - Memory system settings (enabled, storage, debounce, facts limits)
Provider note:
- `models[*].use` references provider classes by module path (for example `langchain_openai:ChatOpenAI`).
- If a provider module is missing, DeerFlow now returns an actionable error with install guidance (for example `uv add langchain-google-genai`).
### Extensions Configuration (`extensions_config.json`)
MCP servers and skill states in a single file:
```json
{
"mcpServers": {
"github": {
"enabled": true,
"type": "stdio",
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-github"],
"env": {"GITHUB_TOKEN": "$GITHUB_TOKEN"}
},
"secure-http": {
"enabled": true,
"type": "http",
"url": "https://api.example.com/mcp",
"oauth": {
"enabled": true,
"token_url": "https://auth.example.com/oauth/token",
"grant_type": "client_credentials",
"client_id": "$MCP_OAUTH_CLIENT_ID",
"client_secret": "$MCP_OAUTH_CLIENT_SECRET"
}
}
},
"skills": {
"pdf-processing": {"enabled": true}
}
}
```
### Environment Variables
- `DEER_FLOW_CONFIG_PATH` - Override config.yaml location
- `DEER_FLOW_EXTENSIONS_CONFIG_PATH` - Override extensions_config.json location
- Model API keys: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `DEEPSEEK_API_KEY`, etc.
- Tool API keys: `TAVILY_API_KEY`, `GITHUB_TOKEN`, etc.
### LangSmith Tracing
DeerFlow has built-in [LangSmith](https://smith.langchain.com) integration for observability. When enabled, all LLM calls, agent runs, tool executions, and middleware processing are traced and visible in the LangSmith dashboard.
**Setup:**
1. Sign up at [smith.langchain.com](https://smith.langchain.com) and create a project.
2. Add the following to your `.env` file in the project root:
```bash
LANGSMITH_TRACING=true
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
LANGSMITH_PROJECT=xxx
```
**Legacy variables:** The `LANGCHAIN_TRACING_V2`, `LANGCHAIN_API_KEY`, `LANGCHAIN_PROJECT`, and `LANGCHAIN_ENDPOINT` variables are also supported for backward compatibility. `LANGSMITH_*` variables take precedence when both are set.
### Langfuse Tracing
DeerFlow also supports [Langfuse](https://langfuse.com) observability for LangChain-compatible runs.
Add the following to your `.env` file:
```bash
LANGFUSE_TRACING=true
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
LANGFUSE_BASE_URL=https://cloud.langfuse.com
```
If you are using a self-hosted Langfuse deployment, set `LANGFUSE_BASE_URL` to your Langfuse host.
### Dual Provider Behavior
If both LangSmith and Langfuse are enabled, DeerFlow initializes and attaches both callbacks so the same run data is reported to both systems.
If a provider is explicitly enabled but required credentials are missing, or the provider callback cannot be initialized, DeerFlow raises an error when tracing is initialized during model creation instead of silently disabling tracing.
**Docker:** In `docker-compose.yaml`, tracing is disabled by default (`LANGSMITH_TRACING=false`). Set `LANGSMITH_TRACING=true` and/or `LANGFUSE_TRACING=true` in your `.env`, together with the required credentials, to enable tracing in containerized deployments.
---
## Development
### Commands
```bash
make install # Install dependencies
make dev # Run LangGraph server (port 2024)
make gateway # Run Gateway API (port 8001)
make lint # Run linter (ruff)
make format # Format code (ruff)
```
### Code Style
- **Linter/Formatter**: `ruff`
- **Line length**: 240 characters
- **Python**: 3.12+ with type hints
- **Quotes**: Double quotes
- **Indentation**: 4 spaces
### Testing
```bash
uv run pytest
```
---
## Technology Stack
- **LangGraph** (1.0.6+) - Agent framework and multi-agent orchestration
- **LangChain** (1.2.3+) - LLM abstractions and tool system
- **FastAPI** (0.115.0+) - Gateway REST API
- **langchain-mcp-adapters** - Model Context Protocol support
- **agent-sandbox** - Sandboxed code execution
- **markitdown** - Multi-format document conversion
- **tavily-python** / **firecrawl-py** - Web search and scraping
---
## Documentation
- [Configuration Guide](docs/CONFIGURATION.md)
- [Architecture Details](docs/ARCHITECTURE.md)
- [API Reference](docs/API.md)
- [File Upload](docs/FILE_UPLOAD.md)
- [Path Examples](docs/PATH_EXAMPLES.md)
- [Context Summarization](docs/summarization.md)
- [Plan Mode](docs/plan_mode_usage.md)
- [Setup Guide](docs/SETUP.md)
---
## License
See the [LICENSE](../LICENSE) file in the project root.
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines.
View File
+16
View File
@@ -0,0 +1,16 @@
"""IM Channel integration for DeerFlow.
Provides a pluggable channel system that connects external messaging platforms
(Feishu/Lark, Slack, Telegram) to the DeerFlow agent via the ChannelManager,
which uses ``langgraph-sdk`` to communicate with Gateway's LangGraph-compatible API.
"""
from app.channels.base import Channel
from app.channels.message_bus import InboundMessage, MessageBus, OutboundMessage
__all__ = [
"Channel",
"InboundMessage",
"MessageBus",
"OutboundMessage",
]
+130
View File
@@ -0,0 +1,130 @@
"""Abstract base class for IM channels."""
from __future__ import annotations
import logging
from abc import ABC, abstractmethod
from typing import Any
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
logger = logging.getLogger(__name__)
class Channel(ABC):
"""Base class for all IM channel implementations.
Each channel connects to an external messaging platform and:
1. Receives messages, wraps them as InboundMessage, publishes to the bus.
2. Subscribes to outbound messages and sends replies back to the platform.
Subclasses must implement ``start``, ``stop``, and ``send``.
"""
def __init__(self, name: str, bus: MessageBus, config: dict[str, Any]) -> None:
self.name = name
self.bus = bus
self.config = config
self._running = False
@property
def is_running(self) -> bool:
return self._running
@property
def supports_streaming(self) -> bool:
return False
# -- lifecycle ---------------------------------------------------------
@abstractmethod
async def start(self) -> None:
"""Start listening for messages from the external platform."""
@abstractmethod
async def stop(self) -> None:
"""Gracefully stop the channel."""
# -- outbound ----------------------------------------------------------
@abstractmethod
async def send(self, msg: OutboundMessage) -> None:
"""Send a message back to the external platform.
The implementation should use ``msg.chat_id`` and ``msg.thread_ts``
to route the reply to the correct conversation/thread.
"""
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
"""Upload a single file attachment to the platform.
Returns True if the upload succeeded, False otherwise.
Default implementation returns False (no file upload support).
"""
return False
# -- helpers -----------------------------------------------------------
def _make_inbound(
self,
chat_id: str,
user_id: str,
text: str,
*,
msg_type: InboundMessageType = InboundMessageType.CHAT,
thread_ts: str | None = None,
files: list[dict[str, Any]] | None = None,
metadata: dict[str, Any] | None = None,
) -> InboundMessage:
"""Convenience factory for creating InboundMessage instances."""
return InboundMessage(
channel_name=self.name,
chat_id=chat_id,
user_id=user_id,
text=text,
msg_type=msg_type,
thread_ts=thread_ts,
files=files or [],
metadata=metadata or {},
)
async def _on_outbound(self, msg: OutboundMessage) -> None:
"""Outbound callback registered with the bus.
Only forwards messages targeted at this channel.
Sends the text message first, then uploads any file attachments.
File uploads are skipped entirely when the text send fails to avoid
partial deliveries (files without accompanying text).
"""
if msg.channel_name == self.name:
try:
await self.send(msg)
except Exception:
logger.exception("Failed to send outbound message on channel %s", self.name)
return # Do not attempt file uploads when the text message failed
for attachment in msg.attachments:
try:
success = await self.send_file(msg, attachment)
if not success:
logger.warning("[%s] file upload skipped for %s", self.name, attachment.filename)
except Exception:
logger.exception("[%s] failed to upload file %s", self.name, attachment.filename)
async def receive_file(self, msg: InboundMessage, thread_id: str) -> InboundMessage:
"""
Optionally process and materialize inbound file attachments for this channel.
By default, this method does nothing and simply returns the original message.
Subclasses (e.g. FeishuChannel) may override this to download files (images, documents, etc)
referenced in msg.files, save them to the sandbox, and update msg.text to include
the sandbox file paths for downstream model consumption.
Args:
msg: The inbound message, possibly containing file metadata in msg.files.
thread_id: The resolved DeerFlow thread ID for sandbox path context.
Returns:
The (possibly modified) InboundMessage, with text and/or files updated as needed.
"""
return msg
+20
View File
@@ -0,0 +1,20 @@
"""Shared command definitions used by all channel implementations.
Keeping the authoritative command set in one place ensures that channel
parsers (e.g. Feishu) and the ChannelManager dispatcher stay in sync
automatically — adding or removing a command here is the single edit
required.
"""
from __future__ import annotations
KNOWN_CHANNEL_COMMANDS: frozenset[str] = frozenset(
{
"/bootstrap",
"/new",
"/status",
"/models",
"/memory",
"/help",
}
)
+740
View File
@@ -0,0 +1,740 @@
"""DingTalk channel implementation."""
from __future__ import annotations
import asyncio
import json
import logging
import re
import threading
import time
from pathlib import Path
from typing import Any
import httpx
from app.channels.base import Channel
from app.channels.commands import KNOWN_CHANNEL_COMMANDS
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
logger = logging.getLogger(__name__)
DINGTALK_API_BASE = "https://api.dingtalk.com"
_TOKEN_REFRESH_MARGIN_SECONDS = 300
_CONVERSATION_TYPE_P2P = "1"
_CONVERSATION_TYPE_GROUP = "2"
_MAX_UPLOAD_SIZE_BYTES = 20 * 1024 * 1024
def _normalize_conversation_type(raw: Any) -> str:
"""Normalize ``conversationType`` to ``"1"`` (P2P) or ``"2"`` (group).
Stream payloads may send int or string values.
"""
if raw is None:
return _CONVERSATION_TYPE_P2P
s = str(raw).strip()
if s == _CONVERSATION_TYPE_GROUP:
return _CONVERSATION_TYPE_GROUP
return _CONVERSATION_TYPE_P2P
def _normalize_allowed_users(allowed_users: Any) -> set[str]:
if allowed_users is None:
return set()
if isinstance(allowed_users, str):
values = [allowed_users]
elif isinstance(allowed_users, (list, tuple, set)):
values = allowed_users
else:
logger.warning(
"DingTalk allowed_users should be a list of user IDs; treating %s as one string value",
type(allowed_users).__name__,
)
values = [allowed_users]
return {str(uid) for uid in values if str(uid)}
def _is_dingtalk_command(text: str) -> bool:
if not text.startswith("/"):
return False
return text.split(maxsplit=1)[0].lower() in KNOWN_CHANNEL_COMMANDS
def _extract_text_from_rich_text(rich_text_list: list) -> str:
parts: list[str] = []
for item in rich_text_list:
if isinstance(item, dict) and "text" in item:
parts.append(item["text"])
return " ".join(parts)
_FENCED_CODE_BLOCK_RE = re.compile(r"```(\w*)\n(.*?)```", re.DOTALL)
_INLINE_CODE_RE = re.compile(r"`([^`\n]+)`")
_HORIZONTAL_RULE_RE = re.compile(r"^-{3,}$", re.MULTILINE)
_TABLE_SEPARATOR_RE = re.compile(r"^\|[-:| ]+\|$", re.MULTILINE)
def _convert_markdown_table(text: str) -> str:
# DingTalk sampleMarkdown does not render pipe-delimited tables.
lines = text.split("\n")
result: list[str] = []
i = 0
while i < len(lines):
line = lines[i]
# Detect table: header row followed by separator row
if i + 1 < len(lines) and line.strip().startswith("|") and _TABLE_SEPARATOR_RE.match(lines[i + 1].strip()):
headers = [h.strip() for h in line.strip().strip("|").split("|")]
i += 2 # skip header + separator
while i < len(lines) and lines[i].strip().startswith("|"):
cells = [c.strip() for c in lines[i].strip().strip("|").split("|")]
for h, c in zip(headers, cells):
result.append(f"> **{h}**: {c}")
result.append("")
i += 1
else:
result.append(line)
i += 1
return "\n".join(result)
def _adapt_markdown_for_dingtalk(text: str) -> str:
"""Adapt markdown for DingTalk's limited sampleMarkdown renderer."""
def _code_block_to_quote(match: re.Match) -> str:
lang = match.group(1)
code = match.group(2).rstrip("\n")
prefix = f"> **{lang}**\n" if lang else ""
quoted_lines = "\n".join(f"> {line}" for line in code.split("\n"))
return f"{prefix}{quoted_lines}\n"
text = _FENCED_CODE_BLOCK_RE.sub(_code_block_to_quote, text)
text = _INLINE_CODE_RE.sub(r"**\1**", text)
text = _convert_markdown_table(text)
text = _HORIZONTAL_RULE_RE.sub("───────────", text)
return text
class DingTalkChannel(Channel):
"""DingTalk IM channel using Stream Push (WebSocket, no public IP needed)."""
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
super().__init__(name="dingtalk", bus=bus, config=config)
self._thread: threading.Thread | None = None
self._main_loop: asyncio.AbstractEventLoop | None = None
self._client_id: str = ""
self._client_secret: str = ""
self._allowed_users: set[str] = _normalize_allowed_users(config.get("allowed_users"))
self._cached_token: str = ""
self._token_expires_at: float = 0.0
self._token_lock = asyncio.Lock()
self._card_template_id: str = config.get("card_template_id", "")
self._card_track_ids: dict[str, str] = {}
self._dingtalk_client: Any = None
self._stream_client: Any = None
self._incoming_messages: dict[str, Any] = {}
self._incoming_messages_lock = threading.Lock()
self._card_repliers: dict[str, Any] = {}
@property
def supports_streaming(self) -> bool:
return bool(self._card_template_id)
async def start(self) -> None:
if self._running:
return
try:
import dingtalk_stream # noqa: F401
except ImportError:
logger.error("dingtalk-stream is not installed. Install it with: uv add dingtalk-stream")
return
client_id = self.config.get("client_id", "")
client_secret = self.config.get("client_secret", "")
if not client_id or not client_secret:
logger.error("DingTalk channel requires client_id and client_secret")
return
self._client_id = client_id
self._client_secret = client_secret
self._main_loop = asyncio.get_running_loop()
if self._card_template_id:
logger.info("[DingTalk] AI Card mode enabled (template=%s)", self._card_template_id)
self._running = True
self.bus.subscribe_outbound(self._on_outbound)
self._thread = threading.Thread(
target=self._run_stream,
args=(client_id, client_secret),
daemon=True,
)
self._thread.start()
logger.info("DingTalk channel started")
async def stop(self) -> None:
self._running = False
self.bus.unsubscribe_outbound(self._on_outbound)
stream_client = self._stream_client
if stream_client is not None:
try:
if hasattr(stream_client, "disconnect"):
stream_client.disconnect()
except Exception:
logger.debug("[DingTalk] error disconnecting stream client", exc_info=True)
self._dingtalk_client = None
self._stream_client = None
with self._incoming_messages_lock:
self._incoming_messages.clear()
self._card_repliers.clear()
self._card_track_ids.clear()
if self._thread:
self._thread.join(timeout=5)
self._thread = None
logger.info("DingTalk channel stopped")
def _resolve_routing(self, msg: OutboundMessage) -> tuple[str, str, str]:
"""Return (conversation_type, sender_staff_id, conversation_id).
Uses msg.chat_id as the primary routing key; metadata as fallback.
"""
conversation_type = _normalize_conversation_type(msg.metadata.get("conversation_type"))
sender_staff_id = msg.metadata.get("sender_staff_id", "")
conversation_id = msg.metadata.get("conversation_id", "")
if conversation_type == _CONVERSATION_TYPE_GROUP:
conversation_id = msg.chat_id or conversation_id
else:
sender_staff_id = msg.chat_id or sender_staff_id
return conversation_type, sender_staff_id, conversation_id
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
conversation_type, sender_staff_id, conversation_id = self._resolve_routing(msg)
robot_code = self._client_id
# Card mode: stream update to existing AI card
source_key = self._make_card_source_key_from_outbound(msg)
out_track_id = self._card_track_ids.get(source_key)
# ``card_template_id`` enables ``runs.stream`` (non-final + final outbounds).
# If card creation failed, skip non-final chunks to avoid duplicate messages.
if self._card_template_id and not out_track_id and not msg.is_final:
return
if out_track_id:
try:
await self._stream_update_card(
out_track_id,
msg.text,
is_finalize=msg.is_final,
)
except Exception:
logger.warning("[DingTalk] card stream failed, falling back to sampleMarkdown")
if msg.is_final:
self._card_track_ids.pop(source_key, None)
self._card_repliers.pop(out_track_id, None)
await self._send_markdown_fallback(robot_code, conversation_type, sender_staff_id, conversation_id, msg.text)
return
if msg.is_final:
self._card_track_ids.pop(source_key, None)
self._card_repliers.pop(out_track_id, None)
return
# Non-card mode: send sampleMarkdown with retry
last_exc: Exception | None = None
for attempt in range(_max_retries):
try:
if conversation_type == _CONVERSATION_TYPE_GROUP:
await self._send_group_message(robot_code, conversation_id, msg.text, at_user_ids=[sender_staff_id] if sender_staff_id else None)
else:
await self._send_p2p_message(robot_code, sender_staff_id, msg.text)
return
except Exception as exc:
last_exc = exc
if attempt < _max_retries - 1:
delay = 2**attempt
logger.warning(
"[DingTalk] send failed (attempt %d/%d), retrying in %ds: %s",
attempt + 1,
_max_retries,
delay,
exc,
)
await asyncio.sleep(delay)
logger.error("[DingTalk] send failed after %d attempts: %s", _max_retries, last_exc)
if last_exc is None:
raise RuntimeError("DingTalk send failed without an exception from any attempt")
raise last_exc
async def _send_markdown_fallback(
self,
robot_code: str,
conversation_type: str,
sender_staff_id: str,
conversation_id: str,
text: str,
) -> None:
try:
if conversation_type == _CONVERSATION_TYPE_GROUP:
await self._send_group_message(robot_code, conversation_id, text)
else:
await self._send_p2p_message(robot_code, sender_staff_id, text)
except Exception:
logger.exception("[DingTalk] markdown fallback also failed")
raise
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
if attachment.size > _MAX_UPLOAD_SIZE_BYTES:
logger.warning("[DingTalk] file too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
return False
conversation_type, sender_staff_id, conversation_id = self._resolve_routing(msg)
robot_code = self._client_id
try:
media_id = await self._upload_media(attachment.actual_path, "image" if attachment.is_image else "file")
if not media_id:
return False
if attachment.is_image:
msg_key = "sampleImageMsg"
msg_param = json.dumps({"photoURL": media_id})
else:
msg_key = "sampleFile"
msg_param = json.dumps(
{
"fileUrl": media_id,
"fileName": attachment.filename,
"fileSize": str(attachment.size),
}
)
token = await self._get_access_token()
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
if conversation_type == _CONVERSATION_TYPE_GROUP:
response = await client.post(
f"{DINGTALK_API_BASE}/v1.0/robot/groupMessages/send",
headers=self._api_headers(token),
json={
"msgKey": msg_key,
"msgParam": msg_param,
"robotCode": robot_code,
"openConversationId": conversation_id,
},
)
else:
response = await client.post(
f"{DINGTALK_API_BASE}/v1.0/robot/oToMessages/batchSend",
headers=self._api_headers(token),
json={
"msgKey": msg_key,
"msgParam": msg_param,
"robotCode": robot_code,
"userIds": [sender_staff_id],
},
)
response.raise_for_status()
logger.info("[DingTalk] file sent: %s", attachment.filename)
return True
except (httpx.HTTPError, OSError, ValueError, TypeError, AttributeError):
logger.exception("[DingTalk] failed to send file: %s", attachment.filename)
return False
# -- stream client (runs in dedicated thread) --------------------------
def _run_stream(self, client_id: str, client_secret: str) -> None:
try:
import dingtalk_stream
credential = dingtalk_stream.Credential(client_id, client_secret)
client = dingtalk_stream.DingTalkStreamClient(credential)
self._stream_client = client
client.register_callback_handler(
dingtalk_stream.chatbot.ChatbotMessage.TOPIC,
_DingTalkMessageHandler(self),
)
client.start_forever()
except Exception:
if self._running:
logger.exception("DingTalk Stream Push error")
finally:
self._stream_client = None
def _on_chatbot_message(self, message: Any) -> None:
if not self._running:
return
try:
sender_staff_id = message.sender_staff_id or ""
conversation_type = _normalize_conversation_type(message.conversation_type)
conversation_id = message.conversation_id or ""
msg_id = message.message_id or ""
sender_nick = message.sender_nick or ""
if self._allowed_users and sender_staff_id not in self._allowed_users:
logger.debug("[DingTalk] ignoring message from non-allowed user: %s", sender_staff_id)
return
text = self._extract_text(message)
if not text:
logger.info("[DingTalk] empty text, ignoring message")
return
logger.info(
"[DingTalk] parsed message: conv_type=%s, msg_id=%s, sender=%s(%s), text=%r",
conversation_type,
msg_id,
sender_staff_id,
sender_nick,
text[:100],
)
if _is_dingtalk_command(text):
msg_type = InboundMessageType.COMMAND
else:
msg_type = InboundMessageType.CHAT
# P2P: topic_id=None (single thread per user, like Telegram private chat)
# Group: topic_id=msg_id (each new message starts a new topic, like Feishu)
topic_id: str | None = msg_id if conversation_type == _CONVERSATION_TYPE_GROUP else None
# chat_id uses conversation_id for groups, sender_staff_id for P2P
chat_id = conversation_id if conversation_type == _CONVERSATION_TYPE_GROUP else sender_staff_id
inbound = self._make_inbound(
chat_id=chat_id,
user_id=sender_staff_id,
text=text,
msg_type=msg_type,
thread_ts=msg_id,
metadata={
"conversation_type": conversation_type,
"conversation_id": conversation_id,
"sender_staff_id": sender_staff_id,
"sender_nick": sender_nick,
"message_id": msg_id,
},
)
inbound.topic_id = topic_id
if self._card_template_id:
source_key = self._make_card_source_key(inbound)
with self._incoming_messages_lock:
self._incoming_messages[source_key] = message
if self._main_loop and self._main_loop.is_running():
logger.info("[DingTalk] publishing inbound message to bus (type=%s, msg_id=%s)", msg_type.value, msg_id)
fut = asyncio.run_coroutine_threadsafe(
self._prepare_inbound(chat_id, inbound),
self._main_loop,
)
fut.add_done_callback(lambda f, mid=msg_id: self._log_future_error(f, "prepare_inbound", mid))
else:
logger.warning("[DingTalk] main loop not running, cannot publish inbound message")
except Exception:
logger.exception("[DingTalk] error processing chatbot message")
@staticmethod
def _extract_text(message: Any) -> str:
msg_type = message.message_type
if msg_type == "text" and message.text:
return message.text.content.strip()
if msg_type == "richText" and message.rich_text_content:
return _extract_text_from_rich_text(message.rich_text_content.rich_text_list).strip()
return ""
async def _prepare_inbound(self, chat_id: str, inbound: InboundMessage) -> None:
# Running reply must finish before publish_inbound so AI card tracks are
# registered before the manager emits streaming outbounds.
await self._send_running_reply(chat_id, inbound)
await self.bus.publish_inbound(inbound)
async def _send_running_reply(self, chat_id: str, inbound: InboundMessage) -> None:
conversation_type = inbound.metadata.get("conversation_type", _CONVERSATION_TYPE_P2P)
sender_staff_id = inbound.metadata.get("sender_staff_id", "")
conversation_id = inbound.metadata.get("conversation_id", "")
text = "\u23f3 Working on it..."
try:
if self._card_template_id:
source_key = self._make_card_source_key(inbound)
with self._incoming_messages_lock:
chatbot_message = self._incoming_messages.pop(source_key, None)
out_track_id = await self._create_and_deliver_card(
text,
chatbot_message=chatbot_message,
)
if out_track_id:
self._card_track_ids[source_key] = out_track_id
logger.info("[DingTalk] AI card running reply sent for chat=%s", chat_id)
return
robot_code = self._client_id
if conversation_type == _CONVERSATION_TYPE_GROUP:
await self._send_text_message_to_group(robot_code, conversation_id, text)
else:
await self._send_text_message_to_user(robot_code, sender_staff_id, text)
logger.info("[DingTalk] 'Working on it...' reply sent for chat=%s", chat_id)
except Exception:
logger.exception("[DingTalk] failed to send running reply for chat=%s", chat_id)
# -- DingTalk API helpers ----------------------------------------------
async def _get_access_token(self) -> str:
if self._cached_token and time.monotonic() < self._token_expires_at:
return self._cached_token
async with self._token_lock:
if self._cached_token and time.monotonic() < self._token_expires_at:
return self._cached_token
async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client:
response = await client.post(
f"{DINGTALK_API_BASE}/v1.0/oauth2/accessToken",
json={"appKey": self._client_id, "appSecret": self._client_secret}, # DingTalk API field names
)
response.raise_for_status()
data = response.json()
if not isinstance(data, dict):
raise ValueError(f"DingTalk access token response must be a JSON object, got {type(data).__name__}")
access_token = data.get("accessToken")
if not isinstance(access_token, str) or not access_token.strip():
raise ValueError("DingTalk access token response did not contain a usable accessToken")
raw_expires_in = data.get("expireIn", 7200)
try:
expires_in = int(raw_expires_in)
except (TypeError, ValueError):
logger.warning("[DingTalk] invalid expireIn value %r, using default 7200s", raw_expires_in)
expires_in = 7200
self._cached_token = access_token.strip()
self._token_expires_at = time.monotonic() + expires_in - _TOKEN_REFRESH_MARGIN_SECONDS
return self._cached_token
@staticmethod
def _api_headers(token: str) -> dict[str, str]:
return {
"x-acs-dingtalk-access-token": token,
"Content-Type": "application/json",
}
async def _send_text_message_to_user(self, robot_code: str, user_id: str, text: str) -> None:
token = await self._get_access_token()
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
response = await client.post(
f"{DINGTALK_API_BASE}/v1.0/robot/oToMessages/batchSend",
headers=self._api_headers(token),
json={
"msgKey": "sampleText",
"msgParam": json.dumps({"content": text}),
"robotCode": robot_code,
"userIds": [user_id],
},
)
response.raise_for_status()
async def _send_text_message_to_group(self, robot_code: str, conversation_id: str, text: str) -> None:
token = await self._get_access_token()
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
response = await client.post(
f"{DINGTALK_API_BASE}/v1.0/robot/groupMessages/send",
headers=self._api_headers(token),
json={
"msgKey": "sampleText",
"msgParam": json.dumps({"content": text}),
"robotCode": robot_code,
"openConversationId": conversation_id,
},
)
response.raise_for_status()
async def _send_p2p_message(self, robot_code: str, user_id: str, text: str) -> None:
text = _adapt_markdown_for_dingtalk(text)
token = await self._get_access_token()
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
response = await client.post(
f"{DINGTALK_API_BASE}/v1.0/robot/oToMessages/batchSend",
headers=self._api_headers(token),
json={
"msgKey": "sampleMarkdown",
"msgParam": json.dumps({"title": "DeerFlow", "text": text}),
"robotCode": robot_code,
"userIds": [user_id],
},
)
response.raise_for_status()
data = response.json()
if data.get("processQueryKey"):
logger.info("[DingTalk] P2P message sent to user=%s", user_id)
else:
logger.warning("[DingTalk] P2P send response: %s", data)
async def _send_group_message(
self,
robot_code: str,
conversation_id: str,
text: str,
*,
at_user_ids: list[str] | None = None, # noqa: ARG002
) -> None:
# at_user_ids accepted for call-site compatibility but not passed to the API
# (sampleMarkdown does not support @mentions).
text = _adapt_markdown_for_dingtalk(text)
token = await self._get_access_token()
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
response = await client.post(
f"{DINGTALK_API_BASE}/v1.0/robot/groupMessages/send",
headers=self._api_headers(token),
json={
"msgKey": "sampleMarkdown",
"msgParam": json.dumps({"title": "DeerFlow", "text": text}),
"robotCode": robot_code,
"openConversationId": conversation_id,
},
)
response.raise_for_status()
data = response.json()
if data.get("processQueryKey"):
logger.info("[DingTalk] group message sent to conversation=%s", conversation_id)
else:
logger.warning("[DingTalk] group send response: %s", data)
# -- AI Card streaming helpers -------------------------------------------
def _make_card_source_key(self, inbound: InboundMessage) -> str:
m = inbound.metadata
return f"{m.get('conversation_type', '')}:{m.get('sender_staff_id', '')}:{m.get('conversation_id', '')}:{m.get('message_id', '')}"
def _make_card_source_key_from_outbound(self, msg: OutboundMessage) -> str:
m = msg.metadata
correlation_id = m.get("message_id") or msg.thread_ts or ""
return f"{m.get('conversation_type', '')}:{m.get('sender_staff_id', '')}:{m.get('conversation_id', '')}:{correlation_id}"
async def _create_and_deliver_card(
self,
initial_text: str,
*,
chatbot_message: Any = None,
) -> str | None:
if self._dingtalk_client is None or chatbot_message is None:
logger.warning("[DingTalk] SDK client or chatbot_message unavailable, skipping AI card")
return None
try:
from dingtalk_stream.card_replier import AICardReplier
except ImportError:
logger.warning("[DingTalk] dingtalk-stream card_replier not available")
return None
try:
replier = AICardReplier(self._dingtalk_client, chatbot_message)
card_instance_id = await replier.async_create_and_deliver_card(
card_template_id=self._card_template_id,
card_data={"content": initial_text},
)
if not card_instance_id:
return None
self._card_repliers[card_instance_id] = replier
logger.info("[DingTalk] AI card created: outTrackId=%s", card_instance_id)
return card_instance_id
except Exception:
logger.exception("[DingTalk] failed to create AI card")
return None
async def _stream_update_card(
self,
out_track_id: str,
content: str,
*,
is_finalize: bool = False,
is_error: bool = False,
) -> None:
replier = self._card_repliers.get(out_track_id)
if not replier:
raise RuntimeError(f"No AICardReplier found for track ID {out_track_id}")
await replier.async_streaming(
card_instance_id=out_track_id,
content_key="content",
content_value=content,
append=False,
finished=is_finalize,
failed=is_error,
)
# -- media upload --------------------------------------------------------
async def _upload_media(self, file_path: str | Path, media_type: str) -> str | None:
try:
file_bytes = await asyncio.to_thread(Path(file_path).read_bytes)
token = await self._get_access_token()
async with httpx.AsyncClient(timeout=httpx.Timeout(60.0)) as client:
response = await client.post(
f"{DINGTALK_API_BASE}/v1.0/files/upload",
headers={"x-acs-dingtalk-access-token": token},
files={"file": ("upload", file_bytes)},
data={"type": media_type},
)
response.raise_for_status()
try:
payload = response.json()
except json.JSONDecodeError:
logger.exception("[DingTalk] failed to decode upload response JSON: %s", file_path)
return None
if not isinstance(payload, dict):
logger.warning("[DingTalk] unexpected upload response type %s for %s", type(payload).__name__, file_path)
return None
return payload.get("mediaId")
except (httpx.HTTPError, OSError):
logger.exception("[DingTalk] failed to upload media: %s", file_path)
return None
@staticmethod
def _log_future_error(fut: Any, name: str, msg_id: str) -> None:
try:
exc = fut.exception()
if exc:
logger.error("[DingTalk] %s failed for msg_id=%s: %s", name, msg_id, exc)
except (asyncio.CancelledError, asyncio.InvalidStateError):
pass
class _DingTalkMessageHandler:
"""Callback handler registered with dingtalk-stream."""
def __init__(self, channel: DingTalkChannel) -> None:
self._channel = channel
def pre_start(self) -> None:
if hasattr(self, "dingtalk_client") and self.dingtalk_client is not None:
self._channel._dingtalk_client = self.dingtalk_client
async def raw_process(self, callback_message: Any) -> Any:
import dingtalk_stream
from dingtalk_stream.frames import Headers
code, message = await self.process(callback_message)
ack_message = dingtalk_stream.AckMessage()
ack_message.code = code
ack_message.headers.message_id = callback_message.headers.message_id
ack_message.headers.content_type = Headers.CONTENT_TYPE_APPLICATION_JSON
ack_message.data = {"response": message}
return ack_message
async def process(self, callback: Any) -> tuple[int, str]:
import dingtalk_stream
incoming_message = dingtalk_stream.ChatbotMessage.from_dict(callback.data)
self._channel._on_chatbot_message(incoming_message)
return dingtalk_stream.AckMessage.STATUS_OK, "OK"
+273
View File
@@ -0,0 +1,273 @@
"""Discord channel integration using discord.py."""
from __future__ import annotations
import asyncio
import logging
import threading
from typing import Any
from app.channels.base import Channel
from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
logger = logging.getLogger(__name__)
_DISCORD_MAX_MESSAGE_LEN = 2000
class DiscordChannel(Channel):
"""Discord bot channel.
Configuration keys (in ``config.yaml`` under ``channels.discord``):
- ``bot_token``: Discord Bot token.
- ``allowed_guilds``: (optional) List of allowed Discord guild IDs. Empty = allow all.
"""
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
super().__init__(name="discord", bus=bus, config=config)
self._bot_token = str(config.get("bot_token", "")).strip()
self._allowed_guilds: set[int] = set()
for guild_id in config.get("allowed_guilds", []):
try:
self._allowed_guilds.add(int(guild_id))
except (TypeError, ValueError):
continue
self._client = None
self._thread: threading.Thread | None = None
self._discord_loop: asyncio.AbstractEventLoop | None = None
self._main_loop: asyncio.AbstractEventLoop | None = None
self._discord_module = None
async def start(self) -> None:
if self._running:
return
try:
import discord
except ImportError:
logger.error("discord.py is not installed. Install it with: uv add discord.py")
return
if not self._bot_token:
logger.error("Discord channel requires bot_token")
return
intents = discord.Intents.default()
intents.messages = True
intents.guilds = True
intents.message_content = True
client = discord.Client(
intents=intents,
allowed_mentions=discord.AllowedMentions.none(),
)
self._client = client
self._discord_module = discord
self._main_loop = asyncio.get_event_loop()
@client.event
async def on_message(message) -> None:
await self._on_message(message)
self._running = True
self.bus.subscribe_outbound(self._on_outbound)
self._thread = threading.Thread(target=self._run_client, daemon=True)
self._thread.start()
logger.info("Discord channel started")
async def stop(self) -> None:
self._running = False
self.bus.unsubscribe_outbound(self._on_outbound)
if self._client and self._discord_loop and self._discord_loop.is_running():
close_future = asyncio.run_coroutine_threadsafe(self._client.close(), self._discord_loop)
try:
await asyncio.wait_for(asyncio.wrap_future(close_future), timeout=10)
except TimeoutError:
logger.warning("[Discord] client close timed out after 10s")
except Exception:
logger.exception("[Discord] error while closing client")
if self._thread:
self._thread.join(timeout=10)
self._thread = None
self._client = None
self._discord_loop = None
self._discord_module = None
logger.info("Discord channel stopped")
async def send(self, msg: OutboundMessage) -> None:
target = await self._resolve_target(msg)
if target is None:
logger.error("[Discord] target not found for chat_id=%s thread_ts=%s", msg.chat_id, msg.thread_ts)
return
text = msg.text or ""
for chunk in self._split_text(text):
send_future = asyncio.run_coroutine_threadsafe(target.send(chunk), self._discord_loop)
await asyncio.wrap_future(send_future)
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
target = await self._resolve_target(msg)
if target is None:
logger.error("[Discord] target not found for file upload chat_id=%s thread_ts=%s", msg.chat_id, msg.thread_ts)
return False
if self._discord_module is None:
return False
try:
fp = open(str(attachment.actual_path), "rb") # noqa: SIM115
file = self._discord_module.File(fp, filename=attachment.filename)
send_future = asyncio.run_coroutine_threadsafe(target.send(file=file), self._discord_loop)
await asyncio.wrap_future(send_future)
logger.info("[Discord] file uploaded: %s", attachment.filename)
return True
except Exception:
logger.exception("[Discord] failed to upload file: %s", attachment.filename)
return False
async def _on_message(self, message) -> None:
if not self._running or not self._client:
return
if message.author.bot:
return
if self._client.user and message.author.id == self._client.user.id:
return
guild = message.guild
if self._allowed_guilds:
if guild is None or guild.id not in self._allowed_guilds:
return
text = (message.content or "").strip()
if not text:
return
if self._discord_module is None:
return
if isinstance(message.channel, self._discord_module.Thread):
chat_id = str(message.channel.parent_id or message.channel.id)
thread_id = str(message.channel.id)
else:
thread = await self._create_thread(message)
if thread is None:
return
chat_id = str(message.channel.id)
thread_id = str(thread.id)
msg_type = InboundMessageType.COMMAND if text.startswith("/") else InboundMessageType.CHAT
inbound = self._make_inbound(
chat_id=chat_id,
user_id=str(message.author.id),
text=text,
msg_type=msg_type,
thread_ts=thread_id,
metadata={
"guild_id": str(guild.id) if guild else None,
"channel_id": str(message.channel.id),
"message_id": str(message.id),
},
)
inbound.topic_id = thread_id
if self._main_loop and self._main_loop.is_running():
future = asyncio.run_coroutine_threadsafe(self.bus.publish_inbound(inbound), self._main_loop)
future.add_done_callback(lambda f: logger.exception("[Discord] publish_inbound failed", exc_info=f.exception()) if f.exception() else None)
def _run_client(self) -> None:
self._discord_loop = asyncio.new_event_loop()
asyncio.set_event_loop(self._discord_loop)
try:
self._discord_loop.run_until_complete(self._client.start(self._bot_token))
except Exception:
if self._running:
logger.exception("Discord client error")
finally:
try:
if self._client and not self._client.is_closed():
self._discord_loop.run_until_complete(self._client.close())
except Exception:
logger.exception("Error during Discord shutdown")
async def _create_thread(self, message):
try:
thread_name = f"deerflow-{message.author.display_name}-{message.id}"[:100]
return await message.create_thread(name=thread_name)
except Exception:
logger.exception("[Discord] failed to create thread for message=%s (threads may be disabled or missing permissions)", message.id)
try:
await message.channel.send("Could not create a thread for your message. Please check that threads are enabled in this channel.")
except Exception:
pass
return None
async def _resolve_target(self, msg: OutboundMessage):
if not self._client or not self._discord_loop:
return None
target_ids: list[str] = []
if msg.thread_ts:
target_ids.append(msg.thread_ts)
if msg.chat_id and msg.chat_id not in target_ids:
target_ids.append(msg.chat_id)
for raw_id in target_ids:
target = await self._get_channel_or_thread(raw_id)
if target is not None:
return target
return None
async def _get_channel_or_thread(self, raw_id: str):
if not self._client or not self._discord_loop:
return None
try:
target_id = int(raw_id)
except (TypeError, ValueError):
return None
get_future = asyncio.run_coroutine_threadsafe(self._fetch_channel(target_id), self._discord_loop)
try:
return await asyncio.wrap_future(get_future)
except Exception:
logger.exception("[Discord] failed to resolve target id=%s", raw_id)
return None
async def _fetch_channel(self, target_id: int):
if not self._client:
return None
channel = self._client.get_channel(target_id)
if channel is not None:
return channel
try:
return await self._client.fetch_channel(target_id)
except Exception:
return None
@staticmethod
def _split_text(text: str) -> list[str]:
if not text:
return [""]
chunks: list[str] = []
remaining = text
while len(remaining) > _DISCORD_MAX_MESSAGE_LEN:
split_at = remaining.rfind("\n", 0, _DISCORD_MAX_MESSAGE_LEN)
if split_at <= 0:
split_at = _DISCORD_MAX_MESSAGE_LEN
chunks.append(remaining[:split_at])
remaining = remaining[split_at:].lstrip("\n")
if remaining:
chunks.append(remaining)
return chunks
+698
View File
@@ -0,0 +1,698 @@
"""Feishu/Lark channel — connects to Feishu via WebSocket (no public IP needed)."""
from __future__ import annotations
import asyncio
import json
import logging
import re
import threading
from typing import Any, Literal
from app.channels.base import Channel
from app.channels.commands import KNOWN_CHANNEL_COMMANDS
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
from deerflow.config.paths import VIRTUAL_PATH_PREFIX, get_paths
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.sandbox.sandbox_provider import get_sandbox_provider
logger = logging.getLogger(__name__)
def _is_feishu_command(text: str) -> bool:
if not text.startswith("/"):
return False
return text.split(maxsplit=1)[0].lower() in KNOWN_CHANNEL_COMMANDS
class FeishuChannel(Channel):
"""Feishu/Lark IM channel using the ``lark-oapi`` WebSocket client.
Configuration keys (in ``config.yaml`` under ``channels.feishu``):
- ``app_id``: Feishu app ID.
- ``app_secret``: Feishu app secret.
- ``verification_token``: (optional) Event verification token.
The channel uses WebSocket long-connection mode so no public IP is required.
Message flow:
1. User sends a message → bot adds "OK" emoji reaction
2. Bot replies in thread: "Working on it......"
3. Agent processes the message and returns a result
4. Bot replies in thread with the result
5. Bot adds "DONE" emoji reaction to the original message
"""
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
super().__init__(name="feishu", bus=bus, config=config)
self._thread: threading.Thread | None = None
self._main_loop: asyncio.AbstractEventLoop | None = None
self._api_client = None
self._CreateMessageReactionRequest = None
self._CreateMessageReactionRequestBody = None
self._Emoji = None
self._PatchMessageRequest = None
self._PatchMessageRequestBody = None
self._background_tasks: set[asyncio.Task] = set()
self._running_card_ids: dict[str, str] = {}
self._running_card_tasks: dict[str, asyncio.Task] = {}
self._CreateFileRequest = None
self._CreateFileRequestBody = None
self._CreateImageRequest = None
self._CreateImageRequestBody = None
self._GetMessageResourceRequest = None
self._thread_lock = threading.Lock()
@property
def supports_streaming(self) -> bool:
return True
async def start(self) -> None:
if self._running:
return
try:
import lark_oapi as lark
from lark_oapi.api.im.v1 import (
CreateFileRequest,
CreateFileRequestBody,
CreateImageRequest,
CreateImageRequestBody,
CreateMessageReactionRequest,
CreateMessageReactionRequestBody,
CreateMessageRequest,
CreateMessageRequestBody,
Emoji,
GetMessageResourceRequest,
PatchMessageRequest,
PatchMessageRequestBody,
ReplyMessageRequest,
ReplyMessageRequestBody,
)
except ImportError:
logger.error("lark-oapi is not installed. Install it with: uv add lark-oapi")
return
self._lark = lark
self._CreateMessageRequest = CreateMessageRequest
self._CreateMessageRequestBody = CreateMessageRequestBody
self._ReplyMessageRequest = ReplyMessageRequest
self._ReplyMessageRequestBody = ReplyMessageRequestBody
self._CreateMessageReactionRequest = CreateMessageReactionRequest
self._CreateMessageReactionRequestBody = CreateMessageReactionRequestBody
self._Emoji = Emoji
self._PatchMessageRequest = PatchMessageRequest
self._PatchMessageRequestBody = PatchMessageRequestBody
self._CreateFileRequest = CreateFileRequest
self._CreateFileRequestBody = CreateFileRequestBody
self._CreateImageRequest = CreateImageRequest
self._CreateImageRequestBody = CreateImageRequestBody
self._GetMessageResourceRequest = GetMessageResourceRequest
app_id = self.config.get("app_id", "")
app_secret = self.config.get("app_secret", "")
domain = self.config.get("domain", "https://open.feishu.cn")
if not app_id or not app_secret:
logger.error("Feishu channel requires app_id and app_secret")
return
self._api_client = lark.Client.builder().app_id(app_id).app_secret(app_secret).domain(domain).build()
logger.info("[Feishu] using domain: %s", domain)
self._main_loop = asyncio.get_event_loop()
self._running = True
self.bus.subscribe_outbound(self._on_outbound)
# Both ws.Client construction and start() must happen in a dedicated
# thread with its own event loop. lark-oapi caches the running loop
# at construction time and later calls loop.run_until_complete(),
# which conflicts with an already-running uvloop.
self._thread = threading.Thread(
target=self._run_ws,
args=(app_id, app_secret, domain),
daemon=True,
)
self._thread.start()
logger.info("Feishu channel started")
def _run_ws(self, app_id: str, app_secret: str, domain: str) -> None:
"""Construct and run the lark WS client in a thread with a fresh event loop.
The lark-oapi SDK captures a module-level event loop at import time
(``lark_oapi.ws.client.loop``). When uvicorn uses uvloop, that
captured loop is the *main* thread's uvloop — which is already
running, so ``loop.run_until_complete()`` inside ``Client.start()``
raises ``RuntimeError``.
We work around this by creating a plain asyncio event loop for this
thread and patching the SDK's module-level reference before calling
``start()``.
"""
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
import lark_oapi as lark
import lark_oapi.ws.client as _ws_client_mod
# Replace the SDK's module-level loop so Client.start() uses
# this thread's (non-running) event loop instead of the main
# thread's uvloop.
_ws_client_mod.loop = loop
event_handler = lark.EventDispatcherHandler.builder("", "").register_p2_im_message_receive_v1(self._on_message).build()
ws_client = lark.ws.Client(
app_id=app_id,
app_secret=app_secret,
event_handler=event_handler,
log_level=lark.LogLevel.INFO,
domain=domain,
)
ws_client.start()
except Exception:
if self._running:
logger.exception("Feishu WebSocket error")
async def stop(self) -> None:
self._running = False
self.bus.unsubscribe_outbound(self._on_outbound)
for task in list(self._background_tasks):
task.cancel()
self._background_tasks.clear()
for task in list(self._running_card_tasks.values()):
task.cancel()
self._running_card_tasks.clear()
if self._thread:
self._thread.join(timeout=5)
self._thread = None
logger.info("Feishu channel stopped")
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
if not self._api_client:
logger.warning("[Feishu] send called but no api_client available")
return
logger.info(
"[Feishu] sending reply: chat_id=%s, thread_ts=%s, text_len=%d",
msg.chat_id,
msg.thread_ts,
len(msg.text),
)
last_exc: Exception | None = None
for attempt in range(_max_retries):
try:
await self._send_card_message(msg)
return # success
except Exception as exc:
last_exc = exc
if attempt < _max_retries - 1:
delay = 2**attempt # 1s, 2s
logger.warning(
"[Feishu] send failed (attempt %d/%d), retrying in %ds: %s",
attempt + 1,
_max_retries,
delay,
exc,
)
await asyncio.sleep(delay)
logger.error("[Feishu] send failed after %d attempts: %s", _max_retries, last_exc)
if last_exc is None:
raise RuntimeError("Feishu send failed without an exception from any attempt")
raise last_exc
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
if not self._api_client:
return False
# Check size limits (image: 10MB, file: 30MB)
if attachment.is_image and attachment.size > 10 * 1024 * 1024:
logger.warning("[Feishu] image too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
return False
if not attachment.is_image and attachment.size > 30 * 1024 * 1024:
logger.warning("[Feishu] file too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
return False
try:
if attachment.is_image:
file_key = await self._upload_image(attachment.actual_path)
msg_type = "image"
content = json.dumps({"image_key": file_key})
else:
file_key = await self._upload_file(attachment.actual_path, attachment.filename)
msg_type = "file"
content = json.dumps({"file_key": file_key})
if msg.thread_ts:
request = self._ReplyMessageRequest.builder().message_id(msg.thread_ts).request_body(self._ReplyMessageRequestBody.builder().msg_type(msg_type).content(content).reply_in_thread(True).build()).build()
await asyncio.to_thread(self._api_client.im.v1.message.reply, request)
else:
request = self._CreateMessageRequest.builder().receive_id_type("chat_id").request_body(self._CreateMessageRequestBody.builder().receive_id(msg.chat_id).msg_type(msg_type).content(content).build()).build()
await asyncio.to_thread(self._api_client.im.v1.message.create, request)
logger.info("[Feishu] file sent: %s (type=%s)", attachment.filename, msg_type)
return True
except Exception:
logger.exception("[Feishu] failed to upload/send file: %s", attachment.filename)
return False
async def _upload_image(self, path) -> str:
"""Upload an image to Feishu and return the image_key."""
with open(str(path), "rb") as f:
request = self._CreateImageRequest.builder().request_body(self._CreateImageRequestBody.builder().image_type("message").image(f).build()).build()
response = await asyncio.to_thread(self._api_client.im.v1.image.create, request)
if not response.success():
raise RuntimeError(f"Feishu image upload failed: code={response.code}, msg={response.msg}")
return response.data.image_key
async def _upload_file(self, path, filename: str) -> str:
"""Upload a file to Feishu and return the file_key."""
suffix = path.suffix.lower() if hasattr(path, "suffix") else ""
if suffix in (".xls", ".xlsx", ".csv"):
file_type = "xls"
elif suffix in (".ppt", ".pptx"):
file_type = "ppt"
elif suffix == ".pdf":
file_type = "pdf"
elif suffix in (".doc", ".docx"):
file_type = "doc"
else:
file_type = "stream"
with open(str(path), "rb") as f:
request = self._CreateFileRequest.builder().request_body(self._CreateFileRequestBody.builder().file_type(file_type).file_name(filename).file(f).build()).build()
response = await asyncio.to_thread(self._api_client.im.v1.file.create, request)
if not response.success():
raise RuntimeError(f"Feishu file upload failed: code={response.code}, msg={response.msg}")
return response.data.file_key
async def receive_file(self, msg: InboundMessage, thread_id: str) -> InboundMessage:
"""Download a Feishu file into the thread uploads directory.
Returns the sandbox virtual path when the image is persisted successfully.
"""
if not msg.thread_ts:
logger.warning("[Feishu] received file message without thread_ts, cannot associate with conversation: %s", msg)
return msg
files = msg.files
if not files:
logger.warning("[Feishu] received message with no files: %s", msg)
return msg
text = msg.text
for file in files:
if file.get("image_key"):
virtual_path = await self._receive_single_file(msg.thread_ts, file["image_key"], "image", thread_id)
text = text.replace("[image]", virtual_path, 1)
elif file.get("file_key"):
virtual_path = await self._receive_single_file(msg.thread_ts, file["file_key"], "file", thread_id)
text = text.replace("[file]", virtual_path, 1)
msg.text = text
return msg
async def _receive_single_file(self, message_id: str, file_key: str, type: Literal["image", "file"], thread_id: str) -> str:
request = self._GetMessageResourceRequest.builder().message_id(message_id).file_key(file_key).type(type).build()
def inner():
return self._api_client.im.v1.message_resource.get(request)
try:
response = await asyncio.to_thread(inner)
except Exception:
logger.exception("[Feishu] resource get request failed for resource_key=%s type=%s", file_key, type)
return f"Failed to obtain the [{type}]"
if not response.success():
logger.warning(
"[Feishu] resource get failed: resource_key=%s, type=%s, code=%s, msg=%s, log_id=%s ",
file_key,
type,
response.code,
response.msg,
response.get_log_id(),
)
return f"Failed to obtain the [{type}]"
image_stream = getattr(response, "file", None)
if image_stream is None:
logger.warning("[Feishu] resource get returned no file stream: resource_key=%s, type=%s", file_key, type)
return f"Failed to obtain the [{type}]"
try:
content: bytes = await asyncio.to_thread(image_stream.read)
except Exception:
logger.exception("[Feishu] failed to read resource stream: resource_key=%s, type=%s", file_key, type)
return f"Failed to obtain the [{type}]"
if not content:
logger.warning("[Feishu] empty resource content: resource_key=%s, type=%s", file_key, type)
return f"Failed to obtain the [{type}]"
paths = get_paths()
user_id = get_effective_user_id()
paths.ensure_thread_dirs(thread_id, user_id=user_id)
uploads_dir = paths.sandbox_uploads_dir(thread_id, user_id=user_id).resolve()
ext = "png" if type == "image" else "bin"
raw_filename = getattr(response, "file_name", "") or f"feishu_{file_key[-12:]}.{ext}"
# Sanitize filename: preserve extension, replace path chars in name part
if "." in raw_filename:
name_part, ext = raw_filename.rsplit(".", 1)
name_part = re.sub(r"[./\\]", "_", name_part)
filename = f"{name_part}.{ext}"
else:
filename = re.sub(r"[./\\]", "_", raw_filename)
resolved_target = uploads_dir / filename
def down_load():
# use thread_lock to avoid filename conflicts when writing
with self._thread_lock:
resolved_target.write_bytes(content)
try:
await asyncio.to_thread(down_load)
except Exception:
logger.exception("[Feishu] failed to persist downloaded resource: %s, type=%s", resolved_target, type)
return f"Failed to obtain the [{type}]"
virtual_path = f"{VIRTUAL_PATH_PREFIX}/uploads/{resolved_target.name}"
try:
sandbox_provider = get_sandbox_provider()
sandbox_id = sandbox_provider.acquire(thread_id)
if sandbox_id != "local":
sandbox = sandbox_provider.get(sandbox_id)
if sandbox is None:
logger.warning("[Feishu] sandbox not found for thread_id=%s", thread_id)
return f"Failed to obtain the [{type}]"
sandbox.update_file(virtual_path, content)
except Exception:
logger.exception("[Feishu] failed to sync resource into non-local sandbox: %s", virtual_path)
return f"Failed to obtain the [{type}]"
logger.info("[Feishu] downloaded resource mapped: file_key=%s -> %s", file_key, virtual_path)
return virtual_path
# -- message formatting ------------------------------------------------
@staticmethod
def _build_card_content(text: str) -> str:
"""Build a Feishu interactive card with markdown content.
Feishu's interactive card format natively renders markdown, including
headers, bold/italic, code blocks, lists, and links.
"""
card = {
"config": {"wide_screen_mode": True, "update_multi": True},
"elements": [{"tag": "markdown", "content": text}],
}
return json.dumps(card)
# -- reaction helpers --------------------------------------------------
async def _add_reaction(self, message_id: str, emoji_type: str = "THUMBSUP") -> None:
"""Add an emoji reaction to a message."""
if not self._api_client or not self._CreateMessageReactionRequest:
return
try:
request = self._CreateMessageReactionRequest.builder().message_id(message_id).request_body(self._CreateMessageReactionRequestBody.builder().reaction_type(self._Emoji.builder().emoji_type(emoji_type).build()).build()).build()
await asyncio.to_thread(self._api_client.im.v1.message_reaction.create, request)
logger.info("[Feishu] reaction '%s' added to message %s", emoji_type, message_id)
except Exception:
logger.exception("[Feishu] failed to add reaction '%s' to message %s", emoji_type, message_id)
async def _reply_card(self, message_id: str, text: str) -> str | None:
"""Reply with an interactive card and return the created card message ID."""
if not self._api_client:
return None
content = self._build_card_content(text)
request = self._ReplyMessageRequest.builder().message_id(message_id).request_body(self._ReplyMessageRequestBody.builder().msg_type("interactive").content(content).reply_in_thread(True).build()).build()
response = await asyncio.to_thread(self._api_client.im.v1.message.reply, request)
response_data = getattr(response, "data", None)
return getattr(response_data, "message_id", None)
async def _create_card(self, chat_id: str, text: str) -> None:
"""Create a new card message in the target chat."""
if not self._api_client:
return
content = self._build_card_content(text)
request = self._CreateMessageRequest.builder().receive_id_type("chat_id").request_body(self._CreateMessageRequestBody.builder().receive_id(chat_id).msg_type("interactive").content(content).build()).build()
await asyncio.to_thread(self._api_client.im.v1.message.create, request)
async def _update_card(self, message_id: str, text: str) -> None:
"""Patch an existing card message in place."""
if not self._api_client or not self._PatchMessageRequest:
return
content = self._build_card_content(text)
request = self._PatchMessageRequest.builder().message_id(message_id).request_body(self._PatchMessageRequestBody.builder().content(content).build()).build()
await asyncio.to_thread(self._api_client.im.v1.message.patch, request)
def _track_background_task(self, task: asyncio.Task, *, name: str, msg_id: str) -> None:
"""Keep a strong reference to fire-and-forget tasks and surface errors."""
self._background_tasks.add(task)
task.add_done_callback(lambda done_task, task_name=name, mid=msg_id: self._finalize_background_task(done_task, task_name, mid))
def _finalize_background_task(self, task: asyncio.Task, name: str, msg_id: str) -> None:
self._background_tasks.discard(task)
self._log_task_error(task, name, msg_id)
async def _create_running_card(self, source_message_id: str, text: str) -> str | None:
"""Create the running card and cache its message ID when available."""
running_card_id = await self._reply_card(source_message_id, text)
if running_card_id:
self._running_card_ids[source_message_id] = running_card_id
logger.info("[Feishu] running card created: source=%s card=%s", source_message_id, running_card_id)
else:
logger.warning("[Feishu] running card creation returned no message_id for source=%s, subsequent updates will fall back to new replies", source_message_id)
return running_card_id
def _ensure_running_card_started(self, source_message_id: str, text: str = "Working on it...") -> asyncio.Task | None:
"""Start running-card creation once per source message."""
running_card_id = self._running_card_ids.get(source_message_id)
if running_card_id:
return None
running_card_task = self._running_card_tasks.get(source_message_id)
if running_card_task:
return running_card_task
running_card_task = asyncio.create_task(self._create_running_card(source_message_id, text))
self._running_card_tasks[source_message_id] = running_card_task
running_card_task.add_done_callback(lambda done_task, mid=source_message_id: self._finalize_running_card_task(mid, done_task))
return running_card_task
def _finalize_running_card_task(self, source_message_id: str, task: asyncio.Task) -> None:
if self._running_card_tasks.get(source_message_id) is task:
self._running_card_tasks.pop(source_message_id, None)
self._log_task_error(task, "create_running_card", source_message_id)
async def _ensure_running_card(self, source_message_id: str, text: str = "Working on it...") -> str | None:
"""Ensure the in-thread running card exists and track its message ID."""
running_card_id = self._running_card_ids.get(source_message_id)
if running_card_id:
return running_card_id
running_card_task = self._ensure_running_card_started(source_message_id, text)
if running_card_task is None:
return self._running_card_ids.get(source_message_id)
return await running_card_task
async def _send_running_reply(self, message_id: str) -> None:
"""Reply to a message in-thread with a running card."""
try:
await self._ensure_running_card(message_id)
except Exception:
logger.exception("[Feishu] failed to send running reply for message %s", message_id)
async def _send_card_message(self, msg: OutboundMessage) -> None:
"""Send or update the Feishu card tied to the current request."""
source_message_id = msg.thread_ts
if source_message_id:
running_card_id = self._running_card_ids.get(source_message_id)
awaited_running_card_task = False
if not running_card_id:
running_card_task = self._running_card_tasks.get(source_message_id)
if running_card_task:
awaited_running_card_task = True
running_card_id = await running_card_task
if running_card_id:
try:
await self._update_card(running_card_id, msg.text)
except Exception:
if not msg.is_final:
raise
logger.exception(
"[Feishu] failed to patch running card %s, falling back to final reply",
running_card_id,
)
await self._reply_card(source_message_id, msg.text)
else:
logger.info("[Feishu] running card updated: source=%s card=%s", source_message_id, running_card_id)
elif msg.is_final:
await self._reply_card(source_message_id, msg.text)
elif awaited_running_card_task:
logger.warning(
"[Feishu] running card task finished without message_id for source=%s, skipping duplicate non-final creation",
source_message_id,
)
else:
await self._ensure_running_card(source_message_id, msg.text)
if msg.is_final:
self._running_card_ids.pop(source_message_id, None)
await self._add_reaction(source_message_id, "DONE")
return
await self._create_card(msg.chat_id, msg.text)
# -- internal ----------------------------------------------------------
@staticmethod
def _log_future_error(fut, name: str, msg_id: str) -> None:
"""Callback for run_coroutine_threadsafe futures to surface errors."""
try:
exc = fut.exception()
if exc:
logger.error("[Feishu] %s failed for msg_id=%s: %s", name, msg_id, exc)
except Exception:
pass
@staticmethod
def _log_task_error(task: asyncio.Task, name: str, msg_id: str) -> None:
"""Callback for background asyncio tasks to surface errors."""
try:
exc = task.exception()
if exc:
logger.error("[Feishu] %s failed for msg_id=%s: %s", name, msg_id, exc)
except asyncio.CancelledError:
logger.info("[Feishu] %s cancelled for msg_id=%s", name, msg_id)
except Exception:
pass
async def _prepare_inbound(self, msg_id: str, inbound) -> None:
"""Kick off Feishu side effects without delaying inbound dispatch."""
reaction_task = asyncio.create_task(self._add_reaction(msg_id, "OK"))
self._track_background_task(reaction_task, name="add_reaction", msg_id=msg_id)
self._ensure_running_card_started(msg_id)
await self.bus.publish_inbound(inbound)
def _on_message(self, event) -> None:
"""Called by lark-oapi when a message is received (runs in lark thread)."""
try:
logger.info("[Feishu] raw event received: type=%s", type(event).__name__)
message = event.event.message
chat_id = message.chat_id
msg_id = message.message_id
sender_id = event.event.sender.sender_id.open_id
# root_id is set when the message is a reply within a Feishu thread.
# Use it as topic_id so all replies share the same DeerFlow thread.
root_id = getattr(message, "root_id", None) or None
# Parse message content
content = json.loads(message.content)
# files_list store the any-file-key in feishu messages, which can be used to download the file content later
# In Feishu channel, image_keys are independent of file_keys.
# The file_key includes files, videos, and audio, but does not include stickers.
files_list = []
if "text" in content:
# Handle plain text messages
text = content["text"]
elif "file_key" in content:
file_key = content.get("file_key")
if isinstance(file_key, str) and file_key:
files_list.append({"file_key": file_key})
text = "[file]"
else:
text = ""
elif "image_key" in content:
image_key = content.get("image_key")
if isinstance(image_key, str) and image_key:
files_list.append({"image_key": image_key})
text = "[image]"
else:
text = ""
elif "content" in content and isinstance(content["content"], list):
# Handle rich-text messages with a top-level "content" list (e.g., topic groups/posts)
text_paragraphs: list[str] = []
for paragraph in content["content"]:
if isinstance(paragraph, list):
paragraph_text_parts: list[str] = []
for element in paragraph:
if isinstance(element, dict):
# Include both normal text and @ mentions
if element.get("tag") in ("text", "at"):
text_value = element.get("text", "")
if text_value:
paragraph_text_parts.append(text_value)
elif element.get("tag") == "img":
image_key = element.get("image_key")
if isinstance(image_key, str) and image_key:
files_list.append({"image_key": image_key})
paragraph_text_parts.append("[image]")
elif element.get("tag") in ("file", "media"):
file_key = element.get("file_key")
if isinstance(file_key, str) and file_key:
files_list.append({"file_key": file_key})
paragraph_text_parts.append("[file]")
if paragraph_text_parts:
# Join text segments within a paragraph with spaces to avoid "helloworld"
text_paragraphs.append(" ".join(paragraph_text_parts))
# Join paragraphs with blank lines to preserve paragraph boundaries
text = "\n\n".join(text_paragraphs)
else:
text = ""
text = text.strip()
logger.info(
"[Feishu] parsed message: chat_id=%s, msg_id=%s, root_id=%s, sender=%s, text=%r",
chat_id,
msg_id,
root_id,
sender_id,
text[:100] if text else "",
)
if not (text or files_list):
logger.info("[Feishu] empty text, ignoring message")
return
# Only treat known slash commands as commands; absolute paths and
# other slash-prefixed text should be handled as normal chat.
if _is_feishu_command(text):
msg_type = InboundMessageType.COMMAND
else:
msg_type = InboundMessageType.CHAT
# topic_id: use root_id for replies (same topic), msg_id for new messages (new topic)
topic_id = root_id or msg_id
inbound = self._make_inbound(
chat_id=chat_id,
user_id=sender_id,
text=text,
msg_type=msg_type,
thread_ts=msg_id,
files=files_list,
metadata={"message_id": msg_id, "root_id": root_id},
)
inbound.topic_id = topic_id
# Schedule on the async event loop
if self._main_loop and self._main_loop.is_running():
logger.info("[Feishu] publishing inbound message to bus (type=%s, msg_id=%s)", msg_type.value, msg_id)
fut = asyncio.run_coroutine_threadsafe(self._prepare_inbound(msg_id, inbound), self._main_loop)
fut.add_done_callback(lambda f, mid=msg_id: self._log_future_error(f, "prepare_inbound", mid))
else:
logger.warning("[Feishu] main loop not running, cannot publish inbound message")
except Exception:
logger.exception("[Feishu] error processing message")
File diff suppressed because it is too large Load Diff
+173
View File
@@ -0,0 +1,173 @@
"""MessageBus — async pub/sub hub that decouples channels from the agent dispatcher."""
from __future__ import annotations
import asyncio
import logging
import time
from collections.abc import Callable, Coroutine
from dataclasses import dataclass, field
from enum import StrEnum
from pathlib import Path
from typing import Any
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Message types
# ---------------------------------------------------------------------------
class InboundMessageType(StrEnum):
"""Types of messages arriving from IM channels."""
CHAT = "chat"
COMMAND = "command"
@dataclass
class InboundMessage:
"""A message arriving from an IM channel toward the agent dispatcher.
Attributes:
channel_name: Name of the source channel (e.g. "feishu", "slack").
chat_id: Platform-specific chat/conversation identifier.
user_id: Platform-specific user identifier.
text: The message text.
msg_type: Whether this is a regular chat message or a command.
thread_ts: Optional platform thread identifier (for threaded replies).
topic_id: Conversation topic identifier used to map to a DeerFlow thread.
Messages sharing the same ``topic_id`` within a ``chat_id`` will
reuse the same DeerFlow thread. When ``None``, each message
creates a new thread (one-shot Q&A).
files: Optional list of file attachments (platform-specific dicts).
metadata: Arbitrary extra data from the channel.
created_at: Unix timestamp when the message was created.
"""
channel_name: str
chat_id: str
user_id: str
text: str
msg_type: InboundMessageType = InboundMessageType.CHAT
thread_ts: str | None = None
topic_id: str | None = None
files: list[dict[str, Any]] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
created_at: float = field(default_factory=time.time)
@dataclass
class ResolvedAttachment:
"""A file attachment resolved to a host filesystem path, ready for upload.
Attributes:
virtual_path: Original virtual path (e.g. /mnt/user-data/outputs/report.pdf).
actual_path: Resolved host filesystem path.
filename: Basename of the file.
mime_type: MIME type (e.g. "application/pdf").
size: File size in bytes.
is_image: True for image/* MIME types (platforms may handle images differently).
"""
virtual_path: str
actual_path: Path
filename: str
mime_type: str
size: int
is_image: bool
@dataclass
class OutboundMessage:
"""A message from the agent dispatcher back to a channel.
Attributes:
channel_name: Target channel name (used for routing).
chat_id: Target chat/conversation identifier.
thread_id: DeerFlow thread ID that produced this response.
text: The response text.
artifacts: List of artifact paths produced by the agent.
is_final: Whether this is the final message in the response stream.
thread_ts: Optional platform thread identifier for threaded replies.
metadata: Arbitrary extra data.
created_at: Unix timestamp.
"""
channel_name: str
chat_id: str
thread_id: str
text: str
artifacts: list[str] = field(default_factory=list)
attachments: list[ResolvedAttachment] = field(default_factory=list)
is_final: bool = True
thread_ts: str | None = None
metadata: dict[str, Any] = field(default_factory=dict)
created_at: float = field(default_factory=time.time)
# ---------------------------------------------------------------------------
# MessageBus
# ---------------------------------------------------------------------------
OutboundCallback = Callable[[OutboundMessage], Coroutine[Any, Any, None]]
class MessageBus:
"""Async pub/sub hub connecting channels and the agent dispatcher.
Channels publish inbound messages; the dispatcher consumes them.
The dispatcher publishes outbound messages; channels receive them
via registered callbacks.
"""
def __init__(self) -> None:
self._inbound_queue: asyncio.Queue[InboundMessage] = asyncio.Queue()
self._outbound_listeners: list[OutboundCallback] = []
# -- inbound -----------------------------------------------------------
async def publish_inbound(self, msg: InboundMessage) -> None:
"""Enqueue an inbound message from a channel."""
await self._inbound_queue.put(msg)
logger.info(
"[Bus] inbound enqueued: channel=%s, chat_id=%s, type=%s, queue_size=%d",
msg.channel_name,
msg.chat_id,
msg.msg_type.value,
self._inbound_queue.qsize(),
)
async def get_inbound(self) -> InboundMessage:
"""Block until the next inbound message is available."""
return await self._inbound_queue.get()
@property
def inbound_queue(self) -> asyncio.Queue[InboundMessage]:
return self._inbound_queue
# -- outbound ----------------------------------------------------------
def subscribe_outbound(self, callback: OutboundCallback) -> None:
"""Register an async callback for outbound messages."""
self._outbound_listeners.append(callback)
def unsubscribe_outbound(self, callback: OutboundCallback) -> None:
"""Remove a previously registered outbound callback."""
self._outbound_listeners = [cb for cb in self._outbound_listeners if cb is not callback]
async def publish_outbound(self, msg: OutboundMessage) -> None:
"""Dispatch an outbound message to all registered listeners."""
logger.info(
"[Bus] outbound dispatching: channel=%s, chat_id=%s, listeners=%d, text_len=%d",
msg.channel_name,
msg.chat_id,
len(self._outbound_listeners),
len(msg.text),
)
for callback in self._outbound_listeners:
try:
await callback(msg)
except Exception:
logger.exception("Error in outbound callback for channel=%s", msg.channel_name)
+230
View File
@@ -0,0 +1,230 @@
"""ChannelService — manages the lifecycle of all IM channels."""
from __future__ import annotations
import logging
import os
from typing import TYPE_CHECKING, Any
from app.channels.base import Channel
from app.channels.manager import DEFAULT_GATEWAY_URL, DEFAULT_LANGGRAPH_URL, ChannelManager
from app.channels.message_bus import MessageBus
from app.channels.store import ChannelStore
logger = logging.getLogger(__name__)
if TYPE_CHECKING:
from deerflow.config.app_config import AppConfig
# Channel name → import path for lazy loading
_CHANNEL_REGISTRY: dict[str, str] = {
"dingtalk": "app.channels.dingtalk:DingTalkChannel",
"discord": "app.channels.discord:DiscordChannel",
"feishu": "app.channels.feishu:FeishuChannel",
"slack": "app.channels.slack:SlackChannel",
"telegram": "app.channels.telegram:TelegramChannel",
"wechat": "app.channels.wechat:WechatChannel",
"wecom": "app.channels.wecom:WeComChannel",
}
# Keys that indicate a user has configured credentials for a channel.
_CHANNEL_CREDENTIAL_KEYS: dict[str, list[str]] = {
"dingtalk": ["client_id", "client_secret"],
"discord": ["bot_token"],
"feishu": ["app_id", "app_secret"],
"slack": ["bot_token", "app_token"],
"telegram": ["bot_token"],
"wecom": ["bot_id", "bot_secret"],
"wechat": ["bot_token"],
}
_CHANNELS_LANGGRAPH_URL_ENV = "DEER_FLOW_CHANNELS_LANGGRAPH_URL"
_CHANNELS_GATEWAY_URL_ENV = "DEER_FLOW_CHANNELS_GATEWAY_URL"
def _resolve_service_url(config: dict[str, Any], config_key: str, env_key: str, default: str) -> str:
value = config.pop(config_key, None)
if isinstance(value, str) and value.strip():
return value
env_value = os.getenv(env_key, "").strip()
if env_value:
return env_value
return default
class ChannelService:
"""Manages the lifecycle of all configured IM channels.
Reads configuration from ``config.yaml`` under the ``channels`` key,
instantiates enabled channels, and starts the ChannelManager dispatcher.
"""
def __init__(self, channels_config: dict[str, Any] | None = None) -> None:
self.bus = MessageBus()
self.store = ChannelStore()
config = dict(channels_config or {})
langgraph_url = _resolve_service_url(config, "langgraph_url", _CHANNELS_LANGGRAPH_URL_ENV, DEFAULT_LANGGRAPH_URL)
gateway_url = _resolve_service_url(config, "gateway_url", _CHANNELS_GATEWAY_URL_ENV, DEFAULT_GATEWAY_URL)
default_session = config.pop("session", None)
channel_sessions = {name: channel_config.get("session") for name, channel_config in config.items() if isinstance(channel_config, dict)}
self.manager = ChannelManager(
bus=self.bus,
store=self.store,
langgraph_url=langgraph_url,
gateway_url=gateway_url,
default_session=default_session if isinstance(default_session, dict) else None,
channel_sessions=channel_sessions,
)
self._channels: dict[str, Any] = {} # name -> Channel instance
self._config = config
self._running = False
@classmethod
def from_app_config(cls, app_config: AppConfig | None = None) -> ChannelService:
"""Create a ChannelService from the application config."""
if app_config is None:
from deerflow.config.app_config import get_app_config
app_config = get_app_config()
channels_config = {}
# extra fields are allowed by AppConfig (extra="allow")
extra = app_config.model_extra or {}
if "channels" in extra:
channels_config = extra["channels"]
return cls(channels_config=channels_config)
async def start(self) -> None:
"""Start the manager and all enabled channels."""
if self._running:
return
await self.manager.start()
for name, channel_config in self._config.items():
if not isinstance(channel_config, dict):
continue
if not channel_config.get("enabled", False):
cred_keys = _CHANNEL_CREDENTIAL_KEYS.get(name, [])
has_creds = any(not isinstance(channel_config.get(k), bool) and channel_config.get(k) is not None and str(channel_config[k]).strip() for k in cred_keys)
if has_creds:
logger.warning(
"Channel '%s' has credentials configured but is disabled. Set enabled: true under channels.%s in config.yaml to activate it.",
name,
name,
)
else:
logger.info("Channel %s is disabled, skipping", name)
continue
await self._start_channel(name, channel_config)
self._running = True
logger.info("ChannelService started with channels: %s", list(self._channels.keys()))
async def stop(self) -> None:
"""Stop all channels and the manager."""
for name, channel in list(self._channels.items()):
try:
await channel.stop()
logger.info("Channel %s stopped", name)
except Exception:
logger.exception("Error stopping channel %s", name)
self._channels.clear()
await self.manager.stop()
self._running = False
logger.info("ChannelService stopped")
async def restart_channel(self, name: str) -> bool:
"""Restart a specific channel. Returns True if successful."""
if name in self._channels:
try:
await self._channels[name].stop()
except Exception:
logger.exception("Error stopping channel %s for restart", name)
del self._channels[name]
config = self._config.get(name)
if not config or not isinstance(config, dict):
logger.warning("No config for channel %s", name)
return False
return await self._start_channel(name, config)
async def _start_channel(self, name: str, config: dict[str, Any]) -> bool:
"""Instantiate and start a single channel."""
import_path = _CHANNEL_REGISTRY.get(name)
if not import_path:
logger.warning("Unknown channel type: %s", name)
return False
try:
from deerflow.reflection import resolve_class
channel_cls = resolve_class(import_path, base_class=None)
except Exception:
logger.exception("Failed to import channel class for %s", name)
return False
try:
channel = channel_cls(bus=self.bus, config=config)
self._channels[name] = channel
await channel.start()
if not channel.is_running:
self._channels.pop(name, None)
logger.error("Channel %s did not enter a running state after start()", name)
return False
logger.info("Channel %s started", name)
return True
except Exception:
self._channels.pop(name, None)
logger.exception("Failed to start channel %s", name)
return False
def get_status(self) -> dict[str, Any]:
"""Return status information for all channels."""
channels_status = {}
for name in _CHANNEL_REGISTRY:
config = self._config.get(name, {})
enabled = isinstance(config, dict) and config.get("enabled", False)
running = name in self._channels and self._channels[name].is_running
channels_status[name] = {
"enabled": enabled,
"running": running,
}
return {
"service_running": self._running,
"channels": channels_status,
}
def get_channel(self, name: str) -> Channel | None:
"""Return a running channel instance by name when available."""
return self._channels.get(name)
# -- singleton access -------------------------------------------------------
_channel_service: ChannelService | None = None
def get_channel_service() -> ChannelService | None:
"""Get the singleton ChannelService instance (if started)."""
return _channel_service
async def start_channel_service(app_config: AppConfig | None = None) -> ChannelService:
"""Create and start the global ChannelService from app config."""
global _channel_service
if _channel_service is not None:
return _channel_service
_channel_service = ChannelService.from_app_config(app_config)
await _channel_service.start()
return _channel_service
async def stop_channel_service() -> None:
"""Stop the global ChannelService."""
global _channel_service
if _channel_service is not None:
await _channel_service.stop()
_channel_service = None
+264
View File
@@ -0,0 +1,264 @@
"""Slack channel — connects via Socket Mode (no public IP needed)."""
from __future__ import annotations
import asyncio
import logging
from typing import Any
from markdown_to_mrkdwn import SlackMarkdownConverter
from app.channels.base import Channel
from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
logger = logging.getLogger(__name__)
_slack_md_converter = SlackMarkdownConverter()
def _normalize_allowed_users(allowed_users: Any) -> set[str]:
if allowed_users is None:
return set()
if isinstance(allowed_users, str):
values = [allowed_users]
elif isinstance(allowed_users, list | tuple | set):
values = allowed_users
else:
logger.warning(
"Slack allowed_users should be a list of Slack user IDs or a single Slack user ID string; treating %s as one string value",
type(allowed_users).__name__,
)
values = [allowed_users]
return {str(user_id) for user_id in values if str(user_id)}
class SlackChannel(Channel):
"""Slack IM channel using Socket Mode (WebSocket, no public IP).
Configuration keys (in ``config.yaml`` under ``channels.slack``):
- ``bot_token``: Slack Bot User OAuth Token (xoxb-...).
- ``app_token``: Slack App-Level Token (xapp-...) for Socket Mode.
- ``allowed_users``: (optional) List of allowed Slack user IDs, or a
single Slack user ID string as shorthand. Empty = allow all. Other
scalar values are treated as a single string with a warning.
"""
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
super().__init__(name="slack", bus=bus, config=config)
self._socket_client = None
self._web_client = None
self._loop: asyncio.AbstractEventLoop | None = None
self._allowed_users = _normalize_allowed_users(config.get("allowed_users", []))
async def start(self) -> None:
if self._running:
return
try:
from slack_sdk import WebClient
from slack_sdk.socket_mode import SocketModeClient
from slack_sdk.socket_mode.response import SocketModeResponse
except ImportError:
logger.error("slack-sdk is not installed. Install it with: uv add slack-sdk")
return
self._SocketModeResponse = SocketModeResponse
bot_token = self.config.get("bot_token", "")
app_token = self.config.get("app_token", "")
if not bot_token or not app_token:
logger.error("Slack channel requires bot_token and app_token")
return
self._web_client = WebClient(token=bot_token)
self._socket_client = SocketModeClient(
app_token=app_token,
web_client=self._web_client,
)
self._loop = asyncio.get_event_loop()
self._socket_client.socket_mode_request_listeners.append(self._on_socket_event)
self._running = True
self.bus.subscribe_outbound(self._on_outbound)
# Start socket mode in background thread
asyncio.get_event_loop().run_in_executor(None, self._socket_client.connect)
logger.info("Slack channel started")
async def stop(self) -> None:
self._running = False
self.bus.unsubscribe_outbound(self._on_outbound)
if self._socket_client:
self._socket_client.close()
self._socket_client = None
logger.info("Slack channel stopped")
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
if not self._web_client:
return
kwargs: dict[str, Any] = {
"channel": msg.chat_id,
"text": _slack_md_converter.convert(msg.text),
}
if msg.thread_ts:
kwargs["thread_ts"] = msg.thread_ts
last_exc: Exception | None = None
for attempt in range(_max_retries):
try:
await asyncio.to_thread(self._web_client.chat_postMessage, **kwargs)
# Add a completion reaction to the thread root
if msg.thread_ts:
await asyncio.to_thread(
self._add_reaction,
msg.chat_id,
msg.thread_ts,
"white_check_mark",
)
return
except Exception as exc:
last_exc = exc
if attempt < _max_retries - 1:
delay = 2**attempt # 1s, 2s
logger.warning(
"[Slack] send failed (attempt %d/%d), retrying in %ds: %s",
attempt + 1,
_max_retries,
delay,
exc,
)
await asyncio.sleep(delay)
logger.error("[Slack] send failed after %d attempts: %s", _max_retries, last_exc)
# Add failure reaction on error
if msg.thread_ts:
try:
await asyncio.to_thread(
self._add_reaction,
msg.chat_id,
msg.thread_ts,
"x",
)
except Exception:
pass
if last_exc is None:
raise RuntimeError("Slack send failed without an exception from any attempt")
raise last_exc
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
if not self._web_client:
return False
try:
kwargs: dict[str, Any] = {
"channel": msg.chat_id,
"file": str(attachment.actual_path),
"filename": attachment.filename,
"title": attachment.filename,
}
if msg.thread_ts:
kwargs["thread_ts"] = msg.thread_ts
await asyncio.to_thread(self._web_client.files_upload_v2, **kwargs)
logger.info("[Slack] file uploaded: %s to channel=%s", attachment.filename, msg.chat_id)
return True
except Exception:
logger.exception("[Slack] failed to upload file: %s", attachment.filename)
return False
# -- internal ----------------------------------------------------------
def _add_reaction(self, channel_id: str, timestamp: str, emoji: str) -> None:
"""Add an emoji reaction to a message (best-effort, non-blocking)."""
if not self._web_client:
return
try:
self._web_client.reactions_add(
channel=channel_id,
timestamp=timestamp,
name=emoji,
)
except Exception as exc:
if "already_reacted" not in str(exc):
logger.warning("[Slack] failed to add reaction %s: %s", emoji, exc)
def _send_running_reply(self, channel_id: str, thread_ts: str) -> None:
"""Send a 'Working on it......' reply in the thread (called from SDK thread)."""
if not self._web_client:
return
try:
self._web_client.chat_postMessage(
channel=channel_id,
text=":hourglass_flowing_sand: Working on it...",
thread_ts=thread_ts,
)
logger.info("[Slack] 'Working on it...' reply sent in channel=%s, thread_ts=%s", channel_id, thread_ts)
except Exception:
logger.exception("[Slack] failed to send running reply in channel=%s", channel_id)
def _on_socket_event(self, client, req) -> None:
"""Called by slack-sdk for each Socket Mode event."""
try:
# Acknowledge the event
response = self._SocketModeResponse(envelope_id=req.envelope_id)
client.send_socket_mode_response(response)
event_type = req.type
if event_type != "events_api":
return
event = req.payload.get("event", {})
etype = event.get("type", "")
# Handle message events (DM or @mention)
if etype in ("message", "app_mention"):
self._handle_message_event(event)
except Exception:
logger.exception("Error processing Slack event")
def _handle_message_event(self, event: dict) -> None:
# Ignore bot messages
if event.get("bot_id") or event.get("subtype"):
return
user_id = event.get("user", "")
# Check allowed users
if self._allowed_users and user_id not in self._allowed_users:
logger.debug("Ignoring message from non-allowed user: %s", user_id)
return
text = event.get("text", "").strip()
if not text:
return
channel_id = event.get("channel", "")
thread_ts = event.get("thread_ts") or event.get("ts", "")
if text.startswith("/"):
msg_type = InboundMessageType.COMMAND
else:
msg_type = InboundMessageType.CHAT
# topic_id: use thread_ts as the topic identifier.
# For threaded messages, thread_ts is the root message ts (shared topic).
# For non-threaded messages, thread_ts is the message's own ts (new topic).
inbound = self._make_inbound(
chat_id=channel_id,
user_id=user_id,
text=text,
msg_type=msg_type,
thread_ts=thread_ts,
)
inbound.topic_id = thread_ts
if self._loop and self._loop.is_running():
# Acknowledge with an eyes reaction
self._add_reaction(channel_id, event.get("ts", thread_ts), "eyes")
# Send "running" reply first (fire-and-forget from SDK thread)
self._send_running_reply(channel_id, thread_ts)
asyncio.run_coroutine_threadsafe(self.bus.publish_inbound(inbound), self._loop)
+153
View File
@@ -0,0 +1,153 @@
"""ChannelStore — persists IM chat-to-DeerFlow thread mappings."""
from __future__ import annotations
import json
import logging
import tempfile
import threading
import time
from pathlib import Path
from typing import Any
logger = logging.getLogger(__name__)
class ChannelStore:
"""JSON-file-backed store that maps IM conversations to DeerFlow threads.
Data layout (on disk)::
{
"<channel_name>:<chat_id>": {
"thread_id": "<uuid>",
"user_id": "<platform_user>",
"created_at": 1700000000.0,
"updated_at": 1700000000.0
},
...
}
The store is intentionally simple — a single JSON file that is atomically
rewritten on every mutation. For production workloads with high concurrency,
this can be swapped for a proper database backend.
"""
def __init__(self, path: str | Path | None = None) -> None:
if path is None:
from deerflow.config.paths import get_paths
path = Path(get_paths().base_dir) / "channels" / "store.json"
self._path = Path(path)
self._path.parent.mkdir(parents=True, exist_ok=True)
self._data: dict[str, dict[str, Any]] = self._load()
self._lock = threading.Lock()
# -- persistence -------------------------------------------------------
def _load(self) -> dict[str, dict[str, Any]]:
if self._path.exists():
try:
return json.loads(self._path.read_text(encoding="utf-8"))
except (json.JSONDecodeError, OSError):
logger.warning("Corrupt channel store at %s, starting fresh", self._path)
return {}
def _save(self) -> None:
fd = tempfile.NamedTemporaryFile(
mode="w",
dir=self._path.parent,
suffix=".tmp",
delete=False,
)
try:
json.dump(self._data, fd, indent=2)
fd.close()
Path(fd.name).replace(self._path)
except BaseException:
fd.close()
Path(fd.name).unlink(missing_ok=True)
raise
# -- key helpers -------------------------------------------------------
@staticmethod
def _key(channel_name: str, chat_id: str, topic_id: str | None = None) -> str:
if topic_id:
return f"{channel_name}:{chat_id}:{topic_id}"
return f"{channel_name}:{chat_id}"
# -- public API --------------------------------------------------------
def get_thread_id(self, channel_name: str, chat_id: str, topic_id: str | None = None) -> str | None:
"""Look up the DeerFlow thread_id for a given IM conversation/topic."""
entry = self._data.get(self._key(channel_name, chat_id, topic_id))
return entry["thread_id"] if entry else None
def set_thread_id(
self,
channel_name: str,
chat_id: str,
thread_id: str,
*,
topic_id: str | None = None,
user_id: str = "",
) -> None:
"""Create or update the mapping for an IM conversation/topic."""
with self._lock:
key = self._key(channel_name, chat_id, topic_id)
now = time.time()
existing = self._data.get(key)
self._data[key] = {
"thread_id": thread_id,
"user_id": user_id,
"created_at": existing["created_at"] if existing else now,
"updated_at": now,
}
self._save()
def remove(self, channel_name: str, chat_id: str, topic_id: str | None = None) -> bool:
"""Remove a mapping.
If ``topic_id`` is provided, only that specific conversation/topic mapping is removed.
If ``topic_id`` is omitted, all mappings whose key starts with
``"<channel_name>:<chat_id>"`` (including topic-specific ones) are removed.
Returns True if at least one mapping was removed.
"""
with self._lock:
# Remove a specific conversation/topic mapping.
if topic_id is not None:
key = self._key(channel_name, chat_id, topic_id)
if key in self._data:
del self._data[key]
self._save()
return True
return False
# Remove all mappings for this channel/chat_id (base and any topic-specific keys).
prefix = self._key(channel_name, chat_id)
keys_to_delete = [k for k in self._data if k == prefix or k.startswith(prefix + ":")]
if not keys_to_delete:
return False
for k in keys_to_delete:
del self._data[k]
self._save()
return True
def list_entries(self, channel_name: str | None = None) -> list[dict[str, Any]]:
"""List all stored mappings, optionally filtered by channel."""
results = []
for key, entry in self._data.items():
parts = key.split(":", 2)
ch = parts[0]
chat = parts[1] if len(parts) > 1 else ""
topic = parts[2] if len(parts) > 2 else None
if channel_name and ch != channel_name:
continue
item: dict[str, Any] = {"channel_name": ch, "chat_id": chat, **entry}
if topic is not None:
item["topic_id"] = topic
results.append(item)
return results
+317
View File
@@ -0,0 +1,317 @@
"""Telegram channel — connects via long-polling (no public IP needed)."""
from __future__ import annotations
import asyncio
import logging
import threading
from typing import Any
from app.channels.base import Channel
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
logger = logging.getLogger(__name__)
class TelegramChannel(Channel):
"""Telegram bot channel using long-polling.
Configuration keys (in ``config.yaml`` under ``channels.telegram``):
- ``bot_token``: Telegram Bot API token (from @BotFather).
- ``allowed_users``: (optional) List of allowed Telegram user IDs. Empty = allow all.
"""
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
super().__init__(name="telegram", bus=bus, config=config)
self._application = None
self._thread: threading.Thread | None = None
self._tg_loop: asyncio.AbstractEventLoop | None = None
self._main_loop: asyncio.AbstractEventLoop | None = None
self._allowed_users: set[int] = set()
for uid in config.get("allowed_users", []):
try:
self._allowed_users.add(int(uid))
except (ValueError, TypeError):
pass
# chat_id -> last sent message_id for threaded replies
self._last_bot_message: dict[str, int] = {}
async def start(self) -> None:
if self._running:
return
try:
from telegram.ext import ApplicationBuilder, CommandHandler, MessageHandler, filters
except ImportError:
logger.error("python-telegram-bot is not installed. Install it with: uv add python-telegram-bot")
return
bot_token = self.config.get("bot_token", "")
if not bot_token:
logger.error("Telegram channel requires bot_token")
return
self._main_loop = asyncio.get_event_loop()
self._running = True
self.bus.subscribe_outbound(self._on_outbound)
# Build the application
app = ApplicationBuilder().token(bot_token).build()
# Command handlers
app.add_handler(CommandHandler("start", self._cmd_start))
app.add_handler(CommandHandler("new", self._cmd_generic))
app.add_handler(CommandHandler("status", self._cmd_generic))
app.add_handler(CommandHandler("models", self._cmd_generic))
app.add_handler(CommandHandler("memory", self._cmd_generic))
app.add_handler(CommandHandler("help", self._cmd_generic))
# General message handler
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, self._on_text))
self._application = app
# Run polling in a dedicated thread with its own event loop
self._thread = threading.Thread(target=self._run_polling, daemon=True)
self._thread.start()
logger.info("Telegram channel started")
async def stop(self) -> None:
self._running = False
self.bus.unsubscribe_outbound(self._on_outbound)
if self._tg_loop and self._tg_loop.is_running():
self._tg_loop.call_soon_threadsafe(self._tg_loop.stop)
if self._thread:
self._thread.join(timeout=10)
self._thread = None
self._application = None
logger.info("Telegram channel stopped")
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
if not self._application:
return
try:
chat_id = int(msg.chat_id)
except (ValueError, TypeError):
logger.error("Invalid Telegram chat_id: %s", msg.chat_id)
return
kwargs: dict[str, Any] = {"chat_id": chat_id, "text": msg.text}
# Reply to the last bot message in this chat for threading
reply_to = self._last_bot_message.get(msg.chat_id)
if reply_to:
kwargs["reply_to_message_id"] = reply_to
bot = self._application.bot
last_exc: Exception | None = None
for attempt in range(_max_retries):
try:
sent = await bot.send_message(**kwargs)
self._last_bot_message[msg.chat_id] = sent.message_id
return
except Exception as exc:
last_exc = exc
if attempt < _max_retries - 1:
delay = 2**attempt # 1s, 2s
logger.warning(
"[Telegram] send failed (attempt %d/%d), retrying in %ds: %s",
attempt + 1,
_max_retries,
delay,
exc,
)
await asyncio.sleep(delay)
logger.error("[Telegram] send failed after %d attempts: %s", _max_retries, last_exc)
if last_exc is None:
raise RuntimeError("Telegram send failed without an exception from any attempt")
raise last_exc
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
if not self._application:
return False
try:
chat_id = int(msg.chat_id)
except (ValueError, TypeError):
logger.error("[Telegram] Invalid chat_id: %s", msg.chat_id)
return False
# Telegram limits: 10MB for photos, 50MB for documents
if attachment.size > 50 * 1024 * 1024:
logger.warning("[Telegram] file too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
return False
bot = self._application.bot
reply_to = self._last_bot_message.get(msg.chat_id)
try:
if attachment.is_image and attachment.size <= 10 * 1024 * 1024:
with open(attachment.actual_path, "rb") as f:
kwargs: dict[str, Any] = {"chat_id": chat_id, "photo": f}
if reply_to:
kwargs["reply_to_message_id"] = reply_to
sent = await bot.send_photo(**kwargs)
else:
from telegram import InputFile
with open(attachment.actual_path, "rb") as f:
input_file = InputFile(f, filename=attachment.filename)
kwargs = {"chat_id": chat_id, "document": input_file}
if reply_to:
kwargs["reply_to_message_id"] = reply_to
sent = await bot.send_document(**kwargs)
self._last_bot_message[msg.chat_id] = sent.message_id
logger.info("[Telegram] file sent: %s to chat=%s", attachment.filename, msg.chat_id)
return True
except Exception:
logger.exception("[Telegram] failed to send file: %s", attachment.filename)
return False
# -- helpers -----------------------------------------------------------
async def _send_running_reply(self, chat_id: str, reply_to_message_id: int) -> None:
"""Send a 'Working on it...' reply to the user's message."""
if not self._application:
return
try:
bot = self._application.bot
await bot.send_message(
chat_id=int(chat_id),
text="Working on it...",
reply_to_message_id=reply_to_message_id,
)
logger.info("[Telegram] 'Working on it...' reply sent in chat=%s", chat_id)
except Exception:
logger.exception("[Telegram] failed to send running reply in chat=%s", chat_id)
# -- internal ----------------------------------------------------------
@staticmethod
def _log_future_error(fut, name: str, msg_id: str):
try:
exc = fut.exception()
if exc:
logger.error("[Telegram] %s failed for msg_id=%s: %s", name, msg_id, exc)
except Exception:
logger.exception("[Telegram] Failed to inspect future for %s (msg_id=%s)", name, msg_id)
def _run_polling(self) -> None:
"""Run telegram polling in a dedicated thread."""
self._tg_loop = asyncio.new_event_loop()
asyncio.set_event_loop(self._tg_loop)
try:
# Cannot use run_polling() because it calls add_signal_handler(),
# which only works in the main thread. Instead, manually
# initialize the application and start the updater.
self._tg_loop.run_until_complete(self._application.initialize())
self._tg_loop.run_until_complete(self._application.start())
self._tg_loop.run_until_complete(self._application.updater.start_polling())
self._tg_loop.run_forever()
except Exception:
if self._running:
logger.exception("Telegram polling error")
finally:
# Graceful shutdown
try:
if self._application.updater.running:
self._tg_loop.run_until_complete(self._application.updater.stop())
self._tg_loop.run_until_complete(self._application.stop())
self._tg_loop.run_until_complete(self._application.shutdown())
except Exception:
logger.exception("Error during Telegram shutdown")
def _check_user(self, user_id: int) -> bool:
if not self._allowed_users:
return True
return user_id in self._allowed_users
async def _cmd_start(self, update, context) -> None:
"""Handle /start command."""
if not self._check_user(update.effective_user.id):
return
await update.message.reply_text("Welcome to DeerFlow! Send me a message to start a conversation.\nType /help for available commands.")
async def _process_incoming_with_reply(self, chat_id: str, msg_id: int, inbound: InboundMessage) -> None:
await self._send_running_reply(chat_id, msg_id)
await self.bus.publish_inbound(inbound)
async def _cmd_generic(self, update, context) -> None:
"""Forward slash commands to the channel manager."""
if not self._check_user(update.effective_user.id):
return
text = update.message.text
chat_id = str(update.effective_chat.id)
user_id = str(update.effective_user.id)
msg_id = str(update.message.message_id)
# Use the same topic_id logic as _on_text so that commands
# like /new target the correct thread mapping.
if update.effective_chat.type == "private":
topic_id = None
else:
reply_to = update.message.reply_to_message
if reply_to:
topic_id = str(reply_to.message_id)
else:
topic_id = msg_id
inbound = self._make_inbound(
chat_id=chat_id,
user_id=user_id,
text=text,
msg_type=InboundMessageType.COMMAND,
thread_ts=msg_id,
)
inbound.topic_id = topic_id
if self._main_loop and self._main_loop.is_running():
fut = asyncio.run_coroutine_threadsafe(self._process_incoming_with_reply(chat_id, update.message.message_id, inbound), self._main_loop)
fut.add_done_callback(lambda f: self._log_future_error(f, "process_incoming_with_reply", update.message.message_id))
else:
logger.warning("[Telegram] Main loop not running. Cannot publish inbound message.")
async def _on_text(self, update, context) -> None:
"""Handle regular text messages."""
if not self._check_user(update.effective_user.id):
return
text = update.message.text.strip()
if not text:
return
chat_id = str(update.effective_chat.id)
user_id = str(update.effective_user.id)
msg_id = str(update.message.message_id)
# topic_id determines which DeerFlow thread the message maps to.
# In private chats, use None so that all messages share a single
# thread (the store key becomes "channel:chat_id").
# In group chats, use the reply-to message id or the current
# message id to keep separate conversation threads.
if update.effective_chat.type == "private":
topic_id = None
else:
reply_to = update.message.reply_to_message
if reply_to:
topic_id = str(reply_to.message_id)
else:
topic_id = msg_id
inbound = self._make_inbound(
chat_id=chat_id,
user_id=user_id,
text=text,
msg_type=InboundMessageType.CHAT,
thread_ts=msg_id,
)
inbound.topic_id = topic_id
if self._main_loop and self._main_loop.is_running():
fut = asyncio.run_coroutine_threadsafe(self._process_incoming_with_reply(chat_id, update.message.message_id, inbound), self._main_loop)
fut.add_done_callback(lambda f: self._log_future_error(f, "process_incoming_with_reply", update.message.message_id))
else:
logger.warning("[Telegram] Main loop not running. Cannot publish inbound message.")
File diff suppressed because it is too large Load Diff
+398
View File
@@ -0,0 +1,398 @@
from __future__ import annotations
import asyncio
import base64
import hashlib
import logging
from collections.abc import Awaitable, Callable
from typing import Any, cast
from app.channels.base import Channel
from app.channels.message_bus import (
InboundMessageType,
MessageBus,
OutboundMessage,
ResolvedAttachment,
)
logger = logging.getLogger(__name__)
class WeComChannel(Channel):
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
super().__init__(name="wecom", bus=bus, config=config)
self._bot_id: str | None = None
self._bot_secret: str | None = None
self._ws_client = None
self._ws_task: asyncio.Task | None = None
self._ws_frames: dict[str, dict[str, Any]] = {}
self._ws_stream_ids: dict[str, str] = {}
self._working_message = "Working on it..."
@property
def supports_streaming(self) -> bool:
return True
def _clear_ws_context(self, thread_ts: str | None) -> None:
if not thread_ts:
return
self._ws_frames.pop(thread_ts, None)
self._ws_stream_ids.pop(thread_ts, None)
async def _send_ws_upload_command(self, req_id: str, body: dict[str, Any], cmd: str) -> dict[str, Any]:
if not self._ws_client:
raise RuntimeError("WeCom WebSocket client is not available")
ws_manager = getattr(self._ws_client, "_ws_manager", None)
send_reply = getattr(ws_manager, "send_reply", None)
if not callable(send_reply):
raise RuntimeError("Installed wecom-aibot-python-sdk does not expose the WebSocket media upload API expected by DeerFlow. Use wecom-aibot-python-sdk==0.1.6 or update the adapter.")
send_reply_async = cast(Callable[[str, dict[str, Any], str], Awaitable[dict[str, Any]]], send_reply)
return await send_reply_async(req_id, body, cmd)
async def start(self) -> None:
if self._running:
return
bot_id = self.config.get("bot_id")
bot_secret = self.config.get("bot_secret")
working_message = self.config.get("working_message")
self._bot_id = bot_id if isinstance(bot_id, str) and bot_id else None
self._bot_secret = bot_secret if isinstance(bot_secret, str) and bot_secret else None
self._working_message = working_message if isinstance(working_message, str) and working_message else "Working on it..."
if not self._bot_id or not self._bot_secret:
logger.error("WeCom channel requires bot_id and bot_secret")
return
try:
from aibot import WSClient, WSClientOptions
except ImportError:
logger.error("wecom-aibot-python-sdk is not installed. Install it with: uv add wecom-aibot-python-sdk")
return
else:
self._ws_client = WSClient(WSClientOptions(bot_id=self._bot_id, secret=self._bot_secret, logger=logger))
self._ws_client.on("message.text", self._on_ws_text)
self._ws_client.on("message.mixed", self._on_ws_mixed)
self._ws_client.on("message.image", self._on_ws_image)
self._ws_client.on("message.file", self._on_ws_file)
self._ws_task = asyncio.create_task(self._ws_client.connect())
self._running = True
self.bus.subscribe_outbound(self._on_outbound)
logger.info("WeCom channel started")
async def stop(self) -> None:
self._running = False
self.bus.unsubscribe_outbound(self._on_outbound)
if self._ws_task:
try:
self._ws_task.cancel()
except Exception:
pass
self._ws_task = None
if self._ws_client:
try:
self._ws_client.disconnect()
except Exception:
pass
self._ws_client = None
self._ws_frames.clear()
self._ws_stream_ids.clear()
logger.info("WeCom channel stopped")
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
if self._ws_client:
await self._send_ws(msg, _max_retries=_max_retries)
return
logger.warning("[WeCom] send called but WebSocket client is not available")
async def _on_outbound(self, msg: OutboundMessage) -> None:
if msg.channel_name != self.name:
return
try:
await self.send(msg)
except Exception:
logger.exception("Failed to send outbound message on channel %s", self.name)
if msg.is_final:
self._clear_ws_context(msg.thread_ts)
return
for attachment in msg.attachments:
try:
success = await self.send_file(msg, attachment)
if not success:
logger.warning("[%s] file upload skipped for %s", self.name, attachment.filename)
except Exception:
logger.exception("[%s] failed to upload file %s", self.name, attachment.filename)
if msg.is_final:
self._clear_ws_context(msg.thread_ts)
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
if not msg.is_final:
return True
if not self._ws_client:
return False
if not msg.thread_ts:
return False
frame = self._ws_frames.get(msg.thread_ts)
if not frame:
return False
media_type = "image" if attachment.is_image else "file"
size_limit = 2 * 1024 * 1024 if attachment.is_image else 20 * 1024 * 1024
if attachment.size > size_limit:
logger.warning(
"[WeCom] %s too large (%d bytes), skipping: %s",
media_type,
attachment.size,
attachment.filename,
)
return False
try:
media_id = await self._upload_media_ws(
media_type=media_type,
filename=attachment.filename,
path=str(attachment.actual_path),
size=attachment.size,
)
if not media_id:
return False
body = {media_type: {"media_id": media_id}, "msgtype": media_type}
await self._ws_client.reply(frame, body)
logger.debug("[WeCom] %s sent via ws: %s", media_type, attachment.filename)
return True
except Exception:
logger.exception("[WeCom] failed to upload/send file via ws: %s", attachment.filename)
return False
async def _on_ws_text(self, frame: dict[str, Any]) -> None:
body = frame.get("body", {}) or {}
text = ((body.get("text") or {}).get("content") or "").strip()
quote = body.get("quote", {}).get("text", {}).get("content", "").strip()
if not text and not quote:
return
await self._publish_ws_inbound(frame, text + (f"\nQuote message: {quote}" if quote else ""))
async def _on_ws_mixed(self, frame: dict[str, Any]) -> None:
body = frame.get("body", {}) or {}
mixed = body.get("mixed") or {}
items = mixed.get("msg_item") or []
parts: list[str] = []
files: list[dict[str, Any]] = []
for item in items:
item_type = (item or {}).get("msgtype")
if item_type == "text":
content = (((item or {}).get("text") or {}).get("content") or "").strip()
if content:
parts.append(content)
elif item_type in ("image", "file"):
payload = (item or {}).get(item_type) or {}
url = payload.get("url")
aeskey = payload.get("aeskey")
if isinstance(url, str) and url:
files.append(
{
"type": item_type,
"url": url,
"aeskey": (aeskey if isinstance(aeskey, str) and aeskey else None),
}
)
text = "\n\n".join(parts).strip()
if not text and not files:
return
if not text:
text = "receive image/file"
await self._publish_ws_inbound(frame, text, files=files)
async def _on_ws_image(self, frame: dict[str, Any]) -> None:
body = frame.get("body", {}) or {}
image = body.get("image") or {}
url = image.get("url")
aeskey = image.get("aeskey")
if not isinstance(url, str) or not url:
return
await self._publish_ws_inbound(
frame,
"receive image ",
files=[
{
"type": "image",
"url": url,
"aeskey": aeskey if isinstance(aeskey, str) and aeskey else None,
}
],
)
async def _on_ws_file(self, frame: dict[str, Any]) -> None:
body = frame.get("body", {}) or {}
file_obj = body.get("file") or {}
url = file_obj.get("url")
aeskey = file_obj.get("aeskey")
if not isinstance(url, str) or not url:
return
await self._publish_ws_inbound(
frame,
"receive file",
files=[
{
"type": "file",
"url": url,
"aeskey": aeskey if isinstance(aeskey, str) and aeskey else None,
}
],
)
async def _publish_ws_inbound(
self,
frame: dict[str, Any],
text: str,
*,
files: list[dict[str, Any]] | None = None,
) -> None:
if not self._ws_client:
return
try:
from aibot import generate_req_id
except Exception:
return
body = frame.get("body", {}) or {}
msg_id = body.get("msgid")
if not msg_id:
return
user_id = (body.get("from") or {}).get("userid")
inbound_type = InboundMessageType.COMMAND if text.startswith("/") else InboundMessageType.CHAT
inbound = self._make_inbound(
chat_id=user_id, # keep user's conversation in memory
user_id=user_id,
text=text,
msg_type=inbound_type,
thread_ts=msg_id,
files=files or [],
metadata={"aibotid": body.get("aibotid"), "chattype": body.get("chattype")},
)
inbound.topic_id = user_id # keep the same thread
stream_id = generate_req_id("stream")
self._ws_frames[msg_id] = frame
self._ws_stream_ids[msg_id] = stream_id
try:
await self._ws_client.reply_stream(frame, stream_id, self._working_message, False)
except Exception:
pass
await self.bus.publish_inbound(inbound)
async def _send_ws(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
if not self._ws_client:
return
try:
from aibot import generate_req_id
except Exception:
generate_req_id = None
if msg.thread_ts and msg.thread_ts in self._ws_frames:
frame = self._ws_frames[msg.thread_ts]
stream_id = self._ws_stream_ids.get(msg.thread_ts)
if not stream_id and generate_req_id:
stream_id = generate_req_id("stream")
self._ws_stream_ids[msg.thread_ts] = stream_id
if not stream_id:
return
last_exc: Exception | None = None
for attempt in range(_max_retries):
try:
await self._ws_client.reply_stream(frame, stream_id, msg.text, bool(msg.is_final))
return
except Exception as exc:
last_exc = exc
if attempt < _max_retries - 1:
await asyncio.sleep(2**attempt)
if last_exc:
raise last_exc
body = {"msgtype": "markdown", "markdown": {"content": msg.text}}
last_exc = None
for attempt in range(_max_retries):
try:
await self._ws_client.send_message(msg.chat_id, body)
return
except Exception as exc:
last_exc = exc
if attempt < _max_retries - 1:
await asyncio.sleep(2**attempt)
if last_exc:
raise last_exc
async def _upload_media_ws(
self,
*,
media_type: str,
filename: str,
path: str,
size: int,
) -> str | None:
if not self._ws_client:
return None
try:
from aibot import generate_req_id
except Exception:
return None
chunk_size = 512 * 1024
total_chunks = (size + chunk_size - 1) // chunk_size
if total_chunks < 1 or total_chunks > 100:
logger.warning("[WeCom] invalid total_chunks=%d for %s", total_chunks, filename)
return None
md5_hasher = hashlib.md5()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(1024 * 1024), b""):
md5_hasher.update(chunk)
md5 = md5_hasher.hexdigest()
init_req_id = generate_req_id("aibot_upload_media_init")
init_body = {
"type": media_type,
"filename": filename,
"total_size": int(size),
"total_chunks": int(total_chunks),
"md5": md5,
}
init_ack = await self._send_ws_upload_command(init_req_id, init_body, "aibot_upload_media_init")
upload_id = (init_ack.get("body") or {}).get("upload_id")
if not upload_id:
logger.warning("[WeCom] upload init returned no upload_id: %s", init_ack)
return None
with open(path, "rb") as f:
for idx in range(total_chunks):
data = f.read(chunk_size)
if not data:
break
chunk_req_id = generate_req_id("aibot_upload_media_chunk")
chunk_body = {
"upload_id": upload_id,
"chunk_index": int(idx),
"base64_data": base64.b64encode(data).decode("utf-8"),
}
await self._send_ws_upload_command(chunk_req_id, chunk_body, "aibot_upload_media_chunk")
finish_req_id = generate_req_id("aibot_upload_media_finish")
finish_ack = await self._send_ws_upload_command(finish_req_id, {"upload_id": upload_id}, "aibot_upload_media_finish")
media_id = (finish_ack.get("body") or {}).get("media_id")
if not media_id:
logger.warning("[WeCom] upload finish returned no media_id: %s", finish_ack)
return None
return media_id
+4
View File
@@ -0,0 +1,4 @@
from .app import app, create_app
from .config import GatewayConfig, get_gateway_config
__all__ = ["app", "create_app", "GatewayConfig", "get_gateway_config"]
+389
View File
@@ -0,0 +1,389 @@
import asyncio
import logging
import os
from collections.abc import AsyncGenerator
from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from app.gateway.auth_middleware import AuthMiddleware
from app.gateway.config import get_gateway_config
from app.gateway.csrf_middleware import CSRFMiddleware
from app.gateway.deps import langgraph_runtime
from app.gateway.routers import (
agents,
artifacts,
assistants_compat,
auth,
channels,
feedback,
mcp,
memory,
models,
runs,
skills,
suggestions,
thread_runs,
threads,
uploads,
)
from deerflow.config import app_config as deerflow_app_config
from deerflow.config.app_config import apply_logging_level
AppConfig = deerflow_app_config.AppConfig
get_app_config = deerflow_app_config.get_app_config
# Default logging; lifespan overrides from config.yaml log_level.
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger(__name__)
# Upper bound (seconds) each lifespan shutdown hook is allowed to run.
# Bounds worker exit time so uvicorn's reload supervisor does not keep
# firing signals into a worker that is stuck waiting for shutdown cleanup.
_SHUTDOWN_HOOK_TIMEOUT_SECONDS = 5.0
async def _ensure_admin_user(app: FastAPI) -> None:
"""Startup hook: handle first boot and migrate orphan threads otherwise.
After admin creation, migrate orphan threads from the LangGraph
store (metadata.user_id unset) to the admin account. This is the
"no-auth → with-auth" upgrade path: users who ran DeerFlow without
authentication have existing LangGraph thread data that needs an
owner assigned.
First boot (no admin exists):
- Does NOT create any user accounts automatically.
- The operator must visit ``/setup`` to create the first admin.
Subsequent boots (admin already exists):
- Runs the one-time "no-auth → with-auth" orphan thread migration for
existing LangGraph thread metadata that has no owner_id.
No SQL persistence migration is needed: the four user_id columns
(threads_meta, runs, run_events, feedback) only come into existence
alongside the auth module via create_all, so freshly created tables
never contain NULL-owner rows.
"""
from sqlalchemy import select
from app.gateway.deps import get_local_provider
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.user.model import UserRow
try:
provider = get_local_provider()
except RuntimeError:
# Auth persistence may not be initialized in some test/boot paths.
# Skip admin migration work rather than failing gateway startup.
logger.warning("Auth persistence not ready; skipping admin bootstrap check")
return
sf = get_session_factory()
if sf is None:
return
admin_count = await provider.count_admin_users()
if admin_count == 0:
logger.info("=" * 60)
logger.info(" First boot detected — no admin account exists.")
logger.info(" Visit /setup to complete admin account creation.")
logger.info("=" * 60)
return
# Admin already exists — run orphan thread migration for any
# LangGraph thread metadata that pre-dates the auth module.
async with sf() as session:
stmt = select(UserRow).where(UserRow.system_role == "admin").limit(1)
row = (await session.execute(stmt)).scalar_one_or_none()
if row is None:
return # Should not happen (admin_count > 0 above), but be safe.
admin_id = str(row.id)
# LangGraph store orphan migration — non-fatal.
# This covers the "no-auth → with-auth" upgrade path for users
# whose existing LangGraph thread metadata has no user_id set.
store = getattr(app.state, "store", None)
if store is not None:
try:
migrated = await _migrate_orphaned_threads(store, admin_id)
if migrated:
logger.info("Migrated %d orphan LangGraph thread(s) to admin", migrated)
except Exception:
logger.exception("LangGraph thread migration failed (non-fatal)")
async def _iter_store_items(store, namespace, *, page_size: int = 500):
"""Paginated async iterator over a LangGraph store namespace.
Replaces the old hardcoded ``limit=1000`` call with a cursor-style
loop so that environments with more than one page of orphans do
not silently lose data. Terminates when a page is empty OR when a
short page arrives (indicating the last page).
"""
offset = 0
while True:
batch = await store.asearch(namespace, limit=page_size, offset=offset)
if not batch:
return
for item in batch:
yield item
if len(batch) < page_size:
return
offset += page_size
async def _migrate_orphaned_threads(store, admin_user_id: str) -> int:
"""Migrate LangGraph store threads with no user_id to the given admin.
Uses cursor pagination so all orphans are migrated regardless of
count. Returns the number of rows migrated.
"""
migrated = 0
async for item in _iter_store_items(store, ("threads",)):
metadata = item.value.get("metadata", {})
if not metadata.get("user_id"):
metadata["user_id"] = admin_user_id
item.value["metadata"] = metadata
await store.aput(("threads",), item.key, item.value)
migrated += 1
return migrated
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
"""Application lifespan handler."""
# Load config and check necessary environment variables at startup
try:
app.state.config = get_app_config()
apply_logging_level(app.state.config.log_level)
logger.info("Configuration loaded successfully")
except Exception as e:
error_msg = f"Failed to load configuration during gateway startup: {e}"
logger.exception(error_msg)
raise RuntimeError(error_msg) from e
config = get_gateway_config()
logger.info(f"Starting API Gateway on {config.host}:{config.port}")
# Initialize LangGraph runtime components (StreamBridge, RunManager, checkpointer, store)
async with langgraph_runtime(app):
logger.info("LangGraph runtime initialised")
# Ensure admin user exists (auto-create on first boot)
# Must run AFTER langgraph_runtime so app.state.store is available for thread migration
await _ensure_admin_user(app)
# Start IM channel service if any channels are configured
try:
from app.channels.service import start_channel_service
channel_service = await start_channel_service(app.state.config)
logger.info("Channel service started: %s", channel_service.get_status())
except Exception:
logger.exception("No IM channels configured or channel service failed to start")
yield
# Stop channel service on shutdown (bounded to prevent worker hang)
try:
from app.channels.service import stop_channel_service
await asyncio.wait_for(
stop_channel_service(),
timeout=_SHUTDOWN_HOOK_TIMEOUT_SECONDS,
)
except TimeoutError:
logger.warning(
"Channel service shutdown exceeded %.1fs; proceeding with worker exit.",
_SHUTDOWN_HOOK_TIMEOUT_SECONDS,
)
except Exception:
logger.exception("Failed to stop channel service")
logger.info("Shutting down API Gateway")
def create_app() -> FastAPI:
"""Create and configure the FastAPI application.
Returns:
Configured FastAPI application instance.
"""
config = get_gateway_config()
docs_kwargs = {"docs_url": "/docs", "redoc_url": "/redoc", "openapi_url": "/openapi.json"} if config.enable_docs else {"docs_url": None, "redoc_url": None, "openapi_url": None}
app = FastAPI(
title="DeerFlow API Gateway",
description="""
## DeerFlow API Gateway
API Gateway for DeerFlow - A LangGraph-based AI agent backend with sandbox execution capabilities.
### Features
- **Models Management**: Query and retrieve available AI models
- **MCP Configuration**: Manage Model Context Protocol (MCP) server configurations
- **Memory Management**: Access and manage global memory data for personalized conversations
- **Skills Management**: Query and manage skills and their enabled status
- **Artifacts**: Access thread artifacts and generated files
- **Health Monitoring**: System health check endpoints
### Architecture
LangGraph requests are handled by nginx reverse proxy.
This gateway provides custom endpoints for models, MCP configuration, skills, and artifacts.
""",
version="0.1.0",
lifespan=lifespan,
**docs_kwargs,
openapi_tags=[
{
"name": "models",
"description": "Operations for querying available AI models and their configurations",
},
{
"name": "mcp",
"description": "Manage Model Context Protocol (MCP) server configurations",
},
{
"name": "memory",
"description": "Access and manage global memory data for personalized conversations",
},
{
"name": "skills",
"description": "Manage skills and their configurations",
},
{
"name": "artifacts",
"description": "Access and download thread artifacts and generated files",
},
{
"name": "uploads",
"description": "Upload and manage user files for threads",
},
{
"name": "threads",
"description": "Manage DeerFlow thread-local filesystem data",
},
{
"name": "agents",
"description": "Create and manage custom agents with per-agent config and prompts",
},
{
"name": "suggestions",
"description": "Generate follow-up question suggestions for conversations",
},
{
"name": "channels",
"description": "Manage IM channel integrations (Feishu, Slack, Telegram)",
},
{
"name": "assistants-compat",
"description": "LangGraph Platform-compatible assistants API (stub)",
},
{
"name": "runs",
"description": "LangGraph Platform-compatible runs lifecycle (create, stream, cancel)",
},
{
"name": "health",
"description": "Health check and system status endpoints",
},
],
)
# Auth: reject unauthenticated requests to non-public paths (fail-closed safety net)
app.add_middleware(AuthMiddleware)
# CSRF: Double Submit Cookie pattern for state-changing requests
app.add_middleware(CSRFMiddleware)
# CORS: when GATEWAY_CORS_ORIGINS is set (dev without nginx), add CORS middleware.
# In production, nginx handles CORS and no middleware is needed.
cors_origins_env = os.environ.get("GATEWAY_CORS_ORIGINS", "")
if cors_origins_env:
cors_origins = [o.strip() for o in cors_origins_env.split(",") if o.strip()]
# Validate: wildcard origin with credentials is a security misconfiguration
for origin in cors_origins:
if origin == "*":
logger.error("GATEWAY_CORS_ORIGINS contains wildcard '*' with allow_credentials=True. This is a security misconfiguration — browsers will reject the response. Use explicit scheme://host:port origins instead.")
cors_origins = [o for o in cors_origins if o != "*"]
break
if cors_origins:
app.add_middleware(
CORSMiddleware,
allow_origins=cors_origins,
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Include routers
# Models API is mounted at /api/models
app.include_router(models.router)
# MCP API is mounted at /api/mcp
app.include_router(mcp.router)
# Memory API is mounted at /api/memory
app.include_router(memory.router)
# Skills API is mounted at /api/skills
app.include_router(skills.router)
# Artifacts API is mounted at /api/threads/{thread_id}/artifacts
app.include_router(artifacts.router)
# Uploads API is mounted at /api/threads/{thread_id}/uploads
app.include_router(uploads.router)
# Thread cleanup API is mounted at /api/threads/{thread_id}
app.include_router(threads.router)
# Agents API is mounted at /api/agents
app.include_router(agents.router)
# Suggestions API is mounted at /api/threads/{thread_id}/suggestions
app.include_router(suggestions.router)
# Channels API is mounted at /api/channels
app.include_router(channels.router)
# Assistants compatibility API (LangGraph Platform stub)
app.include_router(assistants_compat.router)
# Auth API is mounted at /api/v1/auth
app.include_router(auth.router)
# Feedback API is mounted at /api/threads/{thread_id}/runs/{run_id}/feedback
app.include_router(feedback.router)
# Thread Runs API (LangGraph Platform-compatible runs lifecycle)
app.include_router(thread_runs.router)
# Stateless Runs API (stream/wait without a pre-existing thread)
app.include_router(runs.router)
@app.get("/health", tags=["health"])
async def health_check() -> dict:
"""Health check endpoint.
Returns:
Service health status information.
"""
return {"status": "healthy", "service": "deer-flow-gateway"}
return app
# Create app instance for uvicorn
app = create_app()
+42
View File
@@ -0,0 +1,42 @@
"""Authentication module for DeerFlow.
This module provides:
- JWT-based authentication
- Provider Factory pattern for extensible auth methods
- UserRepository interface for storage backends (SQLite)
"""
from app.gateway.auth.config import AuthConfig, get_auth_config, set_auth_config
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse, TokenError
from app.gateway.auth.jwt import TokenPayload, create_access_token, decode_token
from app.gateway.auth.local_provider import LocalAuthProvider
from app.gateway.auth.models import User, UserResponse
from app.gateway.auth.password import hash_password, verify_password
from app.gateway.auth.providers import AuthProvider
from app.gateway.auth.repositories.base import UserRepository
__all__ = [
# Config
"AuthConfig",
"get_auth_config",
"set_auth_config",
# Errors
"AuthErrorCode",
"AuthErrorResponse",
"TokenError",
# JWT
"TokenPayload",
"create_access_token",
"decode_token",
# Password
"hash_password",
"verify_password",
# Models
"User",
"UserResponse",
# Providers
"AuthProvider",
"LocalAuthProvider",
# Repository
"UserRepository",
]
+57
View File
@@ -0,0 +1,57 @@
"""Authentication configuration for DeerFlow."""
import logging
import os
import secrets
from pydantic import BaseModel, Field
logger = logging.getLogger(__name__)
class AuthConfig(BaseModel):
"""JWT and auth-related configuration. Parsed once at startup.
Note: the ``users`` table now lives in the shared persistence
database managed by ``deerflow.persistence.engine``. The old
``users_db_path`` config key has been removed — user storage is
configured through ``config.database`` like every other table.
"""
jwt_secret: str = Field(
...,
description="Secret key for JWT signing. MUST be set via AUTH_JWT_SECRET.",
)
token_expiry_days: int = Field(default=7, ge=1, le=30)
oauth_github_client_id: str | None = Field(default=None)
oauth_github_client_secret: str | None = Field(default=None)
_auth_config: AuthConfig | None = None
def get_auth_config() -> AuthConfig:
"""Get the global AuthConfig instance. Parses from env on first call."""
global _auth_config
if _auth_config is None:
from dotenv import load_dotenv
load_dotenv()
jwt_secret = os.environ.get("AUTH_JWT_SECRET")
if not jwt_secret:
jwt_secret = secrets.token_urlsafe(32)
os.environ["AUTH_JWT_SECRET"] = jwt_secret
logger.warning(
"⚠ AUTH_JWT_SECRET is not set — using an auto-generated ephemeral secret. "
"Sessions will be invalidated on restart. "
"For production, add AUTH_JWT_SECRET to your .env file: "
'python -c "import secrets; print(secrets.token_urlsafe(32))"'
)
_auth_config = AuthConfig(jwt_secret=jwt_secret)
return _auth_config
def set_auth_config(config: AuthConfig) -> None:
"""Set the global AuthConfig instance (for testing)."""
global _auth_config
_auth_config = config
@@ -0,0 +1,48 @@
"""Write initial admin credentials to a restricted file instead of logs.
Logging secrets to stdout/stderr is a well-known CodeQL finding
(py/clear-text-logging-sensitive-data) — in production those logs
get collected into ELK/Splunk/etc and become a secret sprawl
source. This helper writes the credential to a 0600 file that only
the process user can read, and returns the path so the caller can
log **the path** (not the password) for the operator to pick up.
"""
from __future__ import annotations
import os
from pathlib import Path
from deerflow.config.paths import get_paths
_CREDENTIAL_FILENAME = "admin_initial_credentials.txt"
def write_initial_credentials(email: str, password: str, *, label: str = "initial") -> Path:
"""Write the admin email + password to ``{base_dir}/admin_initial_credentials.txt``.
The file is created **atomically** with mode 0600 via ``os.open``
so the password is never world-readable, even for the single syscall
window between ``write_text`` and ``chmod``.
``label`` distinguishes "initial" (fresh creation) from "reset"
(password reset) in the file header so an operator picking up the
file after a restart can tell which event produced it.
Returns the absolute :class:`Path` to the file.
"""
target = get_paths().base_dir / _CREDENTIAL_FILENAME
target.parent.mkdir(parents=True, exist_ok=True)
content = (
f"# DeerFlow admin {label} credentials\n# This file is generated on first boot or password reset.\n# Change the password after login via Settings -> Account,\n# then delete this file.\n#\nemail: {email}\npassword: {password}\n"
)
# Atomic 0600 create-or-truncate. O_TRUNC (not O_EXCL) so the
# reset-password path can rewrite an existing file without a
# separate unlink-then-create dance.
fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w", encoding="utf-8") as fh:
fh.write(content)
return target.resolve()
+45
View File
@@ -0,0 +1,45 @@
"""Typed error definitions for auth module.
AuthErrorCode: exhaustive enum of all auth failure conditions.
TokenError: exhaustive enum of JWT decode failures.
AuthErrorResponse: structured error payload for HTTP responses.
"""
from enum import StrEnum
from pydantic import BaseModel
class AuthErrorCode(StrEnum):
"""Exhaustive list of auth error conditions."""
INVALID_CREDENTIALS = "invalid_credentials"
TOKEN_EXPIRED = "token_expired"
TOKEN_INVALID = "token_invalid"
USER_NOT_FOUND = "user_not_found"
EMAIL_ALREADY_EXISTS = "email_already_exists"
PROVIDER_NOT_FOUND = "provider_not_found"
NOT_AUTHENTICATED = "not_authenticated"
SYSTEM_ALREADY_INITIALIZED = "system_already_initialized"
class TokenError(StrEnum):
"""Exhaustive list of JWT decode failure reasons."""
EXPIRED = "expired"
INVALID_SIGNATURE = "invalid_signature"
MALFORMED = "malformed"
class AuthErrorResponse(BaseModel):
"""Structured error response — replaces bare `detail` strings."""
code: AuthErrorCode
message: str
def token_error_to_code(err: TokenError) -> AuthErrorCode:
"""Map TokenError to AuthErrorCode — single source of truth."""
if err == TokenError.EXPIRED:
return AuthErrorCode.TOKEN_EXPIRED
return AuthErrorCode.TOKEN_INVALID
+55
View File
@@ -0,0 +1,55 @@
"""JWT token creation and verification."""
from datetime import UTC, datetime, timedelta
import jwt
from pydantic import BaseModel
from app.gateway.auth.config import get_auth_config
from app.gateway.auth.errors import TokenError
class TokenPayload(BaseModel):
"""JWT token payload."""
sub: str # user_id
exp: datetime
iat: datetime | None = None
ver: int = 0 # token_version — must match User.token_version
def create_access_token(user_id: str, expires_delta: timedelta | None = None, token_version: int = 0) -> str:
"""Create a JWT access token.
Args:
user_id: The user's UUID as string
expires_delta: Optional custom expiry, defaults to 7 days
token_version: User's current token_version for invalidation
Returns:
Encoded JWT string
"""
config = get_auth_config()
expiry = expires_delta or timedelta(days=config.token_expiry_days)
now = datetime.now(UTC)
payload = {"sub": user_id, "exp": now + expiry, "iat": now, "ver": token_version}
return jwt.encode(payload, config.jwt_secret, algorithm="HS256")
def decode_token(token: str) -> TokenPayload | TokenError:
"""Decode and validate a JWT token.
Returns:
TokenPayload if valid, or a specific TokenError variant.
"""
config = get_auth_config()
try:
payload = jwt.decode(token, config.jwt_secret, algorithms=["HS256"])
return TokenPayload(**payload)
except jwt.ExpiredSignatureError:
return TokenError.EXPIRED
except jwt.InvalidSignatureError:
return TokenError.INVALID_SIGNATURE
except jwt.PyJWTError:
return TokenError.MALFORMED
+104
View File
@@ -0,0 +1,104 @@
"""Local email/password authentication provider."""
import logging
from app.gateway.auth.models import User
from app.gateway.auth.password import hash_password_async, needs_rehash, verify_password_async
from app.gateway.auth.providers import AuthProvider
from app.gateway.auth.repositories.base import UserRepository
logger = logging.getLogger(__name__)
class LocalAuthProvider(AuthProvider):
"""Email/password authentication provider using local database."""
def __init__(self, repository: UserRepository):
"""Initialize with a UserRepository.
Args:
repository: UserRepository implementation (SQLite)
"""
self._repo = repository
async def authenticate(self, credentials: dict) -> User | None:
"""Authenticate with email and password.
Args:
credentials: dict with 'email' and 'password' keys
Returns:
User if authentication succeeds, None otherwise
"""
email = credentials.get("email")
password = credentials.get("password")
if not email or not password:
return None
user = await self._repo.get_user_by_email(email)
if user is None:
return None
if user.password_hash is None:
# OAuth user without local password
return None
if not await verify_password_async(password, user.password_hash):
return None
if needs_rehash(user.password_hash):
try:
user.password_hash = await hash_password_async(password)
await self._repo.update_user(user)
except Exception:
# Rehash is an opportunistic upgrade; a transient DB error must not
# prevent an otherwise-valid login from succeeding.
logger.warning("Failed to rehash password for user %s; login will still succeed", user.email, exc_info=True)
return user
async def get_user(self, user_id: str) -> User | None:
"""Get user by ID."""
return await self._repo.get_user_by_id(user_id)
async def create_user(self, email: str, password: str | None = None, system_role: str = "user", needs_setup: bool = False) -> User:
"""Create a new local user.
Args:
email: User email address
password: Plain text password (will be hashed)
system_role: Role to assign ("admin" or "user")
needs_setup: If True, user must complete setup on first login
Returns:
Created User instance
"""
password_hash = await hash_password_async(password) if password else None
user = User(
email=email,
password_hash=password_hash,
system_role=system_role,
needs_setup=needs_setup,
)
return await self._repo.create_user(user)
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
"""Get user by OAuth provider and ID."""
return await self._repo.get_user_by_oauth(provider, oauth_id)
async def count_users(self) -> int:
"""Return total number of registered users."""
return await self._repo.count_users()
async def count_admin_users(self) -> int:
"""Return number of admin users."""
return await self._repo.count_admin_users()
async def update_user(self, user: User) -> User:
"""Update an existing user."""
return await self._repo.update_user(user)
async def get_user_by_email(self, email: str) -> User | None:
"""Get user by email."""
return await self._repo.get_user_by_email(email)
+41
View File
@@ -0,0 +1,41 @@
"""User Pydantic models for authentication."""
from datetime import UTC, datetime
from typing import Literal
from uuid import UUID, uuid4
from pydantic import BaseModel, ConfigDict, EmailStr, Field
def _utc_now() -> datetime:
"""Return current UTC time (timezone-aware)."""
return datetime.now(UTC)
class User(BaseModel):
"""Internal user representation."""
model_config = ConfigDict(from_attributes=True)
id: UUID = Field(default_factory=uuid4, description="Primary key")
email: EmailStr = Field(..., description="Unique email address")
password_hash: str | None = Field(None, description="bcrypt hash, nullable for OAuth users")
system_role: Literal["admin", "user"] = Field(default="user")
created_at: datetime = Field(default_factory=_utc_now)
# OAuth linkage (optional)
oauth_provider: str | None = Field(None, description="e.g. 'github', 'google'")
oauth_id: str | None = Field(None, description="User ID from OAuth provider")
# Auth lifecycle
needs_setup: bool = Field(default=False, description="True for auto-created admin until setup completes")
token_version: int = Field(default=0, description="Incremented on password change to invalidate old JWTs")
class UserResponse(BaseModel):
"""Response model for user info endpoint."""
id: str
email: str
system_role: Literal["admin", "user"]
needs_setup: bool = False
+81
View File
@@ -0,0 +1,81 @@
"""Password hashing utilities with versioned hash format.
Hash format: ``$dfv<N>$<bcrypt_hash>`` where ``<N>`` is the version.
- **v1** (legacy): ``bcrypt(password)`` — plain bcrypt, susceptible to
72-byte silent truncation.
- **v2** (current): ``bcrypt(b64(sha256(password)))`` — SHA-256 pre-hash
avoids the 72-byte truncation limit so the full password contributes
to the hash.
Verification auto-detects the version and falls back to v1 for hashes
without a prefix, so existing deployments upgrade transparently on next
login.
"""
import asyncio
import base64
import hashlib
import bcrypt
_CURRENT_VERSION = 2
_PREFIX_V2 = "$dfv2$"
_PREFIX_V1 = "$dfv1$"
def _pre_hash_v2(password: str) -> bytes:
"""SHA-256 pre-hash to bypass bcrypt's 72-byte limit."""
return base64.b64encode(hashlib.sha256(password.encode("utf-8")).digest())
def hash_password(password: str) -> str:
"""Hash a password (current version: v2 — SHA-256 + bcrypt)."""
raw = bcrypt.hashpw(_pre_hash_v2(password), bcrypt.gensalt()).decode("utf-8")
return f"{_PREFIX_V2}{raw}"
def verify_password(plain_password: str, hashed_password: str) -> bool:
"""Verify a password, auto-detecting the hash version.
Accepts v2 (``$dfv2$…``), v1 (``$dfv1$…``), and bare bcrypt hashes
(treated as v1 for backward compatibility with pre-versioning data).
"""
try:
if hashed_password.startswith(_PREFIX_V2):
bcrypt_hash = hashed_password[len(_PREFIX_V2) :]
return bcrypt.checkpw(_pre_hash_v2(plain_password), bcrypt_hash.encode("utf-8"))
if hashed_password.startswith(_PREFIX_V1):
bcrypt_hash = hashed_password[len(_PREFIX_V1) :]
else:
bcrypt_hash = hashed_password
return bcrypt.checkpw(plain_password.encode("utf-8"), bcrypt_hash.encode("utf-8"))
except ValueError:
# bcrypt raises ValueError for malformed or corrupt hashes (e.g., invalid salt).
# Fail closed rather than crashing the request.
return False
def needs_rehash(hashed_password: str) -> bool:
"""Return True if the hash uses an older version and should be rehashed."""
return not hashed_password.startswith(_PREFIX_V2)
async def hash_password_async(password: str) -> str:
"""Hash a password using bcrypt (non-blocking).
Wraps the blocking bcrypt operation in a thread pool to avoid
blocking the event loop during password hashing.
"""
return await asyncio.to_thread(hash_password, password)
async def verify_password_async(plain_password: str, hashed_password: str) -> bool:
"""Verify a password against its hash (non-blocking).
Wraps the blocking bcrypt operation in a thread pool to avoid
blocking the event loop during password verification.
"""
return await asyncio.to_thread(verify_password, plain_password, hashed_password)
+24
View File
@@ -0,0 +1,24 @@
"""Auth provider abstraction."""
from abc import ABC, abstractmethod
class AuthProvider(ABC):
"""Abstract base class for authentication providers."""
@abstractmethod
async def authenticate(self, credentials: dict) -> "User | None":
"""Authenticate user with given credentials.
Returns User if authentication succeeds, None otherwise.
"""
raise NotImplementedError
@abstractmethod
async def get_user(self, user_id: str) -> "User | None":
"""Retrieve user by ID."""
raise NotImplementedError
# Import User at runtime to avoid circular imports
from app.gateway.auth.models import User # noqa: E402
@@ -0,0 +1,102 @@
"""User repository interface for abstracting database operations."""
from abc import ABC, abstractmethod
from app.gateway.auth.models import User
class UserNotFoundError(LookupError):
"""Raised when a user repository operation targets a non-existent row.
Subclass of :class:`LookupError` so callers that already catch
``LookupError`` for "missing entity" can keep working unchanged,
while specific call sites can pin to this class to distinguish
"concurrent delete during update" from other lookups.
"""
class UserRepository(ABC):
"""Abstract interface for user data storage.
Implement this interface to support different storage backends
(SQLite)
"""
@abstractmethod
async def create_user(self, user: User) -> User:
"""Create a new user.
Args:
user: User object to create
Returns:
Created User with ID assigned
Raises:
ValueError: If email already exists
"""
raise NotImplementedError
@abstractmethod
async def get_user_by_id(self, user_id: str) -> User | None:
"""Get user by ID.
Args:
user_id: User UUID as string
Returns:
User if found, None otherwise
"""
raise NotImplementedError
@abstractmethod
async def get_user_by_email(self, email: str) -> User | None:
"""Get user by email.
Args:
email: User email address
Returns:
User if found, None otherwise
"""
raise NotImplementedError
@abstractmethod
async def update_user(self, user: User) -> User:
"""Update an existing user.
Args:
user: User object with updated fields
Returns:
Updated User
Raises:
UserNotFoundError: If no row exists for ``user.id``. This is
a hard failure (not a no-op) so callers cannot mistake a
concurrent-delete race for a successful update.
"""
raise NotImplementedError
@abstractmethod
async def count_users(self) -> int:
"""Return total number of registered users."""
raise NotImplementedError
@abstractmethod
async def count_admin_users(self) -> int:
"""Return number of users with system_role == 'admin'."""
raise NotImplementedError
@abstractmethod
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
"""Get user by OAuth provider and ID.
Args:
provider: OAuth provider name (e.g. 'github', 'google')
oauth_id: User ID from the OAuth provider
Returns:
User if found, None otherwise
"""
raise NotImplementedError
@@ -0,0 +1,127 @@
"""SQLAlchemy-backed UserRepository implementation.
Uses the shared async session factory from
``deerflow.persistence.engine`` — the ``users`` table lives in the
same database as ``threads_meta``, ``runs``, ``run_events``, and
``feedback``.
Constructor takes the session factory directly (same pattern as the
other four repositories in ``deerflow.persistence.*``). Callers
construct this after ``init_engine_from_config()`` has run.
"""
from __future__ import annotations
from datetime import UTC
from uuid import UUID
from sqlalchemy import func, select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from app.gateway.auth.models import User
from app.gateway.auth.repositories.base import UserNotFoundError, UserRepository
from deerflow.persistence.user.model import UserRow
class SQLiteUserRepository(UserRepository):
"""Async user repository backed by the shared SQLAlchemy engine."""
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
self._sf = session_factory
# ── Converters ────────────────────────────────────────────────────
@staticmethod
def _row_to_user(row: UserRow) -> User:
return User(
id=UUID(row.id),
email=row.email,
password_hash=row.password_hash,
system_role=row.system_role, # type: ignore[arg-type]
# SQLite loses tzinfo on read; reattach UTC so downstream
# code can compare timestamps reliably.
created_at=row.created_at if row.created_at.tzinfo else row.created_at.replace(tzinfo=UTC),
oauth_provider=row.oauth_provider,
oauth_id=row.oauth_id,
needs_setup=row.needs_setup,
token_version=row.token_version,
)
@staticmethod
def _user_to_row(user: User) -> UserRow:
return UserRow(
id=str(user.id),
email=user.email,
password_hash=user.password_hash,
system_role=user.system_role,
created_at=user.created_at,
oauth_provider=user.oauth_provider,
oauth_id=user.oauth_id,
needs_setup=user.needs_setup,
token_version=user.token_version,
)
# ── CRUD ──────────────────────────────────────────────────────────
async def create_user(self, user: User) -> User:
"""Insert a new user. Raises ``ValueError`` on duplicate email."""
row = self._user_to_row(user)
async with self._sf() as session:
session.add(row)
try:
await session.commit()
except IntegrityError as exc:
await session.rollback()
raise ValueError(f"Email already registered: {user.email}") from exc
return user
async def get_user_by_id(self, user_id: str) -> User | None:
async with self._sf() as session:
row = await session.get(UserRow, user_id)
return self._row_to_user(row) if row is not None else None
async def get_user_by_email(self, email: str) -> User | None:
stmt = select(UserRow).where(UserRow.email == email)
async with self._sf() as session:
result = await session.execute(stmt)
row = result.scalar_one_or_none()
return self._row_to_user(row) if row is not None else None
async def update_user(self, user: User) -> User:
async with self._sf() as session:
row = await session.get(UserRow, str(user.id))
if row is None:
# Hard fail on concurrent delete: callers (reset_admin,
# password change handlers, _ensure_admin_user) all
# fetched the user just before this call, so a missing
# row here means the row vanished underneath us. Silent
# success would let the caller log "password reset" for
# a row that no longer exists.
raise UserNotFoundError(f"User {user.id} no longer exists")
row.email = user.email
row.password_hash = user.password_hash
row.system_role = user.system_role
row.oauth_provider = user.oauth_provider
row.oauth_id = user.oauth_id
row.needs_setup = user.needs_setup
row.token_version = user.token_version
await session.commit()
return user
async def count_users(self) -> int:
stmt = select(func.count()).select_from(UserRow)
async with self._sf() as session:
return await session.scalar(stmt) or 0
async def count_admin_users(self) -> int:
stmt = select(func.count()).select_from(UserRow).where(UserRow.system_role == "admin")
async with self._sf() as session:
return await session.scalar(stmt) or 0
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
stmt = select(UserRow).where(UserRow.oauth_provider == provider, UserRow.oauth_id == oauth_id)
async with self._sf() as session:
result = await session.execute(stmt)
row = result.scalar_one_or_none()
return self._row_to_user(row) if row is not None else None
+91
View File
@@ -0,0 +1,91 @@
"""CLI tool to reset an admin password.
Usage:
python -m app.gateway.auth.reset_admin
python -m app.gateway.auth.reset_admin --email admin@example.com
Writes the new password to ``.deer-flow/admin_initial_credentials.txt``
(mode 0600) instead of printing it, so CI / log aggregators never see
the cleartext secret.
"""
from __future__ import annotations
import argparse
import asyncio
import secrets
import sys
from sqlalchemy import select
from app.gateway.auth.credential_file import write_initial_credentials
from app.gateway.auth.password import hash_password
from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
from deerflow.persistence.user.model import UserRow
async def _run(email: str | None) -> int:
from deerflow.config import get_app_config
from deerflow.persistence.engine import (
close_engine,
get_session_factory,
init_engine_from_config,
)
config = get_app_config()
await init_engine_from_config(config.database)
try:
sf = get_session_factory()
if sf is None:
print("Error: persistence engine not available (check config.database).", file=sys.stderr)
return 1
repo = SQLiteUserRepository(sf)
if email:
user = await repo.get_user_by_email(email)
else:
# Find first admin via direct SELECT — repository does not
# expose a "first admin" helper and we do not want to add
# one just for this CLI.
async with sf() as session:
stmt = select(UserRow).where(UserRow.system_role == "admin").limit(1)
row = (await session.execute(stmt)).scalar_one_or_none()
if row is None:
user = None
else:
user = await repo.get_user_by_id(row.id)
if user is None:
if email:
print(f"Error: user '{email}' not found.", file=sys.stderr)
else:
print("Error: no admin user found.", file=sys.stderr)
return 1
new_password = secrets.token_urlsafe(16)
user.password_hash = hash_password(new_password)
user.token_version += 1
user.needs_setup = True
await repo.update_user(user)
cred_path = write_initial_credentials(user.email, new_password, label="reset")
print(f"Password reset for: {user.email}")
print(f"Credentials written to: {cred_path} (mode 0600)")
print("Next login will require setup (new email + password).")
return 0
finally:
await close_engine()
def main() -> None:
parser = argparse.ArgumentParser(description="Reset admin password")
parser.add_argument("--email", help="Admin email (default: first admin found)")
args = parser.parse_args()
exit_code = asyncio.run(_run(args.email))
sys.exit(exit_code)
if __name__ == "__main__":
main()
+126
View File
@@ -0,0 +1,126 @@
"""Global authentication middleware — fail-closed safety net.
Rejects unauthenticated requests to non-public paths with 401. When a
request passes the cookie check, resolves the JWT payload to a real
``User`` object and stamps it into both ``request.state.user`` and the
``deerflow.runtime.user_context`` contextvar so that repository-layer
owner filtering works automatically via the sentinel pattern.
Fine-grained permission checks remain in authz.py decorators.
"""
from collections.abc import Callable
from fastapi import HTTPException, Request, Response
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.responses import JSONResponse
from starlette.types import ASGIApp
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
from app.gateway.authz import _ALL_PERMISSIONS, AuthContext
from app.gateway.internal_auth import INTERNAL_AUTH_HEADER_NAME, get_internal_user, is_valid_internal_auth_token
from deerflow.runtime.user_context import reset_current_user, set_current_user
# Paths that never require authentication.
_PUBLIC_PATH_PREFIXES: tuple[str, ...] = (
"/health",
"/docs",
"/redoc",
"/openapi.json",
)
# Exact auth paths that are public (login/register/status check).
# /api/v1/auth/me, /api/v1/auth/change-password etc. are NOT public.
_PUBLIC_EXACT_PATHS: frozenset[str] = frozenset(
{
"/api/v1/auth/login/local",
"/api/v1/auth/register",
"/api/v1/auth/logout",
"/api/v1/auth/setup-status",
"/api/v1/auth/initialize",
}
)
def _is_public(path: str) -> bool:
stripped = path.rstrip("/")
if stripped in _PUBLIC_EXACT_PATHS:
return True
return any(path.startswith(prefix) for prefix in _PUBLIC_PATH_PREFIXES)
class AuthMiddleware(BaseHTTPMiddleware):
"""Strict auth gate: reject requests without a valid session.
Two-stage check for non-public paths:
1. Cookie presence — return 401 NOT_AUTHENTICATED if missing
2. JWT validation via ``get_optional_user_from_request`` — return 401
TOKEN_INVALID if the token is absent, malformed, expired, or the
signed user does not exist / is stale
On success, stamps ``request.state.user`` and the
``deerflow.runtime.user_context`` contextvar so that repository-layer
owner filters work downstream without every route needing a
``@require_auth`` decorator. Routes that need per-resource
authorization (e.g. "user A cannot read user B's thread by guessing
the URL") should additionally use ``@require_permission(...,
owner_check=True)`` for explicit enforcement — but authentication
itself is fully handled here.
"""
def __init__(self, app: ASGIApp) -> None:
super().__init__(app)
async def dispatch(self, request: Request, call_next: Callable) -> Response:
if _is_public(request.url.path):
return await call_next(request)
internal_user = None
if is_valid_internal_auth_token(request.headers.get(INTERNAL_AUTH_HEADER_NAME)):
internal_user = get_internal_user()
# Non-public path: require session cookie
if internal_user is None and not request.cookies.get("access_token"):
return JSONResponse(
status_code=401,
content={
"detail": AuthErrorResponse(
code=AuthErrorCode.NOT_AUTHENTICATED,
message="Authentication required",
).model_dump()
},
)
# Strict JWT validation: reject junk/expired tokens with 401
# right here instead of silently passing through. This closes
# the "junk cookie bypass" gap (AUTH_TEST_PLAN test 7.5.8):
# without this, non-isolation routes like /api/models would
# accept any cookie-shaped string as authentication.
#
# We call the *strict* resolver so that fine-grained error
# codes (token_expired, token_invalid, user_not_found, …)
# propagate from AuthErrorCode, not get flattened into one
# generic code. BaseHTTPMiddleware doesn't let HTTPException
# bubble up, so we catch and render it as JSONResponse here.
from app.gateway.deps import get_current_user_from_request
if internal_user is not None:
user = internal_user
else:
try:
user = await get_current_user_from_request(request)
except HTTPException as exc:
return JSONResponse(status_code=exc.status_code, content={"detail": exc.detail})
# Stamp both request.state.user (for the contextvar pattern)
# and request.state.auth (so @require_permission's "auth is
# None" branch short-circuits instead of running the entire
# JWT-decode + DB-lookup pipeline a second time per request).
request.state.user = user
request.state.auth = AuthContext(user=user, permissions=_ALL_PERMISSIONS)
token = set_current_user(user)
try:
return await call_next(request)
finally:
reset_current_user(token)
+301
View File
@@ -0,0 +1,301 @@
"""Authorization decorators and context for DeerFlow.
Inspired by LangGraph Auth system: https://github.com/langchain-ai/langgraph/blob/main/libs/sdk-py/langgraph_sdk/auth/__init__.py
**Usage:**
1. Use ``@require_auth`` on routes that need authentication
2. Use ``@require_permission("resource", "action", filter_key=...)`` for permission checks
3. The decorator chain processes from bottom to top
**Example:**
@router.get("/{thread_id}")
@require_auth
@require_permission("threads", "read", owner_check=True)
async def get_thread(thread_id: str, request: Request):
# User is authenticated and has threads:read permission
...
**Permission Model:**
- threads:read - View thread
- threads:write - Create/update thread
- threads:delete - Delete thread
- runs:create - Run agent
- runs:read - View run
- runs:cancel - Cancel run
"""
from __future__ import annotations
import functools
import inspect
from collections.abc import Callable
from types import SimpleNamespace
from typing import TYPE_CHECKING, Any, ParamSpec, TypeVar
from fastapi import HTTPException, Request
if TYPE_CHECKING:
from app.gateway.auth.models import User
P = ParamSpec("P")
T = TypeVar("T")
# Permission constants
class Permissions:
"""Permission constants for resource:action format."""
# Threads
THREADS_READ = "threads:read"
THREADS_WRITE = "threads:write"
THREADS_DELETE = "threads:delete"
# Runs
RUNS_CREATE = "runs:create"
RUNS_READ = "runs:read"
RUNS_CANCEL = "runs:cancel"
class AuthContext:
"""Authentication context for the current request.
Stored in request.state.auth after require_auth decoration.
Attributes:
user: The authenticated user, or None if anonymous
permissions: List of permission strings (e.g., "threads:read")
"""
__slots__ = ("user", "permissions")
def __init__(self, user: User | None = None, permissions: list[str] | None = None):
self.user = user
self.permissions = permissions or []
@property
def is_authenticated(self) -> bool:
"""Check if user is authenticated."""
return self.user is not None
def has_permission(self, resource: str, action: str) -> bool:
"""Check if context has permission for resource:action.
Args:
resource: Resource name (e.g., "threads")
action: Action name (e.g., "read")
Returns:
True if user has permission
"""
permission = f"{resource}:{action}"
return permission in self.permissions
def require_user(self) -> User:
"""Get user or raise 401.
Raises:
HTTPException 401 if not authenticated
"""
if not self.user:
raise HTTPException(status_code=401, detail="Authentication required")
return self.user
def get_auth_context(request: Request) -> AuthContext | None:
"""Get AuthContext from request state."""
return getattr(request.state, "auth", None)
_ALL_PERMISSIONS: list[str] = [
Permissions.THREADS_READ,
Permissions.THREADS_WRITE,
Permissions.THREADS_DELETE,
Permissions.RUNS_CREATE,
Permissions.RUNS_READ,
Permissions.RUNS_CANCEL,
]
def _make_test_request_stub() -> Any:
"""Create a minimal request-like object for direct unit calls.
Used when decorated route handlers are invoked without FastAPI's
request injection. Includes fields accessed by auth helpers.
"""
return SimpleNamespace(state=SimpleNamespace(), cookies={}, _deerflow_test_bypass_auth=True)
async def _authenticate(request: Request) -> AuthContext:
"""Authenticate request and return AuthContext.
Delegates to deps.get_optional_user_from_request() for the JWT→User pipeline.
Returns AuthContext with user=None for anonymous requests.
"""
from app.gateway.deps import get_optional_user_from_request
user = await get_optional_user_from_request(request)
if user is None:
return AuthContext(user=None, permissions=[])
# In future, permissions could be stored in user record
return AuthContext(user=user, permissions=_ALL_PERMISSIONS)
def require_auth[**P, T](func: Callable[P, T]) -> Callable[P, T]:
"""Decorator that authenticates the request and enforces authentication.
Independently raises HTTP 401 for unauthenticated requests, regardless of
whether ``AuthMiddleware`` is present in the ASGI stack. Sets the resolved
``AuthContext`` on ``request.state.auth`` for downstream handlers.
Must be placed ABOVE other decorators (executes after them).
Usage:
@router.get("/{thread_id}")
@require_auth # Bottom decorator (executes first after permission check)
@require_permission("threads", "read")
async def get_thread(thread_id: str, request: Request):
auth: AuthContext = request.state.auth
...
Raises:
HTTPException: 401 if the request is unauthenticated.
ValueError: If 'request' parameter is missing.
"""
@functools.wraps(func)
async def wrapper(*args: Any, **kwargs: Any) -> Any:
request = kwargs.get("request")
if request is None:
# Unit tests may call decorated handlers directly without a
# FastAPI Request object. Inject a minimal request stub when
# the wrapped function declares `request`.
if "request" in inspect.signature(func).parameters:
kwargs["request"] = _make_test_request_stub()
else:
raise ValueError("require_auth decorator requires 'request' parameter")
request = kwargs["request"]
if getattr(request, "_deerflow_test_bypass_auth", False):
return await func(*args, **kwargs)
# Authenticate and set context
auth_context = await _authenticate(request)
request.state.auth = auth_context
if not auth_context.is_authenticated:
raise HTTPException(status_code=401, detail="Authentication required")
return await func(*args, **kwargs)
return wrapper
def require_permission(
resource: str,
action: str,
owner_check: bool = False,
require_existing: bool = False,
) -> Callable[[Callable[P, T]], Callable[P, T]]:
"""Decorator that checks permission for resource:action.
Must be used AFTER @require_auth.
Args:
resource: Resource name (e.g., "threads", "runs")
action: Action name (e.g., "read", "write", "delete")
owner_check: If True, validates that the current user owns the resource.
Requires 'thread_id' path parameter and performs ownership check.
require_existing: Only meaningful with ``owner_check=True``. If True, a
missing ``threads_meta`` row counts as a denial (404)
instead of "untracked legacy thread, allow". Use on
**destructive / mutating** routes (DELETE, PATCH,
state-update) so a deleted thread can't be re-targeted
by another user via the missing-row code path.
Usage:
# Read-style: legacy untracked threads are allowed
@require_permission("threads", "read", owner_check=True)
async def get_thread(thread_id: str, request: Request):
...
# Destructive: thread row MUST exist and be owned by caller
@require_permission("threads", "delete", owner_check=True, require_existing=True)
async def delete_thread(thread_id: str, request: Request):
...
Raises:
HTTPException 401: If authentication required but user is anonymous
HTTPException 403: If user lacks permission
HTTPException 404: If owner_check=True but user doesn't own the thread
ValueError: If owner_check=True but 'thread_id' parameter is missing
"""
def decorator(func: Callable[P, T]) -> Callable[P, T]:
@functools.wraps(func)
async def wrapper(*args: Any, **kwargs: Any) -> Any:
request = kwargs.get("request")
if request is None:
# Unit tests may call decorated route handlers directly without
# constructing a FastAPI Request object. Inject a minimal stub
# when the wrapped function declares `request`.
if "request" in inspect.signature(func).parameters:
kwargs["request"] = _make_test_request_stub()
else:
return await func(*args, **kwargs)
request = kwargs["request"]
if getattr(request, "_deerflow_test_bypass_auth", False):
return await func(*args, **kwargs)
auth: AuthContext = getattr(request.state, "auth", None)
if auth is None:
auth = await _authenticate(request)
request.state.auth = auth
if not auth.is_authenticated:
raise HTTPException(status_code=401, detail="Authentication required")
# Check permission
if not auth.has_permission(resource, action):
raise HTTPException(
status_code=403,
detail=f"Permission denied: {resource}:{action}",
)
# Owner check for thread-specific resources.
#
# 2.0-rc moved thread metadata into the SQL persistence layer
# (``threads_meta`` table). We verify ownership via
# ``ThreadMetaStore.check_access``: it returns True for
# missing rows (untracked legacy thread) and for rows whose
# ``user_id`` is NULL (shared / pre-auth data), so this is
# strict-deny rather than strict-allow — only an *existing*
# row with a *different* user_id triggers 404.
if owner_check:
thread_id = kwargs.get("thread_id")
if thread_id is None:
raise ValueError("require_permission with owner_check=True requires 'thread_id' parameter")
from app.gateway.deps import get_thread_store
thread_store = get_thread_store(request)
allowed = await thread_store.check_access(
thread_id,
str(auth.user.id),
require_existing=require_existing,
)
if not allowed:
raise HTTPException(
status_code=404,
detail=f"Thread {thread_id} not found",
)
return await func(*args, **kwargs)
return wrapper
return decorator
+29
View File
@@ -0,0 +1,29 @@
import os
from pydantic import BaseModel, Field
class GatewayConfig(BaseModel):
"""Configuration for the API Gateway."""
host: str = Field(default="0.0.0.0", description="Host to bind the gateway server")
port: int = Field(default=8001, description="Port to bind the gateway server")
cors_origins: list[str] = Field(default_factory=lambda: ["http://localhost:3000"], description="Allowed CORS origins")
enable_docs: bool = Field(default=True, description="Enable Swagger/ReDoc/OpenAPI endpoints")
_gateway_config: GatewayConfig | None = None
def get_gateway_config() -> GatewayConfig:
"""Get gateway config, loading from environment if available."""
global _gateway_config
if _gateway_config is None:
cors_origins_str = os.getenv("CORS_ORIGINS", "http://localhost:3000")
_gateway_config = GatewayConfig(
host=os.getenv("GATEWAY_HOST", "0.0.0.0"),
port=int(os.getenv("GATEWAY_PORT", "8001")),
cors_origins=cors_origins_str.split(","),
enable_docs=os.getenv("GATEWAY_ENABLE_DOCS", "true").lower() == "true",
)
return _gateway_config
+113
View File
@@ -0,0 +1,113 @@
"""CSRF protection middleware for FastAPI.
Per RFC-001:
State-changing operations require CSRF protection.
"""
import secrets
from collections.abc import Callable
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.responses import JSONResponse
from starlette.types import ASGIApp
CSRF_COOKIE_NAME = "csrf_token"
CSRF_HEADER_NAME = "X-CSRF-Token"
CSRF_TOKEN_LENGTH = 64 # bytes
def is_secure_request(request: Request) -> bool:
"""Detect whether the original client request was made over HTTPS."""
return request.headers.get("x-forwarded-proto", request.url.scheme) == "https"
def generate_csrf_token() -> str:
"""Generate a secure random CSRF token."""
return secrets.token_urlsafe(CSRF_TOKEN_LENGTH)
def should_check_csrf(request: Request) -> bool:
"""Determine if a request needs CSRF validation.
CSRF is checked for state-changing methods (POST, PUT, DELETE, PATCH).
GET, HEAD, OPTIONS, and TRACE are exempt per RFC 7231.
"""
if request.method not in ("POST", "PUT", "DELETE", "PATCH"):
return False
path = request.url.path.rstrip("/")
# Exempt /api/v1/auth/me endpoint
if path == "/api/v1/auth/me":
return False
return True
_AUTH_EXEMPT_PATHS: frozenset[str] = frozenset(
{
"/api/v1/auth/login/local",
"/api/v1/auth/logout",
"/api/v1/auth/register",
"/api/v1/auth/initialize",
}
)
def is_auth_endpoint(request: Request) -> bool:
"""Check if the request is to an auth endpoint.
Auth endpoints don't need CSRF validation on first call (no token).
"""
return request.url.path.rstrip("/") in _AUTH_EXEMPT_PATHS
class CSRFMiddleware(BaseHTTPMiddleware):
"""Middleware that implements CSRF protection using Double Submit Cookie pattern."""
def __init__(self, app: ASGIApp) -> None:
super().__init__(app)
async def dispatch(self, request: Request, call_next: Callable) -> Response:
_is_auth = is_auth_endpoint(request)
if should_check_csrf(request) and not _is_auth:
cookie_token = request.cookies.get(CSRF_COOKIE_NAME)
header_token = request.headers.get(CSRF_HEADER_NAME)
if not cookie_token or not header_token:
return JSONResponse(
status_code=403,
content={"detail": "CSRF token missing. Include X-CSRF-Token header."},
)
if not secrets.compare_digest(cookie_token, header_token):
return JSONResponse(
status_code=403,
content={"detail": "CSRF token mismatch."},
)
response = await call_next(request)
# For auth endpoints that set up session, also set CSRF cookie
if _is_auth and request.method == "POST":
# Generate a new CSRF token for the session
csrf_token = generate_csrf_token()
is_https = is_secure_request(request)
response.set_cookie(
key=CSRF_COOKIE_NAME,
value=csrf_token,
httponly=False, # Must be JS-readable for Double Submit Cookie pattern
secure=is_https,
samesite="strict",
)
return response
def get_csrf_token(request: Request) -> str | None:
"""Get the CSRF token from the current request's cookies.
This is useful for server-side rendering where you need to embed
token in forms or headers.
"""
return request.cookies.get(CSRF_COOKIE_NAME)
+245
View File
@@ -0,0 +1,245 @@
"""Centralized accessors for singleton objects stored on ``app.state``.
**Getters** (used by routers): raise 503 when a required dependency is
missing, except ``get_store`` which returns ``None``.
Initialization is handled directly in ``app.py`` via :class:`AsyncExitStack`.
"""
from __future__ import annotations
from collections.abc import AsyncGenerator, Callable
from contextlib import AsyncExitStack, asynccontextmanager
from typing import TYPE_CHECKING, TypeVar, cast
from fastapi import FastAPI, HTTPException, Request
from langgraph.types import Checkpointer
from deerflow.config.app_config import AppConfig
from deerflow.persistence.feedback import FeedbackRepository
from deerflow.runtime import RunContext, RunManager, StreamBridge
from deerflow.runtime.events.store.base import RunEventStore
from deerflow.runtime.runs.store.base import RunStore
if TYPE_CHECKING:
from app.gateway.auth.local_provider import LocalAuthProvider
from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
from deerflow.persistence.thread_meta.base import ThreadMetaStore
T = TypeVar("T")
def get_config(request: Request) -> AppConfig:
"""Return the app-scoped ``AppConfig`` stored on ``app.state``."""
config = getattr(request.app.state, "config", None)
if config is None:
raise HTTPException(status_code=503, detail="Configuration not available")
return config
@asynccontextmanager
async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
"""Bootstrap and tear down all LangGraph runtime singletons.
Usage in ``app.py``::
async with langgraph_runtime(app):
yield
"""
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine_from_config
from deerflow.runtime import make_store, make_stream_bridge
from deerflow.runtime.checkpointer.async_provider import make_checkpointer
from deerflow.runtime.events.store import make_run_event_store
async with AsyncExitStack() as stack:
config = getattr(app.state, "config", None)
if config is None:
raise RuntimeError("langgraph_runtime() requires app.state.config to be initialized")
app.state.stream_bridge = await stack.enter_async_context(make_stream_bridge(config))
# Initialize persistence engine BEFORE checkpointer so that
# auto-create-database logic runs first (postgres backend).
await init_engine_from_config(config.database)
app.state.checkpointer = await stack.enter_async_context(make_checkpointer(config))
app.state.store = await stack.enter_async_context(make_store(config))
# Initialize repositories — one get_session_factory() call for all.
sf = get_session_factory()
if sf is not None:
from deerflow.persistence.feedback import FeedbackRepository
from deerflow.persistence.run import RunRepository
app.state.run_store = RunRepository(sf)
app.state.feedback_repo = FeedbackRepository(sf)
else:
from deerflow.runtime.runs.store.memory import MemoryRunStore
app.state.run_store = MemoryRunStore()
app.state.feedback_repo = None
from deerflow.persistence.thread_meta import make_thread_store
app.state.thread_store = make_thread_store(sf, app.state.store)
# Run event store (has its own factory with config-driven backend selection)
run_events_config = getattr(config, "run_events", None)
app.state.run_event_store = make_run_event_store(run_events_config)
# RunManager with store backing for persistence
app.state.run_manager = RunManager(store=app.state.run_store)
try:
yield
finally:
await close_engine()
# ---------------------------------------------------------------------------
# Getters called by routers per-request
# ---------------------------------------------------------------------------
def _require(attr: str, label: str) -> Callable[[Request], T]:
"""Create a FastAPI dependency that returns ``app.state.<attr>`` or 503."""
def dep(request: Request) -> T:
val = getattr(request.app.state, attr, None)
if val is None:
raise HTTPException(status_code=503, detail=f"{label} not available")
return cast(T, val)
dep.__name__ = dep.__qualname__ = f"get_{attr}"
return dep
get_stream_bridge: Callable[[Request], StreamBridge] = _require("stream_bridge", "Stream bridge")
get_run_manager: Callable[[Request], RunManager] = _require("run_manager", "Run manager")
get_checkpointer: Callable[[Request], Checkpointer] = _require("checkpointer", "Checkpointer")
get_run_event_store: Callable[[Request], RunEventStore] = _require("run_event_store", "Run event store")
get_feedback_repo: Callable[[Request], FeedbackRepository] = _require("feedback_repo", "Feedback")
get_run_store: Callable[[Request], RunStore] = _require("run_store", "Run store")
def get_store(request: Request):
"""Return the global store (may be ``None`` if not configured)."""
return getattr(request.app.state, "store", None)
def get_thread_store(request: Request) -> ThreadMetaStore:
"""Return the thread metadata store (SQL or memory-backed)."""
val = getattr(request.app.state, "thread_store", None)
if val is None:
raise HTTPException(status_code=503, detail="Thread metadata store not available")
return val
def get_run_context(request: Request) -> RunContext:
"""Build a :class:`RunContext` from ``app.state`` singletons.
Returns a *base* context with infrastructure dependencies.
"""
config = get_config(request)
return RunContext(
checkpointer=get_checkpointer(request),
store=get_store(request),
event_store=get_run_event_store(request),
run_events_config=getattr(config, "run_events", None),
thread_store=get_thread_store(request),
app_config=config,
)
# ---------------------------------------------------------------------------
# Auth helpers (used by authz.py and auth middleware)
# ---------------------------------------------------------------------------
# Cached singletons to avoid repeated instantiation per request
_cached_local_provider: LocalAuthProvider | None = None
_cached_repo: SQLiteUserRepository | None = None
def get_local_provider() -> LocalAuthProvider:
"""Get or create the cached LocalAuthProvider singleton.
Must be called after ``init_engine_from_config()`` — the shared
session factory is required to construct the user repository.
"""
global _cached_local_provider, _cached_repo
if _cached_repo is None:
from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
from deerflow.persistence.engine import get_session_factory
sf = get_session_factory()
if sf is None:
raise RuntimeError("get_local_provider() called before init_engine_from_config(); cannot access users table")
_cached_repo = SQLiteUserRepository(sf)
if _cached_local_provider is None:
from app.gateway.auth.local_provider import LocalAuthProvider
_cached_local_provider = LocalAuthProvider(repository=_cached_repo)
return _cached_local_provider
async def get_current_user_from_request(request: Request):
"""Get the current authenticated user from the request cookie.
Raises HTTPException 401 if not authenticated.
"""
from app.gateway.auth import decode_token
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse, TokenError, token_error_to_code
access_token = request.cookies.get("access_token")
if not access_token:
raise HTTPException(
status_code=401,
detail=AuthErrorResponse(code=AuthErrorCode.NOT_AUTHENTICATED, message="Not authenticated").model_dump(),
)
payload = decode_token(access_token)
if isinstance(payload, TokenError):
raise HTTPException(
status_code=401,
detail=AuthErrorResponse(code=token_error_to_code(payload), message=f"Token error: {payload.value}").model_dump(),
)
provider = get_local_provider()
user = await provider.get_user(payload.sub)
if user is None:
raise HTTPException(
status_code=401,
detail=AuthErrorResponse(code=AuthErrorCode.USER_NOT_FOUND, message="User not found").model_dump(),
)
# Token version mismatch → password was changed, token is stale
if user.token_version != payload.ver:
raise HTTPException(
status_code=401,
detail=AuthErrorResponse(code=AuthErrorCode.TOKEN_INVALID, message="Token revoked (password changed)").model_dump(),
)
return user
async def get_optional_user_from_request(request: Request):
"""Get optional authenticated user from request.
Returns None if not authenticated.
"""
try:
return await get_current_user_from_request(request)
except HTTPException:
return None
async def get_current_user(request: Request) -> str | None:
"""Extract user_id from request cookie, or None if not authenticated.
Thin adapter that returns the string id for callers that only need
identification (e.g., ``feedback.py``). Full-user callers should use
``get_current_user_from_request`` or ``get_optional_user_from_request``.
"""
user = await get_optional_user_from_request(request)
return str(user.id) if user else None
+26
View File
@@ -0,0 +1,26 @@
"""Process-local authentication for Gateway internal callers."""
from __future__ import annotations
import secrets
from types import SimpleNamespace
from deerflow.runtime.user_context import DEFAULT_USER_ID
INTERNAL_AUTH_HEADER_NAME = "X-DeerFlow-Internal-Token"
_INTERNAL_AUTH_TOKEN = secrets.token_urlsafe(32)
def create_internal_auth_headers() -> dict[str, str]:
"""Return headers that authenticate same-process Gateway internal calls."""
return {INTERNAL_AUTH_HEADER_NAME: _INTERNAL_AUTH_TOKEN}
def is_valid_internal_auth_token(token: str | None) -> bool:
"""Return True when *token* matches the process-local internal token."""
return bool(token) and secrets.compare_digest(token, _INTERNAL_AUTH_TOKEN)
def get_internal_user():
"""Return the synthetic user used for trusted internal channel calls."""
return SimpleNamespace(id=DEFAULT_USER_ID, system_role="internal")
+106
View File
@@ -0,0 +1,106 @@
"""LangGraph Server auth handler — shares JWT logic with Gateway.
Loaded by LangGraph Server via langgraph.json ``auth.path``.
Reuses the same ``decode_token`` / ``get_auth_config`` as Gateway,
so both modes validate tokens with the same secret and rules.
Two layers:
1. @auth.authenticate — validates JWT cookie, extracts user_id,
and enforces CSRF on state-changing methods (POST/PUT/DELETE/PATCH)
2. @auth.on — returns metadata filter so each user only sees own threads
"""
import secrets
from langgraph_sdk import Auth
from app.gateway.auth.errors import TokenError
from app.gateway.auth.jwt import decode_token
from app.gateway.deps import get_local_provider
auth = Auth()
# Methods that require CSRF validation (state-changing per RFC 7231).
_CSRF_METHODS = frozenset({"POST", "PUT", "DELETE", "PATCH"})
def _check_csrf(request) -> None:
"""Enforce Double Submit Cookie CSRF check for state-changing requests.
Mirrors Gateway's CSRFMiddleware logic so that LangGraph routes
proxied directly by nginx have the same CSRF protection.
"""
method = getattr(request, "method", "") or ""
if method.upper() not in _CSRF_METHODS:
return
cookie_token = request.cookies.get("csrf_token")
header_token = request.headers.get("x-csrf-token")
if not cookie_token or not header_token:
raise Auth.exceptions.HTTPException(
status_code=403,
detail="CSRF token missing. Include X-CSRF-Token header.",
)
if not secrets.compare_digest(cookie_token, header_token):
raise Auth.exceptions.HTTPException(
status_code=403,
detail="CSRF token mismatch.",
)
@auth.authenticate
async def authenticate(request):
"""Validate the session cookie, decode JWT, and check token_version.
Same validation chain as Gateway's get_current_user_from_request:
cookie → decode JWT → DB lookup → token_version match
Also enforces CSRF on state-changing methods.
"""
# CSRF check before authentication so forged cross-site requests
# are rejected early, even if the cookie carries a valid JWT.
_check_csrf(request)
token = request.cookies.get("access_token")
if not token:
raise Auth.exceptions.HTTPException(
status_code=401,
detail="Not authenticated",
)
payload = decode_token(token)
if isinstance(payload, TokenError):
raise Auth.exceptions.HTTPException(
status_code=401,
detail="Invalid token",
)
user = await get_local_provider().get_user(payload.sub)
if user is None:
raise Auth.exceptions.HTTPException(
status_code=401,
detail="User not found",
)
if user.token_version != payload.ver:
raise Auth.exceptions.HTTPException(
status_code=401,
detail="Token revoked (password changed)",
)
return payload.sub
@auth.on
async def add_owner_filter(ctx: Auth.types.AuthContext, value: dict):
"""Inject user_id metadata on writes; filter by user_id on reads.
Gateway stores thread ownership as ``metadata.user_id``.
This handler ensures LangGraph Server enforces the same isolation.
"""
# On create/update: stamp user_id into metadata
metadata = value.setdefault("metadata", {})
metadata["user_id"] = ctx.user.identity
# Return filter dict — LangGraph applies it to search/read/delete
return {"user_id": ctx.user.identity}
+29
View File
@@ -0,0 +1,29 @@
"""Shared path resolution for thread virtual paths (e.g. mnt/user-data/outputs/...)."""
from pathlib import Path
from fastapi import HTTPException
from deerflow.config.paths import get_paths
from deerflow.runtime.user_context import get_effective_user_id
def resolve_thread_virtual_path(thread_id: str, virtual_path: str) -> Path:
"""Resolve a virtual path to the actual filesystem path under thread user-data.
Args:
thread_id: The thread ID.
virtual_path: The virtual path as seen inside the sandbox
(e.g., /mnt/user-data/outputs/file.txt).
Returns:
The resolved filesystem path.
Raises:
HTTPException: If the path is invalid or outside allowed directories.
"""
try:
return get_paths().resolve_virtual_path(thread_id, virtual_path, user_id=get_effective_user_id())
except ValueError as e:
status = 403 if "traversal" in str(e) else 400
raise HTTPException(status_code=status, detail=str(e))
+3
View File
@@ -0,0 +1,3 @@
from . import artifacts, assistants_compat, mcp, models, skills, suggestions, thread_runs, threads, uploads
__all__ = ["artifacts", "assistants_compat", "mcp", "models", "skills", "suggestions", "threads", "thread_runs", "uploads"]
+421
View File
@@ -0,0 +1,421 @@
"""CRUD API for custom agents."""
import logging
import re
import shutil
import yaml
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel, Field
from deerflow.config.agents_api_config import get_agents_api_config
from deerflow.config.agents_config import AgentConfig, list_custom_agents, load_agent_config, load_agent_soul
from deerflow.config.paths import get_paths
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api", tags=["agents"])
AGENT_NAME_PATTERN = re.compile(r"^[A-Za-z0-9-]+$")
class AgentResponse(BaseModel):
"""Response model for a custom agent."""
name: str = Field(..., description="Agent name (hyphen-case)")
description: str = Field(default="", description="Agent description")
model: str | None = Field(default=None, description="Optional model override")
tool_groups: list[str] | None = Field(default=None, description="Optional tool group whitelist")
skills: list[str] | None = Field(default=None, description="Optional skill whitelist (None=all, []=none)")
soul: str | None = Field(default=None, description="SOUL.md content")
class AgentsListResponse(BaseModel):
"""Response model for listing all custom agents."""
agents: list[AgentResponse]
class AgentCreateRequest(BaseModel):
"""Request body for creating a custom agent."""
name: str = Field(..., description="Agent name (must match ^[A-Za-z0-9-]+$, stored as lowercase)")
description: str = Field(default="", description="Agent description")
model: str | None = Field(default=None, description="Optional model override")
tool_groups: list[str] | None = Field(default=None, description="Optional tool group whitelist")
skills: list[str] | None = Field(default=None, description="Optional skill whitelist (None=all enabled, []=none)")
soul: str = Field(default="", description="SOUL.md content — agent personality and behavioral guardrails")
class AgentUpdateRequest(BaseModel):
"""Request body for updating a custom agent."""
description: str | None = Field(default=None, description="Updated description")
model: str | None = Field(default=None, description="Updated model override")
tool_groups: list[str] | None = Field(default=None, description="Updated tool group whitelist")
skills: list[str] | None = Field(default=None, description="Updated skill whitelist (None=all, []=none)")
soul: str | None = Field(default=None, description="Updated SOUL.md content")
def _validate_agent_name(name: str) -> None:
"""Validate agent name against allowed pattern.
Args:
name: The agent name to validate.
Raises:
HTTPException: 422 if the name is invalid.
"""
if not AGENT_NAME_PATTERN.match(name):
raise HTTPException(
status_code=422,
detail=f"Invalid agent name '{name}'. Must match ^[A-Za-z0-9-]+$ (letters, digits, and hyphens only).",
)
def _normalize_agent_name(name: str) -> str:
"""Normalize agent name to lowercase for filesystem storage."""
return name.lower()
def _require_agents_api_enabled() -> None:
"""Reject access unless the custom-agent management API is explicitly enabled."""
if not get_agents_api_config().enabled:
raise HTTPException(
status_code=403,
detail=("Custom-agent management API is disabled. Set agents_api.enabled=true to expose agent and user-profile routes over HTTP."),
)
def _agent_config_to_response(agent_cfg: AgentConfig, include_soul: bool = False) -> AgentResponse:
"""Convert AgentConfig to AgentResponse."""
soul: str | None = None
if include_soul:
soul = load_agent_soul(agent_cfg.name) or ""
return AgentResponse(
name=agent_cfg.name,
description=agent_cfg.description,
model=agent_cfg.model,
tool_groups=agent_cfg.tool_groups,
skills=agent_cfg.skills,
soul=soul,
)
@router.get(
"/agents",
response_model=AgentsListResponse,
summary="List Custom Agents",
description="List all custom agents available in the agents directory, including their soul content.",
)
async def list_agents() -> AgentsListResponse:
"""List all custom agents.
Returns:
List of all custom agents with their metadata and soul content.
"""
_require_agents_api_enabled()
try:
agents = list_custom_agents()
return AgentsListResponse(agents=[_agent_config_to_response(a, include_soul=True) for a in agents])
except Exception as e:
logger.error(f"Failed to list agents: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to list agents: {str(e)}")
@router.get(
"/agents/check",
summary="Check Agent Name",
description="Validate an agent name and check if it is available (case-insensitive).",
)
async def check_agent_name(name: str) -> dict:
"""Check whether an agent name is valid and not yet taken.
Args:
name: The agent name to check.
Returns:
``{"available": true/false, "name": "<normalized>"}``
Raises:
HTTPException: 422 if the name is invalid.
"""
_require_agents_api_enabled()
_validate_agent_name(name)
normalized = _normalize_agent_name(name)
available = not get_paths().agent_dir(normalized).exists()
return {"available": available, "name": normalized}
@router.get(
"/agents/{name}",
response_model=AgentResponse,
summary="Get Custom Agent",
description="Retrieve details and SOUL.md content for a specific custom agent.",
)
async def get_agent(name: str) -> AgentResponse:
"""Get a specific custom agent by name.
Args:
name: The agent name.
Returns:
Agent details including SOUL.md content.
Raises:
HTTPException: 404 if agent not found.
"""
_require_agents_api_enabled()
_validate_agent_name(name)
name = _normalize_agent_name(name)
try:
agent_cfg = load_agent_config(name)
return _agent_config_to_response(agent_cfg, include_soul=True)
except FileNotFoundError:
raise HTTPException(status_code=404, detail=f"Agent '{name}' not found")
except Exception as e:
logger.error(f"Failed to get agent '{name}': {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to get agent: {str(e)}")
@router.post(
"/agents",
response_model=AgentResponse,
status_code=201,
summary="Create Custom Agent",
description="Create a new custom agent with its config and SOUL.md.",
)
async def create_agent_endpoint(request: AgentCreateRequest) -> AgentResponse:
"""Create a new custom agent.
Args:
request: The agent creation request.
Returns:
The created agent details.
Raises:
HTTPException: 409 if agent already exists, 422 if name is invalid.
"""
_require_agents_api_enabled()
_validate_agent_name(request.name)
normalized_name = _normalize_agent_name(request.name)
agent_dir = get_paths().agent_dir(normalized_name)
if agent_dir.exists():
raise HTTPException(status_code=409, detail=f"Agent '{normalized_name}' already exists")
try:
agent_dir.mkdir(parents=True, exist_ok=True)
# Write config.yaml
config_data: dict = {"name": normalized_name}
if request.description:
config_data["description"] = request.description
if request.model is not None:
config_data["model"] = request.model
if request.tool_groups is not None:
config_data["tool_groups"] = request.tool_groups
if request.skills is not None:
config_data["skills"] = request.skills
config_file = agent_dir / "config.yaml"
with open(config_file, "w", encoding="utf-8") as f:
yaml.dump(config_data, f, default_flow_style=False, allow_unicode=True)
# Write SOUL.md
soul_file = agent_dir / "SOUL.md"
soul_file.write_text(request.soul, encoding="utf-8")
logger.info(f"Created agent '{normalized_name}' at {agent_dir}")
agent_cfg = load_agent_config(normalized_name)
return _agent_config_to_response(agent_cfg, include_soul=True)
except HTTPException:
raise
except Exception as e:
# Clean up on failure
if agent_dir.exists():
shutil.rmtree(agent_dir)
logger.error(f"Failed to create agent '{request.name}': {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to create agent: {str(e)}")
@router.put(
"/agents/{name}",
response_model=AgentResponse,
summary="Update Custom Agent",
description="Update an existing custom agent's config and/or SOUL.md.",
)
async def update_agent(name: str, request: AgentUpdateRequest) -> AgentResponse:
"""Update an existing custom agent.
Args:
name: The agent name.
request: The update request (all fields optional).
Returns:
The updated agent details.
Raises:
HTTPException: 404 if agent not found.
"""
_require_agents_api_enabled()
_validate_agent_name(name)
name = _normalize_agent_name(name)
try:
agent_cfg = load_agent_config(name)
except FileNotFoundError:
raise HTTPException(status_code=404, detail=f"Agent '{name}' not found")
agent_dir = get_paths().agent_dir(name)
try:
# Update config if any config fields changed
# Use model_fields_set to distinguish "field omitted" from "explicitly set to null".
# This is critical for skills where None means "inherit all" (not "don't change").
fields_set = request.model_fields_set
config_changed = bool(fields_set & {"description", "model", "tool_groups", "skills"})
if config_changed:
updated: dict = {
"name": agent_cfg.name,
"description": request.description if "description" in fields_set else agent_cfg.description,
}
new_model = request.model if "model" in fields_set else agent_cfg.model
if new_model is not None:
updated["model"] = new_model
new_tool_groups = request.tool_groups if "tool_groups" in fields_set else agent_cfg.tool_groups
if new_tool_groups is not None:
updated["tool_groups"] = new_tool_groups
# skills: None = inherit all, [] = no skills, ["a","b"] = whitelist
if "skills" in fields_set:
new_skills = request.skills
else:
new_skills = agent_cfg.skills
if new_skills is not None:
updated["skills"] = new_skills
config_file = agent_dir / "config.yaml"
with open(config_file, "w", encoding="utf-8") as f:
yaml.dump(updated, f, default_flow_style=False, allow_unicode=True)
# Update SOUL.md if provided
if request.soul is not None:
soul_path = agent_dir / "SOUL.md"
soul_path.write_text(request.soul, encoding="utf-8")
logger.info(f"Updated agent '{name}'")
refreshed_cfg = load_agent_config(name)
return _agent_config_to_response(refreshed_cfg, include_soul=True)
except HTTPException:
raise
except Exception as e:
logger.error(f"Failed to update agent '{name}': {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to update agent: {str(e)}")
class UserProfileResponse(BaseModel):
"""Response model for the global user profile (USER.md)."""
content: str | None = Field(default=None, description="USER.md content, or null if not yet created")
class UserProfileUpdateRequest(BaseModel):
"""Request body for setting the global user profile."""
content: str = Field(default="", description="USER.md content — describes the user's background and preferences")
@router.get(
"/user-profile",
response_model=UserProfileResponse,
summary="Get User Profile",
description="Read the global USER.md file that is injected into all custom agents.",
)
async def get_user_profile() -> UserProfileResponse:
"""Return the current USER.md content.
Returns:
UserProfileResponse with content=None if USER.md does not exist yet.
"""
_require_agents_api_enabled()
try:
user_md_path = get_paths().user_md_file
if not user_md_path.exists():
return UserProfileResponse(content=None)
raw = user_md_path.read_text(encoding="utf-8").strip()
return UserProfileResponse(content=raw or None)
except Exception as e:
logger.error(f"Failed to read user profile: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to read user profile: {str(e)}")
@router.put(
"/user-profile",
response_model=UserProfileResponse,
summary="Update User Profile",
description="Write the global USER.md file that is injected into all custom agents.",
)
async def update_user_profile(request: UserProfileUpdateRequest) -> UserProfileResponse:
"""Create or overwrite the global USER.md.
Args:
request: The update request with the new USER.md content.
Returns:
UserProfileResponse with the saved content.
"""
_require_agents_api_enabled()
try:
paths = get_paths()
paths.base_dir.mkdir(parents=True, exist_ok=True)
paths.user_md_file.write_text(request.content, encoding="utf-8")
logger.info(f"Updated USER.md at {paths.user_md_file}")
return UserProfileResponse(content=request.content or None)
except Exception as e:
logger.error(f"Failed to update user profile: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to update user profile: {str(e)}")
@router.delete(
"/agents/{name}",
status_code=204,
summary="Delete Custom Agent",
description="Delete a custom agent and all its files (config, SOUL.md, memory).",
)
async def delete_agent(name: str) -> None:
"""Delete a custom agent.
Args:
name: The agent name.
Raises:
HTTPException: 404 if agent not found.
"""
_require_agents_api_enabled()
_validate_agent_name(name)
name = _normalize_agent_name(name)
agent_dir = get_paths().agent_dir(name)
if not agent_dir.exists():
raise HTTPException(status_code=404, detail=f"Agent '{name}' not found")
try:
shutil.rmtree(agent_dir)
logger.info(f"Deleted agent '{name}' from {agent_dir}")
except Exception as e:
logger.error(f"Failed to delete agent '{name}': {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to delete agent: {str(e)}")
+183
View File
@@ -0,0 +1,183 @@
import logging
import mimetypes
import zipfile
from pathlib import Path
from urllib.parse import quote
from fastapi import APIRouter, HTTPException, Request
from fastapi.responses import FileResponse, PlainTextResponse, Response
from app.gateway.authz import require_permission
from app.gateway.path_utils import resolve_thread_virtual_path
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api", tags=["artifacts"])
ACTIVE_CONTENT_MIME_TYPES = {
"text/html",
"application/xhtml+xml",
"image/svg+xml",
}
def _build_content_disposition(disposition_type: str, filename: str) -> str:
"""Build an RFC 5987 encoded Content-Disposition header value."""
return f"{disposition_type}; filename*=UTF-8''{quote(filename)}"
def _build_attachment_headers(filename: str, extra_headers: dict[str, str] | None = None) -> dict[str, str]:
headers = {"Content-Disposition": _build_content_disposition("attachment", filename)}
if extra_headers:
headers.update(extra_headers)
return headers
def is_text_file_by_content(path: Path, sample_size: int = 8192) -> bool:
"""Check if file is text by examining content for null bytes."""
try:
with open(path, "rb") as f:
chunk = f.read(sample_size)
# Text files shouldn't contain null bytes
return b"\x00" not in chunk
except Exception:
return False
def _extract_file_from_skill_archive(zip_path: Path, internal_path: str) -> bytes | None:
"""Extract a file from a .skill ZIP archive.
Args:
zip_path: Path to the .skill file (ZIP archive).
internal_path: Path to the file inside the archive (e.g., "SKILL.md").
Returns:
The file content as bytes, or None if not found.
"""
if not zipfile.is_zipfile(zip_path):
return None
try:
with zipfile.ZipFile(zip_path, "r") as zip_ref:
# List all files in the archive
namelist = zip_ref.namelist()
# Try direct path first
if internal_path in namelist:
return zip_ref.read(internal_path)
# Try with any top-level directory prefix (e.g., "skill-name/SKILL.md")
for name in namelist:
if name.endswith("/" + internal_path) or name == internal_path:
return zip_ref.read(name)
# Not found
return None
except (zipfile.BadZipFile, KeyError):
return None
@router.get(
"/threads/{thread_id}/artifacts/{path:path}",
summary="Get Artifact File",
description="Retrieve an artifact file generated by the AI agent. Text and binary files can be viewed inline, while active web content is always downloaded.",
)
@require_permission("threads", "read", owner_check=True)
async def get_artifact(thread_id: str, path: str, request: Request, download: bool = False) -> Response:
"""Get an artifact file by its path.
The endpoint automatically detects file types and returns appropriate content types.
Use the `download` query parameter to force file download for non-active content.
Args:
thread_id: The thread ID.
path: The artifact path with virtual prefix (e.g., mnt/user-data/outputs/file.txt).
request: FastAPI request object (automatically injected).
Returns:
The file content as a FileResponse with appropriate content type:
- Active content (HTML/XHTML/SVG): Served as download attachment
- Text files: Plain text with proper MIME type
- Binary files: Inline display with download option
Raises:
HTTPException:
- 400 if path is invalid or not a file
- 403 if access denied (path traversal detected)
- 404 if file not found
Query Parameters:
download (bool): If true, forces attachment download for file types that are
otherwise returned inline or as plain text. Active HTML/XHTML/SVG content
is always downloaded regardless of this flag.
Example:
- Get text file inline: `/api/threads/abc123/artifacts/mnt/user-data/outputs/notes.txt`
- Download file: `/api/threads/abc123/artifacts/mnt/user-data/outputs/data.csv?download=true`
- Active web content such as `.html`, `.xhtml`, and `.svg` artifacts is always downloaded
"""
# Check if this is a request for a file inside a .skill archive (e.g., xxx.skill/SKILL.md)
if ".skill/" in path:
# Split the path at ".skill/" to get the ZIP file path and internal path
skill_marker = ".skill/"
marker_pos = path.find(skill_marker)
skill_file_path = path[: marker_pos + len(".skill")] # e.g., "mnt/user-data/outputs/my-skill.skill"
internal_path = path[marker_pos + len(skill_marker) :] # e.g., "SKILL.md"
actual_skill_path = resolve_thread_virtual_path(thread_id, skill_file_path)
if not actual_skill_path.exists():
raise HTTPException(status_code=404, detail=f"Skill file not found: {skill_file_path}")
if not actual_skill_path.is_file():
raise HTTPException(status_code=400, detail=f"Path is not a file: {skill_file_path}")
# Extract the file from the .skill archive
content = _extract_file_from_skill_archive(actual_skill_path, internal_path)
if content is None:
raise HTTPException(status_code=404, detail=f"File '{internal_path}' not found in skill archive")
# Determine MIME type based on the internal file
mime_type, _ = mimetypes.guess_type(internal_path)
# Add cache headers to avoid repeated ZIP extraction (cache for 5 minutes)
cache_headers = {"Cache-Control": "private, max-age=300"}
download_name = Path(internal_path).name or actual_skill_path.stem
if download or mime_type in ACTIVE_CONTENT_MIME_TYPES:
return Response(content=content, media_type=mime_type or "application/octet-stream", headers=_build_attachment_headers(download_name, cache_headers))
if mime_type and mime_type.startswith("text/"):
return PlainTextResponse(content=content.decode("utf-8"), media_type=mime_type, headers=cache_headers)
# Default to plain text for unknown types that look like text
try:
return PlainTextResponse(content=content.decode("utf-8"), media_type="text/plain", headers=cache_headers)
except UnicodeDecodeError:
return Response(content=content, media_type=mime_type or "application/octet-stream", headers=cache_headers)
actual_path = resolve_thread_virtual_path(thread_id, path)
logger.info(f"Resolving artifact path: thread_id={thread_id}, requested_path={path}, actual_path={actual_path}")
if not actual_path.exists():
raise HTTPException(status_code=404, detail=f"Artifact not found: {path}")
if not actual_path.is_file():
raise HTTPException(status_code=400, detail=f"Path is not a file: {path}")
mime_type, _ = mimetypes.guess_type(actual_path)
if download:
return FileResponse(path=actual_path, filename=actual_path.name, media_type=mime_type, headers=_build_attachment_headers(actual_path.name))
# Always force download for active content types to prevent script execution
# in the application origin when users open generated artifacts.
if mime_type in ACTIVE_CONTENT_MIME_TYPES:
return FileResponse(path=actual_path, filename=actual_path.name, media_type=mime_type, headers=_build_attachment_headers(actual_path.name))
if mime_type and mime_type.startswith("text/"):
return PlainTextResponse(content=actual_path.read_text(encoding="utf-8"), media_type=mime_type)
if is_text_file_by_content(actual_path):
return PlainTextResponse(content=actual_path.read_text(encoding="utf-8"), media_type=mime_type)
return Response(content=actual_path.read_bytes(), media_type=mime_type, headers={"Content-Disposition": _build_content_disposition("inline", actual_path.name)})
@@ -0,0 +1,149 @@
"""Assistants compatibility endpoints.
Provides LangGraph Platform-compatible assistants API backed by the
``langgraph.json`` graph registry and ``config.yaml`` agent definitions.
This is a minimal stub that satisfies the ``useStream`` React hook's
initialization requirements (``assistants.search()`` and ``assistants.get()``).
"""
from __future__ import annotations
import logging
from datetime import UTC, datetime
from typing import Any
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel, Field
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/assistants", tags=["assistants-compat"])
class AssistantResponse(BaseModel):
assistant_id: str
graph_id: str
name: str
config: dict[str, Any] = Field(default_factory=dict)
metadata: dict[str, Any] = Field(default_factory=dict)
description: str | None = None
created_at: str = ""
updated_at: str = ""
version: int = 1
class AssistantSearchRequest(BaseModel):
graph_id: str | None = None
name: str | None = None
metadata: dict[str, Any] | None = None
limit: int = 10
offset: int = 0
def _get_default_assistant() -> AssistantResponse:
"""Return the default lead_agent assistant."""
now = datetime.now(UTC).isoformat()
return AssistantResponse(
assistant_id="lead_agent",
graph_id="lead_agent",
name="lead_agent",
config={},
metadata={"created_by": "system"},
description="DeerFlow lead agent",
created_at=now,
updated_at=now,
version=1,
)
def _list_assistants() -> list[AssistantResponse]:
"""List all available assistants from config."""
assistants = [_get_default_assistant()]
# Also include custom agents from config.yaml agents directory
try:
from deerflow.config.agents_config import list_custom_agents
for agent_cfg in list_custom_agents():
now = datetime.now(UTC).isoformat()
assistants.append(
AssistantResponse(
assistant_id=agent_cfg.name,
graph_id="lead_agent", # All agents use the same graph
name=agent_cfg.name,
config={},
metadata={"created_by": "user"},
description=agent_cfg.description or "",
created_at=now,
updated_at=now,
version=1,
)
)
except Exception:
logger.debug("Could not load custom agents for assistants list")
return assistants
@router.post("/search", response_model=list[AssistantResponse])
async def search_assistants(body: AssistantSearchRequest | None = None) -> list[AssistantResponse]:
"""Search assistants.
Returns all registered assistants (lead_agent + custom agents from config).
"""
assistants = _list_assistants()
if body and body.graph_id:
assistants = [a for a in assistants if a.graph_id == body.graph_id]
if body and body.name:
assistants = [a for a in assistants if body.name.lower() in a.name.lower()]
offset = body.offset if body else 0
limit = body.limit if body else 10
return assistants[offset : offset + limit]
@router.get("/{assistant_id}", response_model=AssistantResponse)
async def get_assistant_compat(assistant_id: str) -> AssistantResponse:
"""Get an assistant by ID."""
for a in _list_assistants():
if a.assistant_id == assistant_id:
return a
raise HTTPException(status_code=404, detail=f"Assistant {assistant_id} not found")
@router.get("/{assistant_id}/graph")
async def get_assistant_graph(assistant_id: str) -> dict:
"""Get the graph structure for an assistant.
Returns a minimal graph description. Full graph introspection is
not supported in the Gateway this stub satisfies SDK validation.
"""
found = any(a.assistant_id == assistant_id for a in _list_assistants())
if not found:
raise HTTPException(status_code=404, detail=f"Assistant {assistant_id} not found")
return {
"graph_id": "lead_agent",
"nodes": [],
"edges": [],
}
@router.get("/{assistant_id}/schemas")
async def get_assistant_schemas(assistant_id: str) -> dict:
"""Get JSON schemas for an assistant's input/output/state.
Returns empty schemas full introspection not supported in Gateway.
"""
found = any(a.assistant_id == assistant_id for a in _list_assistants())
if not found:
raise HTTPException(status_code=404, detail=f"Assistant {assistant_id} not found")
return {
"graph_id": "lead_agent",
"input_schema": {},
"output_schema": {},
"state_schema": {},
"config_schema": {},
}
+493
View File
@@ -0,0 +1,493 @@
"""Authentication endpoints."""
import logging
import os
import time
from ipaddress import ip_address, ip_network
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
from fastapi.security import OAuth2PasswordRequestForm
from pydantic import BaseModel, EmailStr, Field, field_validator
from app.gateway.auth import (
UserResponse,
create_access_token,
)
from app.gateway.auth.config import get_auth_config
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
from app.gateway.csrf_middleware import is_secure_request
from app.gateway.deps import get_current_user_from_request, get_local_provider
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1/auth", tags=["auth"])
# ── Request/Response Models ──────────────────────────────────────────────
class LoginResponse(BaseModel):
"""Response model for login — token only lives in HttpOnly cookie."""
expires_in: int # seconds
needs_setup: bool = False
# Top common-password blocklist. Drawn from the public SecLists "10k worst
# passwords" set, lowercased + length>=8 only (shorter ones already fail
# the min_length check). Kept tight on purpose: this is the **lower bound**
# defense, not a full HIBP / passlib check, and runs in-process per request.
_COMMON_PASSWORDS: frozenset[str] = frozenset(
{
"password",
"password1",
"password12",
"password123",
"password1234",
"12345678",
"123456789",
"1234567890",
"qwerty12",
"qwertyui",
"qwerty123",
"abc12345",
"abcd1234",
"iloveyou",
"letmein1",
"welcome1",
"welcome123",
"admin123",
"administrator",
"passw0rd",
"p@ssw0rd",
"monkey12",
"trustno1",
"sunshine",
"princess",
"football",
"baseball",
"superman",
"batman123",
"starwars",
"dragon123",
"master123",
"shadow12",
"michael1",
"jennifer",
"computer",
}
)
def _password_is_common(password: str) -> bool:
"""Case-insensitive blocklist check.
Lowercases the input so trivial mutations like ``Password`` /
``PASSWORD`` are also rejected. Does not normalize digit substitutions
(``p@ssw0rd`` is included as a literal entry instead) keeping the
rule cheap and predictable.
"""
return password.lower() in _COMMON_PASSWORDS
def _validate_strong_password(value: str) -> str:
"""Pydantic field-validator body shared by Register + ChangePassword.
Constraint = function, not type-level mixin. The two request models
have no "is-a" relationship; they only share the password-strength
rule. Lifting it into a free function lets each model bind it via
``@field_validator(field_name)`` without inheritance gymnastics.
"""
if _password_is_common(value):
raise ValueError("Password is too common; choose a stronger password.")
return value
class RegisterRequest(BaseModel):
"""Request model for user registration."""
email: EmailStr
password: str = Field(..., min_length=8)
_strong_password = field_validator("password")(classmethod(lambda cls, v: _validate_strong_password(v)))
class ChangePasswordRequest(BaseModel):
"""Request model for password change (also handles setup flow)."""
current_password: str
new_password: str = Field(..., min_length=8)
new_email: EmailStr | None = None
_strong_password = field_validator("new_password")(classmethod(lambda cls, v: _validate_strong_password(v)))
class MessageResponse(BaseModel):
"""Generic message response."""
message: str
# ── Helpers ───────────────────────────────────────────────────────────────
def _set_session_cookie(response: Response, token: str, request: Request) -> None:
"""Set the access_token HttpOnly cookie on the response."""
config = get_auth_config()
is_https = is_secure_request(request)
response.set_cookie(
key="access_token",
value=token,
httponly=True,
secure=is_https,
samesite="lax",
max_age=config.token_expiry_days * 24 * 3600 if is_https else None,
)
# ── Rate Limiting ────────────────────────────────────────────────────────
# In-process dict — not shared across workers.
#
# **Limitation**: with multi-worker deployments (e.g., gunicorn -w N), each
# worker maintains its own lockout table, so an attacker effectively gets
# N × _MAX_LOGIN_ATTEMPTS guesses before being locked out everywhere. For
# production multi-worker setups, replace this with a shared store (Redis,
# database-backed counter) to enforce a true per-IP limit.
_MAX_LOGIN_ATTEMPTS = 5
_LOCKOUT_SECONDS = 300 # 5 minutes
# ip → (fail_count, lock_until_timestamp)
_login_attempts: dict[str, tuple[int, float]] = {}
def _trusted_proxies() -> list:
"""Parse ``AUTH_TRUSTED_PROXIES`` env var into a list of ip_network objects.
Comma-separated CIDR or single-IP entries. Empty / unset = no proxy is
trusted (direct mode). Invalid entries are skipped with a logger warning.
Read live so env-var overrides take effect immediately and tests can
``monkeypatch.setenv`` without poking a module-level cache.
"""
raw = os.getenv("AUTH_TRUSTED_PROXIES", "").strip()
if not raw:
return []
nets = []
for entry in raw.split(","):
entry = entry.strip()
if not entry:
continue
try:
nets.append(ip_network(entry, strict=False))
except ValueError:
logger.warning("AUTH_TRUSTED_PROXIES: ignoring invalid entry %r", entry)
return nets
def _get_client_ip(request: Request) -> str:
"""Extract the real client IP for rate limiting.
Trust model:
- The TCP peer (``request.client.host``) is always the baseline. It is
whatever the kernel reports as the connecting socket unforgeable
by the client itself.
- ``X-Real-IP`` is **only** honored if the TCP peer is in the
``AUTH_TRUSTED_PROXIES`` allowlist (set via env var, comma-separated
CIDR or single IPs). When set, the gateway is assumed to be behind a
reverse proxy (nginx, Cloudflare, ALB, ) that overwrites
``X-Real-IP`` with the original client address.
- With no ``AUTH_TRUSTED_PROXIES`` set, ``X-Real-IP`` is silently
ignored closing the bypass where any client could rotate the
header to dodge per-IP rate limits in dev / direct-gateway mode.
``X-Forwarded-For`` is intentionally NOT used because it is naturally
client-controlled at the *first* hop and the trust chain is harder to
audit per-request.
"""
peer_host = request.client.host if request.client else None
trusted = _trusted_proxies()
if trusted and peer_host:
try:
peer_ip = ip_address(peer_host)
if any(peer_ip in net for net in trusted):
real_ip = request.headers.get("x-real-ip", "").strip()
if real_ip:
return real_ip
except ValueError:
# peer_host wasn't a parseable IP (e.g. "unknown") — fall through
pass
return peer_host or "unknown"
def _check_rate_limit(ip: str) -> None:
"""Raise 429 if the IP is currently locked out."""
record = _login_attempts.get(ip)
if record is None:
return
fail_count, lock_until = record
if fail_count >= _MAX_LOGIN_ATTEMPTS:
if time.time() < lock_until:
raise HTTPException(
status_code=429,
detail="Too many login attempts. Try again later.",
)
del _login_attempts[ip]
_MAX_TRACKED_IPS = 10000
def _record_login_failure(ip: str) -> None:
"""Record a failed login attempt for the given IP."""
# Evict expired lockouts when dict grows too large
if len(_login_attempts) >= _MAX_TRACKED_IPS:
now = time.time()
expired = [k for k, (c, t) in _login_attempts.items() if c >= _MAX_LOGIN_ATTEMPTS and now >= t]
for k in expired:
del _login_attempts[k]
# If still too large, evict cheapest-to-lose half: below-threshold
# IPs (lock_until=0.0) sort first, then earliest-expiring lockouts.
if len(_login_attempts) >= _MAX_TRACKED_IPS:
by_time = sorted(_login_attempts.items(), key=lambda kv: kv[1][1])
for k, _ in by_time[: len(by_time) // 2]:
del _login_attempts[k]
record = _login_attempts.get(ip)
if record is None:
_login_attempts[ip] = (1, 0.0)
else:
new_count = record[0] + 1
lock_until = time.time() + _LOCKOUT_SECONDS if new_count >= _MAX_LOGIN_ATTEMPTS else 0.0
_login_attempts[ip] = (new_count, lock_until)
def _record_login_success(ip: str) -> None:
"""Clear failure counter for the given IP on successful login."""
_login_attempts.pop(ip, None)
# ── Endpoints ─────────────────────────────────────────────────────────────
@router.post("/login/local", response_model=LoginResponse)
async def login_local(
request: Request,
response: Response,
form_data: OAuth2PasswordRequestForm = Depends(),
):
"""Local email/password login."""
client_ip = _get_client_ip(request)
_check_rate_limit(client_ip)
user = await get_local_provider().authenticate({"email": form_data.username, "password": form_data.password})
if user is None:
_record_login_failure(client_ip)
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="Incorrect email or password").model_dump(),
)
_record_login_success(client_ip)
token = create_access_token(str(user.id), token_version=user.token_version)
_set_session_cookie(response, token, request)
return LoginResponse(
expires_in=get_auth_config().token_expiry_days * 24 * 3600,
needs_setup=user.needs_setup,
)
@router.post("/register", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
async def register(request: Request, response: Response, body: RegisterRequest):
"""Register a new user account (always 'user' role).
Admin is auto-created on first boot. This endpoint creates regular users.
Auto-login by setting the session cookie.
"""
try:
user = await get_local_provider().create_user(email=body.email, password=body.password, system_role="user")
except ValueError:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=AuthErrorResponse(code=AuthErrorCode.EMAIL_ALREADY_EXISTS, message="Email already registered").model_dump(),
)
token = create_access_token(str(user.id), token_version=user.token_version)
_set_session_cookie(response, token, request)
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role)
@router.post("/logout", response_model=MessageResponse)
async def logout(request: Request, response: Response):
"""Logout current user by clearing the cookie."""
response.delete_cookie(key="access_token", secure=is_secure_request(request), samesite="lax")
return MessageResponse(message="Successfully logged out")
@router.post("/change-password", response_model=MessageResponse)
async def change_password(request: Request, response: Response, body: ChangePasswordRequest):
"""Change password for the currently authenticated user.
Also handles the first-boot setup flow:
- If new_email is provided, updates email (checks uniqueness)
- If user.needs_setup is True and new_email is given, clears needs_setup
- Always increments token_version to invalidate old sessions
- Re-issues session cookie with new token_version
"""
from app.gateway.auth.password import hash_password_async, verify_password_async
user = await get_current_user_from_request(request)
if user.password_hash is None:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="OAuth users cannot change password").model_dump())
if not await verify_password_async(body.current_password, user.password_hash):
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="Current password is incorrect").model_dump())
provider = get_local_provider()
# Update email if provided
if body.new_email is not None:
existing = await provider.get_user_by_email(body.new_email)
if existing and str(existing.id) != str(user.id):
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=AuthErrorResponse(code=AuthErrorCode.EMAIL_ALREADY_EXISTS, message="Email already in use").model_dump())
user.email = body.new_email
# Update password + bump version
user.password_hash = await hash_password_async(body.new_password)
user.token_version += 1
# Clear setup flag if this is the setup flow
if user.needs_setup and body.new_email is not None:
user.needs_setup = False
await provider.update_user(user)
# Re-issue cookie with new token_version
token = create_access_token(str(user.id), token_version=user.token_version)
_set_session_cookie(response, token, request)
return MessageResponse(message="Password changed successfully")
@router.get("/me", response_model=UserResponse)
async def get_me(request: Request):
"""Get current authenticated user info."""
user = await get_current_user_from_request(request)
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role, needs_setup=user.needs_setup)
_SETUP_STATUS_COOLDOWN: dict[str, float] = {}
_SETUP_STATUS_COOLDOWN_SECONDS = 60
_MAX_TRACKED_SETUP_STATUS_IPS = 10000
@router.get("/setup-status")
async def setup_status(request: Request):
"""Check if an admin account exists. Returns needs_setup=True when no admin exists."""
client_ip = _get_client_ip(request)
now = time.time()
last_check = _SETUP_STATUS_COOLDOWN.get(client_ip, 0)
elapsed = now - last_check
if elapsed < _SETUP_STATUS_COOLDOWN_SECONDS:
retry_after = max(1, int(_SETUP_STATUS_COOLDOWN_SECONDS - elapsed))
raise HTTPException(
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
detail="Setup status check is rate limited",
headers={"Retry-After": str(retry_after)},
)
# Evict stale entries when dict grows too large to bound memory usage.
if len(_SETUP_STATUS_COOLDOWN) >= _MAX_TRACKED_SETUP_STATUS_IPS:
cutoff = now - _SETUP_STATUS_COOLDOWN_SECONDS
stale = [k for k, t in _SETUP_STATUS_COOLDOWN.items() if t < cutoff]
for k in stale:
del _SETUP_STATUS_COOLDOWN[k]
# If still too large after evicting expired entries, remove oldest half.
if len(_SETUP_STATUS_COOLDOWN) >= _MAX_TRACKED_SETUP_STATUS_IPS:
by_time = sorted(_SETUP_STATUS_COOLDOWN.items(), key=lambda kv: kv[1])
for k, _ in by_time[: len(by_time) // 2]:
del _SETUP_STATUS_COOLDOWN[k]
_SETUP_STATUS_COOLDOWN[client_ip] = now
admin_count = await get_local_provider().count_admin_users()
return {"needs_setup": admin_count == 0}
class InitializeAdminRequest(BaseModel):
"""Request model for first-boot admin account creation."""
email: EmailStr
password: str = Field(..., min_length=8)
_strong_password = field_validator("password")(classmethod(lambda cls, v: _validate_strong_password(v)))
@router.post("/initialize", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
async def initialize_admin(request: Request, response: Response, body: InitializeAdminRequest):
"""Create the first admin account on initial system setup.
Only callable when no admin exists. Returns 409 Conflict if an admin
already exists.
On success, the admin account is created with ``needs_setup=False`` and
the session cookie is set.
"""
admin_count = await get_local_provider().count_admin_users()
if admin_count > 0:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail=AuthErrorResponse(code=AuthErrorCode.SYSTEM_ALREADY_INITIALIZED, message="System already initialized").model_dump(),
)
try:
user = await get_local_provider().create_user(email=body.email, password=body.password, system_role="admin", needs_setup=False)
except ValueError:
# DB unique-constraint race: another concurrent request beat us.
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail=AuthErrorResponse(code=AuthErrorCode.SYSTEM_ALREADY_INITIALIZED, message="System already initialized").model_dump(),
)
token = create_access_token(str(user.id), token_version=user.token_version)
_set_session_cookie(response, token, request)
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role)
# ── OAuth Endpoints (Future/Placeholder) ─────────────────────────────────
@router.get("/oauth/{provider}")
async def oauth_login(provider: str):
"""Initiate OAuth login flow.
Redirects to the OAuth provider's authorization URL.
Currently a placeholder - requires OAuth provider implementation.
"""
if provider not in ["github", "google"]:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Unsupported OAuth provider: {provider}",
)
raise HTTPException(
status_code=status.HTTP_501_NOT_IMPLEMENTED,
detail="OAuth login not yet implemented",
)
@router.get("/callback/{provider}")
async def oauth_callback(provider: str, code: str, state: str):
"""OAuth callback endpoint.
Handles the OAuth provider's callback after user authorization.
Currently a placeholder.
"""
raise HTTPException(
status_code=status.HTTP_501_NOT_IMPLEMENTED,
detail="OAuth callback not yet implemented",
)
+52
View File
@@ -0,0 +1,52 @@
"""Gateway router for IM channel management."""
from __future__ import annotations
import logging
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/channels", tags=["channels"])
class ChannelStatusResponse(BaseModel):
service_running: bool
channels: dict[str, dict]
class ChannelRestartResponse(BaseModel):
success: bool
message: str
@router.get("/", response_model=ChannelStatusResponse)
async def get_channels_status() -> ChannelStatusResponse:
"""Get the status of all IM channels."""
from app.channels.service import get_channel_service
service = get_channel_service()
if service is None:
return ChannelStatusResponse(service_running=False, channels={})
status = service.get_status()
return ChannelStatusResponse(**status)
@router.post("/{name}/restart", response_model=ChannelRestartResponse)
async def restart_channel(name: str) -> ChannelRestartResponse:
"""Restart a specific IM channel."""
from app.channels.service import get_channel_service
service = get_channel_service()
if service is None:
raise HTTPException(status_code=503, detail="Channel service is not running")
success = await service.restart_channel(name)
if success:
logger.info("Channel %s restarted successfully", name)
return ChannelRestartResponse(success=True, message=f"Channel {name} restarted successfully")
else:
logger.warning("Failed to restart channel %s", name)
return ChannelRestartResponse(success=False, message=f"Failed to restart channel {name}")
+188
View File
@@ -0,0 +1,188 @@
"""Feedback endpoints — create, list, stats, delete.
Allows users to submit thumbs-up/down feedback on runs,
optionally scoped to a specific message.
"""
from __future__ import annotations
import logging
from typing import Any
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field
from app.gateway.authz import require_permission
from app.gateway.deps import get_current_user, get_feedback_repo, get_run_store
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/threads", tags=["feedback"])
# ---------------------------------------------------------------------------
# Request / response models
# ---------------------------------------------------------------------------
class FeedbackCreateRequest(BaseModel):
rating: int = Field(..., description="Feedback rating: +1 (positive) or -1 (negative)")
comment: str | None = Field(default=None, description="Optional text feedback")
message_id: str | None = Field(default=None, description="Optional: scope feedback to a specific message")
class FeedbackUpsertRequest(BaseModel):
rating: int = Field(..., description="Feedback rating: +1 (positive) or -1 (negative)")
comment: str | None = Field(default=None, description="Optional text feedback")
class FeedbackResponse(BaseModel):
feedback_id: str
run_id: str
thread_id: str
user_id: str | None = None
message_id: str | None = None
rating: int
comment: str | None = None
created_at: str = ""
class FeedbackStatsResponse(BaseModel):
run_id: str
total: int = 0
positive: int = 0
negative: int = 0
# ---------------------------------------------------------------------------
# Endpoints
# ---------------------------------------------------------------------------
@router.put("/{thread_id}/runs/{run_id}/feedback", response_model=FeedbackResponse)
@require_permission("threads", "write", owner_check=True, require_existing=True)
async def upsert_feedback(
thread_id: str,
run_id: str,
body: FeedbackUpsertRequest,
request: Request,
) -> dict[str, Any]:
"""Create or update feedback for a run (idempotent)."""
if body.rating not in (1, -1):
raise HTTPException(status_code=400, detail="rating must be +1 or -1")
user_id = await get_current_user(request)
run_store = get_run_store(request)
run = await run_store.get(run_id)
if run is None:
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
if run.get("thread_id") != thread_id:
raise HTTPException(status_code=404, detail=f"Run {run_id} not found in thread {thread_id}")
feedback_repo = get_feedback_repo(request)
return await feedback_repo.upsert(
run_id=run_id,
thread_id=thread_id,
rating=body.rating,
user_id=user_id,
comment=body.comment,
)
@router.delete("/{thread_id}/runs/{run_id}/feedback")
@require_permission("threads", "delete", owner_check=True, require_existing=True)
async def delete_run_feedback(
thread_id: str,
run_id: str,
request: Request,
) -> dict[str, bool]:
"""Delete the current user's feedback for a run."""
user_id = await get_current_user(request)
feedback_repo = get_feedback_repo(request)
deleted = await feedback_repo.delete_by_run(
thread_id=thread_id,
run_id=run_id,
user_id=user_id,
)
if not deleted:
raise HTTPException(status_code=404, detail="No feedback found for this run")
return {"success": True}
@router.post("/{thread_id}/runs/{run_id}/feedback", response_model=FeedbackResponse)
@require_permission("threads", "write", owner_check=True, require_existing=True)
async def create_feedback(
thread_id: str,
run_id: str,
body: FeedbackCreateRequest,
request: Request,
) -> dict[str, Any]:
"""Submit feedback (thumbs-up/down) for a run."""
if body.rating not in (1, -1):
raise HTTPException(status_code=400, detail="rating must be +1 or -1")
user_id = await get_current_user(request)
# Validate run exists and belongs to thread
run_store = get_run_store(request)
run = await run_store.get(run_id)
if run is None:
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
if run.get("thread_id") != thread_id:
raise HTTPException(status_code=404, detail=f"Run {run_id} not found in thread {thread_id}")
feedback_repo = get_feedback_repo(request)
return await feedback_repo.create(
run_id=run_id,
thread_id=thread_id,
rating=body.rating,
user_id=user_id,
message_id=body.message_id,
comment=body.comment,
)
@router.get("/{thread_id}/runs/{run_id}/feedback", response_model=list[FeedbackResponse])
@require_permission("threads", "read", owner_check=True)
async def list_feedback(
thread_id: str,
run_id: str,
request: Request,
) -> list[dict[str, Any]]:
"""List all feedback for a run."""
feedback_repo = get_feedback_repo(request)
return await feedback_repo.list_by_run(thread_id, run_id)
@router.get("/{thread_id}/runs/{run_id}/feedback/stats", response_model=FeedbackStatsResponse)
@require_permission("threads", "read", owner_check=True)
async def feedback_stats(
thread_id: str,
run_id: str,
request: Request,
) -> dict[str, Any]:
"""Get aggregated feedback stats (positive/negative counts) for a run."""
feedback_repo = get_feedback_repo(request)
return await feedback_repo.aggregate_by_run(thread_id, run_id)
@router.delete("/{thread_id}/runs/{run_id}/feedback/{feedback_id}")
@require_permission("threads", "delete", owner_check=True, require_existing=True)
async def delete_feedback(
thread_id: str,
run_id: str,
feedback_id: str,
request: Request,
) -> dict[str, bool]:
"""Delete a feedback record."""
feedback_repo = get_feedback_repo(request)
# Verify feedback belongs to the specified thread/run before deleting
existing = await feedback_repo.get(feedback_id)
if existing is None:
raise HTTPException(status_code=404, detail=f"Feedback {feedback_id} not found")
if existing.get("thread_id") != thread_id or existing.get("run_id") != run_id:
raise HTTPException(status_code=404, detail=f"Feedback {feedback_id} not found in run {run_id}")
deleted = await feedback_repo.delete(feedback_id)
if not deleted:
raise HTTPException(status_code=404, detail=f"Feedback {feedback_id} not found")
return {"success": True}
+169
View File
@@ -0,0 +1,169 @@
import json
import logging
from pathlib import Path
from typing import Literal
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel, Field
from deerflow.config.extensions_config import ExtensionsConfig, get_extensions_config, reload_extensions_config
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api", tags=["mcp"])
class McpOAuthConfigResponse(BaseModel):
"""OAuth configuration for an MCP server."""
enabled: bool = Field(default=True, description="Whether OAuth token injection is enabled")
token_url: str = Field(default="", description="OAuth token endpoint URL")
grant_type: Literal["client_credentials", "refresh_token"] = Field(default="client_credentials", description="OAuth grant type")
client_id: str | None = Field(default=None, description="OAuth client ID")
client_secret: str | None = Field(default=None, description="OAuth client secret")
refresh_token: str | None = Field(default=None, description="OAuth refresh token")
scope: str | None = Field(default=None, description="OAuth scope")
audience: str | None = Field(default=None, description="OAuth audience")
token_field: str = Field(default="access_token", description="Token response field containing access token")
token_type_field: str = Field(default="token_type", description="Token response field containing token type")
expires_in_field: str = Field(default="expires_in", description="Token response field containing expires-in seconds")
default_token_type: str = Field(default="Bearer", description="Default token type when response omits token_type")
refresh_skew_seconds: int = Field(default=60, description="Refresh this many seconds before expiry")
extra_token_params: dict[str, str] = Field(default_factory=dict, description="Additional form params sent to token endpoint")
class McpServerConfigResponse(BaseModel):
"""Response model for MCP server configuration."""
enabled: bool = Field(default=True, description="Whether this MCP server is enabled")
type: str = Field(default="stdio", description="Transport type: 'stdio', 'sse', or 'http'")
command: str | None = Field(default=None, description="Command to execute to start the MCP server (for stdio type)")
args: list[str] = Field(default_factory=list, description="Arguments to pass to the command (for stdio type)")
env: dict[str, str] = Field(default_factory=dict, description="Environment variables for the MCP server")
url: str | None = Field(default=None, description="URL of the MCP server (for sse or http type)")
headers: dict[str, str] = Field(default_factory=dict, description="HTTP headers to send (for sse or http type)")
oauth: McpOAuthConfigResponse | None = Field(default=None, description="OAuth configuration for MCP HTTP/SSE servers")
description: str = Field(default="", description="Human-readable description of what this MCP server provides")
class McpConfigResponse(BaseModel):
"""Response model for MCP configuration."""
mcp_servers: dict[str, McpServerConfigResponse] = Field(
default_factory=dict,
description="Map of MCP server name to configuration",
)
class McpConfigUpdateRequest(BaseModel):
"""Request model for updating MCP configuration."""
mcp_servers: dict[str, McpServerConfigResponse] = Field(
...,
description="Map of MCP server name to configuration",
)
@router.get(
"/mcp/config",
response_model=McpConfigResponse,
summary="Get MCP Configuration",
description="Retrieve the current Model Context Protocol (MCP) server configurations.",
)
async def get_mcp_configuration() -> McpConfigResponse:
"""Get the current MCP configuration.
Returns:
The current MCP configuration with all servers.
Example:
```json
{
"mcp_servers": {
"github": {
"enabled": true,
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-github"],
"env": {"GITHUB_TOKEN": "ghp_xxx"},
"description": "GitHub MCP server for repository operations"
}
}
}
```
"""
config = get_extensions_config()
return McpConfigResponse(mcp_servers={name: McpServerConfigResponse(**server.model_dump()) for name, server in config.mcp_servers.items()})
@router.put(
"/mcp/config",
response_model=McpConfigResponse,
summary="Update MCP Configuration",
description="Update Model Context Protocol (MCP) server configurations and save to file.",
)
async def update_mcp_configuration(request: McpConfigUpdateRequest) -> McpConfigResponse:
"""Update the MCP configuration.
This will:
1. Save the new configuration to the mcp_config.json file
2. Reload the configuration cache
3. Reset MCP tools cache to trigger reinitialization
Args:
request: The new MCP configuration to save.
Returns:
The updated MCP configuration.
Raises:
HTTPException: 500 if the configuration file cannot be written.
Example Request:
```json
{
"mcp_servers": {
"github": {
"enabled": true,
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-github"],
"env": {"GITHUB_TOKEN": "$GITHUB_TOKEN"},
"description": "GitHub MCP server for repository operations"
}
}
}
```
"""
try:
# Get the current config path (or determine where to save it)
config_path = ExtensionsConfig.resolve_config_path()
# If no config file exists, create one in the parent directory (project root)
if config_path is None:
config_path = Path.cwd().parent / "extensions_config.json"
logger.info(f"No existing extensions config found. Creating new config at: {config_path}")
# Load current config to preserve skills configuration
current_config = get_extensions_config()
# Convert request to dict format for JSON serialization
config_data = {
"mcpServers": {name: server.model_dump() for name, server in request.mcp_servers.items()},
"skills": {name: {"enabled": skill.enabled} for name, skill in current_config.skills.items()},
}
# Write the configuration to file
with open(config_path, "w", encoding="utf-8") as f:
json.dump(config_data, f, indent=2)
logger.info(f"MCP configuration updated and saved to: {config_path}")
# NOTE: No need to reload/reset cache here - LangGraph Server (separate process)
# will detect config file changes via mtime and reinitialize MCP tools automatically
# Reload the configuration and update the global cache
reloaded_config = reload_extensions_config()
return McpConfigResponse(mcp_servers={name: McpServerConfigResponse(**server.model_dump()) for name, server in reloaded_config.mcp_servers.items()})
except Exception as e:
logger.error(f"Failed to update MCP configuration: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to update MCP configuration: {str(e)}")
+356
View File
@@ -0,0 +1,356 @@
"""Memory API router for retrieving and managing global memory data."""
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel, Field
from deerflow.agents.memory.updater import (
clear_memory_data,
create_memory_fact,
delete_memory_fact,
get_memory_data,
import_memory_data,
reload_memory_data,
update_memory_fact,
)
from deerflow.config.memory_config import get_memory_config
from deerflow.runtime.user_context import get_effective_user_id
router = APIRouter(prefix="/api", tags=["memory"])
class ContextSection(BaseModel):
"""Model for context sections (user and history)."""
summary: str = Field(default="", description="Summary content")
updatedAt: str = Field(default="", description="Last update timestamp")
class UserContext(BaseModel):
"""Model for user context."""
workContext: ContextSection = Field(default_factory=ContextSection)
personalContext: ContextSection = Field(default_factory=ContextSection)
topOfMind: ContextSection = Field(default_factory=ContextSection)
class HistoryContext(BaseModel):
"""Model for history context."""
recentMonths: ContextSection = Field(default_factory=ContextSection)
earlierContext: ContextSection = Field(default_factory=ContextSection)
longTermBackground: ContextSection = Field(default_factory=ContextSection)
class Fact(BaseModel):
"""Model for a memory fact."""
id: str = Field(..., description="Unique identifier for the fact")
content: str = Field(..., description="Fact content")
category: str = Field(default="context", description="Fact category")
confidence: float = Field(default=0.5, description="Confidence score (0-1)")
createdAt: str = Field(default="", description="Creation timestamp")
source: str = Field(default="unknown", description="Source thread ID")
sourceError: str | None = Field(default=None, description="Optional description of the prior mistake or wrong approach")
class MemoryResponse(BaseModel):
"""Response model for memory data."""
version: str = Field(default="1.0", description="Memory schema version")
lastUpdated: str = Field(default="", description="Last update timestamp")
user: UserContext = Field(default_factory=UserContext)
history: HistoryContext = Field(default_factory=HistoryContext)
facts: list[Fact] = Field(default_factory=list)
def _map_memory_fact_value_error(exc: ValueError) -> HTTPException:
"""Convert updater validation errors into stable API responses."""
if exc.args and exc.args[0] == "confidence":
detail = "Invalid confidence value; must be between 0 and 1."
else:
detail = "Memory fact content cannot be empty."
return HTTPException(status_code=400, detail=detail)
class FactCreateRequest(BaseModel):
"""Request model for creating a memory fact."""
content: str = Field(..., min_length=1, description="Fact content")
category: str = Field(default="context", description="Fact category")
confidence: float = Field(default=0.5, ge=0.0, le=1.0, description="Confidence score (0-1)")
class FactPatchRequest(BaseModel):
"""PATCH request model that preserves existing values for omitted fields."""
content: str | None = Field(default=None, min_length=1, description="Fact content")
category: str | None = Field(default=None, description="Fact category")
confidence: float | None = Field(default=None, ge=0.0, le=1.0, description="Confidence score (0-1)")
class MemoryConfigResponse(BaseModel):
"""Response model for memory configuration."""
enabled: bool = Field(..., description="Whether memory is enabled")
storage_path: str = Field(..., description="Path to memory storage file")
debounce_seconds: int = Field(..., description="Debounce time for memory updates")
max_facts: int = Field(..., description="Maximum number of facts to store")
fact_confidence_threshold: float = Field(..., description="Minimum confidence threshold for facts")
injection_enabled: bool = Field(..., description="Whether memory injection is enabled")
max_injection_tokens: int = Field(..., description="Maximum tokens for memory injection")
class MemoryStatusResponse(BaseModel):
"""Response model for memory status."""
config: MemoryConfigResponse
data: MemoryResponse
@router.get(
"/memory",
response_model=MemoryResponse,
response_model_exclude_none=True,
summary="Get Memory Data",
description="Retrieve the current global memory data including user context, history, and facts.",
)
async def get_memory() -> MemoryResponse:
"""Get the current global memory data.
Returns:
The current memory data with user context, history, and facts.
Example Response:
```json
{
"version": "1.0",
"lastUpdated": "2024-01-15T10:30:00Z",
"user": {
"workContext": {"summary": "Working on DeerFlow project", "updatedAt": "..."},
"personalContext": {"summary": "Prefers concise responses", "updatedAt": "..."},
"topOfMind": {"summary": "Building memory API", "updatedAt": "..."}
},
"history": {
"recentMonths": {"summary": "Recent development activities", "updatedAt": "..."},
"earlierContext": {"summary": "", "updatedAt": ""},
"longTermBackground": {"summary": "", "updatedAt": ""}
},
"facts": [
{
"id": "fact_abc123",
"content": "User prefers TypeScript over JavaScript",
"category": "preference",
"confidence": 0.9,
"createdAt": "2024-01-15T10:30:00Z",
"source": "thread_xyz"
}
]
}
```
"""
memory_data = get_memory_data(user_id=get_effective_user_id())
return MemoryResponse(**memory_data)
@router.post(
"/memory/reload",
response_model=MemoryResponse,
response_model_exclude_none=True,
summary="Reload Memory Data",
description="Reload memory data from the storage file, refreshing the in-memory cache.",
)
async def reload_memory() -> MemoryResponse:
"""Reload memory data from file.
This forces a reload of the memory data from the storage file,
useful when the file has been modified externally.
Returns:
The reloaded memory data.
"""
memory_data = reload_memory_data(user_id=get_effective_user_id())
return MemoryResponse(**memory_data)
@router.delete(
"/memory",
response_model=MemoryResponse,
response_model_exclude_none=True,
summary="Clear All Memory Data",
description="Delete all saved memory data and reset the memory structure to an empty state.",
)
async def clear_memory() -> MemoryResponse:
"""Clear all persisted memory data."""
try:
memory_data = clear_memory_data(user_id=get_effective_user_id())
except OSError as exc:
raise HTTPException(status_code=500, detail="Failed to clear memory data.") from exc
return MemoryResponse(**memory_data)
@router.post(
"/memory/facts",
response_model=MemoryResponse,
response_model_exclude_none=True,
summary="Create Memory Fact",
description="Create a single saved memory fact manually.",
)
async def create_memory_fact_endpoint(request: FactCreateRequest) -> MemoryResponse:
"""Create a single fact manually."""
try:
memory_data = create_memory_fact(
content=request.content,
category=request.category,
confidence=request.confidence,
user_id=get_effective_user_id(),
)
except ValueError as exc:
raise _map_memory_fact_value_error(exc) from exc
except OSError as exc:
raise HTTPException(status_code=500, detail="Failed to create memory fact.") from exc
return MemoryResponse(**memory_data)
@router.delete(
"/memory/facts/{fact_id}",
response_model=MemoryResponse,
response_model_exclude_none=True,
summary="Delete Memory Fact",
description="Delete a single saved memory fact by its fact id.",
)
async def delete_memory_fact_endpoint(fact_id: str) -> MemoryResponse:
"""Delete a single fact from memory by fact id."""
try:
memory_data = delete_memory_fact(fact_id, user_id=get_effective_user_id())
except KeyError as exc:
raise HTTPException(status_code=404, detail=f"Memory fact '{fact_id}' not found.") from exc
except OSError as exc:
raise HTTPException(status_code=500, detail="Failed to delete memory fact.") from exc
return MemoryResponse(**memory_data)
@router.patch(
"/memory/facts/{fact_id}",
response_model=MemoryResponse,
response_model_exclude_none=True,
summary="Patch Memory Fact",
description="Partially update a single saved memory fact by its fact id while preserving omitted fields.",
)
async def update_memory_fact_endpoint(fact_id: str, request: FactPatchRequest) -> MemoryResponse:
"""Partially update a single fact manually."""
try:
memory_data = update_memory_fact(
fact_id=fact_id,
content=request.content,
category=request.category,
confidence=request.confidence,
user_id=get_effective_user_id(),
)
except ValueError as exc:
raise _map_memory_fact_value_error(exc) from exc
except KeyError as exc:
raise HTTPException(status_code=404, detail=f"Memory fact '{fact_id}' not found.") from exc
except OSError as exc:
raise HTTPException(status_code=500, detail="Failed to update memory fact.") from exc
return MemoryResponse(**memory_data)
@router.get(
"/memory/export",
response_model=MemoryResponse,
response_model_exclude_none=True,
summary="Export Memory Data",
description="Export the current global memory data as JSON for backup or transfer.",
)
async def export_memory() -> MemoryResponse:
"""Export the current memory data."""
memory_data = get_memory_data(user_id=get_effective_user_id())
return MemoryResponse(**memory_data)
@router.post(
"/memory/import",
response_model=MemoryResponse,
response_model_exclude_none=True,
summary="Import Memory Data",
description="Import and overwrite the current global memory data from a JSON payload.",
)
async def import_memory(request: MemoryResponse) -> MemoryResponse:
"""Import and persist memory data."""
try:
memory_data = import_memory_data(request.model_dump(), user_id=get_effective_user_id())
except OSError as exc:
raise HTTPException(status_code=500, detail="Failed to import memory data.") from exc
return MemoryResponse(**memory_data)
@router.get(
"/memory/config",
response_model=MemoryConfigResponse,
summary="Get Memory Configuration",
description="Retrieve the current memory system configuration.",
)
async def get_memory_config_endpoint() -> MemoryConfigResponse:
"""Get the memory system configuration.
Returns:
The current memory configuration settings.
Example Response:
```json
{
"enabled": true,
"storage_path": ".deer-flow/memory.json",
"debounce_seconds": 30,
"max_facts": 100,
"fact_confidence_threshold": 0.7,
"injection_enabled": true,
"max_injection_tokens": 2000
}
```
"""
config = get_memory_config()
return MemoryConfigResponse(
enabled=config.enabled,
storage_path=config.storage_path,
debounce_seconds=config.debounce_seconds,
max_facts=config.max_facts,
fact_confidence_threshold=config.fact_confidence_threshold,
injection_enabled=config.injection_enabled,
max_injection_tokens=config.max_injection_tokens,
)
@router.get(
"/memory/status",
response_model=MemoryStatusResponse,
response_model_exclude_none=True,
summary="Get Memory Status",
description="Retrieve both memory configuration and current data in a single request.",
)
async def get_memory_status() -> MemoryStatusResponse:
"""Get the memory system status including configuration and data.
Returns:
Combined memory configuration and current data.
"""
config = get_memory_config()
memory_data = get_memory_data(user_id=get_effective_user_id())
return MemoryStatusResponse(
config=MemoryConfigResponse(
enabled=config.enabled,
storage_path=config.storage_path,
debounce_seconds=config.debounce_seconds,
max_facts=config.max_facts,
fact_confidence_threshold=config.fact_confidence_threshold,
injection_enabled=config.injection_enabled,
max_injection_tokens=config.max_injection_tokens,
),
data=MemoryResponse(**memory_data),
)

Some files were not shown because too many files have changed in this diff Show More