# HYD Full public corpus Name: HYD Bio: Class of 2024, Data Science and Big Data Technology at Qingdao Institute of Technology. Rather than leaving what I learn in notes, I like pushing it to the point where it runs — from scripts and small tools to products people can actually use. Most days I move between code, papers, and small tools, with an eye on AI agents and AI-native products. Open source, music, and anime fill the rest; anything worth keeping goes into the notes. --- # How can AI agents read this site? Author: HYD Source: https://hydblog.xyz/en/answers/agent-ready/ Summary: The build generates llms.txt and llms-full.txt indexes, a Markdown mirror per page, JSON endpoints for articles, profile, and search, plus an OpenAPI description — so agents can cite material without parsing page layouts. **One corpus, two read paths: pages for people and interfaces for machines.** The index is [llms.txt](/en/llms.txt), and the full corpus sits at [llms-full.txt](/en/llms-full.txt). Articles, notes, and answers each have a Markdown mirror and a JSON endpoint — /api/articles.json, /api/profile.json, /api/search-index.json — with field descriptions in [openapi.json](/openapi.json) and a sitemap at /sitemap-index.xml. The Markdown mirror is the source text that was published, so citations do not drift from the rendered page. Asking on this site runs in three layers: curated answers, full-text retrieval, and live AI answers (see "How do search and asking work here?"). The live answers come from a same-origin worker that retrieves this site's public content before generating a source-cited, single-turn answer, without long-term memory. --- # How can I collaborate with HYD? Author: HYD Source: https://hydblog.xyz/en/answers/collaborate/ Summary: Email HYDhyd0505@gmail.com, or open an issue or a pull request as HYDtomako on GitHub — AI agent practice, open-source collaboration, and publishing setups are all welcome. **The fastest route is email: HYDhyd0505@gmail.com.** You can also open an issue or a pull request on GitHub under HYDtomako, or reach out on QQ at 2804956879. Worth talking about: AI agent practice — tool calling, long-term memory, multi-agent collaboration; splitting work and reviewing code on open-source projects; and personal sites and publishing. What exists today is listed on the [projects page](/en/projects/), so if something there looks worth doing together, just say so. Still a student, so replies may take a while, but every message gets read. --- # How do search and asking work here? Author: HYD Source: https://hydblog.xyz/en/answers/live-ask/ Summary: A question matches pre-generated curated answers first; then it searches the published articles, notes, and answers; anything still unmatched goes to a live service that answers from retrieved public content, single-turn and source-cited. **Asking works without a model.** The common questions have pre-generated curated answers that appear as soon as they match; anything else runs a full-text search across the published articles, notes, and answers. If nothing matches, the question goes to a live same-origin service: it retrieves this site's public content, hands the retrieved passages to a model for a source-grounded answer, and gates the browser entry point with a challenge and quotas. Answers stay single-turn and cite the pages they came from. Either way there is no long-term memory and no speaking on the author's behalf; when in doubt, trust the linked public page. --- # What AI practices does this site document? Author: HYD Source: https://hydblog.xyz/en/answers/ai-practice/ Summary: Three layers — study notes on transformers and deep learning, source-reading notes on agent mechanics (agent loop, tools, context compression, multi-agent collaboration), and small tools built from them, such as EnCoder and QQbot. **AI is the main line here, and the writing sits in three layers, from mechanism to something that runs.** The first layer is fundamentals: the [transformer notes](/en/2026/04/04/transformer-learning-notes/) walk through tokens, positional encoding, Q/K/V attention, multi-head attention, and layer norm in data-flow order, while Simple_CNN reproduces the path from convolution and pooling to a full training run. The second layer is agent engineering: reading three Claude Code implementations side by side ([the notebook](/en/2026/08/23/agent-engineering-from-claude-code/)) lines up the agent loop, tool definitions, context injection and compression, parallel tools and sub-agents, multi-agent collaboration, and security boundaries. Minimind-notes covers model architecture and training methods. The third layer is tooling that runs: EnCoder extends a minimal agent with cron scheduling, long-term memory, an agent team, and task management; QQbot uses pi as its brain and NapCatQQ as its connection, covering knowledge Q&A, timetable lookup, group management, long-term memory, and hot model switching. The project list is on the [projects page](/en/projects/). --- # What is this site? Author: HYD Source: https://hydblog.xyz/en/answers/what-is-this-site/ Summary: HYD's personal site, recording the path through AI engineering and LLMs — articles, notes, projects, and a curated FAQ, built with Astro, Starlight, and the Refined-X starter so one corpus serves readers, search engines, and AI agents. **This is HYD's personal site.** Most of it is a record of learning: notes on AI engineering, LLMs, and agents, alongside the projects that came out of them and shorter pieces written along the way. It runs on Astro and Starlight over the Refined-X publishing starter. Articles, notes, answers, and projects are written in Markdown and YAML, and each build produces both the pages people read and the same corpus as Markdown mirrors, llms.txt, and JSON endpoints, so search engines and AI agents can read it directly. For the person, see the [about page](/en/about/); for the work, the [projects page](/en/projects/); for direct answers, ask on this site or browse the [FAQ](/en/answers/). --- # Where were the earlier posts published? Author: HYD Source: https://hydblog.xyz/en/answers/where-articles-were/ Summary: On CSDN from the summer of 2025 (blog.csdn.net/2401_87876529) — Python and data analysis first, then transformers, LLMs, and agents. New writing moved here in September 2026; the old posts stay up but are no longer updated. **From the summer of 2025 until September 2026 the writing lived on CSDN — about twenty posts:** https://blog.csdn.net/2401_87876529 The first half was fundamentals: Python data structures, NumPy and pandas, then data analysis and visualization, with a few competitions and small projects in between. The second half turned to AI — from transformers through tool calling, the agent loop, multi-agent setups, and Claude Code, plus engineering questions like prompt-injection defences and agent checkpoints. New writing happens here now; the CSDN archive stays online but is no longer updated. Moving also meant re-ordering the old notes: [the transformer piece](/en/2026/04/04/transformer-learning-notes/) was rewritten into its current shape after the move, and the move itself is written up in the note [From CSDN to a site of my own](/en/notes/from-csdn/). --- # Which open source projects are here? Author: HYD Source: https://hydblog.xyz/en/answers/open-source-projects/ Summary: Six so far — EnCoder, QQbot, Minimind-notes, Simple_CNN, interview-skill, and homework-web_blog — mostly agent tooling and reproductions of deep-learning basics. **The public work comes down to two things: agent tooling, and reproductions of deep-learning basics.** Repositories are on GitHub: https://github.com/HYDtomako **EnCoder** extends CoreCoder — a minimal agent — with cron scheduling, long-term memory, an agent team, and task management. **QQbot** pairs pi as the brain with NapCatQQ as the connection, covering knowledge Q&A, timetable lookup, group management, scheduled tasks, long-term memory, and hot model switching. **Minimind-notes** works through model architecture and training methods, and **Simple_CNN** reproduces convolution and pooling through a full training run. The other two are practice: **interview-skill** pairs a job description with your resume for targeted interview prep, and **homework-web_blog** is the final project for a Python web course. Status, tags, and repository links for each are on the [projects page](/en/projects/). --- # Who is the author? Author: HYD Source: https://hydblog.xyz/en/answers/who-is-author/ Summary: HYD is a Data Science and Big Data Technology student at Qingdao Institute of Technology (class of 2024), focused on AI agents and AI-native products — he likes pushing what he learns into small tools that actually run, then open-sourcing them. **HYD** studies Data Science and Big Data Technology at Qingdao Institute of Technology (class of 2024) and works on AI agents and AI-native products, moving between code, papers, and small tools. Rather than leaving what he learns in notes, he pushes it to the point where it runs: read the source until it can be restated, turn the idea into a tool, then come back here to write it up. Public projects live on GitHub (https://github.com/HYDtomako), from agent tooling to reproductions of deep-learning basics. The rest is on the [about page](/en/about/): anime, music, and landscapes picked up on walks. --- # Three Claude Code implementations: an agent engineering notebook Author: HYD Source: https://hydblog.xyz/en/2026/08/23/agent-engineering-from-claude-code/ Published: 2026-08-23T13:00:00.000Z Updated: 2026-08-23T13:00:00.000Z Summary: Agent engineering notes drawn from three implementations. CoreCoder is a minimal implementation written by hand in Python, Claude Code is the production implementation, and learn-claude-code is the teaching implementation. Coverage spans the Agent Loop's transition condition and interrupt repair, tool definitions and file-editing strategy, context injection and three tiers of compression, the long-lived state of QueryEngine, parallel tools and sub-agents, the Hook and Task extension points, multi-agent collaboration, memory / MCP / background tasks / scheduled tasks, and the security defenses around paths and commands. Reading agent source code, you tend to get stuck in one of two places: either you are dragged along by the interaction details of the CLI layer, or you never move past the conclusion that "the LLM will call tools." Putting three implementations side by side is actually clearer — which parts are irreducible and which are just a matter of style falls out of the comparison. - **[CoreCoder](https://github.com/he-yufeng/CoreCoder)**: a minimal agent written by hand in Python. The loop, tool, and context layers are all written from scratch, so the skeleton is visible. - **[Claude Code](https://www.xuanyuancode.com/learn-claude-code)**: the production implementation, with a full serialized source walkthrough on xuanyuancode. What is worth reading is the startup assembly, the long-lived session object, context engineering, and how tools like BashTool get "governed." - **[learn-claude-code](https://github.com/shareAI-lab/learn-claude-code)**: the teaching implementation. hook, task, multi-agent, memory, and MCP are broken into small pieces and covered one at a time. What follows is organized by mechanism, not by project. ## Agent Loop: the heart of the whole flow CoreCoder holds up the entire flow with a single `for` loop, `max_rounds = 50`, and **exactly one transition condition: does this round still have a tool call**. Tool calls follow OpenAI's function calling paradigm: the assistant message carries a `tool_call`, each call has an id, and the execution result comes back into the history under that same id, paired as `tool → tool_call_id → tool_content`. That is all the skeleton amounts to; what actually causes trouble are two "unclean" moments inside the loop. **First, tool errors have to be attributable.** The same error has to be split into two cases: it went wrong when the arguments were passed into the tool function, or it went wrong while the tool was executing. The former is the model's fault (bad schema or arguments), the latter is the environment's fault (command, path, permissions). Without separating them, the feedback you hand the model is wrong, and the next round is wrong again. **Second, the history has to be made legal again after an interrupt.** When the user hits Ctrl+C, if it lands exactly in the gap where "the model has already emitted a tool_call but the tool has not finished executing," the history now contains a tool_call with no result. To the model that is an illegal state, and it affects the whole session. CoreCoder's move is to patch that gap with `content: ["interrupt"]` — restore the history to a legal state first, then rethrow the exception up to the caller. This idea of "patching in one legal piece of content" shows up once more later on — during context compression. ## The tool layer: definition, editing, and execution Tool definitions are plain: subclass the `Tool` base class, hold `name`, `description`, and `parameters`, implement `execute()`, then let `schema()` assemble it into OpenAI function calling format and register it into `tool_list`. Where the real work goes is the file-editing tool, because it is the hand the agent uses most. All three routes have been tried: 1. **Let the model rewrite the file wholesale** — too many tokens wasted. 2. **Let the model locate by line number** — the model is very fuzzy about line numbers, and line counts cannot drift. 3. **Let the model produce a unified diff** (the kind with a `@@ -42,7 +42,8 @@` header) — the error rate is maddening; it cannot get the context line counts and offsets right. So CoreCoder took a fourth route: **locate a uniquely identified edit**. The snippet to be replaced must appear in the file exactly once; if it appears more than once, make it carry more context until it is unique; after the edit, the tool hands the diff of that change back to the model so the model can judge for itself whether it got it right. Bash is a different discipline: every command passes through a detection table of dangerous commands like `rm` before it runs, and chained commands such as `cd a && cd b` have to be read in order, because `b` is relative to `a`. Claude Code's BashTool turns this into a system capability. The key is not that it "can execute commands" but that it builds command execution into a first-class capability with **semantic classification, permission constraints, task management, and UI presentation**: - **It provides the execution capability you need to verify code.** Without BashTool, an Agent can only make static edits; with it, the agent can run tests (`pytest`, `cargo test`), watch builds (`npm run build`), drive project scripts (lint, formatting), and connect to toolchains like Git and package managers to get the full project state. The distance between "understanding the code" and "verifying the code" is exactly this layer. - **It tries to understand a command's semantics instead of executing it as a black box.** It distinguishes whether a command is a search, a read, or something else, rather than dumping everything into the shell. The payoff is threefold: UI presentation is more sensible (a different interface per command type), security policy is finer-grained (read-only commands can be auto-approved with more confidence), and result handling is better (search and read results can be collapsed). - **It has strict security controls.** When a command comes in, it additionally cares about: whether it is a read-only operation, whether the operating path is legal, whether it should run inside a sandbox, whether it needs a user permission confirmation, and whether it is suitable for running in the background. One more note: in the CLI, the agent's actual output and the UI display are two different things. The display layer can truncate, collapse, and colorize — but that is the display layer's business. ## Context engineering: what to inject, what to cut Context comes down to two things: what you put in and what you cut out. **Injection.** Files like `CLAUDE.md` are the core — they are what you get after filtering and deduplicating repo information, project branches, memory information, and so on. They turn project experience from "something a person explains on the fly" into "injectable system knowledge," which directly affects three things: output style is more consistent, edits match repo conventions more closely, and the same mistakes stop recurring. Alongside them, git status goes into context as well. **Compression.** CoreCoder prepared three tiers of safety net, applying the next one when the previous is not enough: 1. A `tool_call`'s return value is only useful to the task at hand and there is little worth keeping in later tasks, so **keep the head and tail, delete the middle**. 2. Compress the history messages into a summary, but **keep the structured content** rather than turning them into a blob of background text. 3. If the first two tiers are still not enough, keep only the first few rounds plus the summary and **delete aggressively**. There is a trap here, from the same root as the interrupt problem at the start: **if the split point happens to land on a tool_call, `tool_id` and its `tool_content` are cut apart**, and the history is illegal again. So when deleting, you have to move the split boundary; you cannot cut straight through the middle. Claude Code's ordering is an extension of the same idea: **keep structured context first, compress and trim the low-value content, and do not rush to turn it into a summary**; if that is not enough, project a collapsed view of the context and let the model compress on its own; only then fall back to last-resort compression. A summary is not the first step, it is the last. ## Session and state: why QueryEngine is needed `main.tsx` is the application's "assembly root": it has to get everything ready before the REPL comes up. 1. Use profile checkpoints to measure initialization time per stage and fight for startup time (loading config, the API). 2. Pull context, command, skill, and tool, initialize external capabilities (MCP, LSP), and pin down the current session's state: whether it is interactive mode, whether there is a remote session or bridge mode, whether an old session should be restored, and what the model / permissions / prompt style / working directory are. 3. All of this is done before the REPL starts — by the time the user sees the interface, everything that can be used is already in place. `QueryEngine` is an object that **lives for the duration of a session**, and that is why it can span many rounds. The state it holds for the long term is exactly the set of things that "remain after a session ends": - message history - known permission denials - file read cache - usage statistics - discovery state for certain memory / skill `submitMessage()` is essentially "start one agent run": receive user input → set directory and session → filter tools → configure prompt and context → open a query to interact with the model → tool call output is appended to history → tally usage and state. That is the whole chain; there is no other magic. **Why does it have reflection and error correction?** Because it is not simply "the model calls a tool" — it is **tool results flowing back**: the model decides based on system_prompt / history → the decision may include tools → tools go through permission checks → the model calls a tool and gets a result → the result is injected into history → the next round begins. Every round the model sees the real result of the previous round; the ability to correct itself grows out of that loop, it is not a feature bolted on top. ## Concurrency: parallel tools and sub-agents In Claude Code, tools are already executing while the user input is being parsed, so it does not have to wait for the whole response to finish and replies are fast; CoreCoder executes in order, honestly. The difference is not "who is more advanced" but whether it is worth paying in complexity for that bit of latency. CoreCoder's trade-off: when a task has multiple tool responses, a single one just runs, and multiple ones go to `_exec_tools_parallel`, which is a thread pool. But parallelism is not unbridled — > Thread 1 is saving directory A while thread 2 saves B in parallel; come back later and A's contents have been overwritten. So threads have to be isolated from one another. Python has a ready-made tool for this: `threading.local()`, which gives each thread its own copy, invisible to the others. **Sub-agent handling is more worth reading.** CoreCoder has no explicitly defined "sub-agent" entity; it is defined through `AgentTool`: AgentTool points back to the main agent, and the sub-agent shares resources with the main agent; at the same time the sub-agent gets a separate context of its own, keeping the main agent's window clean — the main agent does not need to know what the sub-agent did in between, only its execution result. The one constraint: **a sub-agent must not call AgentTool again**, or it is infinite nesting. learn-claude-code draws this boundary more clearly: ![Sub-agent context boundary: the parent Agent receives only a task description, and the sub-agent returns only the final text](/asset/subagent-architecture.png) - The sub-agent gets a **brand-new `messages[]`**, `fresh`, with no inheritance from the parent conversation. - It runs its own `while` loop, at most 30 rounds, with a tool table of the basics — bash / read / write / edit / glob — and **no task**: only one level of delegation is allowed. - Its internal tool calls and results are not copied back into the parent's `messages[]`; only the final text goes back, through something like `extract_text()`, as the parent Agent's `tool_result`. Round-trip accounting: the parent Agent's window gains only "a short task description" and "a conclusion." ## Extension points: Hook and the task system **Hooks are written outside the loop.** If extension logic is shoved into the loop line by line, it soon becomes unrecognizable: ```python def agent_loop(messages): while True: # ... LLM call ... for block in response.content: if block.type != "tool_use": continue log_to_file(block) # add one line check_permission(block) # add one line notify_slack(block) # and another output = execute(block) auto_git_add(block) # one more # ... before long the loop is unrecognizable ``` Hence the HOOK table: `UserPromptSubmit` (context injection), `PreToolUse` (permissions, logging), `PostToolUse` (large-output handling), `Stop`… covering three kinds of needs — permissions on a tool (user consent, high-risk commands), printing the work log of a tool, and judging a tool's output (is it a large file). The key point: **the execution order could stay the same anyway; the value of a Hook is not changing the order but decoupling the concrete extension logic from the Agent Loop**, so extensions can be registered, added, and removed independently without touching the core loop. **Todo and Task are two different things.** If `todo_write` tells the agent "what to do," then `task_system` tells it **how the tasks depend on each other**. Todo is a list: it gives the Agent planning ability, but not every task needs a todo. States are written `[ ] pending`, `[>] in progress`, `[x] completed`. Task is a persisted file. For a job like "create the db → create the table → test," the whole thing has sequential dependencies: ```python # Create task dependencies, wired up through blockedBy # One task per file, saved across sessions / restorable @dataclass class Task: id: str subject: str description: str status: str # pending | in_progress | completed owner: str | None # the Agent responsible for the current task blockedBy: list[str] # list of task IDs it depends on ``` There are two rules: one task, one file (`.task/.json`); and **only when all the tasks in `blockedBy` (except the first) have `status=complete` can the next one start**. The status moves from `pending` to `in_progress` to `completed`. This also answers "if you quit CC, do the tasks still exist" — they do, because they are files on disk, not variables in memory. ## Multi-agent: Agent Team A subagent is like a tool — it finishes and it is done; **an Agent_team, by contrast, has to hold persistent state** (work, IDLE, shutdown). This part is the most "engineered" piece of the three implementations, and is worth noting item by item: 1. **Whether to use multiple agents is the user's call.** The lead proposes a plan; the user decides. 2. **Assignment runs through files, not contention.** One task, one file (there is a `lead.json` too), with `owner = agent` inside, so a task can be assigned directly instead of being fought over. 3. **Task assignment runs both ways.** Early on the lead assigns agent_task; but after an agent finishes there are still tasks left over (in reality the number of tasks far exceeds the number of agents), plus the lead's own review tasks, so idle agents go pick up new tasks (`new_task > lead_task`). 4. **Messages go through a persisted mailbox.** Agents — and the lead with them — pass information through `.mailboxes/.json`. ```python # Mailbox handles communication between agents, .mailboxes/ # One .json file per agent, recording the messages sent to that agent # Agents have independent context / loop / tool / messages class MessageBus: def send(self, from_agent, to_agent, content, msg_type="message", metadata=None): msg = { "from": from_agent, "to": to_agent, "content": content, "type": msg_type, "metadata": metadata or {}, } ... ``` 5. **Feedback has to be cleared.** When an agent finishes, it reports result / status back to the lead and calls `clear_lead`; without clearing, the next round injects the old agent_team information all over again. 6. **Parallelism needs isolated files.** Agent_team executes in parallel; to keep agents from interfering with each other's edits of the same content, a `worktree` branch is created under the directory (`.git/worktree`). 7. **Only what is observable is executable.** The Runtime exposes args to show whether an agent is really "doing" something, rather than merely having accepted the request. 8. **One agent, one thread.** A thread is created, each agent gets a loop of its own, executing `(task, name)`. ## Long-lived capabilities: memory, MCP, background tasks, scheduled tasks ### memory Memory has to solve four problems, each with a corresponding mechanism: **Storage — how is it persisted?** One memory, one file, kept under `.memory/` in categories (`personal_prefer`, `project`…), with fields `name` / `description` / `type` / `content`; plus a `.memory/MEMORY.md` recording a one-line summary of each memory, one per line, acting as an index. **Recall — how does it get into the conversation?** Four progressive steps: first take the last few rounds of conversation as feedback; then use a small model to semantically match the user input against `MEMORY.md` to get an index; if matching fails, normalize the user input into keywords and match those against a memory's `name` and `description`; finally, memory joins as **background knowledge** — the current request takes priority over historical memory; "taken into account first," not overridden. **Extraction — how is it extracted from context?** Have the LLM parse the user input into json to match against memory; only `scope=persistent` gets stored; the fallback is a temporary memory vocabulary check; missing fields likewise disqualify it from being stored. **Maintenance — how are memories optimized?** When the number of memories reaches a threshold, reorganize the knowledge; when handling duplicate memories, compare fields like `name`, `description`, and `body`; **snapshot the original file before changing anything**, so a failed edit or delete can still be recovered (snapshot first, then change). ### MCP The chain has three layers: **mcp server (where you get tools) → mcp client (executes tools with a handler) → harness**. - **The tool_pool is dynamic.** The first round has only the base tools; after connecting to an mcp server and getting the corresponding tools, the second round assembles them with `assemble_tool_pool()`, `Tool = base tool + mcp_client_tool`. - **Why name them `mcp__{server}__{tool}`?** Because different servers may have tools with the same name, and the server prefix keeps them from colliding. - **Permissions need a choke point.** If some mcp server offers inherently dangerous capabilities like `delete_database` or `trigger_deploy`, the harness or the user has to make the final confirmation. ### Background tasks The idea in one sentence: after the Agent starts a time-consuming task, do not let it sit there waiting — throw the task into the background, let the Agent keep thinking and working, and tell it the result once the task is done. - Currently only bash is opened up: the tool_call's return carries `{"name": …, "run_in_background": …}`, and only when `tool_name == "bash"` and `run_in_background` is true does it go to background execution — a judgement that depends on the LLM's semantic understanding. - So one tool gets **two tool_calls**: `bash → bg_id` on the main thread, `bash → command` in the background. The background result is formatted into a ``, telling the LLM that this is a new, separate event. - Beyond locking (`threading.Lock`), there is another detail: when a task completes it is **not inserted into the agent loop right away** but first put into `result{}` / `_ready[]`, to be consumed by the next loop. ### Cron scheduler Scheduled tasks, such as having the agent push once a day. The mechanism is cruder than you would expect: **update the time every second and compare it against the task time**; when it is due, put the task into `cron.queue` and deliver it into the message once the agent is idle; a lock avoids thread contention. There is a first-fire flag: the first fire puts it into the queue, and later fires do not store it again. ```python # durable=True writes it to json, kept across sessions / saved on task restart @dataclass class CronJob: id: str cron: str prompt: str recurring: bool durable: bool pending_delivery: bool = False last_fired: str | None = None # cron decides when to fire, prompt is the task handed to the Agent once it fires. # pending_delivery means the task is already due but has not been picked up by the model, # last_fired prevents the same minute from being enqueued twice. ``` ## Security boundaries A few security designs have come up in passing; here they are collected. CoreCoder's `/save` is a good example: saving a session asks the user to type a session name, but on the command line that is just typing arbitrary characters — it defends against nothing. **The first line of defense is sanitization.** Take just the trailing name, so it cannot jump to another directory: ```text Input: session_id = "../../etc/passwd" Sanitize: replace("\\", "/") -> "../../etc/passwd" split("/")[-1] -> "passwd" regex replace -> "passwd" strip("._") -> "passwd" return "passwd" Join: /home/user/myapp/sessions/passwd.json Check: parent directory is /home/user/myapp/sessions, passed ``` **The second line of defense is validation.** If sanitization is somehow bypassed, the parent directory is compared once more after `resolve()`: ```text Join: Path("/home/user/myapp/sessions/../../../etc/passwd.json") Resolve: resolve() -> /etc/passwd.json Check: path.parent is /etc, which does not equal root (/home/user/myapp/sessions) Result: immediately raise ValueError("Invalid session id") and block access ``` These two, plus the Bash dangerous-command table and the final confirmation on dangerous MCP tools from earlier, follow one principle: **do not count on the model to police itself — concentrate the danger at the boundary and make crossing it impossible at the code level.** ## Three takeaways from reading these Put the three implementations together and roughly three judgements survive: **The loop itself is the simplest part; what is hard are the "unclean" moments inside it.** Interrupts, errors, compression, concurrency — every one of them makes the history or the state illegal, and the pattern for handling them is surprisingly uniform: make the data legal again first, then talk about anything else. `content: ["interrupt"]`, moving the split boundary of compression, giving each thread its own copy — they are all the same move. **Context is a resource, and compression is not an after-the-fact patch.** It determines where to cut and what to keep when compressing. That is why "keep the structured content" comes before "turn it into a summary" — a summary is irreversible, trimming is reversible. **An Agent's capability boundary is held up by external systems.** Task files, mailboxes, worktrees, memory directories, cron queues — put together they look pretty crude, but they are exactly what determines whether an agent can span sessions, whether it can collaborate, and whether it can be observed. This layer outside the model is where the engineering really has to be written. --- # Transformer learning notes: from the overall architecture to multi-head attention Author: HYD Source: https://hydblog.xyz/en/2026/04/04/transformer-learning-notes/ Published: 2026-04-04T13:00:00.000Z Updated: 2026-04-04T13:00:00.000Z Summary: Transformer study notes ordered whole-first and then part-by-part: the data flow through the encoder and the decoder; the three questions to answer before learning large models (neural networks, attention, what PyTorch is made of); what Token, the word embedding matrix, and positional encoding each solve; the meaning of Q, K, and V, why the Q·K dot product stands for relevance, and why the dot product is divided by the square root of the dimension before softmax; the difference between self-attention and cross-attention, and why cross-attention needs no mask; training predicts several tokens in parallel while inference generates one token at a time, which is why a causal mask is needed to prevent peeking, implemented by setting later positions to negative infinity so that softmax turns them into 0; the definitions of the feed-forward network, multi-head attention, and layer normalization, including how layer norm and batch norm differ in which dimension they normalize over. These notes are what I wrote down while learning the Transformer; they were first posted on CSDN. Now that they have moved back to my own site, I have reworked the order and the data flow along the way — the conclusions are unchanged, the explanation just flows a little better. It is not really a tutorial, more a record of studying: look at the whole first, then take the parts apart one by one. ## The big picture first: how the data flows The input first passes through the word embedding matrix, where it is fused with the positional encoding, then enters multi-head attention (Q, K, V), then goes through layer norm and a residual connection, into the feed-forward network, and once more through layer norm and a residual connection. Up to this point we have the input plus the encoder — and the encoder can be stacked N layers deep with that same structure. On the decoder side, the input starts from ``, again goes through word embedding and positional encoding first, and then passes through, in order: ```text causal mask attention → layer norm + residual → cross-attention → layer norm + residual → feed-forward network → layer norm + residual → linear layer → softmax → output ``` So the whole architecture is nothing more than these few modules, and we can just understand them one by one. ![The overall Transformer architecture: the encoder on the left, the decoder on the right, with layer norm and a residual connection after every sublayer](/asset/transformer-architecture.webp) ## Getting to know large models We have all used plenty of large models by now, so you will have noticed that after you ask a question, the answer does not come out all at once — it pops out one word at a time, like a word-chain game. Once you have learned the fundamentals of large models and then look back at this process, your understanding of AI becomes much more concrete. In learning large models and learning about agents, I set myself three questions to answer first. **1. What is a neural network?** The most basic answer is that it is the foundation of learning: `x -> f(x) -> y`. **2. What is attention?** For your input, I need to allocate different amounts of attention (that is, weight parameters) so that the output moves toward what you want. Looked at spatially: a word (token) has many vector directions, and attention is what turns that word into the vector direction that fits the current meaning. ![A sketch of word-vector space: semantic relationships are encoded as directions](/asset/transformer-embedding-space.webp) **3. What is PyTorch made of?** Remembering these few parts is enough: Tensor, Parameter (the optimizable parameters, that is, the initialized parameters), Model, Autograd (computing gradients), Optimizer (optimizing the parameters). Now we can get into the Transformer. ## Model components ### 1. Token First we need to split a sentence into many words, and such a word is called a token. Take `I love you` — that is 3 tokens; for now just think of a token as a word. A computer does not know these words, so they still have to be turned into a language it can recognize — vectors. Each token contains many numbers, and those numbers are hard for us to read directly, but roughly they can represent complex information such as a word's meaning, part of speech, and position. The same word has different vectors in different sentences. ![The same word points in different vector directions in different contexts: mole points along different semantic directions under its two meanings](/asset/transformer-polysemy-embedding.webp) ### 2. The word embedding matrix The first approach to handling a token was one-hot encoding (0/1). Take the Chinese sentence 我是一只狗 ("I am a dog"): ```text 我 -> [1, 0, 0, 0, 0, …] 是 -> [0, 1, 0, 0, 0, …] 一 -> [0, 0, 1, 0, 0, …] ``` The problem is obvious: the dimensionality is huge, and only one position in each vector is 1. So word embeddings come in: through the word embedding matrix (a weight table learned during training), words that are close in meaning cluster together and words that differ greatly spread apart, and dimensionality is reduced at the same time — separating natural language by numbers. - The word vector dimension `d` is typically 512; it holds the basic semantic information of the token and is a weight table that can be tuned during model training. - `V` is the vocabulary size, that is, the number of tokens. - The word embedding matrix is `d × V`; a single token's one-hot vector is `V × 1`, and multiplying them gives `d × 1`. The tokens of a whole sentence combined give `V × d`. ![A diagram of one-hot encoding, the word embedding matrix, and word vectors reduced to two dimensions](/asset/transformer-word-embedding.webp) ### 3. Positional encoding At this point the data has in fact already been processed, so why process it a second time? Because what we are dealing with is not data like tables or images but information that carries order and semantic logic. Take 狗咬人 and 人咬狗 ("dog bites man" and "man bites dog"): the words sit in different positions and the meaning is completely different, so we need a piece of information that measures this property. Positional encoding in the Transformer is a formula built on trigonometric functions; its parameters are the embedding dimension `d` and the position `pos` of the current token in the sequence, and the result is added element-wise to the word embeddings. ### 4. Attention: Q, K, V At the outset, a token contains only its own vector, but once it enters the Transformer, attention moves that vector to the semantic position that fits the context. When we predict the next token, the basis is the embedding vector of the previous token; because it already contains the semantics of the context, its parameter count is far larger than that of a single token on its own. Next come the three most important parameters: Q, K, V. **Q (Query)**: in an English sentence, for instance, a noun asks whether there is an adjective in front of it, and that question is encoded into another vector, the query. In effect, every word queries its own context for information. `Q = Wq · E`, where `Wq` is a parameter the model learns; the dimension of Q is far smaller than the dimension of E, which amounts to going from high dimensions down to low ones. ![Query illustrated: every word puts a question to its context — is there an adjective in front of me](/asset/transformer-query.webp) **K (Key)**: if there really is an adjective in front, then the adjective's answer becomes the key. `Wk` is also a learnable parameter, `K = Wk · E`. So when the noun's query for an adjective really exists, the relevance is high: the dot product of Q and K is large, and the query and the answer line up. ![The attention matrix produced by the dot product of Q and K: the relevance score between every two tokens in the sentence](/asset/transformer-qk-dot-product.webp) To turn that into probabilities we then pass it through softmax; for numerical stability we first divide by the square root of the dimension. ![softmax(QKᵀ / √d_k): after the dot product, divide by the square root of the dimension and then take the softmax to get probabilities](/asset/transformer-softmax-formula.webp) **V (Value)**: the vector you need to add in if you want to change the meaning of a particular word. ![Self-attention in matrix form: X multiplied by Wq, Wk, and Wv gives Q, K, and V](/asset/transformer-self-attention-matrix.webp) I may not have explained this very well, so I would suggest reading up on it elsewhere as well. What I described above is self-attention; in machine translation there is also cross-attention: words in one language query and answer words in another language, and no mask is used here (more on that later), because there is no "peeking" problem. ![Cross-attention in matrix form: Q comes from the decoder, K and V come from the encoder](/asset/transformer-cross-attention-matrix.webp) ### 5. Training and inference Take a translation task as an example: the input is 我是一条狗 and the labels are `I am a dog`. Inference first: the Chinese input goes through the encoder and yields the encoded information; the decoder starts from ``, combines the neural network's computation with that encoded information, and produces the highest-probability English word `I`; then from ` I` it produces `am`; then from ` I am` it produces `a`… and so on until the `` terminator. So it takes each output together with the previous outputs as input and infers the next output from them. Training is a little different. First, the model makes full use of the data: it predicts several tokens at once, so one sample trains many times over, and the loss is computed in parallel, which speeds things up enormously. At the same time, we need the model to train in the right direction. Back to the translation task: if the model translates the very first step wrongly, everything after it only goes more wrong. So we tell the model the "correct answer", making sure it learns under the right conditions. And that is where the problem shows up — the answer has been given to the model, so it can simply read along; what is left to train then? So we introduce a mask into attention, to stop the model from "peeking" and to stop later words from influencing the prediction of earlier ones. ![During training several positions are predicted at once, and each position can only see the words before it](/asset/transformer-masked-prediction.webp) ### 6. The causal mask attention mechanism We want this "query" not to be influenced by later words; and since probabilities are what come out in the end, the method is to set the later positions to negative infinity first, which softmax then turns into 0. ![The causal mask in matrix form: the masked upper-triangular region is set to negative infinity before softmax](/asset/transformer-causal-mask.webp) ![The attention matrix after masking: the upper-triangular region is forced to 0](/asset/transformer-attention-pattern.webp) ### 7. The feed-forward network A kind of network structure: no recurrence, one-directional flow, multiple layers, mostly fully connected layers. (An RNN is not one; AlexNet is.) This part is relatively simple. ### 8. Multi-head attention For a set of tokens, several independent attention computations are run on each token. Different attention weights W focus on different angles — the meaning within the sentence, punctuation, syntax, and so on. ![Multi-head attention: several independent sets of Wq, Wk, and Wv computed in parallel, then concatenated](/asset/transformer-multi-head.webp) ### 9. Layer normalization Layer normalization: for one sample, normalize its outputs across all neurons together (y1, y2, y3, y4, …). Batch normalization (the kind ResNet uses): within the same layer, normalize the outputs of different samples at the same neuron (y1, y1, y1, y1, …). | Normalization | What it normalizes over | Typical use | | --- | --- | --- | | Layer normalization LayerNorm | The outputs of one sample across all neuron dimensions | After every sublayer in the Transformer | | Batch normalization BatchNorm | The outputs at the same neuron across different samples in a batch | Convolutional networks such as ResNet | ![The LayerNorm formula: subtract the current sample's mean, divide by the standard deviation, then multiply by a scale factor and add a bias](/asset/transformer-layernorm-formula.webp) ## In closing I am a learner too — at the time I followed the courses by Paoge and 3BB on Bilibili. This article is my own understanding of the Transformer, written only to record my learning process and output; there may be one-sided or wrong points in it, and I warmly welcome corrections so that we can learn together. The architecture diagrams, formula figures, and matrix screenshots in this article come from the course slides; the word-vector-space illustration comes from Paoge's channel and 3Blue1Brown's attention series.