memcache

memcache gives Copilot CLI a small, repo-local memory. It writes durable notes to .agents/.memcache/, searches them fast, builds cited context packs, and keeps noisy tool output from flooding future chats.

Think of it as a project notebook for agents: record what mattered, recall it later, and keep the source of truth in the repo instead of in one chat session.

What it tries to be

• Boring to run: one Go binary, stdio MCP transport, no database.
• Easy to trust: append-only JSONL journal, deterministic rollups, repair from source.
• Cheap in tokens: compact MCP responses by default, with opt-in detail when needed.
• Repo-native: state lives under tracked .agents/.memcache/, never root .memcache.
• Dependency-light: Go stdlib first; golang.org/x/... only when explicitly justified.

Set it up with Copilot CLI

Build the server once, then point Copilot CLI at the binary.

git clone <this-repo-url> ~/src/memcache
cd ~/src/memcache
go test ./...
go build -o ~/.local/bin/memcache ./cmd/memcache

On Windows PowerShell, build an .exe:

go test ./...
go build -o "$env:USERPROFILE\bin\memcache.exe" ./cmd/memcache

Sanity-check the binary from any repo:

cd /path/to/your/repo
~/.local/bin/memcache inspect --repo .

Add one global MCP entry to Copilot CLI. Start copilot, run /mcp add, then fill:

name: memcache
type: local
command: /home/you/.local/bin/memcache
args: serve

On Windows, choose the Copilot CLI STDIO transport if the UI asks for a transport type. serve and stdio are equivalent stdio MCP entrypoints; the stdio alias exists for clients and setup screens that name the transport explicitly:

name: memcache
type: stdio
command: C:\Users\you\bin\memcache.exe
args: stdio

Some Copilot CLI versions serialize local stdio servers as type: local in JSON even when the UI labels the transport STDIO; either spelling is fine as long as the command runs memcache.exe stdio or memcache.exe serve.

Press Ctrl+S to save. Copilot stores this in ~/.copilot/mcp-config.json by default, or $COPILOT_HOME/mcp-config.json if COPILOT_HOME is set. You can also edit that file directly:

{
  "mcpServers": {
    "memcache": {
      "type": "local",
      "command": "/home/you/.local/bin/memcache",
      "args": ["serve"],
      "tools": [
        "memory.record",
        "memory.search",
        "memory.context",
        "memory.timeline",
        "memory.repair"
      ]
    }
  }
}

Windows JSON example for CLIs that store the transport as stdio:

{
  "mcpServers": {
    "memcache": {
      "type": "stdio",
      "command": "C:\\Users\\you\\bin\\memcache.exe",
      "args": ["stdio"],
      "tools": [
        "memory.record",
        "memory.search",
        "memory.context",
        "memory.timeline",
        "memory.repair"
      ]
    }
  }
}

Use an absolute path for command. serve/stdio speaks MCP over stdio, so it keeps stdout clean and sends diagnostics to stderr. In MCP mode it asks Copilot for the current workspace roots and writes to that repo's .agents/.memcache/; if roots are unavailable, it falls back to the process working directory. If you really want to pin one store, use args: serve --repo /path/to/repo.

Restart copilot in that repo, check /mcp, then try:

Use memory.context for "what did we decide about the cache?"
Record a memory: kind decision, summary "Keep rollups deterministic"

State lands in the active workspace's .agents/.memcache/. Track that directory in the repo so future agents inherit memory.

Local CLI

If you install memcache on your PATH, use memcache .... While hacking on this repo, go run ./cmd/memcache ... does the same thing.

Command	What it is for
serve [--repo PATH]	Start the MCP server over stdio. This is what Copilot usually runs.
stdio [--repo PATH]	Same as serve, named for clients that explicitly ask for a STDIO transport.
inspect [--repo PATH] [--json] [--repair]	Show store health. Add --json for scripts. Add --repair to fix derived state first, then print health.
repair [--repo PATH] [--json]	Fix .agents/.memcache/: update layout metadata, prune legacy caches, rebuild the index, regenerate rollups, then report health.
reindex [--repo PATH]	Rebuild only the search index from the journal. Use this when search looks stale but the rest of the store is fine.
record --kind KIND --summary TEXT [--repo PATH]	Add one memory entry. Use this after meaningful decisions, fixes, tests, or handoff points.
submit --kind KIND --summary TEXT [--repo PATH]	Alias for record, named for agents that say "submit to memcache."
import-jsonl [--repo PATH] [FILE...]	Import a batch of JSONL memory entries. Validation happens before any writes.
compact [--repo PATH] [--dry-run] [--older-than DURATION] [--limit N]	Summarize older noisy entries without deleting originals. Use --dry-run first.
search --query TEXT [--repo PATH]	Search memory from the terminal.
context --task TEXT [--repo PATH]	Build a cited memory pack for a task.
timeline [--repo PATH]	Review recent memory chronologically, optionally grouped by day or kind.

Common commands:

go run ./cmd/memcache inspect --repo .
go run ./cmd/memcache inspect --repo . --json
go run ./cmd/memcache inspect --repo . --repair
go run ./cmd/memcache repair --repo .
go run ./cmd/memcache reindex --repo .
go run ./cmd/memcache serve
go run ./cmd/memcache stdio
go run ./cmd/memcache record --repo . --kind decision --summary "Keep memory compact"
go run ./cmd/memcache submit --repo . --kind checkpoint --summary "Finished cache fix"
go run ./cmd/memcache import-jsonl --repo . events.jsonl
go run ./cmd/memcache compact --repo . --dry-run --older-than 720h
go run ./cmd/memcache search --repo . --query "kind:decision memory"
go run ./cmd/memcache context --repo . --task "fix nil pointer" --include-recent
go run ./cmd/memcache timeline --repo . --group-by day --limit 20

State-changing and maintenance commands print a final done: <command> line on success. --json output stays machine-readable without that text footer.

serve and stdio are special: they are reserved for MCP stdio mode. They must not write non-MCP logs to stdout; diagnostics go to stderr.

If an agent needs to write memory, it must make one real write: call MCP memory.record, or run memcache submit --repo . --kind note --summary "..." / memcache record ... from the shell. Do not simulate memory with no-op shell commands such as true, false, or echo cannot-call-memcache-tool-from-schema; if neither MCP nor the CLI is available, say that once and continue.

Implemented MCP methods:

• initialize
• notifications/initialized
• notifications/resources/list_changed
• notifications/roots/list_changed
• outgoing roots/list request when the client advertises workspace roots
• ping
• tools/list
• tools/call
• prompts/list
• prompts/get
• resources/list
• resources/read
• resources/templates/list
• completion/complete

MCP tools:

• memory.record: append a memory entry. Required: summary. kind defaults to note; use note when unsure. Duplicate content and same-summary retry loops are suppressed and return the existing entry with duplicate: true.
• memory.search: ranked recall. Required: query. Optional view is full or compact.
• memory.context: cited task memory pack. Required: task.
• memory.timeline: chronological/grouped memory view. Optional view is full or compact.
• memory.repair: runtime self-repair for derived state. Rebuilds index/rollups, prunes legacy object cache, updates relocated manifest paths, and leaves journal records untouched.

Successful tool calls start text with done: <tool>, set _meta.done: true, _meta.status: "done", and _meta.operation, and include matching done, status, and operation fields in structuredContent. Treat that as terminal: do not retry or run no-op shell placeholders after a done result.

memory.search and memory.timeline default to view: "compact" to save tokens. Compact view returns shorter text, smaller structured hits/entries, and no per-hit resource links; each compact item keeps id, uri, kind, ts, and summary so clients can fetch mem://entry/{id} for detail. Clients that need legacy rich arrays, snippets, files, tags, or default resource links should send view: "full". This is output-only and works with existing journal records.

Tools that can cite entries (memory.record, memory.search, memory.context, and memory.timeline) accept links: "auto" | "none" | "all". auto preserves existing behavior: full search/timeline/context/record include resource_link blocks, compact search/timeline omit them. none suppresses resource links to save tokens; structured outputs still carry entry IDs and/or mem://entry/{id} URIs so clients can fetch details with resources/read. all forces resource links, including compact views.

memory.search and memory.timeline accept fields to project structured hit/entry fields. Search always keeps id, uri, and score; timeline always keeps id and uri. Allowed search fields are id, uri, score, kind, ts, summary, snippet, text, files, refs, tags, session_id, importance, and terms. Timeline allows the same fields except score, snippet, and terms.

tools/list keeps JSON schemas terse to reduce startup tokens; this README is the detailed argument reference.

MCP resources:

• mem://repo/overview: repo memory card.
• mem://repo/recent: recent timeline.
• mem://entry/{id}: one entry.
• mem://file/{path}: file memory card.

resources/list returns static repo resources plus compact known entry and file-card descriptors from the journal. Dynamic descriptors keep only fetch-critical fields so discovery stays small; use resources/read for detail and annotations. List methods support MCP opaque cursor pagination with nextCursor; invalid cursors return JSON-RPC -32602.

The server advertises resources.listChanged. After initialization, successful memory.record calls emit notifications/resources/list_changed because resource discovery gains a new entry and may gain a new file card.

MCP prompts:

• memory-workflow: teach agents when to call context, search, timeline, and record.
• memory-context: guide the client to build cited task memory with memory.context.
• memory-search: guide the client to search prior work before answering or editing.
• memory-record: guide the client to capture one durable event with memory.record.
• memory-timeline: guide the client to review chronological memory for handoff or recent work.

Tool results include compact text plus structuredContent matching each tool's outputSchema. By default, search, context, timeline, and record responses include resource_link blocks for cited mem://entry/{id} resources so clients can fetch full entries only when needed; links: "none" suppresses those blocks.

Search and timeline limits are capped at 100 items. token_budget is capped at 16000 approximate tokens so MCP and CLI responses stay bounded.

Completions are implemented for resource templates:

• mem://entry/{id} completes the id argument from known memory entry IDs.
• mem://file/{path} completes the path argument from repo-relative files seen in memory.
• Prompt completions cover enum-like prompt arguments such as kind, group_by, and booleans; prompt file arguments reuse file-path completion.

Static resources, resource templates, resource read contents, and entry resource links include MCP annotations for audience/priority. Dynamic resources/list entry/file descriptors intentionally omit annotations to save tokens. Entry resources and file/repo memory reads include lastModified when the source timestamp is known.

Unknown tools return JSON-RPC -32602. Tool validation/runtime failures return isError: true. Missing resources return MCP resource error -32002. Reserved MCP _meta fields in params or tool arguments are ignored; other unknown fields stay validation errors.

Store layout

.agents/.memcache/
  manifest.json
  journal/
  index/
    manifest.json
    segments/
  rollups/
    daily/
  tmp/
  locks/

manifest.json records the current checkout path and a stable repo_id used in journal records; when a tracked .agents/.memcache/ moves to another machine, repo_id stays fixed while repo_root/store_root update to the new checkout. Legacy stores derive repo_id from their original repo_root so existing entry IDs keep validating. journal/ holds canonical append-only JSONL memory records at journal/YYYY/MM/DD.jsonl and is the only per-entry persistent store. index/ holds dependency-free search structures. rollups/ holds compact scannable repo and daily memory cards; file cards are served on demand from the journal rather than materialized as one file per touched repo file. Older stores may contain legacy objects/ or rollups/files/ caches; current writes do not create them, and inspect --repair prunes them.

Memory entries use these fields:

• id: stable base32 SHA-256-derived identifier
• repo: stable store repo identity from manifest.json (repo_id); omit on writes to use the current store identity
• ts: RFC3339Nano timestamp
• kind: turn, command, edit, test, error, decision, checkpoint, note, or summary; MCP record defaults to note
• summary: required compact human line
• text, files, refs, tags, importance, session_id, actor, source, metadata: optional detail

Writes use a repo-local lock file, then append to the journal. Single-record writes append a small immutable index segment after the journal write, and automatically rebuild the index when incremental segments would exceed 16 files; batch imports rebuild the index once after the batch. The JSONL journal stays source of truth for rebuilds, and the index manifest stores a cheap journal file signature for fast freshness checks. Corrupt segment caches are rebuilt from the journal before recall or after the next write. Write, repair, reindex, rollup, resource, and context paths reject malformed, duplicate, wrong-repo, or tampered entries before using journal data; search skips invalid journal records with a warning and indexes valid records only. Duplicate protection includes exact content dedup independent of timestamps plus a same-kind/same-summary retry guard for volatile tool-output loops: decision, checkpoint, note, and summary duplicates are suppressed globally, while event-like duplicates are suppressed inside a short retry window to stop tool loops without losing later recurring events.

Harness integration

Harnesses should emit compact, useful records instead of raw noisy transcripts:

• Task start/end: turn or checkpoint.
• Commands: command only when command affects debugging, build, test, or repo state.
• File edits: edit with repo-relative paths and short summary; do not store full diffs by default.
• Test/build failures: error or test with command, exit status, and compact failure snippet.
• Durable choices: decision.
• If memory.record returns duplicate: true, treat the existing entry as the citation and do not retry the same record.

CLI ingestion uses the same record validation, redaction, indexing, and rollup path as MCP memory.record. submit is just an alias for record, for agents and scripts that phrase the action as "submit to memcache." Batch JSONL import validates every line before writing, prepares/redacts accepted entries once, appends the prepared entries under one store write path, then rebuilds index/rollups once.

Record one event:

go run ./cmd/memcache record \
  --repo . \
  --kind edit \
  --summary "Add ranked search tests" \
  --file internal/index/index_test.go \
  --tag test \
  --ref master \
  --session-id "$SESSION_ID"

Import JSONL events:

{"kind":"command","summary":"Run go test","text":"go test ./... passed","tags":["test"]}
{"kind":"decision","summary":"Keep store under .agents/.memcache","files":["README.md"],"tags":["architecture"]}

go run ./cmd/memcache import-jsonl --repo . events.jsonl

Invalid JSONL lines, invalid records, wrong-repo-identity records, duplicate import entries, and entries already present in the journal are rejected with source and line diagnostics before any import writes happen. Successful imports report imported: N and redacted: N when any records had secret-shaped fields redacted.

Compact old noisy memories:

go run ./cmd/memcache compact --repo . --older-than 720h --limit 50

compact is non-destructive: it appends one summary record with memcache_compaction metadata listing compacted entry IDs, then rebuilds index/rollups. Original journal entries remain for audit and can be searched with --include-compacted, but normal search/context/timeline/rollups suppress compacted originals. By default, candidates are low-importance turn, command, edit, test, error, and note entries older than the cutoff; durable decision, checkpoint, and existing summary entries stay active. Use --dry-run to preview candidates without writing.

Search from CLI:

go run ./cmd/memcache search --repo . --query "nil pointer kind:error session:$SESSION_ID" --file internal/index

Build a cited task context pack from CLI:

go run ./cmd/memcache context \
  --repo . \
  --task "fix nil pointer in memory loader" \
  --file internal/memory \
  --session "$SESSION_ID" \
  --include-recent \
  --token-budget 800

Review timeline memory from CLI:

go run ./cmd/memcache timeline \
  --repo . \
  --since 2026-05-10 \
  --group-by kind \
  --session "$SESSION_ID" \
  --limit 50

Session-aware recall:

• Record harness/session identity with session_id (--session-id in CLI).
• Filter search queries with session: or session_id:.
• Use --session filters on CLI search, context, and timeline.
• Use MCP sessions filters on memory.search, memory.context, and memory.timeline.

Best-effort redaction replaces obvious secret-shaped values before write and reports affected fields in command/tool output. Redaction is conservative and not a substitute for avoiding secret input.

Reliability and maintenance

memcache inspect reports store health:

• manifest schema and repo/store identity status
• journal entry and issue counts
• index manifest entry count and segment count
• index freshness via index_current, index_lag, and index_issue
• segment count and byte size
• quarantine file count
• active lock files and stale-lock status

Use memcache inspect --json for harness-readable health fields with stable snake_case names. manifest_ok is true when the store manifest schema is supported and its tracked paths/repo_id are normalized; moving a tracked store across machines is valid, and the next write updates repo_root/store_root while preserving repo_id. index_current is true only when the index manifest is valid, segment files are readable and match the manifest segment count, the journal has no malformed records, indexed entry count matches journal entry count, and the index manifest journal signature matches current journal file metadata. Derived index corruption or drift sets index_issue instead of making inspect fail.

Use memcache repair, memcache inspect --repair, or MCP memory.repair to repair derived state from the journal. Repair prunes legacy object-cache files, removes legacy file-rollup caches, rebuilds search index, regenerates rollups, and updates relocated manifest paths/schema metadata. It does not mutate journal files and stops if journal integrity checks report issues. memcache repair --json and inspect --repair --json return both repair and post-repair health.

memcache reindex rebuilds from the JSONL journal into a staged temporary index, writes the manifest last, then swaps the staged index into place. It also compacts many incremental record segments back into one full segment. If journal lines are malformed, denormalized, or have entry IDs that no longer match their canonical content, reindex leaves the previous index in place and writes the bad records to .agents/.memcache/quarantine/ for inspection. Journals are never auto-deleted.

On-disk schema changes must go through the migration hook in internal/store/migrate.go; unknown schema versions and mismatched store manifests are rejected instead of guessed or silently rewritten.

Search index

memcache reindex rebuilds search state from the JSONL journal. The index uses immutable JSON segment files under .agents/.memcache/index/segments/. MCP/CLI single-record writes append inc-* segments; reindex writes a compact full-* segment. Search repairs stale, missing, or corrupt index segments before returning hits: it appends journal entries missing from the index, but rebuilds when segment JSON is invalid, segment contents fail validation, or indexed documents are no longer present in the journal. Fresh indexes compare the manifest journal signature with current journal file metadata, then read segment data without parsing the full journal or rewriting the manifest. If malformed journal records are present, search skips them, reports journal_issues/a warning, and keeps serving hits from valid records; use memcache inspect and memcache reindex to diagnose and quarantine the bad lines.

Analyzer behavior:

• Unicode lowercase.
• Split text on non-letter/non-digit characters.
• Drop a small built-in stopword set.
• Index weighted fields: summary, text, file paths, refs, tags, session id, and kind.
• File paths add full path plus segment/base/ext tokens, so internal/store/entry.go can match internal, store, entry, go, or the full path.

Query behavior:

• Plain terms use BM25-style scoring with field weights.
• Quoted phrases must occur in the entry text/search fields.
• Filters: kind:, file:, tag:, ref:, session:, session_id:, since:, until:, limit:, token_budget:, include_compacted:.
• Recency and importance apply small boosts after term scoring.
• Result snippets use [[term]] markers, never ANSI.

Rollups and context packs

Rollups are deterministic markdown views rebuilt from active journal entries. Each rebuild removes stale generated daily markdown and prunes legacy per-file rollup caches.

• rollups/repo.md: durable decisions/summaries plus recent activity.
• rollups/daily/YYYY-MM-DD.md: chronological daily memory.
• mem://file/{path} resources: compact file memory cards rendered on demand from active journal entries.

Context packs combine durable facts, ranked search hits, and optional recent activity. Every memory-derived line cites an entry ID in [id] form. Token budgets use a conservative tokens * 4 character cap and truncate at line boundaries. CLI context and timeline use the same memory service as MCP tools.

Dependency policy

All Go imports must be one of:

• standard library packages
• this module: git.sharkk.net/go/memcache/...
• explicitly approved golang.org/x/... packages

go test ./... includes a guard that rejects other external imports.

Benchmarks and fuzzing

Core parsers and hot paths have Go benchmark/fuzz coverage:

• MCP wire decode/encode and raw server input.
• Query parsing, result formatting, analyzer, and search.
• Entry preparation, append path, redaction, rollup generation, and context packs.

Useful commands:

go test ./...
go test ./... -run=^$ -bench=. -benchmem
go test ./internal/mcpwire -run=^$ -fuzz='^FuzzDecodeLine$' -fuzztime=15s
go test ./internal/mcpserver -run=^$ -fuzz='^FuzzServeRawInput$' -fuzztime=15s
go test ./internal/index -run=^$ -fuzz='^FuzzParseQuery$' -fuzztime=15s
go test ./internal/index -run=^$ -fuzz='^FuzzFormatResults$' -fuzztime=15s
go test ./internal/store -run=^$ -fuzz='^FuzzPrepareEntry$' -fuzztime=15s
go test ./internal/memory -run=^$ -fuzz='^FuzzRedactEntry$' -fuzztime=15s
go test ./internal/memory -run=^$ -fuzz='^FuzzCompactSummary$' -fuzztime=15s

Fuzzing found and fixed:

• composite JSON-RPC IDs were accepted; now only string/number IDs are valid.
• limit: and token_budget: query filters bypassed final bounds checks.
• mem://file/... resource reads could traverse outside file rollups.
• Windows-style backslash traversal in file inputs was not normalized before validation.
• adjacent GitHub-style tokens were not redacted.