RFC-005 — Keyboard-driven region capture (multi-source)

• Status: Draft
• Authors: @yiidtw
• Created: 2026-05-06
• Related: RFC-001 (bridge), RFC-003 (grounding — this RFC supplies the citable units), RFC-007 (planned: macOS Accessibility hint mode)

TL;DR

Make every visual region a user can point at — a figure on a webpage, a table on a PDF, a frame in a design tool — addressable by a stable amem:// URI. Reference once, reference forever, even after the source artifact is renamed, mutated, or deleted by an agent.

Two sources ship together in v0:

1. Chrome hint mode — keyboard hotkey on amem Clipper draws vimium-style hints over <figure>, <img>, <table>, headings, and selectable blocks; user types the hint; region is captured + assigned amem://<uuid>.
2. Sioyek annotation watcher — amem-sh tails Sioyek’s local SQLite db; every keyboard rectangle-mark (r mode) the user makes auto-becomes an amem://<uuid> capture without changing Sioyek’s UX.

Unified output: every captured region surfaces in the user’s session with a short alias like @fig-A that paste-able into agent prompts. The agent reads the region image + metadata via the existing amem_recall MCP tool.

Motivation — the fig 4 → fig 2 problem

A scenario the operator hits weekly:

User:  "Claude, change fig 4's color to blue."
Claude: <does the modification, but renames it to fig 2 in the process>
User:  "fig 4 is now where? was it deleted?"
Claude: <confused about identifier history>
User:  Spends 5 minutes verbally re-anchoring "the figure showing X-axis Y"
       …or gives up and re-screenshots.

The bug isn’t the agent. The bug is that fig 4 is a name binding, not an identity reference. As soon as the agent mutates the underlying artifact, the binding breaks but the user’s mental model still points at “that thing.”

The solved-problem analogues live elsewhere in software:

• Git’s commit hash (content-addressable; renaming the branch doesn’t invalidate the commit)
• Notion’s per-block immutable IDs
• Roam’s ((block-ref))
• amem’s existing amem://<uuid> (already content-addressable for whole-document captures)

This RFC extends amem:// from document-grade addressing to region-grade addressing — and gives the user a keyboard-only path to mint such IDs from anywhere they read.

The killer secondary effect: every amem:// minted this way is automatically a citable unit for RFC-003 fact-check. Grounding gets concrete targets (“amem://b3f29... shows X”) rather than vague text recalls.

Proposal

1. amem:// URI extended for regions

Existing scheme:

amem://<item-id>                  # whole captured document
amem://<item-id>#chunk=<n>        # specific text chunk

Add region addressing:

amem://<region-id>                # always a 16-char base32 of UUID
                                  # mints a NEW item, not a sub-ref of an
                                  # existing one — content-addressable, so a
                                  # region you saved last week resolves to
                                  # the same id even if the source is gone

A region is a first-class amem item with these required fields:

# example: ~/.amem/raw/regions/b3f29x7q4t8m6w2n.toml
uri        = "amem://b3f29x7q4t8m6w2n"
short      = "fig-A"             # session-scoped alias, may collide across sessions
source_app = "amem-clipper"      # or "sioyek", "macos-ax", ...
captured_at = 2026-05-06T11:23:08Z

# What was captured
image_path  = "raw/regions/b3f29x7q4t8m6w2n.png"
ocr_text    = "Figure 4: Self-attention scores across heads…"

# Where it came from (best-effort, may be partial for some sources)
source_url  = "https://arxiv.org/pdf/1706.03762"
source_doc  = "Attention Is All You Need (Vaswani 2017)"
source_page = 7
source_bbox = [120, 340, 480, 620]    # x, y, w, h in source coordinates
source_dom  = "main > article > figure:nth-of-type(4)"  # if web

# Optional, for grounding
parent_item = "amem://efb1...."  # if region was extracted from an existing
                                  # whole-doc capture

The point is: the URI alone is enough to recover the image + context even after the source is offline / mutated / removed. amem caches the image bytes locally; the source metadata is provenance, not the source of truth.

2. New MCP tool: amem_capture_region

amem_capture_region(target?: TargetSpec, mode?: "hint" | "wait")
  -> { uri: "amem://...", short: "fig-A", image_path, source_meta }

TargetSpec :=
  | { app: "chrome", tab: "active" | <tabId> }
  | { app: "sioyek", current_doc: true }
  | (omitted)            // amem decides based on which source is "live"

Behaviour by mode:

• mode: "hint" (default for Chrome) — agent calls the tool; bridge sends enter_hint_mode to amem Clipper; user types a hint; the call blocks until the user picks (or 60s timeout aborts).
• mode: "wait" (default for Sioyek) — agent calls the tool; amem waits for the next new entry in Sioyek’s db (or 60s timeout); returns the region that landed.

Both are blocking from MCP’s perspective — clients (Claude Code, Cursor, etc.) already render “tool running” UI for long-running calls. The user isn’t surprised; they expect the tool to wait for their selection.

3. Source 1 — Chrome hint mode (amem-clipper)

Keyboard flow:

User in Chrome: presses ⌥G  (or invoked via amem_capture_region MCP call)
       ↓
Content script enumerates candidate elements:
  • <figure>, <img>, <table>, <video>, <pre>, <code>
  • <h1>–<h4>
  • Block elements with computed area > 8000 px²
  • PDF.js text-layer divs (when PDF rendered in Chrome)
       ↓
Each element gets a 2-letter hint label: aa, ab, ac, …
       ↓
Overlay rendered at element's top-left corner with high z-index
       ↓
User types: "ac"
       ↓
Content script captures the region:
  • Screenshot via chrome.tabs.captureVisibleTab + crop to bbox
  • Plus: outerHTML of the element (DOM range archive)
  • Plus: element's computed CSS rect, page URL, page title
       ↓
Posts to bridge: { action: "region_captured", payload: <region-record> }
       ↓
amem-sh receives, mints amem://<uuid>, stores PNG + metadata, returns short alias

Hint allocation: depth-first DOM traversal, skipping invisible elements (zero size or display:none). Two-letter hints support up to 676 elements per page; in practice ~50–100 visible candidates is plenty.

Activation: extension’s existing capture button gets a long-press / shift modifier for “hint mode,” or a new dedicated icon. The hotkey ⌥G is a reserved keyboard shortcut declared in manifest.json.

4. Source 2 — Sioyek annotation watcher (amem-sh)

Sioyek already provides keyboard-driven rectangle selection via the r command — and it stores the result in a stable SQLite database. This RFC does not require changes to Sioyek; we just listen.

~/.config/sioyek/local.db       ← Sioyek writes (highlights, bookmarks, marks)
       │
       ▼ fsnotify
amem-sh sioyek-watcher loop:
  every change event ↓
       ▼
  SELECT * FROM highlights
  WHERE creation_time > last_seen_creation_time
       ▼
  for each new row:
    • read (begin_x, begin_y, end_x, end_y, page, document_path, type, creation_time)
    • render that bbox of <document_path>:<page> via mupdf/pdfium  → PNG
    • OCR the rendered region via Vision/tesseract → text
    • mint amem://<uuid>, store under raw/regions/
    • update last_seen_creation_time
  end
       ▼
  emit MCP notification (if any client subscribed) so the agent can refresh

Trade-offs:

• ✅ Zero Sioyek modification, zero plugin maintenance
• ✅ Keyboard-only (Sioyek’s r mode already is)
• ✅ Cross-platform (SQLite is portable; mupdf works on Mac/Linux)
• ⚠️ Couples to Sioyek’s db schema. We pin ~/.config/sioyek/local.db at SQLite version we tested with; on Sioyek upgrade we re-validate.
• ⚠️ User has to be using Sioyek for this source to fire — non-Sioyek PDF users go via Chrome PDF viewer (see RFC-005 §3 / “PDF in Chrome” path)

5. Short alias system (@fig-A)

The full URI amem://b3f29x7q4t8m6w2n is unwieldy in chat. amem maintains a per-session alias table:

~/.amem/state/aliases.json
{
  "session_id": "2026-05-06-am",
  "started_at": "...",
  "aliases": {
    "fig-A": "amem://b3f29x7q4t8m6w2n",
    "fig-B": "amem://4ek38u2t1y9q5w0v",
    "tbl-A": "amem://nz0p7qm6r3s4d2f8"
  }
}

Alias scheme:

• Prefix indicates type: fig- (figure / image), tbl- (table), txt- (text block), pg- (whole page screenshot), dom- (DOM range)
• Suffix is alphabetic, in capture order this session
• Resets at session start (user can amem alias persist to keep them)

CLI surface:

amem aliases                    # list current session aliases
amem alias persist               # promote session aliases to permanent
amem alias resolve @fig-A        # → amem://b3f29x7q4t8m6w2n
amem alias forget @fig-A         # remove from current session

In agent conversations, the amem_recall tool already resolves amem:// URIs. Aliases get resolved client-side by amem-sh before the MCP call: the user pastes @fig-A, the agent’s amem_recall sees the full URI.

6. Normalized capture record (cross-source schema)

All sources produce records of the same shape, regardless of provenance:

#![allow(unused)]
fn main() {
struct RegionCapture {
    uri:           String,         // "amem://<uuid>"
    short_alias:   Option<String>, // "fig-A"
    captured_at:   DateTime,

    // Always present
    image_path:    PathBuf,        // PNG, full-resolution
    image_bytes:   u64,            // for budget tracking

    // Optional but encouraged
    ocr_text:      Option<String>,
    image_caption: Option<String>, // alt text, figure caption, etc.

    // Source provenance (one of these blocks present)
    source: SourceProvenance,
}

enum SourceProvenance {
    Web {
        url:        String,
        title:      String,
        dom_path:   String,
        bbox_css:   [f32; 4],
        outer_html: Option<String>,  // archived for posterity
    },
    Sioyek {
        document_path: String,
        document_hash: String,       // for tracking renames / moves
        page:          u32,
        bbox_pdf:      [f32; 4],     // PDF coordinates
        document_title: Option<String>,
    },
    MacOSAX {                         // RFC-007, placeholder
        bundle_id:     String,
        window_title:  String,
        bbox_screen:   [f32; 4],
    },
}
}

Every consumer (amem_recall, the side panel, amem cite) handles a RegionCapture regardless of source. New sources just add new SourceProvenance variants.

Privacy

• All captures land only in ~/.amem/raw/regions/ on the user’s machine. Never uploaded.
• Source provenance metadata is verbose by design (DOM path, PDF coordinates) so the user can audit. Verbosity stays local.
• The Sioyek watcher reads ~/.config/sioyek/local.db only; it never writes to it. Sioyek’s own data integrity is unaffected.
• amem Clipper’s hint mode uses the same chrome.tabs.captureVisibleTab permission already declared. No new permissions.
• Region OCR runs locally (Vision on macOS, tesseract on Linux). No cloud OCR.

Failure modes

Concrete work

In rough order of dependency:

1. (amem-sh) Extend amem:// URI parser to accept region IDs (compat with existing item IDs — same UUID space)
2. (amem-sh) Add RegionCapture data model + storage layout under raw/regions/
3. (amem-sh) Add amem_capture_region MCP tool (blocking, mode parameterised)
4. (amem-sh) Add amem alias CLI subcommands + session alias state
5. (amem-bridge) Add enter_hint_mode + region_captured verbs to bridge protocol
6. (amem-clipper) Add hint overlay content script (~3 days; see §3)
7. (amem-clipper) Wire ⌥G hotkey + side-panel “hint mode” toggle
8. (amem-sh) Add Sioyek watcher (~/.config/sioyek/local.db poller + PDF region renderer via mupdf)
9. (amem-sh) OCR pipeline (Vision + tesseract fallback) for region images
10. (docs.amem.sh) User guide page: “Capturing regions for AI references”

Estimated total: two weeks of focused work, parallelisable across amem-sh / amem-bridge / amem-clipper. Sioyek source ships in week 1 (simpler — db reads only); Chrome hint mode ships in week 2.

Rejected alternatives

• “Write a Sioyek Lua plugin to bind a hotkey to amem capture” — duplicates Sioyek’s built-in r mode, requires per-Sioyek-version maintenance, and breaks if Sioyek loses Lua support. The fs-watch approach decouples completely.
• “Mouse-driven rectangle selector” — operator’s stated preference is hand-stays-on-keyboard. Mouse mode could be a v2 nice-to-have but isn’t the design center.
• “Just screenshot and let the agent OCR/describe” — loses stable identity. Two screenshots of the same figure get different IDs; agent can’t track “the same thing” across calls.
• “Push everything through Chrome only (drop Sioyek)” — operator uses Sioyek for daily PDF reading. Forcing them into Chrome is a UX regression for the work Sioyek is good at.
• “Build native macOS overlay with Accessibility API now” — would cover all native apps generically, but is multi-week macOS work and is unjustified before Chrome + Sioyek prove the model. Deferred to RFC-007.

Open questions

• Alias naming policy — should aliases be fig-A style (semantic prefix + letter), or pure @a1, @a2, …? Soft preference: semantic prefix; users immediately know @fig- ≠ @tbl-. But it requires classification at capture time (heuristic on element tag / PDF caption detection).
• Cross-session alias persistence — should aliases auto-persist if the user uses them in a chat that the agent also persists into amem (closing the loop)? Probably yes; soft preference: aliases referenced in any captured conversation get auto-persisted.
• Multi-monitor / multi-window — when the user has two Chrome windows on different monitors, which one gets the hint overlay? Soft preference: only the focused window. If the agent calls amem_capture_region(target={app: chrome, tab: ...}) with no tab specified, route to the active window’s active tab.
• PDF.js inside Chrome — should the hint mode treat PDF text-layer divs as targets (currently yes per §3), or should it route to the Sioyek source instead? The user’s choice of viewer is the answer; if the PDF is in Chrome, hint mode handles it.
• Should amem_factcheck (RFC-003) accept a region URI as the claim context? Soft preference: yes — amem_factcheck(claim, context_uri: "amem://<region>") makes grounding richer.

Roll-out

• Week 1 — RFC-005 implementation kickoff. Sioyek watcher first (cleanest path, no extension changes). Ships behind feature flag features.sioyek_capture = true.
• Week 2 — amem Clipper hint mode lands. Side panel shows live capture log. Ships behind features.region_capture = true until stable.
• Week 3 — wire amem_factcheck to accept region URIs (RFC-003 Phase B integration). Now grounding loop is end-to-end.
• Post-CWS — promote both flags from beta to default. Update docs.amem.sh/clipper and add a new docs.amem.sh/sioyek page.
• Future — RFC-007 macOS Accessibility hint mode, when first user asks “I want this for Sketch / Figma desktop / Adobe.”