Phase 1 of the improvement plan (PR 3 of the sequence). The docs claimed
"fully offline"/"never talks to the network," but text-intel makes HTTP
requests to a configurable Ollama host that could be remote, with no timeout,
no cancellation, and no size limit; the Windows hook logged raw keystrokes
into capture metadata that could then be sent to that host.
Privacy — raw keystroke capture:
- New capture.captureTypedText setting, default false. With it off, printable
characters are never buffered in JS and the Windows keyboard hook never even
emits them across the process boundary (the flag is threaded into the C#).
Shortcut/navigation detection (Ctrl+T, Enter, …) is unaffected.
AI network hardening (app/text-intel.js):
- Every Ollama call goes through fetchJson with an AbortController deadline
(ai.timeoutMs, default 60s): a dead endpoint fails fast instead of leaving
UI actions pending forever.
- Cancellation: in-flight requests are tracked and cancelInflight(guideId)
aborts them; new ai:cancel IPC + api.ai.cancel are called when the editor
closes, and shutdown cancels everything.
- Bounded concurrency (2) for AI network work.
- Screenshots are only attached when allowed (ai.attachScreenshots), the model
is vision-capable, and the image is within ai.maxImageBytes — no more
unbounded base64-expanded 4K bodies.
Local-first host policy (core/text-intel.js):
- New isLoopbackHost + validateOllamaHost. By default only a loopback Ollama
endpoint is contacted; a remote host is refused with a clear message unless
ai.allowRemoteHost is explicitly enabled. Blocked hosts are never contacted.
Honest documentation:
- README, package.json, and the welcome screen drop "fully offline"/"never
talks to the network"/"Electron is the only dependency" for an accurate
local-first contract that discloses the optional AI path and the bundled
Tesseract OCR dependency.
- New docs/PRIVACY.md details exactly what is collected locally and the one
outbound (opt-in, loopback-by-default) AI feature.
Tests: loopback/remote host matrix, remote-blocked-without-opt-in (and never
contacted), remote-allowed-with-opt-in, request timeout, explicit cancel vs
timeout, typed-text off-by-default vs opted-in, shortcut detection still works,
and a source guard that the C# CHAR emission stays behind the opt-in. 224 unit
tests pass; startup smoke and workflow E2E pass.
Co-Authored-By: Claude Fable 5 <[email protected]>
The model was producing Text and Code blocks on every 'generate all
fields' action — uninstructed, generic, and never useful for the user.
Blocks are only appropriate when the user explicitly edits an existing
block (target === 'block'). For title/description/all generation:
- Schema collapses to { "title", "description" } — no blocks key
- targetText no longer mentions blocks
- allowedBlockNote is null (omitted from prompt)
- Rule explicitly says "Do NOT add any blocks array"
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
**Root cause**
When OCR fails, the title fell back to the browser window title. For a
click on a search-results page, the window title reflects the *previous*
search query ("oracle - Google Search"), producing "Search for Oracle"
even though the user is clicking a link *on* that page.
**Fix 1: UIAutomation element label from the click watcher**
The C# click-watcher hook now enriches each click in a background thread
(ClickProcessorLoop) rather than in the hook callback:
- MouseHookCallback captures window title synchronously (fast Win32),
then queues a PendingClick and returns immediately.
- ClickProcessorLoop calls AutomationElement.FromPoint() via reflection
(no compile-time assembly reference → no startup failure if UIA is
absent). Wrapped in a 300ms timeout thread so slow UIA calls don't
delay the click event past the frame buffer window.
- Emits CTX + ELEM (label/role/value) + CLICK as an atomic batch.
Node.js:
- Parses ELEM events, merges element info into _pendingWindowContext.
- clickMeta.windowContext now carries elementLabel/elementRole/elementValue
in addition to windowTitle/appName.
- buildCaptureTitle priority-5 (element label) now fires from click-watcher
data, giving "Select Oracle | Cloud Applications…" without OCR.
**Fix 2: Wider OCR crop**
ocrAroundClick now uses a full-display-width × 100px horizontal strip at
the click height. The previous 420 px crop cropped through long link text
(e.g. "Oracle | Cloud Applications and Cloud Platform"), causing fragments
to be scored lower than the complete text.
**Fix 3: Search-results window title fallback**
extractSearchQuery now only produces "Search for Oracle" when recentTyped
is non-empty (the user was actually typing a query). For a pure click on
the search-results page (no recent typing), the fallback is "Select a
Oracle result in Chrome" — honest about what we know without implying the
user performed the search in this step.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
**AI button: dynamic tooltip hints**
- titleAiBtn and descAiBtn now show "Rewrite with AI" when the field
already has user content, "Generate with AI" when empty.
- updateAiButtonHints() fires on every title/description input event
and whenever syncStepFields() runs to keep hints current.
**Stronger rewrite prompt**
- When the user has typed a draft title or description, the prompt now
shows it explicitly: "User's draft title (rewrite this): '...'"
- Rules changed from "improve its wording" to "Your only job is to
polish its grammar and phrasing. Do NOT replace it with something
different." — prevents the model from ignoring the user's text and
generating fresh content from capture context.
- The suggested-title hint is suppressed when a draft title exists so
the model doesn't silently swap the user's text for the auto-title.
**Title quality: generic window title filter**
- GENERIC_WINDOW_TITLES Set filters "New Tab", "Untitled", "Loading"
etc. from the window-title path so they no longer produce titles
like "Open New Tab in Chrome".
**Title quality: app name stripping for non-browser apps**
- stripBrowserNameSuffix now accepts an optional appName; it strips
the app's display name and process name from the window title suffix
using the same pattern as browser names.
- "Document1.docx - Word" with appName "winword" → "Document1.docx".
- buildCaptureTitle passes metadata.appName into the strip call.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
**Keyboard hook (Windows)**
- Extends the existing C# WH_MOUSE_LL process to also install
WH_KEYBOARD_LL alongside it (keyboard hook is optional — failure
does not break mouse capture).
- Emits CHAR <code> <ts> for printable keystrokes, KEY <name> <ts>
for modifier combos (Ctrl+T) and special keys (Backspace, Enter).
**Text accumulation in capture session**
- CaptureService tracks _keyBuffer (typed chars since last step) and
_lastShortcut (last modifier combo) using the new onKeyboardEvent()
method.
- snapshotKeyContext() is called at enqueueClickCapture time so each
step's clickMeta.keyContext carries { recentTyped, recentShortcut }.
- Buffer resets after each snapshot; stale input (>8s gap) is dropped.
**UIAutomation element value**
- collectWindowsWindowContext now reads ValuePattern.Current.Value
from the clicked element — giving us what's actually typed in a
search box or text field without needing the keyboard buffer.
**Smart title generation (core/text-intel.js)**
- Priority chain: keyboard shortcut → element value → typed text
→ OCR → element label → page title → app name.
- SHORTCUT_TITLES maps 50+ common shortcuts (Ctrl+T, Ctrl+S, F5 …)
to natural language descriptions: "Open new tab", "Save", etc.
- qualifyTitleWithApp() appends "in Chrome / VS Code / Terminal / …"
to every title when the app is known: "Click Save in VS Code",
"Search for oracle in Chrome", "Open new tab in Chrome".
- APP_DISPLAY_NAMES covers browsers, editors, terminals, office apps.
Six new unit tests cover shortcuts, typed-text search, element value,
and app-qualified OCR titles. Capture test updated for keyContext.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The title engine was falling back to "Screen capture" for all browser
captures because it treated the entire browser window title as noise.
Changes:
- stripBrowserNameSuffix: removes "- Google Chrome" / "| Firefox" etc.
from the end of window titles, leaving just the page title.
"oracle - Google Search - Google Chrome" → "oracle - Google Search"
"Oracle | Cloud Applications - Google Chrome" → page title only
- extractSearchQuery: detects "[query] - Google Search" / Bing / etc.
patterns after stripping the browser suffix, and formats the result
as "Search for oracle".
- buildCaptureTitle: uses stripped page title + search detection before
falling through to the "Screen capture" fallback.
- pickBestOcrPhrase: considers the full OCR line (≤80 chars) as a
candidate with a +35 completeness bonus before splitting on | or ·.
This preserves "Oracle | Cloud Applications and Cloud Platform" as a
single phrase instead of breaking it into fragments.
- candidateWords: filters out standalone punctuation tokens (|, ·, •)
so they don't inflate word-count penalties for compound brand names.
- verbForElementRole: hyperlinks and links now produce "Select" instead
of "Click"; search box / search field produces "Search for".
Five new unit tests cover: browser title stripping, search query
extraction, full pipe-separated link text, link and search box verbs.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Remove the standalone "AI Rewrite" section from the editor panel.
The existing title and description AI buttons already flush the step
first, so they rewrite whatever the user has typed in those fields.
- Never pass a generic fallback title ("Screen capture", "Window
capture", "Region capture", "Capture") as the AI title candidate or
as step content. It is now treated as "(not set — generate a
specific action title)" so the AI always produces something real.
- hasRichCaptureContext now counts any non-trivial app name or window
title as sufficient context, instead of requiring non-browser noise.
- Prompt rules updated: "NEVER output Screen/Window/Region capture",
separate paths for improving a user draft vs generating from context,
and clearer guidance when context is limited (use app/window name).
- isPlaceholderTitle helper guards summarizeStepForAi so a
default-titled step presents itself as empty to the AI.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Store captureMetadata (OCR text, window/app/element info) with each
step at capture time so AI always has the original rich context.
- Add buildCaptureContext() to TextIntelService; capture.js uses it
instead of buildCaptureTitle() so both title and metadata come from
one pass.
- generateStepPatch() prefers stored captureMetadata over re-running
OCR, giving the AI the best possible context when the user clicks
an AI button later.
- Add autoDoc setting: when enabled every capture (shoot, region, and
session hotkey/click) is automatically documented by AI. Manual
captures await AI before returning; session captures fire-and-forget
and push a step:updated event so the renderer reloads seamlessly.
- Add ai:rewriteText IPC and rewriteText() method for plain-text
polishing via a separate callOllamaText() that skips JSON mode.
- Add "AI Rewrite" section in the editor right panel: textarea + AI
button that rewrites whatever the user types in place.
- Improve buildAiPrompt() rules: action-focused title instructions,
explicit anti-junk rules (no "Capture the screen / OCR" blocks),
and a context-quality gate that suppresses blocks when context is
thin.
- Add autoDoc checkbox to AI settings dialog.
- Renderer handles step:updated to reload the selected step after
background auto-doc finishes.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
TylerandClaude Fable 5