**Root cause**
When OCR fails, the title fell back to the browser window title. For a
click on a search-results page, the window title reflects the *previous*
search query ("oracle - Google Search"), producing "Search for Oracle"
even though the user is clicking a link *on* that page.
**Fix 1: UIAutomation element label from the click watcher**
The C# click-watcher hook now enriches each click in a background thread
(ClickProcessorLoop) rather than in the hook callback:
- MouseHookCallback captures window title synchronously (fast Win32),
then queues a PendingClick and returns immediately.
- ClickProcessorLoop calls AutomationElement.FromPoint() via reflection
(no compile-time assembly reference → no startup failure if UIA is
absent). Wrapped in a 300ms timeout thread so slow UIA calls don't
delay the click event past the frame buffer window.
- Emits CTX + ELEM (label/role/value) + CLICK as an atomic batch.
Node.js:
- Parses ELEM events, merges element info into _pendingWindowContext.
- clickMeta.windowContext now carries elementLabel/elementRole/elementValue
in addition to windowTitle/appName.
- buildCaptureTitle priority-5 (element label) now fires from click-watcher
data, giving "Select Oracle | Cloud Applications…" without OCR.
**Fix 2: Wider OCR crop**
ocrAroundClick now uses a full-display-width × 100px horizontal strip at
the click height. The previous 420 px crop cropped through long link text
(e.g. "Oracle | Cloud Applications and Cloud Platform"), causing fragments
to be scored lower than the complete text.
**Fix 3: Search-results window title fallback**
extractSearchQuery now only produces "Search for Oracle" when recentTyped
is non-empty (the user was actually typing a query). For a pure click on
the search-results page (no recent typing), the fallback is "Select a
Oracle result in Chrome" — honest about what we know without implying the
user performed the search in this step.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
**Keyboard hook (Windows)**
- Extends the existing C# WH_MOUSE_LL process to also install
WH_KEYBOARD_LL alongside it (keyboard hook is optional — failure
does not break mouse capture).
- Emits CHAR <code> <ts> for printable keystrokes, KEY <name> <ts>
for modifier combos (Ctrl+T) and special keys (Backspace, Enter).
**Text accumulation in capture session**
- CaptureService tracks _keyBuffer (typed chars since last step) and
_lastShortcut (last modifier combo) using the new onKeyboardEvent()
method.
- snapshotKeyContext() is called at enqueueClickCapture time so each
step's clickMeta.keyContext carries { recentTyped, recentShortcut }.
- Buffer resets after each snapshot; stale input (>8s gap) is dropped.
**UIAutomation element value**
- collectWindowsWindowContext now reads ValuePattern.Current.Value
from the clicked element — giving us what's actually typed in a
search box or text field without needing the keyboard buffer.
**Smart title generation (core/text-intel.js)**
- Priority chain: keyboard shortcut → element value → typed text
→ OCR → element label → page title → app name.
- SHORTCUT_TITLES maps 50+ common shortcuts (Ctrl+T, Ctrl+S, F5 …)
to natural language descriptions: "Open new tab", "Save", etc.
- qualifyTitleWithApp() appends "in Chrome / VS Code / Terminal / …"
to every title when the app is known: "Click Save in VS Code",
"Search for oracle in Chrome", "Open new tab in Chrome".
- APP_DISPLAY_NAMES covers browsers, editors, terminals, office apps.
Six new unit tests cover shortcuts, typed-text search, element value,
and app-qualified OCR titles. Capture test updated for keyContext.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The title engine was falling back to "Screen capture" for all browser
captures because it treated the entire browser window title as noise.
Changes:
- stripBrowserNameSuffix: removes "- Google Chrome" / "| Firefox" etc.
from the end of window titles, leaving just the page title.
"oracle - Google Search - Google Chrome" → "oracle - Google Search"
"Oracle | Cloud Applications - Google Chrome" → page title only
- extractSearchQuery: detects "[query] - Google Search" / Bing / etc.
patterns after stripping the browser suffix, and formats the result
as "Search for oracle".
- buildCaptureTitle: uses stripped page title + search detection before
falling through to the "Screen capture" fallback.
- pickBestOcrPhrase: considers the full OCR line (≤80 chars) as a
candidate with a +35 completeness bonus before splitting on | or ·.
This preserves "Oracle | Cloud Applications and Cloud Platform" as a
single phrase instead of breaking it into fragments.
- candidateWords: filters out standalone punctuation tokens (|, ·, •)
so they don't inflate word-count penalties for compound brand names.
- verbForElementRole: hyperlinks and links now produce "Select" instead
of "Click"; search box / search field produces "Search for".
Five new unit tests cover: browser title stripping, search query
extraction, full pipe-separated link text, link and search box verbs.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Store captureMetadata (OCR text, window/app/element info) with each
step at capture time so AI always has the original rich context.
- Add buildCaptureContext() to TextIntelService; capture.js uses it
instead of buildCaptureTitle() so both title and metadata come from
one pass.
- generateStepPatch() prefers stored captureMetadata over re-running
OCR, giving the AI the best possible context when the user clicks
an AI button later.
- Add autoDoc setting: when enabled every capture (shoot, region, and
session hotkey/click) is automatically documented by AI. Manual
captures await AI before returning; session captures fire-and-forget
and push a step:updated event so the renderer reloads seamlessly.
- Add ai:rewriteText IPC and rewriteText() method for plain-text
polishing via a separate callOllamaText() that skips JSON mode.
- Add "AI Rewrite" section in the editor right panel: textarea + AI
button that rewrites whatever the user types in place.
- Improve buildAiPrompt() rules: action-focused title instructions,
explicit anti-junk rules (no "Capture the screen / OCR" blocks),
and a context-quality gate that suppresses blocks when context is
thin.
- Add autoDoc checkbox to AI settings dialog.
- Renderer handles step:updated to reload the selected step after
background auto-doc finishes.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Previously the "Contents" page just emitted a TOC field whose cached
result was the literal text "Update contents in Word" - so unless a
user manually ran Update Field in Word, the TOC showed nothing useful
(and many viewers never run that update at all).
Now each step heading is wrapped in a bookmark, and the TOC is built
as real TOC1/2/3-styled paragraphs with hyperlinks to those bookmarks
and PAGEREF fields for page numbers, matching the entry list other
exporters already produce via tocEntries(). The whole thing stays
wrapped in the original `TOC \o "1-3" \h \z \u` field so Word can still
refresh page numbers, but the document is correct on first open even
without that step.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Adds PdfBuilder.linkRect() for /Subtype /Link annotations with a
/Dest pointing at another page's destination. TOC entries record a
target placeholder that the per-step loop fills in once it knows
which page the step landed on, so clicking a Contents line jumps the
reader straight to that step.
Lets users record author/co-authors/organization for a guide via a new
"Guide information…" dialog; this metadata renders below the title on
the PDF cover (title now sits above the accent rule). PDF export also
paginates so each step fits its own page where possible, keeps a
step's title/image/lead-in together, and forces the next step onto a
fresh page after an oversized step overflows. Exports now run in a
forked helper process so large guides no longer freeze the UI.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- The canvas Undo/Redo buttons now also undo/redo step deletion
(single or multi-select), restoring the step's data, images, and
position in the order. Backed by a new step:restore IPC call and
GuideStore.restoreStep().
- Remove "Finish capture session" from the capture menu. The
top-right recording bar in the guide editor is now the only place
to start/stop recording, with its toggle relabeled
"Start recording" / "Stop recording".
Markdown export now emits > [!NOTE]/[!TIP]/[!WARNING]/[!IMPORTANT] for
text-block callouts, giving them the same colored/icon-labeled treatment
as the PDF/HTML/DOCX exports on renderers that support GFM alerts
(GitHub, Azure DevOps wikis, etc.), while degrading gracefully to a
plain blockquote elsewhere.
Root cause: warm-before-hide kept the window visible during backend warmup,
and on a restart that warmup could take several seconds (the stream backend
start waits up to 8s). During that visible window, clicks over the app were
skipped by the userIsInApp guard and clicks elsewhere were shot post-click,
so a restarted session looked like it stopped after one click.
- Recording is now 'armed' only after the window is hidden and the buffer is
primed. A new warmingUp flag makes onOsClick ignore clicks during warmup
(the window is covering the user's work anyway) instead of mishandling
them. Cleared on pause/finish.
- armRecording caps the warmup wait (WARMUP_MAX_MS=1500): the window hides
and the session arms even if the backend start hangs, so it can never sit
visible for seconds dropping clicks. The backend keeps coming up in the
background; the first click or two may take the fresh-shot fallback.
- A generation token invalidates an in-flight backend start whose session
has since finished, so a slow start can't install into a new session or
leave the starting-guard stuck and block the restart from starting one.
Tests: 4 new behavioral capture tests (warmup ignores clicks; pause/finish
clear it; armRecording warms-then-hides-then-arms; a hung start still arms
within the cap; a stale start is discarded and frees the guard) plus a new
end-to-end self-test scenario (warmup click ignored, first armed click
captured). 152 unit tests + all repo checks pass.
Co-Authored-By: Claude Fable 5 <[email protected]>
Per request: clicks of the same button closer together than
capture.clickDebounceMs (default 200ms) now collapse into a single step, so
accidental fast/double clicks don't each become a step. It is a leading-edge
debounce measured from the last *accepted* click, so a run of fast clicks
can't push the next deliberate click out — two clicks spaced beyond the
window (e.g. the reported 400-500ms apart) always register.
Replaces the prior 8ms duplicate-delivery suppression (subsumed by the
window). Configurable; 0 captures every click.
Tests (the point of this change is that it can't silently regress):
- 13 behavioral unit tests in capture.test.js that drive real onOsClick
calls with controlled timestamps and assert which clicks survive — the
reported 400/450/500ms cases, sub-window collapse, the 200ms boundary,
per-button independence, configurability, debounce=0, last-accepted (not
last-dropped) reference, session reset, and a full onOsClick -> queue ->
store integration check. No keyword/comment assertions.
- A fourth end-to-end self-test scenario (burst of 40ms clicks collapses to
1; three 300ms-apart clicks each register => 4 total). The marker/drain
scenarios set debounce to 0 so they keep stressing the frame pipeline.
147 unit tests + all repo checks pass.
Co-Authored-By: Claude Fable 5 <[email protected]>
The remaining 'captured slightly after the click' reports came from the
fresh-shot fallback, which grabs the screen when the click is processed
(after it). The previous lead change made that fallback *more* likely: a
frame now had to be >=120ms before the click to qualify, so on machines
where the capture stream can't always keep a frame that old buffered, more
clicks fell through to the post-click shot.
Make the click-lead a two-tier preference instead of a hard gate in
selectFrameForClick:
1. newest frame captured at least leadMs before the click (ideal margin), else
2. newest frame captured before the click at all.
Only when no pre-click frame exists does the caller fresh-shot. leadMs is
threaded through the stream backend to the worker so both selection paths
agree. Verified end to end: frames land ~120-170ms before each click,
markers stay at 0.00%, and the 8-click burst still saves all 8.
Co-Authored-By: Claude Fable 5 <[email protected]>
Root cause of 'I clicked many times but only got two screenshots':
finishing/pausing a session called backend.stop(), which cancelled every
in-flight frame request to null. Clicks whose PNG had not finished
encoding yet were then dropped — only the first few survived.
Fixes:
- Stream backend now *drains* on stop: it stops accepting new requests but
keeps the worker alive until frames already selected for queued clicks
finish encoding. stop({ immediate: true }) keeps the old abandon-now
behavior for an unhealthy worker.
- Two-stage worker reply: a fast 'frame-selected' ack pins the pairing and
proves liveness; the slow PNG payload follows. A slow encode (seconds on
software-rendered hosts) is no longer mistaken for a dead worker, which
had been forcing the post-click fresh-shot fallback (late screenshots).
- Queued clicks carry their guide id and are stored even if the session
ends while they wait in the queue.
- The tray gesture that stops a session is discarded by matching its
recorded screen position, not a time window — a fast workflow click near
the stop is no longer collateral damage. (Replaces the earlier grace
window, which dropped whole bursts.)
- A click on a display with no ready stream resolves null so the caller
fresh-shots the correct monitor instead of returning another screen.
- STEPFORGE_CAPTURE_LOG=1 prints one line per click decision; the
second-instance handler now surfaces the running window instead of
exiting silently.
- Self-test gains a fast-burst-then-finish scenario (8/8 saved) and the
marker/coordinate checks remain at 0.00% offset.
Tests: 133 unit + all repo checks passing.
Co-Authored-By: Claude Fable 5 <[email protected]>
Implements the architecture change from ai_prompts/prompt3.md:
- New app/click-frames.js: shared timestamped frame ring + strict
click-to-frame pairing (never a frame whose grab started after the
click); legacy slack behavior kept behind capture.strictClickFrames=false.
- New stream capture backend (app/stream-backend.js + hidden worker
window): per-display desktop media streams sampled into ring buffers
and PNG-encoded entirely off the main process, so click delivery is
never starved by capture work. Auto-degrades to the legacy in-process
frame loop when streams cannot start or the worker stops answering.
- Clicks are paired with their frame at event time (eager pairing in
enqueueClickCapture); only the storing is serialized, so slow encodes
cannot skew later clicks in a fast burst.
- Linux watcher: restored event-time root coordinates from
xinput test-xi2 and merge raw/regular twin events structurally.
- Replaced the 40ms time debounce with source-aware duplicate
suppression: fast legitimate clicks are never dropped.
- New app/coords.js: physical-to-DIP conversion with multi-monitor and
scale-factor handling; Windows keeps screenToDipPoint.
- STEPFORGE_CLICK_SELFTEST end-to-end hook: 3/3 clicks become steps via
the stream backend with 0.00% marker offset on this host.
- Tests rewritten/added: strict selection, coords, stream backend,
Linux coordinate parsing, twin merge, burst clicking (126 passing).
Co-Authored-By: Claude Fable 5 <[email protected]>
Tyler
