fix step-behind titles: UIAutomation element label from click watcher + smarter search fallback

**Root cause**
When OCR fails, the title fell back to the browser window title.  For a
click on a search-results page, the window title reflects the *previous*
search query ("oracle - Google Search"), producing "Search for Oracle"
even though the user is clicking a link *on* that page.

**Fix 1: UIAutomation element label from the click watcher**
The C# click-watcher hook now enriches each click in a background thread
(ClickProcessorLoop) rather than in the hook callback:
- MouseHookCallback captures window title synchronously (fast Win32),
  then queues a PendingClick and returns immediately.
- ClickProcessorLoop calls AutomationElement.FromPoint() via reflection
  (no compile-time assembly reference → no startup failure if UIA is
  absent).  Wrapped in a 300ms timeout thread so slow UIA calls don't
  delay the click event past the frame buffer window.
- Emits CTX + ELEM (label/role/value) + CLICK as an atomic batch.

Node.js:
- Parses ELEM events, merges element info into _pendingWindowContext.
- clickMeta.windowContext now carries elementLabel/elementRole/elementValue
  in addition to windowTitle/appName.
- buildCaptureTitle priority-5 (element label) now fires from click-watcher
  data, giving "Select Oracle | Cloud Applications…" without OCR.

**Fix 2: Wider OCR crop**
ocrAroundClick now uses a full-display-width × 100px horizontal strip at
the click height.  The previous 420 px crop cropped through long link text
(e.g. "Oracle | Cloud Applications and Cloud Platform"), causing fragments
to be scored lower than the complete text.

**Fix 3: Search-results window title fallback**
extractSearchQuery now only produces "Search for Oracle" when recentTyped
is non-empty (the user was actually typing a query).  For a pure click on
the search-results page (no recent typing), the fallback is "Select a
Oracle result in Chrome" — honest about what we know without implying the
user performed the search in this step.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
2026-06-24 11:05:34 -05:00
co-authored by Claude Sonnet 4.6
parent dfa65c894b
commit d0b1816175
4 changed files with 154 additions and 21 deletions
+9 -1
View File
@@ -500,7 +500,15 @@ function buildCaptureTitle({ mode = 'fullscreen', metadata = {}, ocrText = '', r
if (strippedWindowTitle) {
const searchQuery = extractSearchQuery(strippedWindowTitle);
if (searchQuery) {
const base = `Search for ${sentenceCase(searchQuery)}`;
// Only claim this step IS the search action when the user was actually typing
// (recentTyped). Without typing context, the search page title is from the
// PREVIOUS step — the current step is a click ON the search results page.
if (recentTyped) {
const base = `Search for ${sentenceCase(searchQuery)}`;
return app ? qualifyTitleWithApp(base, metadata.appName) : base;
}
// User is clicking something on the search results page — don't claim they searched.
const base = `Select a ${sentenceCase(searchQuery)} result`;
return app ? qualifyTitleWithApp(base, metadata.appName) : base;
}
const windowPhrase = pickBestTitleFragment(strippedWindowTitle, { source: 'window', metadata });