fix step-behind titles: UIAutomation element label from click watcher + smarter search fallback
**Root cause**
When OCR fails, the title fell back to the browser window title. For a
click on a search-results page, the window title reflects the *previous*
search query ("oracle - Google Search"), producing "Search for Oracle"
even though the user is clicking a link *on* that page.
**Fix 1: UIAutomation element label from the click watcher**
The C# click-watcher hook now enriches each click in a background thread
(ClickProcessorLoop) rather than in the hook callback:
- MouseHookCallback captures window title synchronously (fast Win32),
then queues a PendingClick and returns immediately.
- ClickProcessorLoop calls AutomationElement.FromPoint() via reflection
(no compile-time assembly reference → no startup failure if UIA is
absent). Wrapped in a 300ms timeout thread so slow UIA calls don't
delay the click event past the frame buffer window.
- Emits CTX + ELEM (label/role/value) + CLICK as an atomic batch.
Node.js:
- Parses ELEM events, merges element info into _pendingWindowContext.
- clickMeta.windowContext now carries elementLabel/elementRole/elementValue
in addition to windowTitle/appName.
- buildCaptureTitle priority-5 (element label) now fires from click-watcher
data, giving "Select Oracle | Cloud Applications…" without OCR.
**Fix 2: Wider OCR crop**
ocrAroundClick now uses a full-display-width × 100px horizontal strip at
the click height. The previous 420 px crop cropped through long link text
(e.g. "Oracle | Cloud Applications and Cloud Platform"), causing fragments
to be scored lower than the complete text.
**Fix 3: Search-results window title fallback**
extractSearchQuery now only produces "Search for Oracle" when recentTyped
is non-empty (the user was actually typing a query). For a pure click on
the search-results page (no recent typing), the fallback is "Select a
Oracle result in Chrome" — honest about what we know without implying the
user performed the search in this step.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
@@ -102,18 +102,26 @@ test('browser window title strips browser name and falls back to page title', ()
|
||||
assert.ok(title.toLowerCase().includes('oracle') || title.toLowerCase().includes('cloud'), `Expected oracle/cloud in title, got: ${title}`);
|
||||
});
|
||||
|
||||
test('search query is extracted from browser window title pattern', () => {
|
||||
test('search query is extracted when user was typing (search step)', () => {
|
||||
const title = buildCaptureTitle({
|
||||
mode: 'fullscreen',
|
||||
metadata: {
|
||||
windowTitle: 'oracle - Google Search - Google Chrome',
|
||||
appName: 'chrome',
|
||||
},
|
||||
metadata: { windowTitle: 'oracle - Google Search - Google Chrome', appName: 'chrome' },
|
||||
ocrText: '',
|
||||
recentTyped: 'oracle', // user was typing → this IS the search step
|
||||
});
|
||||
assert.equal(title, 'Search for Oracle in Chrome');
|
||||
});
|
||||
|
||||
test('search results window title produces select-result title when no typing (click on results page)', () => {
|
||||
const title = buildCaptureTitle({
|
||||
mode: 'fullscreen',
|
||||
metadata: { windowTitle: 'oracle - Google Search - Google Chrome', appName: 'chrome' },
|
||||
ocrText: '',
|
||||
recentTyped: '', // no recent typing → user is clicking a result, not searching
|
||||
});
|
||||
assert.equal(title, 'Select a Oracle result in Chrome');
|
||||
});
|
||||
|
||||
test('full link text with pipe separator is preserved in OCR phrases', () => {
|
||||
const title = buildCaptureTitle({
|
||||
mode: 'fullscreen',
|
||||
|
||||
Reference in New Issue
Block a user