add keyboard tracking and app-qualified smart titles
**Keyboard hook (Windows)**
- Extends the existing C# WH_MOUSE_LL process to also install
WH_KEYBOARD_LL alongside it (keyboard hook is optional — failure
does not break mouse capture).
- Emits CHAR <code> <ts> for printable keystrokes, KEY <name> <ts>
for modifier combos (Ctrl+T) and special keys (Backspace, Enter).
**Text accumulation in capture session**
- CaptureService tracks _keyBuffer (typed chars since last step) and
_lastShortcut (last modifier combo) using the new onKeyboardEvent()
method.
- snapshotKeyContext() is called at enqueueClickCapture time so each
step's clickMeta.keyContext carries { recentTyped, recentShortcut }.
- Buffer resets after each snapshot; stale input (>8s gap) is dropped.
**UIAutomation element value**
- collectWindowsWindowContext now reads ValuePattern.Current.Value
from the clicked element — giving us what's actually typed in a
search box or text field without needing the keyboard buffer.
**Smart title generation (core/text-intel.js)**
- Priority chain: keyboard shortcut → element value → typed text
→ OCR → element label → page title → app name.
- SHORTCUT_TITLES maps 50+ common shortcuts (Ctrl+T, Ctrl+S, F5 …)
to natural language descriptions: "Open new tab", "Save", etc.
- qualifyTitleWithApp() appends "in Chrome / VS Code / Terminal / …"
to every title when the app is known: "Click Save in VS Code",
"Search for oracle in Chrome", "Open new tab in Chrome".
- APP_DISPLAY_NAMES covers browsers, editors, terminals, office apps.
Six new unit tests cover shortcuts, typed-text search, element value,
and app-qualified OCR titles. Capture test updated for keyContext.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
@@ -375,7 +375,7 @@ test('queued click captures preserve the original event time and button', async
|
||||
assert.deepEqual(seen, [{
|
||||
trigger: 'click',
|
||||
clickPos: { x: 7, y: 8 },
|
||||
clickMeta: { at: 1770000000456, button: 'left' },
|
||||
clickMeta: { at: 1770000000456, button: 'left', keyContext: { recentTyped: '', recentShortcut: '' } },
|
||||
}]);
|
||||
});
|
||||
|
||||
|
||||
@@ -61,7 +61,8 @@ test('capture titles ignore browser chrome noise in favor of OCR', () => {
|
||||
},
|
||||
ocrText: 'New tab',
|
||||
});
|
||||
assert.equal(title, 'Click New tab');
|
||||
// OCR wins over the noisy window title; app name is appended for context.
|
||||
assert.equal(title, 'Click New tab in Google Chrome');
|
||||
});
|
||||
|
||||
test('tab-like roles use select when OCR identifies a tab label', () => {
|
||||
@@ -110,7 +111,7 @@ test('search query is extracted from browser window title pattern', () => {
|
||||
},
|
||||
ocrText: '',
|
||||
});
|
||||
assert.equal(title, 'Search for Oracle');
|
||||
assert.equal(title, 'Search for Oracle in Chrome');
|
||||
});
|
||||
|
||||
test('full link text with pipe separator is preserved in OCR phrases', () => {
|
||||
@@ -140,6 +141,56 @@ test('search box element role uses Search for verb', () => {
|
||||
assert.equal(title, 'Search for Oracle');
|
||||
});
|
||||
|
||||
test('keyboard shortcut produces action title qualified with app', () => {
|
||||
const title = buildCaptureTitle({
|
||||
mode: 'fullscreen',
|
||||
metadata: { appName: 'chrome' },
|
||||
ocrText: '',
|
||||
recentShortcut: 'Ctrl+T',
|
||||
});
|
||||
assert.equal(title, 'Open new tab in Chrome');
|
||||
});
|
||||
|
||||
test('keyboard shortcut title without app name', () => {
|
||||
const title = buildCaptureTitle({
|
||||
mode: 'fullscreen',
|
||||
metadata: {},
|
||||
ocrText: '',
|
||||
recentShortcut: 'Ctrl+S',
|
||||
});
|
||||
assert.equal(title, 'Save');
|
||||
});
|
||||
|
||||
test('typed text with search input role produces Search for title', () => {
|
||||
const title = buildCaptureTitle({
|
||||
mode: 'fullscreen',
|
||||
metadata: { elementRole: 'search box', appName: 'chrome' },
|
||||
ocrText: '',
|
||||
recentTyped: 'oracle',
|
||||
});
|
||||
assert.equal(title, 'Search for "oracle" in Chrome');
|
||||
});
|
||||
|
||||
test('UIAutomation element value takes priority over keyboard buffer', () => {
|
||||
const title = buildCaptureTitle({
|
||||
mode: 'fullscreen',
|
||||
metadata: { elementRole: 'edit', elementValue: 'oracle', appName: 'chrome' },
|
||||
ocrText: '',
|
||||
recentTyped: 'ignored',
|
||||
});
|
||||
// elementValue (from UIAutomation) wins over the keyboard buffer
|
||||
assert.ok(title.includes('oracle'), `expected oracle in title, got: ${title}`);
|
||||
});
|
||||
|
||||
test('app-qualified OCR title includes app name', () => {
|
||||
const title = buildCaptureTitle({
|
||||
mode: 'fullscreen',
|
||||
metadata: { appName: 'code' },
|
||||
ocrText: 'Save',
|
||||
});
|
||||
assert.equal(title, 'Click Save in VS Code');
|
||||
});
|
||||
|
||||
test('ai prompts include the deterministic OCR-backed title candidate', () => {
|
||||
const { prompt } = buildAiPrompt({
|
||||
captureContext: {
|
||||
|
||||
Reference in New Issue
Block a user