Phase 1 of the improvement plan (PR 2 of the sequence). A remote page could
previously inherit the privileged preload bridge: the main window had no
navigation guard or popup handler, IPC handlers accepted any sender, every
Electron permission was granted to everyone, and shell:openPath accepted an
arbitrary renderer-supplied target.
- New app/security.js (plain-Node testable) centralizes the policy:
app-page identity, navigation/popup denial, deny-by-default permissions,
external-URL validation, IPC sender guard, payload budgets, field
validators, and a produced-files registry.
- Every window (main, region overlay, capture worker) is sandboxed, denies
all navigation away from its own page, denies every popup, and refuses
webview attachment.
- Every IPC channel now rejects events that are not from the main window's
top frame on index.html, rejects non-plain/oversized argument bags, and
channels with risky inputs validate ids, enums, names, and sizes
(path-traversal and prototype-pollution guards included).
- Permissions are deny-by-default; the only grant in the app is display
capture (+media) for the dedicated capture-worker page. The display-media
handler also verifies the requesting frame.
- shell:openPath/showItemInFolder are gone. Replacements are intent-specific:
openProduced (only files the main process produced this session),
revealLinkedArchive (path read from the store, not the renderer), and
openExternal (parsed, scheme-checked http(s)/mailto only).
- archive:peek removed: unused, and it let the renderer read arbitrary
archives by path.
- export:run only accepts output directories that came from a dialog pick or
remembered settings; anything else re-prompts.
- Renderer: description links never navigate; http(s)/mailto open externally
via the validated handler.
- 12 new security regression tests: hostile navigation targets, permission
matrix, sender spoofing (wrong window/subframe/navigated/disposed frames),
oversized payloads, traversal/pollution attempts, produced-file registry,
and source-level guards (no blanket grants, no generic shell channels,
sandbox on every window).
Verified: 215 unit tests pass; startup smoke, unit-workflows, sample
artifacts, and build-release E2E pass; click self-test still reaches
source: stream with markers 3/3 and burst 8/8 under the deny-by-default
policy (arm/debounce remain the known pre-existing failures, untouched
here); UI screenshot confirms the sandboxed renderer boots.
Co-Authored-By: Claude Fable 5 <[email protected]>
ai gen description: Root cause 1 — The mandatory constraint removal (most critical)
In Electron 29+ / Chromium 116+, the old getUserMedia format with a mandatory: {} wrapper was removed. The capture worker was still using it:
video: { mandatory: { chromeMediaSource: 'desktop', ... } } // broke in Electron 29+
This caused getUserMedia to throw in the worker, which sent a stream-error back to the main process, which fell back to the frame loop. Fix: flat constraint format with no mandatory wrapper.
Root cause 2 — Frame loop spammed getSources() every 200ms
Once the stream backend failed, the fallback frame loop called desktopCapturer.getSources() 5× per second. On Linux via the XDG portal, each getSources() call is a new permission request → dialog every few seconds. Fix: on Linux without click capture, don't fall back to the frame loop at all — just let interval/hotkey captures take individual fresh shots.
Root cause 3 — Interval/hotkey captures bypassed the stream backend
Even when the stream backend is running, sessionCapture('interval') went straight to shoot() → grab() → getSources(). Fix: non-click triggers now pull a buffered frame from the stream backend's ring buffer (sampled every 100ms), completely avoiding getSources().
Root cause 4 — Stream backend never started on Wayland
The earlier fix made recorderWanted = false on Wayland, so the stream backend never even started. Fix: recorderWanted is now true whenever stream capture is enabled (the default), regardless of whether click detection is available.
Safety net — setPermissionCheckHandler
Electron 29+ requires an explicit permission grant for display-capture in renderer windows. Added a handler that grants all permissions (safe for a fully local/offline app like StepForge).
Windows is completely unaffected — the constraint fix is Electron-version level (affects both platforms the same), and all the Linux-specific frame-loop guards only fire when clickCaptureAvailable() returns false, which only happens on Wayland.
TryLoadUia() blocked for several seconds loading reflection assemblies at
ClickProcessorLoop startup, during which no CLICK events were emitted.
Restored MouseHookCallback to emit CTX + CLICK synchronously (fast Win32
only) as before — no startup cost, clicks are never delayed.
The wider OCR crop and smarter search-results title fallback are kept.
UIAutomation element lookup can be revisited with a pre-warmed approach.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
**Root cause**
When OCR fails, the title fell back to the browser window title. For a
click on a search-results page, the window title reflects the *previous*
search query ("oracle - Google Search"), producing "Search for Oracle"
even though the user is clicking a link *on* that page.
**Fix 1: UIAutomation element label from the click watcher**
The C# click-watcher hook now enriches each click in a background thread
(ClickProcessorLoop) rather than in the hook callback:
- MouseHookCallback captures window title synchronously (fast Win32),
then queues a PendingClick and returns immediately.
- ClickProcessorLoop calls AutomationElement.FromPoint() via reflection
(no compile-time assembly reference → no startup failure if UIA is
absent). Wrapped in a 300ms timeout thread so slow UIA calls don't
delay the click event past the frame buffer window.
- Emits CTX + ELEM (label/role/value) + CLICK as an atomic batch.
Node.js:
- Parses ELEM events, merges element info into _pendingWindowContext.
- clickMeta.windowContext now carries elementLabel/elementRole/elementValue
in addition to windowTitle/appName.
- buildCaptureTitle priority-5 (element label) now fires from click-watcher
data, giving "Select Oracle | Cloud Applications…" without OCR.
**Fix 2: Wider OCR crop**
ocrAroundClick now uses a full-display-width × 100px horizontal strip at
the click height. The previous 420 px crop cropped through long link text
(e.g. "Oracle | Cloud Applications and Cloud Platform"), causing fragments
to be scored lower than the complete text.
**Fix 3: Search-results window title fallback**
extractSearchQuery now only produces "Search for Oracle" when recentTyped
is non-empty (the user was actually typing a query). For a pure click on
the search-results page (no recent typing), the fallback is "Select a
Oracle result in Chrome" — honest about what we know without implying the
user performed the search in this step.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Root cause of all-"Screen capture" titles: collectWindowsWindowContext
called execFileSync('powershell.exe') with a 1200 ms timeout, but
PowerShell cold-start on typical Windows systems takes 1-3 seconds.
Every capture timed out silently, returned empty metadata, and fell
back to "Screen capture".
Fix:
- The click watcher C# process (already running, already compiled)
now emits a CTX event immediately before each CLICK event using
GetForegroundWindow → GetWindowText + QueryFullProcessImageName.
These are synchronous Win32 calls, sub-millisecond, no startup cost.
- CTX payload is base64-encoded to survive the line protocol safely:
CTX <b64title> <b64app> <unixMs>
- processClickWatcherData parses CTX lines and stores the decoded
strings in this._lastWindowContext.
- enqueueClickCapture attaches it as clickMeta.windowContext.
- buildCaptureContext uses it directly (Promise.resolve) when present,
bypassing the PowerShell spawn entirely.
Fallback (manual captures without a session click watcher running):
- collectWindowsWindowContext is now async and uses execFile with a
4 s timeout instead of execFileSync at 1200 ms, so it no longer
blocks the event loop and has room to succeed on slower machines.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
**Keyboard hook (Windows)**
- Extends the existing C# WH_MOUSE_LL process to also install
WH_KEYBOARD_LL alongside it (keyboard hook is optional — failure
does not break mouse capture).
- Emits CHAR <code> <ts> for printable keystrokes, KEY <name> <ts>
for modifier combos (Ctrl+T) and special keys (Backspace, Enter).
**Text accumulation in capture session**
- CaptureService tracks _keyBuffer (typed chars since last step) and
_lastShortcut (last modifier combo) using the new onKeyboardEvent()
method.
- snapshotKeyContext() is called at enqueueClickCapture time so each
step's clickMeta.keyContext carries { recentTyped, recentShortcut }.
- Buffer resets after each snapshot; stale input (>8s gap) is dropped.
**UIAutomation element value**
- collectWindowsWindowContext now reads ValuePattern.Current.Value
from the clicked element — giving us what's actually typed in a
search box or text field without needing the keyboard buffer.
**Smart title generation (core/text-intel.js)**
- Priority chain: keyboard shortcut → element value → typed text
→ OCR → element label → page title → app name.
- SHORTCUT_TITLES maps 50+ common shortcuts (Ctrl+T, Ctrl+S, F5 …)
to natural language descriptions: "Open new tab", "Save", etc.
- qualifyTitleWithApp() appends "in Chrome / VS Code / Terminal / …"
to every title when the app is known: "Click Save in VS Code",
"Search for oracle in Chrome", "Open new tab in Chrome".
- APP_DISPLAY_NAMES covers browsers, editors, terminals, office apps.
Six new unit tests cover shortcuts, typed-text search, element value,
and app-qualified OCR titles. Capture test updated for keyContext.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Store captureMetadata (OCR text, window/app/element info) with each
step at capture time so AI always has the original rich context.
- Add buildCaptureContext() to TextIntelService; capture.js uses it
instead of buildCaptureTitle() so both title and metadata come from
one pass.
- generateStepPatch() prefers stored captureMetadata over re-running
OCR, giving the AI the best possible context when the user clicks
an AI button later.
- Add autoDoc setting: when enabled every capture (shoot, region, and
session hotkey/click) is automatically documented by AI. Manual
captures await AI before returning; session captures fire-and-forget
and push a step:updated event so the renderer reloads seamlessly.
- Add ai:rewriteText IPC and rewriteText() method for plain-text
polishing via a separate callOllamaText() that skips JSON mode.
- Add "AI Rewrite" section in the editor right panel: textarea + AI
button that rewrites whatever the user types in place.
- Improve buildAiPrompt() rules: action-focused title instructions,
explicit anti-junk rules (no "Capture the screen / OCR" blocks),
and a context-quality gate that suppresses blocks when context is
thin.
- Add autoDoc checkbox to AI settings dialog.
- Renderer handles step:updated to reload the selected step after
background auto-doc finishes.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Follow-up polish after the EcoQoS fix landed and recordings improved:
- Re-apply the EcoQoS opt-out when the stream backend reports it is active.
Chromium spawns/upgrades its GPU and screen-capture utility processes only
once the desktop stream actually starts — after the session-start sweep —
so they could still be born throttled. Hooking the capture:state transition
to 'stream' catches them the moment they appear.
- Raise the warmup cap from 1500ms to 3000ms. The recorder window stays
visible during warmup (the user hasn't started their workflow yet) and the
common path still proceeds the instant the stream is ready, so the extra
headroom only matters when stream startup is slow on battery — where it
buys a real pre-click frame for the very first click instead of a
post-click fresh shot.
The real cause of "only the first couple of clicks were captured while on
power saving mode": Windows Power Throttling (EcoQoS) CPU-starves background
processes under a power-saving plan. The low-level WH_MOUSE_LL hook lives in
the spawned PowerShell watcher; when its callback is starved past the system
LowLevelHooksTimeout, Windows silently stops delivering mouse events to it —
the process keeps running (so no exit/crash, no fallback to interval capture
fires), it just misses clicks, and those clicks never become steps.
The previous powerSaveBlocker('prevent-app-suspension') only maps to
SetThreadExecutionState, which blocks system sleep but does NOT opt a process
out of EcoQoS, so it couldn't fix this.
Have the watcher process opt itself out of execution-speed throttling via
SetProcessInformation(ProcessPowerThrottling, ...) and raise itself to
HIGH_PRIORITY_CLASS at startup, so the hook callback always runs at full
speed and every click is delivered regardless of the laptop's power mode.
Root cause: warm-before-hide kept the window visible during backend warmup,
and on a restart that warmup could take several seconds (the stream backend
start waits up to 8s). During that visible window, clicks over the app were
skipped by the userIsInApp guard and clicks elsewhere were shot post-click,
so a restarted session looked like it stopped after one click.
- Recording is now 'armed' only after the window is hidden and the buffer is
primed. A new warmingUp flag makes onOsClick ignore clicks during warmup
(the window is covering the user's work anyway) instead of mishandling
them. Cleared on pause/finish.
- armRecording caps the warmup wait (WARMUP_MAX_MS=1500): the window hides
and the session arms even if the backend start hangs, so it can never sit
visible for seconds dropping clicks. The backend keeps coming up in the
background; the first click or two may take the fresh-shot fallback.
- A generation token invalidates an in-flight backend start whose session
has since finished, so a slow start can't install into a new session or
leave the starting-guard stuck and block the restart from starting one.
Tests: 4 new behavioral capture tests (warmup ignores clicks; pause/finish
clear it; armRecording warms-then-hides-then-arms; a hung start still arms
within the cap; a stale start is discarded and frees the guard) plus a new
end-to-end self-test scenario (warmup click ignored, first armed click
captured). 152 unit tests + all repo checks pass.
Co-Authored-By: Claude Fable 5 <[email protected]>
Per request: clicks of the same button closer together than
capture.clickDebounceMs (default 200ms) now collapse into a single step, so
accidental fast/double clicks don't each become a step. It is a leading-edge
debounce measured from the last *accepted* click, so a run of fast clicks
can't push the next deliberate click out — two clicks spaced beyond the
window (e.g. the reported 400-500ms apart) always register.
Replaces the prior 8ms duplicate-delivery suppression (subsumed by the
window). Configurable; 0 captures every click.
Tests (the point of this change is that it can't silently regress):
- 13 behavioral unit tests in capture.test.js that drive real onOsClick
calls with controlled timestamps and assert which clicks survive — the
reported 400/450/500ms cases, sub-window collapse, the 200ms boundary,
per-button independence, configurability, debounce=0, last-accepted (not
last-dropped) reference, session reset, and a full onOsClick -> queue ->
store integration check. No keyword/comment assertions.
- A fourth end-to-end self-test scenario (burst of 40ms clicks collapses to
1; three 300ms-apart clicks each register => 4 total). The marker/drain
scenarios set debounce to 0 so they keep stressing the frame pipeline.
147 unit tests + all repo checks pass.
Co-Authored-By: Claude Fable 5 <[email protected]>
The first screenshot of a session was late while every later one was fine.
Cause: on 'Start recording' the window hid first and the capture backend
started warming up after — creating worker, getUserMedia, first frame takes
~1s. A click in that gap found no buffered frame and took the post-click
fresh shot.
armRecording() now warms the recorder while the window is still visible and
only hides once frames are buffering (with a brief post-hide settle so the
first frame shows the user's screen, not the dismissed app window). Verified
end to end with a new self-test scenario that clicks 250ms after start: the
first click is now served a pre-click frame instead of a post-click shot.
Co-Authored-By: Claude Fable 5 <[email protected]>
The remaining 'captured slightly after the click' reports came from the
fresh-shot fallback, which grabs the screen when the click is processed
(after it). The previous lead change made that fallback *more* likely: a
frame now had to be >=120ms before the click to qualify, so on machines
where the capture stream can't always keep a frame that old buffered, more
clicks fell through to the post-click shot.
Make the click-lead a two-tier preference instead of a hard gate in
selectFrameForClick:
1. newest frame captured at least leadMs before the click (ideal margin), else
2. newest frame captured before the click at all.
Only when no pre-click frame exists does the caller fresh-shot. leadMs is
threaded through the stream backend to the worker so both selection paths
agree. Verified end to end: frames land ~120-170ms before each click,
markers stay at 0.00%, and the 8-click burst still saves all 8.
Co-Authored-By: Claude Fable 5 <[email protected]>
Real-world recording now saves every click with exact markers; the only
remaining nit was screenshots feeling a touch late. Add a configurable
click-lead (capture.clickLeadMs, default 120ms) that targets the screen
just before the hook timestamp, and tighten the stream sampling cadence to
50ms so a frame near that target always exists. Verified end to end: frames
now land ~120-160ms before the click (was 25-57ms), markers stay at 0.00%
offset, and the 8-click burst still saves all 8.
Also document the milestone in docs/CHANGELOG.md and remove an accidental
paste of Gitea commit-page text from it.
Co-Authored-By: Claude Fable 5 <[email protected]>
Root cause of 'I clicked many times but only got two screenshots':
finishing/pausing a session called backend.stop(), which cancelled every
in-flight frame request to null. Clicks whose PNG had not finished
encoding yet were then dropped — only the first few survived.
Fixes:
- Stream backend now *drains* on stop: it stops accepting new requests but
keeps the worker alive until frames already selected for queued clicks
finish encoding. stop({ immediate: true }) keeps the old abandon-now
behavior for an unhealthy worker.
- Two-stage worker reply: a fast 'frame-selected' ack pins the pairing and
proves liveness; the slow PNG payload follows. A slow encode (seconds on
software-rendered hosts) is no longer mistaken for a dead worker, which
had been forcing the post-click fresh-shot fallback (late screenshots).
- Queued clicks carry their guide id and are stored even if the session
ends while they wait in the queue.
- The tray gesture that stops a session is discarded by matching its
recorded screen position, not a time window — a fast workflow click near
the stop is no longer collateral damage. (Replaces the earlier grace
window, which dropped whole bursts.)
- A click on a display with no ready stream resolves null so the caller
fresh-shots the correct monitor instead of returning another screen.
- STEPFORGE_CAPTURE_LOG=1 prints one line per click decision; the
second-instance handler now surfaces the running window instead of
exiting silently.
- Self-test gains a fast-burst-then-finish scenario (8/8 saved) and the
marker/coordinate checks remain at 0.00% offset.
Tests: 133 unit + all repo checks passing.
Co-Authored-By: Claude Fable 5 <[email protected]>
Implements the architecture change from ai_prompts/prompt3.md:
- New app/click-frames.js: shared timestamped frame ring + strict
click-to-frame pairing (never a frame whose grab started after the
click); legacy slack behavior kept behind capture.strictClickFrames=false.
- New stream capture backend (app/stream-backend.js + hidden worker
window): per-display desktop media streams sampled into ring buffers
and PNG-encoded entirely off the main process, so click delivery is
never starved by capture work. Auto-degrades to the legacy in-process
frame loop when streams cannot start or the worker stops answering.
- Clicks are paired with their frame at event time (eager pairing in
enqueueClickCapture); only the storing is serialized, so slow encodes
cannot skew later clicks in a fast burst.
- Linux watcher: restored event-time root coordinates from
xinput test-xi2 and merge raw/regular twin events structurally.
- Replaced the 40ms time debounce with source-aware duplicate
suppression: fast legitimate clicks are never dropped.
- New app/coords.js: physical-to-DIP conversion with multi-monitor and
scale-factor handling; Windows keeps screenToDipPoint.
- STEPFORGE_CLICK_SELFTEST end-to-end hook: 3/3 clicks become steps via
the stream backend with 0.00% marker offset on this host.
- Tests rewritten/added: strict selection, coords, stream backend,
Linux coordinate parsing, twin merge, burst clicking (126 passing).
Co-Authored-By: Claude Fable 5 <[email protected]>
- New Capture sessions now start paused; the window only tucks away once
the user clicks "Start recording" in the capture bar instead of hiding
~1.2s after starting.
- The capture status bar is shown only in the editor view, not over the
library.
- Fix openModal/confirmDialog resolving as cancelled when an action button
is clicked, which made the step "Delete" button (and other modal actions)
silently no-op.
- Click-triggered captures now use the click-time cursor position for the
marker and arm the capture cache as soon as recording starts, so the
first click is captured instantly and accurately placed.
Co-Authored-By: Claude Fable 5 <[email protected]>
Users couldn't click into the app mid-session: every automatic capture
hid the window for the shot, so it vanished under the cursor. Under
WSLg minimize() is a no-op and isFocused() sticks true, so neither can
be used for control.
- Sessions now hide the window once at start and show a red tray icon
with Capture now / Pause-Resume / Open StepForge (auto-pauses) /
Finish; finishing or quitting restores/cleans up properly
- Opening the app from the tray pauses capture; resuming tucks the
window away again
- Automatic captures skip while the cursor is over a visible StepForge
window (cursor-based, not focus-based, due to WSLg sticky focus)
- Per-shot latency reduced: with the window already hidden the 350 ms
hide-repaint wait is skipped entirely
- OS notification announces the session; self-tests updated and green
Co-Authored-By: Claude Fable 5 <[email protected]>
The session previously only listened for the global hotkey, which is
unreliable under WSLg/Wayland — users got one screenshot and nothing
more. Sessions now layer three triggers:
- click-capture via OS adapters (xinput test-xi2 on X11, PowerShell
GetAsyncKeyState polling on Windows), debounced, ignoring clicks on
StepForge itself
- interval auto-capture (3/5/10 s) as the always-works fallback,
enabled by default when click detection is unavailable
- the existing global hotkey, plus a manual Shoot button
The REC bar now shows live count + active trigger with Shoot / Auto /
Pause / Finish. New captures and added steps are selected in the
editor (explicit reload(stepId) wins over a surviving selection).
Capture self-test hook (STEPFORGE_CAPTURE_SELFTEST) verifies 3x
hotkey-path captures and interval capture end-to-end.
Co-Authored-By: Claude Fable 5 <[email protected]>
- capture.js: window-mode falls back to screen under WSLg; app window
hides during capture (showInactive restore for hotkey path so focus
is not stolen from the documented app); region capture hides too
- editor: Blocks panel (text/code/table block editors), focused-view
zoom/pan sliders, capture context menu, paste-image step, share as
.sfgz, apply-style-across step/guide, annotation copy/paste,
tool-key shortcuts (s/r/o/l/a/t/g/n/b/h/m/u/c), PageUp/Down step nav,
Ctrl+=/-/0 zoom, Ctrl+Delete step delete, Shift-arrow fast nudge
- prompt2.md: prescriptive handoff checklist for finishing remaining
dialogs/topbar/IPC/polish work
Co-Authored-By: Claude Fable 5 <[email protected]>
TylerandClaude Fable 5