Phase 2 of the improvement plan (PR 5 of the sequence). Several confirmed
capture defects from the audit.
Region capture:
- regionCapture returned { ok, step } wrapping storeFrameAsStep's own
{ ok, step }, so the real step was at result.step.step — region selection
and region auto-doc (which read result.step.stepId) broke. Return the
storeFrameAsStep result directly.
- pickRegion leaked the region:picked IPC listener (and the overlay/image
refs) whenever the overlay was cancelled/closed rather than picked: cleanup
only ran on a pick. Cleanup is now idempotent and runs on pick, close, load
failure, and settle. The received rectangle is validated and clamped to the
image (new overlayRectToImageRect handles negative-size drags and
out-of-bounds selections) so image.crop can never read out of bounds.
Power blocker ownership:
- New single owner: CaptureService.syncPower() holds the blocker iff a session
is actively recording, called on start/pause/resume/finish. A new session
starts paused and no longer holds the blocker while idle; tray and
second-instance pauses that previously bypassed main.js's stop closure now
release it correctly. main.js provides the power policy (blocker + EcoQoS
opt-out) via dependency injection.
Explicit click-trigger source:
- startEvdevWatcher never set clickWatcher, so state().clickCapture read false
while evdev was actively capturing clicks. Replaced the boolean with an
explicit clickSource (windows-hook | x11 | evdev-x11 | evdev-wayland |
unavailable); clickCapture is now derived from it. evdev device-stream
errors/closes now fall back via handleClickWatcherLoss instead of being
swallowed.
Strict click timing:
- When no pre-click frame qualifies, strict mode previously fell through to a
fresh (post-click) shot and stored it — contradicting the strict promise.
It now skips with a capture:diagnostic instead. Non-strict (balanced) mode
keeps the fresh-shot fallback. Existing tests that exercised the fallback
now run in balanced mode; the strict test asserts the skip.
Other:
- pathToFileURL replaces file://${p} concatenation for step image URLs and
export previews (correct for spaces, #, %, drive letters).
- Best-effort click-queue drain on app shutdown (before-quit) so a burst just
before quit is not lost; bounded so quit never hangs.
Tests: region rect clamping/normalization, power-held-only-while-recording
(incl. finish releases), click-source reporting incl. evdev, drain deadline.
230 unit tests pass. Click self-test: markers 3/3 (strict) and burst 8/8
deterministic across runs; the burst scenario runs in balanced mode because
it tests the drain, not strict timing. (arm/debounce remain the pre-existing
Linux capture failures untouched by this PR.)
Co-Authored-By: Claude Fable 5 <[email protected]>
Phase 1 of the improvement plan (PR 3 of the sequence). The docs claimed
"fully offline"/"never talks to the network," but text-intel makes HTTP
requests to a configurable Ollama host that could be remote, with no timeout,
no cancellation, and no size limit; the Windows hook logged raw keystrokes
into capture metadata that could then be sent to that host.
Privacy — raw keystroke capture:
- New capture.captureTypedText setting, default false. With it off, printable
characters are never buffered in JS and the Windows keyboard hook never even
emits them across the process boundary (the flag is threaded into the C#).
Shortcut/navigation detection (Ctrl+T, Enter, …) is unaffected.
AI network hardening (app/text-intel.js):
- Every Ollama call goes through fetchJson with an AbortController deadline
(ai.timeoutMs, default 60s): a dead endpoint fails fast instead of leaving
UI actions pending forever.
- Cancellation: in-flight requests are tracked and cancelInflight(guideId)
aborts them; new ai:cancel IPC + api.ai.cancel are called when the editor
closes, and shutdown cancels everything.
- Bounded concurrency (2) for AI network work.
- Screenshots are only attached when allowed (ai.attachScreenshots), the model
is vision-capable, and the image is within ai.maxImageBytes — no more
unbounded base64-expanded 4K bodies.
Local-first host policy (core/text-intel.js):
- New isLoopbackHost + validateOllamaHost. By default only a loopback Ollama
endpoint is contacted; a remote host is refused with a clear message unless
ai.allowRemoteHost is explicitly enabled. Blocked hosts are never contacted.
Honest documentation:
- README, package.json, and the welcome screen drop "fully offline"/"never
talks to the network"/"Electron is the only dependency" for an accurate
local-first contract that discloses the optional AI path and the bundled
Tesseract OCR dependency.
- New docs/PRIVACY.md details exactly what is collected locally and the one
outbound (opt-in, loopback-by-default) AI feature.
Tests: loopback/remote host matrix, remote-blocked-without-opt-in (and never
contacted), remote-allowed-with-opt-in, request timeout, explicit cancel vs
timeout, typed-text off-by-default vs opted-in, shortcut detection still works,
and a source guard that the C# CHAR emission stays behind the opt-in. 224 unit
tests pass; startup smoke and workflow E2E pass.
Co-Authored-By: Claude Fable 5 <[email protected]>
Phase 1 of the improvement plan (PR 2 of the sequence). A remote page could
previously inherit the privileged preload bridge: the main window had no
navigation guard or popup handler, IPC handlers accepted any sender, every
Electron permission was granted to everyone, and shell:openPath accepted an
arbitrary renderer-supplied target.
- New app/security.js (plain-Node testable) centralizes the policy:
app-page identity, navigation/popup denial, deny-by-default permissions,
external-URL validation, IPC sender guard, payload budgets, field
validators, and a produced-files registry.
- Every window (main, region overlay, capture worker) is sandboxed, denies
all navigation away from its own page, denies every popup, and refuses
webview attachment.
- Every IPC channel now rejects events that are not from the main window's
top frame on index.html, rejects non-plain/oversized argument bags, and
channels with risky inputs validate ids, enums, names, and sizes
(path-traversal and prototype-pollution guards included).
- Permissions are deny-by-default; the only grant in the app is display
capture (+media) for the dedicated capture-worker page. The display-media
handler also verifies the requesting frame.
- shell:openPath/showItemInFolder are gone. Replacements are intent-specific:
openProduced (only files the main process produced this session),
revealLinkedArchive (path read from the store, not the renderer), and
openExternal (parsed, scheme-checked http(s)/mailto only).
- archive:peek removed: unused, and it let the renderer read arbitrary
archives by path.
- export:run only accepts output directories that came from a dialog pick or
remembered settings; anything else re-prompts.
- Renderer: description links never navigate; http(s)/mailto open externally
via the validated handler.
- 12 new security regression tests: hostile navigation targets, permission
matrix, sender spoofing (wrong window/subframe/navigated/disposed frames),
oversized payloads, traversal/pollution attempts, produced-file registry,
and source-level guards (no blanket grants, no generic shell channels,
sandbox on every window).
Verified: 215 unit tests pass; startup smoke, unit-workflows, sample
artifacts, and build-release E2E pass; click self-test still reaches
source: stream with markers 3/3 and burst 8/8 under the deny-by-default
policy (arm/debounce remain the known pre-existing failures, untouched
here); UI screenshot confirms the sandboxed renderer boots.
Co-Authored-By: Claude Fable 5 <[email protected]>
Phase 0 of the improvement plan (ai_prompts/prompt4.md): make the baseline
reproducible and stop the test runner from masking real failures.
- Pin Node >= 22.12 (engines + .nvmrc + engine-strict); every entry point
fails fast with clear guidance instead of dying late with ERR_REQUIRE_ESM
inside the packaging dependency graph.
- electron-launcher.js is diagnostics-only: all runtime npm install/rebuild/
repair paths are removed. npm ci on the pinned toolchain is the only
supported install path (README + GETTING_STARTED updated).
- Refuse to silently launch unsandboxed on Linux: --no-sandbox now requires
an explicit STEPFORGE_ALLOW_NO_SANDBOX/ELECTRON_DISABLE_SANDBOX opt-in and
is otherwise a hard error with actionable fixes; user-namespace sandboxing
is detected and preferred.
- Click-capture E2E no longer converts startup crashes into "SKIPPED": the
only allowed skip is the upfront absence of a display server. A missing
shared library or crash now fails with the startup log. Same guard added
to the startup smoke check.
- GitHub CI: run on pull_request, pin Node from .nvmrc, drop the macOS matrix
entry (not a support target), and audit production and full dependency
trees as separate signals. Gitea CI: pull_request trigger + pinned Node.
- Refresh package-lock on Node 22/npm 10 and remediate the form-data and
undici advisories (npm audit: 0 vulnerabilities, prod and full tree).
- Stop tracking generated machine-specific build reports
(build/build_report.md, build/artifacts_manifest.json).
Verified: 203 unit tests pass; repo-structure, startup-smoke, unit-workflows,
sample-artifacts, and build-release checks pass locally with a real Electron
launch. The click self-test now truthfully reports the pre-existing Linux
arm/debounce capture failures (also red on Gitea CI main run 177) instead of
hiding behind SKIPPED; that defect is scheduled for the capture-fix PR.
Co-Authored-By: Claude Fable 5 <[email protected]>
ai gen description: Root cause 1 — The mandatory constraint removal (most critical)
In Electron 29+ / Chromium 116+, the old getUserMedia format with a mandatory: {} wrapper was removed. The capture worker was still using it:
video: { mandatory: { chromeMediaSource: 'desktop', ... } } // broke in Electron 29+
This caused getUserMedia to throw in the worker, which sent a stream-error back to the main process, which fell back to the frame loop. Fix: flat constraint format with no mandatory wrapper.
Root cause 2 — Frame loop spammed getSources() every 200ms
Once the stream backend failed, the fallback frame loop called desktopCapturer.getSources() 5× per second. On Linux via the XDG portal, each getSources() call is a new permission request → dialog every few seconds. Fix: on Linux without click capture, don't fall back to the frame loop at all — just let interval/hotkey captures take individual fresh shots.
Root cause 3 — Interval/hotkey captures bypassed the stream backend
Even when the stream backend is running, sessionCapture('interval') went straight to shoot() → grab() → getSources(). Fix: non-click triggers now pull a buffered frame from the stream backend's ring buffer (sampled every 100ms), completely avoiding getSources().
Root cause 4 — Stream backend never started on Wayland
The earlier fix made recorderWanted = false on Wayland, so the stream backend never even started. Fix: recorderWanted is now true whenever stream capture is enabled (the default), regardless of whether click detection is available.
Safety net — setPermissionCheckHandler
Electron 29+ requires an explicit permission grant for display-capture in renderer windows. Added a handler that grants all permissions (safe for a fully local/offline app like StepForge).
Windows is completely unaffected — the constraint fix is Electron-version level (affects both platforms the same), and all the Linux-specific frame-loop guards only fire when clickCaptureAvailable() returns false, which only happens on Wayland.
The previous version called target:'all' on every step unconditionally,
which caused the AI to rewrite steps that already had good titles and
descriptions — often making them worse or inaccurate.
New behaviour:
- Build a queue of only the steps that are actually missing content
(placeholder title and/or empty description).
- Determine target per step: 'title', 'description', or 'all' — only
fill what is blank; leave existing user-written text completely alone.
- If all steps already have titles and descriptions, show a toast and stop.
- Call reload() once at the end instead of patching this.steps mid-loop,
so the editor and step list both update atomically from the store.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Previously the menu action only ran AI generation on the selected step.
Now it loops through all steps sequentially, showing progress ("AI:
filling step N of M…"), then reloads the visible step when done.
Partial failures are reported in the final toast.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The model was producing Text and Code blocks on every 'generate all
fields' action — uninstructed, generic, and never useful for the user.
Blocks are only appropriate when the user explicitly edits an existing
block (target === 'block'). For title/description/all generation:
- Schema collapses to { "title", "description" } — no blocks key
- targetText no longer mentions blocks
- allowedBlockNote is null (omitted from prompt)
- Rule explicitly says "Do NOT add any blocks array"
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
TryLoadUia() blocked for several seconds loading reflection assemblies at
ClickProcessorLoop startup, during which no CLICK events were emitted.
Restored MouseHookCallback to emit CTX + CLICK synchronously (fast Win32
only) as before — no startup cost, clicks are never delayed.
The wider OCR crop and smarter search-results title fallback are kept.
UIAutomation element lookup can be revisited with a pre-warmed approach.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
**Root cause**
When OCR fails, the title fell back to the browser window title. For a
click on a search-results page, the window title reflects the *previous*
search query ("oracle - Google Search"), producing "Search for Oracle"
even though the user is clicking a link *on* that page.
**Fix 1: UIAutomation element label from the click watcher**
The C# click-watcher hook now enriches each click in a background thread
(ClickProcessorLoop) rather than in the hook callback:
- MouseHookCallback captures window title synchronously (fast Win32),
then queues a PendingClick and returns immediately.
- ClickProcessorLoop calls AutomationElement.FromPoint() via reflection
(no compile-time assembly reference → no startup failure if UIA is
absent). Wrapped in a 300ms timeout thread so slow UIA calls don't
delay the click event past the frame buffer window.
- Emits CTX + ELEM (label/role/value) + CLICK as an atomic batch.
Node.js:
- Parses ELEM events, merges element info into _pendingWindowContext.
- clickMeta.windowContext now carries elementLabel/elementRole/elementValue
in addition to windowTitle/appName.
- buildCaptureTitle priority-5 (element label) now fires from click-watcher
data, giving "Select Oracle | Cloud Applications…" without OCR.
**Fix 2: Wider OCR crop**
ocrAroundClick now uses a full-display-width × 100px horizontal strip at
the click height. The previous 420 px crop cropped through long link text
(e.g. "Oracle | Cloud Applications and Cloud Platform"), causing fragments
to be scored lower than the complete text.
**Fix 3: Search-results window title fallback**
extractSearchQuery now only produces "Search for Oracle" when recentTyped
is non-empty (the user was actually typing a query). For a pure click on
the search-results page (no recent typing), the fallback is "Select a
Oracle result in Chrome" — honest about what we know without implying the
user performed the search in this step.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
**AI button: dynamic tooltip hints**
- titleAiBtn and descAiBtn now show "Rewrite with AI" when the field
already has user content, "Generate with AI" when empty.
- updateAiButtonHints() fires on every title/description input event
and whenever syncStepFields() runs to keep hints current.
**Stronger rewrite prompt**
- When the user has typed a draft title or description, the prompt now
shows it explicitly: "User's draft title (rewrite this): '...'"
- Rules changed from "improve its wording" to "Your only job is to
polish its grammar and phrasing. Do NOT replace it with something
different." — prevents the model from ignoring the user's text and
generating fresh content from capture context.
- The suggested-title hint is suppressed when a draft title exists so
the model doesn't silently swap the user's text for the auto-title.
**Title quality: generic window title filter**
- GENERIC_WINDOW_TITLES Set filters "New Tab", "Untitled", "Loading"
etc. from the window-title path so they no longer produce titles
like "Open New Tab in Chrome".
**Title quality: app name stripping for non-browser apps**
- stripBrowserNameSuffix now accepts an optional appName; it strips
the app's display name and process name from the window title suffix
using the same pattern as browser names.
- "Document1.docx - Word" with appName "winword" → "Document1.docx".
- buildCaptureTitle passes metadata.appName into the strip call.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Root cause of all-"Screen capture" titles: collectWindowsWindowContext
called execFileSync('powershell.exe') with a 1200 ms timeout, but
PowerShell cold-start on typical Windows systems takes 1-3 seconds.
Every capture timed out silently, returned empty metadata, and fell
back to "Screen capture".
Fix:
- The click watcher C# process (already running, already compiled)
now emits a CTX event immediately before each CLICK event using
GetForegroundWindow → GetWindowText + QueryFullProcessImageName.
These are synchronous Win32 calls, sub-millisecond, no startup cost.
- CTX payload is base64-encoded to survive the line protocol safely:
CTX <b64title> <b64app> <unixMs>
- processClickWatcherData parses CTX lines and stores the decoded
strings in this._lastWindowContext.
- enqueueClickCapture attaches it as clickMeta.windowContext.
- buildCaptureContext uses it directly (Promise.resolve) when present,
bypassing the PowerShell spawn entirely.
Fallback (manual captures without a session click watcher running):
- collectWindowsWindowContext is now async and uses execFile with a
4 s timeout instead of execFileSync at 1200 ms, so it no longer
blocks the event loop and has room to succeed on slower machines.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
**Keyboard hook (Windows)**
- Extends the existing C# WH_MOUSE_LL process to also install
WH_KEYBOARD_LL alongside it (keyboard hook is optional — failure
does not break mouse capture).
- Emits CHAR <code> <ts> for printable keystrokes, KEY <name> <ts>
for modifier combos (Ctrl+T) and special keys (Backspace, Enter).
**Text accumulation in capture session**
- CaptureService tracks _keyBuffer (typed chars since last step) and
_lastShortcut (last modifier combo) using the new onKeyboardEvent()
method.
- snapshotKeyContext() is called at enqueueClickCapture time so each
step's clickMeta.keyContext carries { recentTyped, recentShortcut }.
- Buffer resets after each snapshot; stale input (>8s gap) is dropped.
**UIAutomation element value**
- collectWindowsWindowContext now reads ValuePattern.Current.Value
from the clicked element — giving us what's actually typed in a
search box or text field without needing the keyboard buffer.
**Smart title generation (core/text-intel.js)**
- Priority chain: keyboard shortcut → element value → typed text
→ OCR → element label → page title → app name.
- SHORTCUT_TITLES maps 50+ common shortcuts (Ctrl+T, Ctrl+S, F5 …)
to natural language descriptions: "Open new tab", "Save", etc.
- qualifyTitleWithApp() appends "in Chrome / VS Code / Terminal / …"
to every title when the app is known: "Click Save in VS Code",
"Search for oracle in Chrome", "Open new tab in Chrome".
- APP_DISPLAY_NAMES covers browsers, editors, terminals, office apps.
Six new unit tests cover shortcuts, typed-text search, element value,
and app-qualified OCR titles. Capture test updated for keyContext.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The title engine was falling back to "Screen capture" for all browser
captures because it treated the entire browser window title as noise.
Changes:
- stripBrowserNameSuffix: removes "- Google Chrome" / "| Firefox" etc.
from the end of window titles, leaving just the page title.
"oracle - Google Search - Google Chrome" → "oracle - Google Search"
"Oracle | Cloud Applications - Google Chrome" → page title only
- extractSearchQuery: detects "[query] - Google Search" / Bing / etc.
patterns after stripping the browser suffix, and formats the result
as "Search for oracle".
- buildCaptureTitle: uses stripped page title + search detection before
falling through to the "Screen capture" fallback.
- pickBestOcrPhrase: considers the full OCR line (≤80 chars) as a
candidate with a +35 completeness bonus before splitting on | or ·.
This preserves "Oracle | Cloud Applications and Cloud Platform" as a
single phrase instead of breaking it into fragments.
- candidateWords: filters out standalone punctuation tokens (|, ·, •)
so they don't inflate word-count penalties for compound brand names.
- verbForElementRole: hyperlinks and links now produce "Select" instead
of "Click"; search box / search field produces "Search for".
Five new unit tests cover: browser title stripping, search query
extraction, full pipe-separated link text, link and search box verbs.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Remove the standalone "AI Rewrite" section from the editor panel.
The existing title and description AI buttons already flush the step
first, so they rewrite whatever the user has typed in those fields.
- Never pass a generic fallback title ("Screen capture", "Window
capture", "Region capture", "Capture") as the AI title candidate or
as step content. It is now treated as "(not set — generate a
specific action title)" so the AI always produces something real.
- hasRichCaptureContext now counts any non-trivial app name or window
title as sufficient context, instead of requiring non-browser noise.
- Prompt rules updated: "NEVER output Screen/Window/Region capture",
separate paths for improving a user draft vs generating from context,
and clearer guidance when context is limited (use app/window name).
- isPlaceholderTitle helper guards summarizeStepForAi so a
default-titled step presents itself as empty to the AI.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Store captureMetadata (OCR text, window/app/element info) with each
step at capture time so AI always has the original rich context.
- Add buildCaptureContext() to TextIntelService; capture.js uses it
instead of buildCaptureTitle() so both title and metadata come from
one pass.
- generateStepPatch() prefers stored captureMetadata over re-running
OCR, giving the AI the best possible context when the user clicks
an AI button later.
- Add autoDoc setting: when enabled every capture (shoot, region, and
session hotkey/click) is automatically documented by AI. Manual
captures await AI before returning; session captures fire-and-forget
and push a step:updated event so the renderer reloads seamlessly.
- Add ai:rewriteText IPC and rewriteText() method for plain-text
polishing via a separate callOllamaText() that skips JSON mode.
- Add "AI Rewrite" section in the editor right panel: textarea + AI
button that rewrites whatever the user types in place.
- Improve buildAiPrompt() rules: action-focused title instructions,
explicit anti-junk rules (no "Capture the screen / OCR" blocks),
and a context-quality gate that suppresses blocks when context is
thin.
- Add autoDoc checkbox to AI settings dialog.
- Renderer handles step:updated to reload the selected step after
background auto-doc finishes.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
TylerandClaude Fable 5