diff --git a/docs/06-AI拾取模型选型与RPA执行方案.md b/docs/06-AI拾取模型选型与RPA执行方案.md new file mode 100644 index 0000000..a9f2e85 --- /dev/null +++ b/docs/06-AI拾取模型选型与RPA执行方案.md @@ -0,0 +1,119 @@ +# AI 拾取模型选型与 RPA 执行方案 + +## 结论 + +当前模型“只能识别控件”不是解析器写错，而是 OmniParser 的 icon detect 本身就是单类别模型。它适合解决“屏幕上有哪些可交互区域”，不适合直接回答“这是一个登录按钮还是搜索框”。 + +更合适的不是换一个全能小模型，而是分四层： + +1. 视觉定位：YOLO/OmniParser 找 bbox，保证又快又能看到 UIA 看不到的区域。 +2. 视觉语义：OCR 读取框内文本；必要时用小分类模型或图标匹配补充类别。 +3. UIA 增强：用 AI bbox 中心回查 Windows UIA，拿到 `Name / ControlType / AutomationId / ClassName`。 +4. RPA 执行：优先保存 UIA selector，AI bbox 只作为兜底坐标和运行时重定位依据。 + +这才是对齐实在这类产品的方向：AI 负责感知和定位，RPA 引擎负责稳定执行和状态反馈。 + +## 当前可试模型 + +模型已经放在 `D:\code\PRA-only-demo\temp\models`。同名 `*.labels.json` 已经配好，当前实验台会读取标签，不再显示“类别1/类别2”。 + +| 本地文件 | 来源 | 类别数 | 类别 | 速度/体积 | 适用场景 | License 风险 | +|---|---|---:|---|---|---|---| +| `omniparser-v1-fp16.onnx` | `onnx-community/OmniParser-icon_detect` | 1 | `控件` | 约 6 MB，测试约 0.8s | 可交互区域粗定位、基线对比 | 商用前复核 OmniParser/底模授权 | +| `amigo-ui-elements-yolov8.onnx` | `amigodev/ui-elements-detector-test` | 8 | `button/field/heading/iframe/image/label/link/text` | 约 12 MB，测试约 0.4s | 网页截图、轻量多类别试验 | 仓库未标 License，仅试验 | +| `webpage-element.onnx` | `OpenDILabCommunity/webpage_element_detection` | 8 | `button/field/heading/iframe/image/label/link/text` | 约 99 MB，测试约 1.5s | 网页 UI，准确率通常优于小模型 | Apache-2.0 | +| `android-ui-yolov8n.onnx` | `yasirfaizahmed/android_ui_detection_yolov8` | 21 | `TextButton/EditText/CheckBox/Switch/Toolbar` 等 | 约 12 MB，测试约 0.4s | 安卓/移动 UI；桌面也可当轻量 UI 分类基线 | Apache-2.0 | +| `macpaw-ui-elements-yolo11l.onnx` | `macpaw-research/yolov11l-ui-elements-detection` | 5 | `AXButton/AXDisclosureTriangle/AXImage/AXLink/AXTextArea` | 约 102 MB，测试约 1.7s | 桌面/辅助功能导向，语义更接近可访问性控件 | AGPL-3.0，商用谨慎 | +| `resources/models/ui-element-detect/webpage-element.onnx` | OpenDILab webpage | 8 | `button/field/heading/iframe/image/label/link/text` | 当前默认，约 99 MB | 已实测并接入 Pane AI 拾取 | Apache-2.0 | +| `resources/models/ui-icon-detect/model.onnx` | OmniParser v2 icon detect | 1 | `控件` | 旧链路兼容回退 | 仅缺新模型时兜底 | 商用前复核 | + +测试方式：打开“AI 识别实验台”，选择识别目标，点“选择 ONNX”，选择上表中的 `.onnx` 文件。确保同名 `.labels.json` 放在同一目录。 + +## 实测观察 + +同一张简单网页风格测试图： + +- Android UI YOLOv8n：约 431ms，能识别 `TextButton`、`EditText`、`Text`，置信度普遍高。 +- Amigo UI YOLOv8：约 430ms，能识别 `button`、`field`，框更整齐。 +- OpenDILab webpage：约 1533ms，类别与 Amigo 相同，体积和耗时更大。 +- MacPaw YOLO11l：约 1743ms，能识别 `AXButton`、`AXTextArea`，但对该测试图置信度偏低。 +- OmniParser v1：约 781ms，只输出“控件”，定位能力可用但没有类别语义。 + +这组结果说明：轻量 YOLO 完全能做到实时/近实时，速度问题不是模型架构本身，而是模型大小、截图裁剪、进程通信和重复初始化共同决定的。 + +## 当前 RPA 可用性 + +当前 Pane AI 拾取生成： + +```text +screen=x,y,width,height;window=...;name=OCR文本或类别;type=视觉类别;aiType=视觉类别;aiText=OCR文本;ai=1 +``` + +`webpage-element.onnx` 先定位候选；PaddleOCR/PP-OCRv4 对同一张 Pane 截图做一次整图识别，并按几何关系把文本写回候选框。整图结果按截图 SHA-256 缓存，重复识别同一画面不会重复推理。执行层已经支持以 `screen=` 做坐标点击、坐标输入、坐标取值，所以“立刻执行一次”是可用的。 + +但只靠当前 selector 还不能算稳定 RPA 元素，原因是： + +- `screen=` 是拾取瞬间的坐标快照。 +- 窗口移动、页面滚动、分辨率/DPI 变化后会失效。 +- `exists/wait` 目前对 `screen=` 过于宽松。 +- `name/type` 很多时候只是视觉模型类别，不是业务语义。 +- 没有运行时重定位机制。 + +## 目标 selector 结构 + +建议逐步变成： + +```text +window=title:xxx;process=xxx; +type=Button;name=登录;id=loginButton;class=Button; +aiType=button;aiRect=x,y,w,h;aiModel=webpage-element; +template=sha1:...;ai=1 +``` + +定位策略： + +1. 优先按 `window + AutomationId + Name + ControlType` 查 UIA。 +2. UIA 查不到时，限制在原窗口/原 Pane 附近重找。 +3. 再失败时截图并运行视觉模型，在候选中按 `aiType + IoU + 外观相似度` 找回目标。 +4. 找回后更新真实 UIA rect，再执行动作。 + +## AI 类别到 RPA 动作映射 + +| AI/UIA 类别 | RPA 默认动作 | +|---|---| +| `button / AXButton / TextButton` | `Element_Click` | +| `field / EditText / AXTextArea / TextBox` | `Element_SetValue` / `Element_GetValue` | +| `link / AXLink` | `Element_Click` | +| `text / label / heading` | `Element_GetText` / OCR | +| `CheckBox / Switch` | `Element_SetCheck` / `Element_GetCheck` | +| `Spinner / ComboBox` | `Element_SetSelect` / 展开后点击 | +| `image / Icon` | 坐标点击 / 图标匹配 | +| `Toolbar / Navigation / Group` | 作为容器遍历子元素，不直接执行动作 | + +## 最好用的落地路线 + +### 第一阶段：可用 + +- 已完成：本地模型、Pane AI、候选框、悬浮确认。 +- 已完成：候选类别从 sidecar 读取，默认模型切换为 `webpage-element.onnx`。 +- 已完成：PaddleOCR 整图识别、候选框文本分配、OCR 结果缓存和异步回填。 +- 已完成：`F2/F4` 进入 AI；`Esc/F9` 取消 AI；`F8` 切换开发候选框。 +- 补 UIA 回查：选中框后用中心点/范围找最小 UIA 元素，生成稳定 selector。 + +### 第二阶段：稳定 + +- `exists/wait` 真正支持 `screen= + window + aiRect`。 +- 执行前校验窗口和元素仍在预期区域。 +- 增加 AI 重定位：截图 → 检测 → 与保存的 bbox/裁剪图匹配 → 更新坐标。 +- 所有执行结果回写目标快照，方便排查框偏。 + +### 第三阶段：接近实在 + +- 元素检测、元素分类、OCR、图标相似、UISelector 组合。 +- AI 只输出结构化 target/action，不直接控鼠标键盘。 +- RPA 执行器负责窗口激活、等待、重试、截图、异常恢复。 +- 对复杂远程桌面/Canvas/Java 应用保留视觉兜底。 + +## 推荐优先级 + +已选定 `webpage-element.onnx` 作为 Pane AI 默认模型。其他模型继续留在本地 `temp/models` 供实验台对比，不再进入默认链路和安装包。内置 OmniParser 只作为老安装包兼容回退，不建议作为最终业务 selector 的唯一来源。 diff --git a/electron-builder.config.mjs b/electron-builder.config.mjs index c666aed..9b4f704 100644 --- a/electron-builder.config.mjs +++ b/electron-builder.config.mjs @@ -25,7 +25,7 @@ export default { { from: path.join(root, "resources", "excluded.json"), to: "excluded.json" }, { from: path.join(root, "resources", "capability-duplicates.json"), to: "capability-duplicates.json" }, { from: path.join(root, "resources", "desktop-picker.ps1"), to: "desktop-picker.ps1" }, - { from: path.join(root, "resources", "models"), to: "models" }, + { from: path.join(root, "resources", "models", "ui-element-detect"), to: "models/ui-element-detect" }, { from: path.join(root, "resources", "fixture"), to: "fixture" } ], asar: true, diff --git a/resources/desktop-picker.ps1 b/resources/desktop-picker.ps1 index 4f45246..5878134 100644 --- a/resources/desktop-picker.ps1 +++ b/resources/desktop-picker.ps1 @@ -192,6 +192,7 @@ $prevSpace = $false $prevPgUp = $false $prevPgDn = $false $prevF2 = $false +$prevF4 = $false $cacheX = [int]::MinValue $cacheY = [int]::MinValue $cacheEl = $null @@ -213,6 +214,7 @@ while ($true) { $pgUp = (([RpaPick.PickW32]::GetAsyncKeyState(33) -band 0x8001) -ne 0) $pgDn = (([RpaPick.PickW32]::GetAsyncKeyState(34) -band 0x8001) -ne 0) $f2 = (([RpaPick.PickW32]::GetAsyncKeyState(113) -band 0x8001) -ne 0) + $f4 = (([RpaPick.PickW32]::GetAsyncKeyState(115) -band 0x8001) -ne 0) if ($esc) { Write-PickLog "esc"; break } if ($stage -eq "hover") { $ptX = [System.Windows.Forms.Cursor]::Position.X @@ -256,11 +258,12 @@ while ($true) { $el = $cacheEl if ($el) { Set-HighlightBox $cachePaneX $cachePaneY $cachePaneW $cachePaneH - $aiHint = if ($cacheCtrlName -eq "Pane") { " [F2]AI识别" } else { "" } + $aiHint = if ($cacheCtrlName -eq "Pane") { " [F2/F4]AI识别" } else { "" } $infoLabel.Text = "名称: " + $cacheName + " 类型: " + $cacheCtrlName + " [空格]锁定" + $aiHint + " [Esc]取消" } else { Hide-Highlight } - if ($f2 -and -not $prevF2 -and $el -and $cacheCtrlName -eq "Pane") { - Write-PickLog "f2 ai" + $aiTrigger = ($f2 -and -not $prevF2) -or ($f4 -and -not $prevF4) + if ($aiTrigger -and $el -and $cacheCtrlName -eq "Pane") { + Write-PickLog "f2/f4 ai" $walker = [System.Windows.Automation.TreeWalker]::ControlViewWalker $cur = $el $topTitle = "" @@ -312,6 +315,7 @@ while ($true) { $prevPgUp = $pgUp $prevPgDn = $pgDn $prevF2 = $f2 + $prevF4 = $f4 Start-Sleep -Milliseconds 60 } $info.Close() diff --git a/resources/models/ui-element-detect/webpage-element.labels.json b/resources/models/ui-element-detect/webpage-element.labels.json new file mode 100644 index 0000000..625736f --- /dev/null +++ b/resources/models/ui-element-detect/webpage-element.labels.json @@ -0,0 +1 @@ +["button","field","heading","iframe","image","label","link","text"] diff --git a/resources/models/ui-element-detect/webpage-element.onnx b/resources/models/ui-element-detect/webpage-element.onnx new file mode 100644 index 0000000..23485fc Binary files /dev/null and b/resources/models/ui-element-detect/webpage-element.onnx differ diff --git a/src/main/desktop-pick.ts b/src/main/desktop-pick.ts index 4a5c516..08a56d3 100644 --- a/src/main/desktop-pick.ts +++ b/src/main/desktop-pick.ts @@ -6,6 +6,7 @@ import path from "node:path"; import { app, BrowserWindow, desktopCapturer, globalShortcut, screen } from "electron"; import { RuntimeError } from "./engine/types"; import { detectUiElements, localUiModelPath } from "./local-vision/ui-detect-service"; +import { assignPaneOcr, recognizePane } from "./local-vision/pane-ocr-service"; const run = promisify(execFile); @@ -67,6 +68,9 @@ interface AiCandidate { name: string; controlType: string; confidence: number; + ocrText?: string; + ocrState?: "pending" | "done" | "empty" | "error"; + ocrConfidence?: number; } interface AiCaptureOutput { @@ -202,31 +206,53 @@ function logHoverState(state){ window.__toggleDebug=function(){ document.body.classList.toggle('debug'); var debug=document.body.classList.contains('debug'); - showStatus(debug?'开发模式：F8 关闭候选框':(window.__candidatesReady?'AI 完成：移动高亮；单击选中，再次单击或空格回填；Esc 取消':'开发模式已开启，识别完成后显示全部候选框')); + showStatus(debug?'开发模式：F8 关闭候选框':(window.__candidatesReady?'AI 完成：移动高亮；单击选中，再次单击或空格回填；Esc/F9 取消':'开发模式已开启，识别完成后显示全部候选框')); }; +function candidateLabel(c){ + var type=c.controlType||'控件'; + if(c.ocrState==='pending')return 'AI识别 · '+type; + var text=String(c.ocrText||'').trim(); + if(text.length>32)text=text.slice(0,32)+'…'; + return (text?text+' · ':'AI识别 · ')+type; +} +function refreshBadge(cid){ + var c=window.__candidateMap[cid];if(!c)return; + var box=document.querySelector('.candidate[data-cid="'+cid+'"]');if(!box)return; + var label=document.getElementById('badge-'+box.dataset.index);if(!label)return; + label.textContent=candidateLabel(c); + if(label.classList.contains('active')&&window.__selectedCid===cid)showStatus('已选中：'+candidateLabel(c)+'；再次单击或空格回填'); +} +function applyOcrUpdates(items){ + if(!window.__candidateMap||!Array.isArray(items))return; + items.forEach(function(item){ + var c=window.__candidateMap[item.candidateId];if(!c)return; + c.ocrText=item.ocrText||'';c.ocrState=item.ocrState||'done';c.ocrConfidence=item.ocrConfidence; + refreshBadge(c.candidateId); + }); +} function showCandidates(items){ paneW=window.__paneW||document.documentElement.clientWidth;paneH=window.__paneH||document.documentElement.clientHeight; document.getElementById('scan').style.display='none'; document.getElementById('aiFrame').style.display='none'; document.body.classList.remove('scanning');document.body.style.cursor='pointer'; - window.__candidateMap={};window.__candidatesReady=false;clearActive(); + window.__candidateMap={};window.__candidatesReady=false;window.__ocrReady=false;clearActive(); items.sort(function(a,b){return a.y-b.y||a.x-b.x}).forEach(function(c,index){ window.__candidateMap[c.candidateId]=c; var box=document.createElement('div');box.className='candidate';box.dataset.cid=c.candidateId;box.dataset.index=index; box.style.left=(c.x/paneW*100)+'%';box.style.top=(c.y/paneH*100)+'%'; box.style.width=(c.width/paneW*100)+'%';box.style.height=(c.height/paneH*100)+'%'; - var label=document.createElement('div');label.className='badge';label.id='badge-'+index;label.textContent='AI识别 · '+(c.controlType||'控件'); + var label=document.createElement('div');label.className='badge';label.id='badge-'+index;label.textContent=candidateLabel(c); label.style.left=(c.x/paneW*100)+'%';label.style.top=(Math.max(0,c.y-24)/paneH*100)+'%'; box.addEventListener('click',function(event){ event.stopPropagation();if(Date.now() { - // 本地小模型是 Pane AI 的主链路；这里不要求用户配置远程 VLM。 + // 本地小模型是 Pane AI 的主链路；OCR 使用同一张 Pane 截图，不要求用户配置远程 VLM。 const localModelPath = localUiModelPath(); console.log("[pane-ai] local model", localModelPath); const pane = paneAiRect(context); const win = await createPaneAiOverlay(pane); const startedAt = Date.now(); - // F8 用全局快捷键，避免键盘焦点停在目标应用时 overlay 收不到按键。 - const debugHotkeyRegistered = globalShortcut.register("F8", () => { + + // F8 是开发框调试；Esc/F9 是兜底取消。全局注册能解决键盘焦点留在目标应用时 overlay 收不到按键的问题。 + const registeredHotkeys: string[] = []; + const registerHotkey = (accelerator: string, handler: () => void) => { + try { + if (globalShortcut.register(accelerator, handler)) registeredHotkeys.push(accelerator); + } catch (error) { + console.warn("[pane-ai] hotkey register failed", accelerator, error); + } + }; + const requestCancel = () => { + if (win.isDestroyed()) return; + void win.webContents.executeJavaScript("window.__cancelled=true;window.__done=true;", true).catch(() => {}); + }; + registerHotkey("Escape", requestCancel); + registerHotkey("F9", requestCancel); + registerHotkey("F8", () => { + if (win.isDestroyed()) return; void win.webContents.executeJavaScript("window.__toggleDebug && window.__toggleDebug()", true).catch(() => {}); }); try { // 稍等 overlay 完成首轮绘制，但不再等一整套“表演动画”。 - await new Promise(resolve => setTimeout(resolve, 350)); + await new Promise(resolve => setTimeout(resolve, 250)); const detection = capturePaneForAi(pane, win) .then(async capture => { const image = await fs.promises.readFile(capture.imagePath); const inferenceStarted = Date.now(); + // 同一张截图只跑一次整图 OCR；候选框检测先出结果，OCR 完成后异步回填。 + const ocrTask = recognizePane({ imagePath: capture.imagePath, image }) + .catch(error => { + console.warn("[pane-ai] pane ocr failed", error); + return null; + }); const boxes = await detectUiElements({ image, width: capture.cropW, height: capture.cropH }); console.log("[pane-ai] local inference", boxes.length, Date.now() - inferenceStarted + "ms"); - return { capture, boxes }; + return { capture, boxes, ocrTask }; }); let settled = false; const tickStatus = async () => { const elapsed = Date.now() - startedAt; - const label = elapsed < 2600 - ? "AI 识别中…" - : elapsed < 6000 - ? "AI 理解中…" - : "正在整理候选区域…"; + const label = elapsed < 2600 ? "AI 识别中…" : elapsed < 6000 ? "AI 理解中…" : "正在整理候选区域…"; + if (win.isDestroyed()) return; try { await win.webContents.executeJavaScript("showStatus(" + JSON.stringify(label) + ");", true); } catch { @@ -390,6 +435,7 @@ async function runPaneAiPick(context: UiaPickOutput): Promise const cancelWatcher = (async () => { while (!settled) { try { + if (win.isDestroyed()) throw new RuntimeError("pick_cancelled", "AI 拾取窗口已关闭"); const state = await win.webContents.executeJavaScript("JSON.stringify({cancelled:window.__cancelled===true})", true); if (JSON.parse(state).cancelled) throw new RuntimeError("pick_cancelled", "已取消 AI 拾取"); } catch (error) { @@ -399,9 +445,13 @@ async function runPaneAiPick(context: UiaPickOutput): Promise } })(); - let detected: { capture: AiCaptureOutput; boxes: Awaited> }; + let detected: { + capture: AiCaptureOutput; + boxes: Awaited>; + ocrTask: Promise> | null>; + }; try { - detected = await Promise.race([detection, cancelWatcher]) as { capture: AiCaptureOutput; boxes: Awaited> }; + detected = await Promise.race([detection, cancelWatcher]) as typeof detected; } finally { settled = true; clearInterval(statusTimer); @@ -415,13 +465,16 @@ async function runPaneAiPick(context: UiaPickOutput): Promise height: box.height, name: box.category, controlType: box.category, - confidence: box.confidence + confidence: box.confidence, + ocrText: "", + ocrState: "pending" })); if (!candidates.length) { throw new RuntimeError("ai_response_invalid", "本地 UI 模型没有识别到可用候选区域"); } const payload = JSON.stringify(candidates); + if (win.isDestroyed()) throw new RuntimeError("pick_cancelled", "AI 拾取窗口已关闭"); await win.show(); win.focus(); await win.webContents.executeJavaScript( @@ -429,10 +482,46 @@ async function runPaneAiPick(context: UiaPickOutput): Promise true ); + // OCR 不阻塞候选框出现；完成后按框分配、缓存到候选对象，并刷新悬浮标签。 + void detected.ocrTask + .then(async paneOcr => { + if (!paneOcr || win.isDestroyed()) return; + const states = assignPaneOcr(paneOcr.lines, candidates); + for (const candidate of candidates) { + const state = states[candidate.candidateId]; + if (!state) continue; + candidate.ocrText = state.ocrText; + candidate.ocrState = state.ocrState; + candidate.ocrConfidence = state.ocrConfidence; + if (state.ocrText) candidate.name = state.ocrText; + } + console.log("[pane-ai] pane ocr", paneOcr.lines.length, paneOcr.elapsedMs + "ms", paneOcr.cacheHit ? "cache" : "inference"); + const ocrPayload = JSON.stringify(candidates.map(candidate => ({ + candidateId: candidate.candidateId, + ocrText: candidate.ocrText, + ocrState: candidate.ocrState, + ocrConfidence: candidate.ocrConfidence + }))); + await win.webContents.executeJavaScript("applyOcrUpdates(" + ocrPayload + ");", true); + await win.webContents.executeJavaScript( + "if(!window.__done)showStatus('OCR 完成：移动高亮；单击选中，再次单击或空格回填；Esc/F9 取消');", + true + ); + }) + .catch(error => { + console.warn("[pane-ai] ocr update failed", error); + if (win.isDestroyed()) return; + void win.webContents.executeJavaScript( + "applyOcrUpdates(" + JSON.stringify(candidates.map(candidate => ({ candidateId: candidate.candidateId, ocrText: "", ocrState: "error" }))) + ");", + true + ).catch(() => {}); + }); + const deadline = Date.now() + 180000; let selectedId: string | null = null; while (Date.now() < deadline) { await new Promise(resolve => setTimeout(resolve, 80)); + if (win.isDestroyed()) throw new RuntimeError("pick_cancelled", "已取消 AI 拾取"); const state = await win.webContents.executeJavaScript( "JSON.stringify({done:window.__done===true,cancelled:window.__cancelled===true,selectedCid:window.__selectedCid||null})", true @@ -450,20 +539,25 @@ async function runPaneAiPick(context: UiaPickOutput): Promise } if (!selectedId) throw new RuntimeError("pick_cancelled", "AI 拾取超时"); + // 用户确认时 OCR 可能还差几百毫秒。等本轮任务收敛，避免回填名和最终 selector 不一致。 + await detected.ocrTask; + const box = candidates.find(item => item.candidateId === selectedId)!; const x = detected.capture.cropX + box.x; const y = detected.capture.cropY + box.y; const selectorParts = [ "screen=" + [x, y, box.width, box.height].map(Math.round).join(","), context.window ? "window=" + safeSelectorText(context.window) : "", - "name=" + safeSelectorText(box.name), + "name=" + safeSelectorText(box.name || box.ocrText || box.controlType), "type=" + safeSelectorText(box.controlType), + "aiType=" + safeSelectorText(box.controlType), + box.ocrText ? "aiText=" + safeSelectorText(box.ocrText) : "", "ai=1" ].filter(Boolean); return { source: "ai", window: context.window, - name: box.name, + name: box.name || box.ocrText || box.controlType, controlType: box.controlType, x, y, w: box.width, h: box.height, confidence: box.confidence, @@ -471,7 +565,10 @@ async function runPaneAiPick(context: UiaPickOutput): Promise selector: selectorParts.join(";") }; } finally { - if (debugHotkeyRegistered) globalShortcut.unregister("F8"); + // 先注册的后注册的都要清理；Esc 失败时 F9 仍会兜底。 + for (const accelerator of [...registeredHotkeys].reverse()) { + try { globalShortcut.unregister(accelerator); } catch {} + } if (!win.isDestroyed()) win.destroy(); } } diff --git a/src/main/local-vision/pane-ocr-service.test.ts b/src/main/local-vision/pane-ocr-service.test.ts new file mode 100644 index 0000000..2ef4bed --- /dev/null +++ b/src/main/local-vision/pane-ocr-service.test.ts @@ -0,0 +1,33 @@ +import { describe, expect, it } from "vitest"; +import { assignPaneOcr, type PaneOcrLine } from "./pane-ocr-service"; + +describe("pane OCR assignment", () => { + it("assigns OCR lines to the boxes containing their center", () => { + const lines: PaneOcrLine[] = [ + { text: "登录", confidence: 0.9, x: 12, y: 10, width: 24, height: 12 } + ]; + const boxes = [ + { candidateId: "button", x: 8, y: 6, width: 48, height: 24 }, + { candidateId: "container", x: 0, y: 0, width: 120, height: 80 } + ]; + expect(assignPaneOcr(lines, boxes)).toEqual({ + button: { ocrText: "登录", ocrState: "done", ocrConfidence: 0.9 }, + container: { ocrText: "登录", ocrState: "done", ocrConfidence: 0.9 } + }); + }); + + it("uses a strongly overlapping line even when its center falls outside", () => { + const lines: PaneOcrLine[] = [ + { text: "保存", confidence: 0.8, x: 0, y: 10, width: 30, height: 12 } + ]; + const boxes = [{ candidateId: "button", x: 8, y: 8, width: 30, height: 16 }]; + const result = assignPaneOcr(lines, boxes); + expect(result.button.ocrText).toBe("保存"); + expect(result.button.ocrState).toBe("done"); + }); + + it("marks boxes without text as empty", () => { + const result = assignPaneOcr([], [{ candidateId: "image", x: 0, y: 0, width: 20, height: 20 }]); + expect(result.image).toEqual({ ocrText: "", ocrState: "empty" }); + }); +}); diff --git a/src/main/local-vision/pane-ocr-service.ts b/src/main/local-vision/pane-ocr-service.ts new file mode 100644 index 0000000..c3e29d9 --- /dev/null +++ b/src/main/local-vision/pane-ocr-service.ts @@ -0,0 +1,177 @@ +import { createHash } from "node:crypto"; + +export interface PaneOcrLine { + text: string; + confidence: number; + x: number; + y: number; + width: number; + height: number; +} + +export interface PaneOcrResult { + lines: PaneOcrLine[]; + elapsedMs: number; + cacheHit: boolean; +} + +export interface CandidateOcrState { + ocrText: string; + ocrState: "pending" | "done" | "empty" | "error"; + ocrConfidence?: number; +} + +interface GutenLine { + text?: unknown; + mean?: unknown; + score?: unknown; + box?: unknown; +} + +interface GutenOcr { + detect: (image: string) => Promise<{ texts?: GutenLine[] }>; +} + +type GutenOcrModule = { + default: { + create: (options?: unknown) => Promise; + }; +}; + +let ocrPromise: Promise | null = null; +const paneCache = new Map>(); +const PANE_CACHE_LIMIT = 32; + +async function getOcr(): Promise { + if (!ocrPromise) { + ocrPromise = import("@gutenye/ocr-node") + .then(async mod => await (mod as unknown as GutenOcrModule).default.create()) + .catch(error => { + ocrPromise = null; + throw error; + }); + } + return await ocrPromise; +} + +function clamp01(value: unknown): number { + const next = Number(value); + return Number.isFinite(next) ? Math.max(0, Math.min(1, next)) : 0; +} + +function lineRect(line: GutenLine): PaneOcrLine { + const points = Array.isArray(line.box) ? line.box : []; + const xs = points + .map(point => Array.isArray(point) ? Number(point[0]) : Number.NaN) + .filter(value => Number.isFinite(value)); + const ys = points + .map(point => Array.isArray(point) ? Number(point[1]) : Number.NaN) + .filter(value => Number.isFinite(value)); + const x = xs.length ? Math.min(...xs) : 0; + const y = ys.length ? Math.min(...ys) : 0; + const maxX = xs.length ? Math.max(...xs) : x; + const maxY = ys.length ? Math.max(...ys) : y; + return { + text: String(line.text || "").replace(/\s+/g, " ").trim(), + confidence: clamp01(line.mean ?? line.score ?? 0), + x, + y, + width: Math.max(0, maxX - x), + height: Math.max(0, maxY - y) + }; +} + +/** + * 每张 Pane 截图只做一次整图 OCR。候选框只是画面上的矩形， + * 整图结果按几何关系分配给候选框，天然形成缓存并避免每个框重复推理。 + */ +export async function recognizePane(input: { + imagePath: string; + image: Buffer; +}): Promise { + const imageHash = createHash("sha256").update(input.image).digest("hex"); + const cached = paneCache.get(imageHash); + if (cached) { + const result = await cached; + return { ...result, cacheHit: true }; + } + + const task = (async (): Promise => { + const started = Date.now(); + try { + const ocr = await getOcr(); + const response = await ocr.detect(input.imagePath); + const lines = (response.texts || []) + .map(lineRect) + .filter(line => line.text.length > 0); + return { lines, elapsedMs: Date.now() - started, cacheHit: false }; + } catch (error) { + // 初始化或识别失败时清掉缓存任务，下一次悬浮仍有机会恢复。 + paneCache.delete(imageHash); + throw error; + } + })(); + + paneCache.set(imageHash, task); + while (paneCache.size > PANE_CACHE_LIMIT) { + const oldest = paneCache.keys().next().value; + if (!oldest) break; + paneCache.delete(oldest); + } + return await task; +} + +export function clearPaneOcrCache(): void { + paneCache.clear(); +} + +interface AssignableBox { + candidateId: string; + x: number; + y: number; + width: number; + height: number; +} + +function centerInsideBox(line: PaneOcrLine, box: AssignableBox): boolean { + const cx = line.x + line.width / 2; + const cy = line.y + line.height / 2; + return cx >= box.x && cx <= box.x + box.width && cy >= box.y && cy <= box.y + box.height; +} + +function overlapRatio(line: PaneOcrLine, box: AssignableBox): number { + const overlapX = Math.max(0, Math.min(line.x + line.width, box.x + box.width) - Math.max(line.x, box.x)); + const overlapY = Math.max(0, Math.min(line.y + line.height, box.y + box.height) - Math.max(line.y, box.y)); + const lineArea = Math.max(1, line.width * line.height); + return (overlapX * overlapY) / lineArea; +} + +export function assignPaneOcr( + lines: PaneOcrLine[], + boxes: AssignableBox[] +): Record { + const result: Record = {}; + for (const box of boxes) { + const matched = lines + .map(line => ({ + line, + inside: centerInsideBox(line, box), + ratio: overlapRatio(line, box) + })) + .filter(item => item.inside || item.ratio >= 0.55) + .sort((a, b) => Number(b.inside) - Number(a.inside) || b.ratio - a.ratio || a.line.y - b.line.y || a.line.x - b.line.x); + + if (!matched.length) { + result[box.candidateId] = { ocrText: "", ocrState: "empty" }; + continue; + } + const text = matched.map(item => item.line.text).join(" ").replace(/\s+/g, " ").trim(); + const confidence = matched.reduce((sum, item) => sum + item.line.confidence, 0) / matched.length; + result[box.candidateId] = { + ocrText: text, + ocrState: text ? "done" : "empty", + ocrConfidence: Math.round(confidence * 1000) / 1000 + }; + } + return result; +} diff --git a/src/main/local-vision/ui-detect-service.ts b/src/main/local-vision/ui-detect-service.ts index 518023f..0a7495a 100644 --- a/src/main/local-vision/ui-detect-service.ts +++ b/src/main/local-vision/ui-detect-service.ts @@ -36,19 +36,28 @@ let nextRequestId = 1; const MAX_LOADED_WORKERS = 4; export function localUiModelPath(): string { - const candidates = [ - path.join(process.resourcesPath || "", "models", "ui-icon-detect", "model.onnx"), - path.join(app.getAppPath(), "resources", "models", "ui-icon-detect", "model.onnx") + const rootCandidates = [ + process.resourcesPath, + app.getAppPath() ? path.join(app.getAppPath(), "resources") : "", + path.join(app.getAppPath(), "temp", "models") ].filter(Boolean); - const found = candidates.find(candidate => fs.existsSync(candidate)); + // webpage-element 是实测后的首选模型；保留旧模型作为老安装包的兼容回退。 + const fileCandidates = rootCandidates.flatMap(root => [ + path.join(root, "models", "ui-element-detect", "webpage-element.onnx"), + path.join(root, "webpage-element.onnx"), + path.join(root, "models", "ui-icon-detect", "model.onnx") + ]); + const found = fileCandidates.find(candidate => { + try { return fs.existsSync(candidate) && fs.statSync(candidate).isFile(); } + catch { return false; } + }); if (!found) { throw new RuntimeError("local_vision_model_missing", "本地 UI 检测模型缺失", { - recommendation: "重新安装应用，或补齐 resources/models/ui-icon-detect/model.onnx" + recommendation: "重新安装应用，或补齐 resources/models/ui-element-detect/webpage-element.onnx" }); } return found; } - export function defaultUiModelPath(): string { return localUiModelPath(); } @@ -142,4 +151,3 @@ export function closeLocalVisionWorkers(): void { } pending.clear(); } - diff --git a/src/main/local-vision/ui-detect-worker.ts b/src/main/local-vision/ui-detect-worker.ts index 3c6c369..6c8e93a 100644 --- a/src/main/local-vision/ui-detect-worker.ts +++ b/src/main/local-vision/ui-detect-worker.ts @@ -1,3 +1,4 @@ +import fs from "node:fs"; import { parentPort, workerData } from "node:worker_threads"; import { InferenceSession, Tensor } from "onnxruntime-node"; import { createCanvas, loadImage } from "@napi-rs/canvas"; @@ -37,6 +38,9 @@ const defaultScoreThreshold = 0.25; const defaultIouThreshold = 0.42; const maxBoxes = 160; +const startupModelPath = String((workerData as { modelPath?: string }).modelPath || ""); +const modelLabels = readModelLabels(startupModelPath); + let sessionPromise: Promise | null = null; function loadSession(): Promise { @@ -76,6 +80,24 @@ function tensorAccessor(output: TensorLike, anchorCount: number, channelCount: n : output.data[anchor * channelCount + channel]; } +// YOLO 导出的 ONNX 通常不含标签元数据。约定同名 *.labels.json 保存类别： +// 1) ["button", "input"] 2) { "labels": ["button", "input"] } +function readModelLabels(modelPath: string): string[] | undefined { + const sidecarPath = modelPath.replace(/\.onnx$/i, ".labels.json"); + try { + const parsed: unknown = JSON.parse(fs.readFileSync(sidecarPath, "utf8")); + const values = Array.isArray(parsed) ? parsed : (parsed as { labels?: unknown })?.labels; + if (!Array.isArray(values)) return undefined; + const labels = values + .filter((item): item is string | number => typeof item === "string" || typeof item === "number") + .map(item => String(item)) + .filter(Boolean); + return labels.length ? labels : undefined; + } catch { + return undefined; + } +} + function extractBoxes( output: TensorLike, sourceWidth: number, @@ -100,6 +122,7 @@ function extractBoxes( const offsetX = (inputSize - drawnWidth) / 2; const offsetY = (inputSize - drawnHeight) / 2; const minSide = Math.max(8, Math.min(sourceWidth, sourceHeight) * 0.006); + const labels = modelLabels; const raw: LocalUiBox[] = []; for (let index = 0; index < anchorCount; index++) { @@ -137,7 +160,8 @@ function extractBoxes( y: Math.round(y), width: Math.round(boxWidth), height: Math.round(boxHeight), - category: classCount <= 1 ? "控件" : `类别${classIndex + 1}`, + category: labels?.[classIndex] + ?? (classCount <= 1 ? "控件" : `类别${classIndex + 1}`), confidence: Math.round(Math.max(0, Math.min(1, confidence)) * 1000) / 1000 }); }