feat: enhance AI pane picking with OCR and cancellable hotkeys
This commit is contained in:
yu
parent
0132871885
commit
1ddd1f1b35
119
docs/06-AI拾取模型选型与RPA执行方案.md
Normal file
119
docs/06-AI拾取模型选型与RPA执行方案.md
Normal file
@ -0,0 +1,119 @@
|
||||
# AI 拾取模型选型与 RPA 执行方案
|
||||
|
||||
## 结论
|
||||
|
||||
当前模型“只能识别控件”不是解析器写错，而是 OmniParser 的 icon detect 本身就是单类别模型。它适合解决“屏幕上有哪些可交互区域”，不适合直接回答“这是一个登录按钮还是搜索框”。
|
||||
|
||||
更合适的不是换一个全能小模型，而是分四层：
|
||||
|
||||
1. 视觉定位：YOLO/OmniParser 找 bbox，保证又快又能看到 UIA 看不到的区域。
|
||||
2. 视觉语义：OCR 读取框内文本；必要时用小分类模型或图标匹配补充类别。
|
||||
3. UIA 增强：用 AI bbox 中心回查 Windows UIA，拿到 `Name / ControlType / AutomationId / ClassName`。
|
||||
4. RPA 执行：优先保存 UIA selector，AI bbox 只作为兜底坐标和运行时重定位依据。
|
||||
|
||||
这才是对齐实在这类产品的方向：AI 负责感知和定位，RPA 引擎负责稳定执行和状态反馈。
|
||||
|
||||
## 当前可试模型
|
||||
|
||||
模型已经放在 `D:\code\PRA-only-demo\temp\models`。同名 `*.labels.json` 已经配好，当前实验台会读取标签，不再显示“类别1/类别2”。
|
||||
|
||||
| 本地文件 | 来源 | 类别数 | 类别 | 速度/体积 | 适用场景 | License 风险 |
|
||||
|---|---|---:|---|---|---|---|
|
||||
| `omniparser-v1-fp16.onnx` | `onnx-community/OmniParser-icon_detect` | 1 | `控件` | 约 6 MB，测试约 0.8s | 可交互区域粗定位、基线对比 | 商用前复核 OmniParser/底模授权 |
|
||||
| `amigo-ui-elements-yolov8.onnx` | `amigodev/ui-elements-detector-test` | 8 | `button/field/heading/iframe/image/label/link/text` | 约 12 MB，测试约 0.4s | 网页截图、轻量多类别试验 | 仓库未标 License，仅试验 |
|
||||
| `webpage-element.onnx` | `OpenDILabCommunity/webpage_element_detection` | 8 | `button/field/heading/iframe/image/label/link/text` | 约 99 MB，测试约 1.5s | 网页 UI，准确率通常优于小模型 | Apache-2.0 |
|
||||
| `android-ui-yolov8n.onnx` | `yasirfaizahmed/android_ui_detection_yolov8` | 21 | `TextButton/EditText/CheckBox/Switch/Toolbar` 等 | 约 12 MB，测试约 0.4s | 安卓/移动 UI；桌面也可当轻量 UI 分类基线 | Apache-2.0 |
|
||||
| `macpaw-ui-elements-yolo11l.onnx` | `macpaw-research/yolov11l-ui-elements-detection` | 5 | `AXButton/AXDisclosureTriangle/AXImage/AXLink/AXTextArea` | 约 102 MB，测试约 1.7s | 桌面/辅助功能导向，语义更接近可访问性控件 | AGPL-3.0，商用谨慎 |
|
||||
| `resources/models/ui-element-detect/webpage-element.onnx` | OpenDILab webpage | 8 | `button/field/heading/iframe/image/label/link/text` | 当前默认，约 99 MB | 已实测并接入 Pane AI 拾取 | Apache-2.0 |
|
||||
| `resources/models/ui-icon-detect/model.onnx` | OmniParser v2 icon detect | 1 | `控件` | 旧链路兼容回退 | 仅缺新模型时兜底 | 商用前复核 |
|
||||
|
||||
测试方式：打开“AI 识别实验台”，选择识别目标，点“选择 ONNX”，选择上表中的 `.onnx` 文件。确保同名 `.labels.json` 放在同一目录。
|
||||
|
||||
## 实测观察
|
||||
|
||||
同一张简单网页风格测试图：
|
||||
|
||||
- Android UI YOLOv8n：约 431ms，能识别 `TextButton`、`EditText`、`Text`，置信度普遍高。
|
||||
- Amigo UI YOLOv8：约 430ms，能识别 `button`、`field`，框更整齐。
|
||||
- OpenDILab webpage：约 1533ms，类别与 Amigo 相同，体积和耗时更大。
|
||||
- MacPaw YOLO11l：约 1743ms，能识别 `AXButton`、`AXTextArea`，但对该测试图置信度偏低。
|
||||
- OmniParser v1：约 781ms，只输出“控件”，定位能力可用但没有类别语义。
|
||||
|
||||
这组结果说明：轻量 YOLO 完全能做到实时/近实时，速度问题不是模型架构本身，而是模型大小、截图裁剪、进程通信和重复初始化共同决定的。
|
||||
|
||||
## 当前 RPA 可用性
|
||||
|
||||
当前 Pane AI 拾取生成：
|
||||
|
||||
```text
|
||||
screen=x,y,width,height;window=...;name=OCR文本或类别;type=视觉类别;aiType=视觉类别;aiText=OCR文本;ai=1
|
||||
```
|
||||
|
||||
`webpage-element.onnx` 先定位候选；PaddleOCR/PP-OCRv4 对同一张 Pane 截图做一次整图识别，并按几何关系把文本写回候选框。整图结果按截图 SHA-256 缓存，重复识别同一画面不会重复推理。执行层已经支持以 `screen=` 做坐标点击、坐标输入、坐标取值，所以“立刻执行一次”是可用的。
|
||||
|
||||
但只靠当前 selector 还不能算稳定 RPA 元素，原因是：
|
||||
|
||||
- `screen=` 是拾取瞬间的坐标快照。
|
||||
- 窗口移动、页面滚动、分辨率/DPI 变化后会失效。
|
||||
- `exists/wait` 目前对 `screen=` 过于宽松。
|
||||
- `name/type` 很多时候只是视觉模型类别，不是业务语义。
|
||||
- 没有运行时重定位机制。
|
||||
|
||||
## 目标 selector 结构
|
||||
|
||||
建议逐步变成：
|
||||
|
||||
```text
|
||||
window=title:xxx;process=xxx;
|
||||
type=Button;name=登录;id=loginButton;class=Button;
|
||||
aiType=button;aiRect=x,y,w,h;aiModel=webpage-element;
|
||||
template=sha1:...;ai=1
|
||||
```
|
||||
|
||||
定位策略：
|
||||
|
||||
1. 优先按 `window + AutomationId + Name + ControlType` 查 UIA。
|
||||
2. UIA 查不到时，限制在原窗口/原 Pane 附近重找。
|
||||
3. 再失败时截图并运行视觉模型，在候选中按 `aiType + IoU + 外观相似度` 找回目标。
|
||||
4. 找回后更新真实 UIA rect，再执行动作。
|
||||
|
||||
## AI 类别到 RPA 动作映射
|
||||
|
||||
| AI/UIA 类别 | RPA 默认动作 |
|
||||
|---|---|
|
||||
| `button / AXButton / TextButton` | `Element_Click` |
|
||||
| `field / EditText / AXTextArea / TextBox` | `Element_SetValue` / `Element_GetValue` |
|
||||
| `link / AXLink` | `Element_Click` |
|
||||
| `text / label / heading` | `Element_GetText` / OCR |
|
||||
| `CheckBox / Switch` | `Element_SetCheck` / `Element_GetCheck` |
|
||||
| `Spinner / ComboBox` | `Element_SetSelect` / 展开后点击 |
|
||||
| `image / Icon` | 坐标点击 / 图标匹配 |
|
||||
| `Toolbar / Navigation / Group` | 作为容器遍历子元素，不直接执行动作 |
|
||||
|
||||
## 最好用的落地路线
|
||||
|
||||
### 第一阶段：可用
|
||||
|
||||
- 已完成：本地模型、Pane AI、候选框、悬浮确认。
|
||||
- 已完成：候选类别从 sidecar 读取，默认模型切换为 `webpage-element.onnx`。
|
||||
- 已完成：PaddleOCR 整图识别、候选框文本分配、OCR 结果缓存和异步回填。
|
||||
- 已完成：`F2/F4` 进入 AI；`Esc/F9` 取消 AI；`F8` 切换开发候选框。
|
||||
- 补 UIA 回查：选中框后用中心点/范围找最小 UIA 元素，生成稳定 selector。
|
||||
|
||||
### 第二阶段：稳定
|
||||
|
||||
- `exists/wait` 真正支持 `screen= + window + aiRect`。
|
||||
- 执行前校验窗口和元素仍在预期区域。
|
||||
- 增加 AI 重定位：截图 → 检测 → 与保存的 bbox/裁剪图匹配 → 更新坐标。
|
||||
- 所有执行结果回写目标快照，方便排查框偏。
|
||||
|
||||
### 第三阶段：接近实在
|
||||
|
||||
- 元素检测、元素分类、OCR、图标相似、UISelector 组合。
|
||||
- AI 只输出结构化 target/action，不直接控鼠标键盘。
|
||||
- RPA 执行器负责窗口激活、等待、重试、截图、异常恢复。
|
||||
- 对复杂远程桌面/Canvas/Java 应用保留视觉兜底。
|
||||
|
||||
## 推荐优先级
|
||||
|
||||
已选定 `webpage-element.onnx` 作为 Pane AI 默认模型。其他模型继续留在本地 `temp/models` 供实验台对比，不再进入默认链路和安装包。内置 OmniParser 只作为老安装包兼容回退，不建议作为最终业务 selector 的唯一来源。
|
||||
@ -25,7 +25,7 @@ export default {
|
||||
{ from: path.join(root, "resources", "excluded.json"), to: "excluded.json" },
|
||||
{ from: path.join(root, "resources", "capability-duplicates.json"), to: "capability-duplicates.json" },
|
||||
{ from: path.join(root, "resources", "desktop-picker.ps1"), to: "desktop-picker.ps1" },
|
||||
{ from: path.join(root, "resources", "models"), to: "models" },
|
||||
{ from: path.join(root, "resources", "models", "ui-element-detect"), to: "models/ui-element-detect" },
|
||||
{ from: path.join(root, "resources", "fixture"), to: "fixture" }
|
||||
],
|
||||
asar: true,
|
||||
|
||||
@ -192,6 +192,7 @@ $prevSpace = $false
|
||||
$prevPgUp = $false
|
||||
$prevPgDn = $false
|
||||
$prevF2 = $false
|
||||
$prevF4 = $false
|
||||
$cacheX = [int]::MinValue
|
||||
$cacheY = [int]::MinValue
|
||||
$cacheEl = $null
|
||||
@ -213,6 +214,7 @@ while ($true) {
|
||||
$pgUp = (([RpaPick.PickW32]::GetAsyncKeyState(33) -band 0x8001) -ne 0)
|
||||
$pgDn = (([RpaPick.PickW32]::GetAsyncKeyState(34) -band 0x8001) -ne 0)
|
||||
$f2 = (([RpaPick.PickW32]::GetAsyncKeyState(113) -band 0x8001) -ne 0)
|
||||
$f4 = (([RpaPick.PickW32]::GetAsyncKeyState(115) -band 0x8001) -ne 0)
|
||||
if ($esc) { Write-PickLog "esc"; break }
|
||||
if ($stage -eq "hover") {
|
||||
$ptX = [System.Windows.Forms.Cursor]::Position.X
|
||||
@ -256,11 +258,12 @@ while ($true) {
|
||||
$el = $cacheEl
|
||||
if ($el) {
|
||||
Set-HighlightBox $cachePaneX $cachePaneY $cachePaneW $cachePaneH
|
||||
$aiHint = if ($cacheCtrlName -eq "Pane") { " [F2]AI识别" } else { "" }
|
||||
$aiHint = if ($cacheCtrlName -eq "Pane") { " [F2/F4]AI识别" } else { "" }
|
||||
$infoLabel.Text = "名称: " + $cacheName + " 类型: " + $cacheCtrlName + " [空格]锁定" + $aiHint + " [Esc]取消"
|
||||
} else { Hide-Highlight }
|
||||
if ($f2 -and -not $prevF2 -and $el -and $cacheCtrlName -eq "Pane") {
|
||||
Write-PickLog "f2 ai"
|
||||
$aiTrigger = ($f2 -and -not $prevF2) -or ($f4 -and -not $prevF4)
|
||||
if ($aiTrigger -and $el -and $cacheCtrlName -eq "Pane") {
|
||||
Write-PickLog "f2/f4 ai"
|
||||
$walker = [System.Windows.Automation.TreeWalker]::ControlViewWalker
|
||||
$cur = $el
|
||||
$topTitle = ""
|
||||
@ -312,6 +315,7 @@ while ($true) {
|
||||
$prevPgUp = $pgUp
|
||||
$prevPgDn = $pgDn
|
||||
$prevF2 = $f2
|
||||
$prevF4 = $f4
|
||||
Start-Sleep -Milliseconds 60
|
||||
}
|
||||
$info.Close()
|
||||
|
||||
@ -0,0 +1 @@
|
||||
["button","field","heading","iframe","image","label","link","text"]
|
||||
BIN
resources/models/ui-element-detect/webpage-element.onnx
Normal file
BIN
resources/models/ui-element-detect/webpage-element.onnx
Normal file
Binary file not shown.
@ -6,6 +6,7 @@ import path from "node:path";
|
||||
import { app, BrowserWindow, desktopCapturer, globalShortcut, screen } from "electron";
|
||||
import { RuntimeError } from "./engine/types";
|
||||
import { detectUiElements, localUiModelPath } from "./local-vision/ui-detect-service";
|
||||
import { assignPaneOcr, recognizePane } from "./local-vision/pane-ocr-service";
|
||||
|
||||
const run = promisify(execFile);
|
||||
|
||||
@ -67,6 +68,9 @@ interface AiCandidate {
|
||||
name: string;
|
||||
controlType: string;
|
||||
confidence: number;
|
||||
ocrText?: string;
|
||||
ocrState?: "pending" | "done" | "empty" | "error";
|
||||
ocrConfidence?: number;
|
||||
}
|
||||
|
||||
interface AiCaptureOutput {
|
||||
@ -202,31 +206,53 @@ function logHoverState(state){
|
||||
window.__toggleDebug=function(){
|
||||
document.body.classList.toggle('debug');
|
||||
var debug=document.body.classList.contains('debug');
|
||||
showStatus(debug?'开发模式：F8 关闭候选框':(window.__candidatesReady?'AI 完成：移动高亮；单击选中，再次单击或空格回填；Esc 取消':'开发模式已开启，识别完成后显示全部候选框'));
|
||||
showStatus(debug?'开发模式：F8 关闭候选框':(window.__candidatesReady?'AI 完成：移动高亮；单击选中，再次单击或空格回填；Esc/F9 取消':'开发模式已开启，识别完成后显示全部候选框'));
|
||||
};
|
||||
function candidateLabel(c){
|
||||
var type=c.controlType||'控件';
|
||||
if(c.ocrState==='pending')return 'AI识别 · '+type;
|
||||
var text=String(c.ocrText||'').trim();
|
||||
if(text.length>32)text=text.slice(0,32)+'…';
|
||||
return (text?text+' · ':'AI识别 · ')+type;
|
||||
}
|
||||
function refreshBadge(cid){
|
||||
var c=window.__candidateMap[cid];if(!c)return;
|
||||
var box=document.querySelector('.candidate[data-cid="'+cid+'"]');if(!box)return;
|
||||
var label=document.getElementById('badge-'+box.dataset.index);if(!label)return;
|
||||
label.textContent=candidateLabel(c);
|
||||
if(label.classList.contains('active')&&window.__selectedCid===cid)showStatus('已选中：'+candidateLabel(c)+'；再次单击或空格回填');
|
||||
}
|
||||
function applyOcrUpdates(items){
|
||||
if(!window.__candidateMap||!Array.isArray(items))return;
|
||||
items.forEach(function(item){
|
||||
var c=window.__candidateMap[item.candidateId];if(!c)return;
|
||||
c.ocrText=item.ocrText||'';c.ocrState=item.ocrState||'done';c.ocrConfidence=item.ocrConfidence;
|
||||
refreshBadge(c.candidateId);
|
||||
});
|
||||
}
|
||||
function showCandidates(items){
|
||||
paneW=window.__paneW||document.documentElement.clientWidth;paneH=window.__paneH||document.documentElement.clientHeight;
|
||||
document.getElementById('scan').style.display='none';
|
||||
document.getElementById('aiFrame').style.display='none';
|
||||
document.body.classList.remove('scanning');document.body.style.cursor='pointer';
|
||||
window.__candidateMap={};window.__candidatesReady=false;clearActive();
|
||||
window.__candidateMap={};window.__candidatesReady=false;window.__ocrReady=false;clearActive();
|
||||
items.sort(function(a,b){return a.y-b.y||a.x-b.x}).forEach(function(c,index){
|
||||
window.__candidateMap[c.candidateId]=c;
|
||||
var box=document.createElement('div');box.className='candidate';box.dataset.cid=c.candidateId;box.dataset.index=index;
|
||||
box.style.left=(c.x/paneW*100)+'%';box.style.top=(c.y/paneH*100)+'%';
|
||||
box.style.width=(c.width/paneW*100)+'%';box.style.height=(c.height/paneH*100)+'%';
|
||||
var label=document.createElement('div');label.className='badge';label.id='badge-'+index;label.textContent='AI识别 · '+(c.controlType||'控件');
|
||||
var label=document.createElement('div');label.className='badge';label.id='badge-'+index;label.textContent=candidateLabel(c);
|
||||
label.style.left=(c.x/paneW*100)+'%';label.style.top=(Math.max(0,c.y-24)/paneH*100)+'%';
|
||||
box.addEventListener('click',function(event){
|
||||
event.stopPropagation();if(Date.now()<window.__readyAt)return;
|
||||
activateBox(box);
|
||||
if(window.__selectedCid!==c.candidateId){window.__selectedCid=c.candidateId;showStatus('已选中：'+(c.controlType||'控件')+'；再次单击或空格回填');return}
|
||||
if(window.__selectedCid!==c.candidateId){window.__selectedCid=c.candidateId;showStatus('已选中：'+candidateLabel(c)+'；再次单击或空格回填');return}
|
||||
window.__confirmedCid=c.candidateId;window.__done=true;
|
||||
});
|
||||
document.body.appendChild(box);document.body.appendChild(label);
|
||||
});
|
||||
window.__candidatesReady=true;window.__readyAt=Date.now()+450;
|
||||
showStatus(document.body.classList.contains('debug')?'开发模式：F8 关闭候选框':'AI 完成：移动高亮；单击选中，再次单击或空格回填；Esc 取消');
|
||||
showStatus(document.body.classList.contains('debug')?'开发模式：F8 关闭候选框':'AI 完成：移动高亮；单击选中，再次单击或空格回填；Esc/F9 取消');
|
||||
syncHover();
|
||||
}
|
||||
document.addEventListener('pointermove',function(event){lastMouse=event;syncHover()});
|
||||
@ -348,37 +374,56 @@ async function capturePaneForAi(rect: { x: number; y: number; w: number; h: numb
|
||||
}
|
||||
|
||||
async function runPaneAiPick(context: UiaPickOutput): Promise<DesktopPickResult> {
|
||||
// 本地小模型是 Pane AI 的主链路；这里不要求用户配置远程 VLM。
|
||||
// 本地小模型是 Pane AI 的主链路；OCR 使用同一张 Pane 截图，不要求用户配置远程 VLM。
|
||||
const localModelPath = localUiModelPath();
|
||||
console.log("[pane-ai] local model", localModelPath);
|
||||
const pane = paneAiRect(context);
|
||||
const win = await createPaneAiOverlay(pane);
|
||||
const startedAt = Date.now();
|
||||
// F8 用全局快捷键，避免键盘焦点停在目标应用时 overlay 收不到按键。
|
||||
const debugHotkeyRegistered = globalShortcut.register("F8", () => {
|
||||
|
||||
// F8 是开发框调试；Esc/F9 是兜底取消。全局注册能解决键盘焦点留在目标应用时 overlay 收不到按键的问题。
|
||||
const registeredHotkeys: string[] = [];
|
||||
const registerHotkey = (accelerator: string, handler: () => void) => {
|
||||
try {
|
||||
if (globalShortcut.register(accelerator, handler)) registeredHotkeys.push(accelerator);
|
||||
} catch (error) {
|
||||
console.warn("[pane-ai] hotkey register failed", accelerator, error);
|
||||
}
|
||||
};
|
||||
const requestCancel = () => {
|
||||
if (win.isDestroyed()) return;
|
||||
void win.webContents.executeJavaScript("window.__cancelled=true;window.__done=true;", true).catch(() => {});
|
||||
};
|
||||
registerHotkey("Escape", requestCancel);
|
||||
registerHotkey("F9", requestCancel);
|
||||
registerHotkey("F8", () => {
|
||||
if (win.isDestroyed()) return;
|
||||
void win.webContents.executeJavaScript("window.__toggleDebug && window.__toggleDebug()", true).catch(() => {});
|
||||
});
|
||||
|
||||
try {
|
||||
// 稍等 overlay 完成首轮绘制，但不再等一整套“表演动画”。
|
||||
await new Promise(resolve => setTimeout(resolve, 350));
|
||||
await new Promise(resolve => setTimeout(resolve, 250));
|
||||
const detection = capturePaneForAi(pane, win)
|
||||
.then(async capture => {
|
||||
const image = await fs.promises.readFile(capture.imagePath);
|
||||
const inferenceStarted = Date.now();
|
||||
// 同一张截图只跑一次整图 OCR；候选框检测先出结果，OCR 完成后异步回填。
|
||||
const ocrTask = recognizePane({ imagePath: capture.imagePath, image })
|
||||
.catch(error => {
|
||||
console.warn("[pane-ai] pane ocr failed", error);
|
||||
return null;
|
||||
});
|
||||
const boxes = await detectUiElements({ image, width: capture.cropW, height: capture.cropH });
|
||||
console.log("[pane-ai] local inference", boxes.length, Date.now() - inferenceStarted + "ms");
|
||||
return { capture, boxes };
|
||||
return { capture, boxes, ocrTask };
|
||||
});
|
||||
|
||||
let settled = false;
|
||||
const tickStatus = async () => {
|
||||
const elapsed = Date.now() - startedAt;
|
||||
const label = elapsed < 2600
|
||||
? "AI 识别中…"
|
||||
: elapsed < 6000
|
||||
? "AI 理解中…"
|
||||
: "正在整理候选区域…";
|
||||
const label = elapsed < 2600 ? "AI 识别中…" : elapsed < 6000 ? "AI 理解中…" : "正在整理候选区域…";
|
||||
if (win.isDestroyed()) return;
|
||||
try {
|
||||
await win.webContents.executeJavaScript("showStatus(" + JSON.stringify(label) + ");", true);
|
||||
} catch {
|
||||
@ -390,6 +435,7 @@ async function runPaneAiPick(context: UiaPickOutput): Promise<DesktopPickResult>
|
||||
const cancelWatcher = (async () => {
|
||||
while (!settled) {
|
||||
try {
|
||||
if (win.isDestroyed()) throw new RuntimeError("pick_cancelled", "AI 拾取窗口已关闭");
|
||||
const state = await win.webContents.executeJavaScript("JSON.stringify({cancelled:window.__cancelled===true})", true);
|
||||
if (JSON.parse(state).cancelled) throw new RuntimeError("pick_cancelled", "已取消 AI 拾取");
|
||||
} catch (error) {
|
||||
@ -399,9 +445,13 @@ async function runPaneAiPick(context: UiaPickOutput): Promise<DesktopPickResult>
|
||||
}
|
||||
})();
|
||||
|
||||
let detected: { capture: AiCaptureOutput; boxes: Awaited<ReturnType<typeof detectUiElements>> };
|
||||
let detected: {
|
||||
capture: AiCaptureOutput;
|
||||
boxes: Awaited<ReturnType<typeof detectUiElements>>;
|
||||
ocrTask: Promise<Awaited<ReturnType<typeof recognizePane>> | null>;
|
||||
};
|
||||
try {
|
||||
detected = await Promise.race([detection, cancelWatcher]) as { capture: AiCaptureOutput; boxes: Awaited<ReturnType<typeof detectUiElements>> };
|
||||
detected = await Promise.race([detection, cancelWatcher]) as typeof detected;
|
||||
} finally {
|
||||
settled = true;
|
||||
clearInterval(statusTimer);
|
||||
@ -415,13 +465,16 @@ async function runPaneAiPick(context: UiaPickOutput): Promise<DesktopPickResult>
|
||||
height: box.height,
|
||||
name: box.category,
|
||||
controlType: box.category,
|
||||
confidence: box.confidence
|
||||
confidence: box.confidence,
|
||||
ocrText: "",
|
||||
ocrState: "pending"
|
||||
}));
|
||||
if (!candidates.length) {
|
||||
throw new RuntimeError("ai_response_invalid", "本地 UI 模型没有识别到可用候选区域");
|
||||
}
|
||||
|
||||
const payload = JSON.stringify(candidates);
|
||||
if (win.isDestroyed()) throw new RuntimeError("pick_cancelled", "AI 拾取窗口已关闭");
|
||||
await win.show();
|
||||
win.focus();
|
||||
await win.webContents.executeJavaScript(
|
||||
@ -429,10 +482,46 @@ async function runPaneAiPick(context: UiaPickOutput): Promise<DesktopPickResult>
|
||||
true
|
||||
);
|
||||
|
||||
// OCR 不阻塞候选框出现；完成后按框分配、缓存到候选对象，并刷新悬浮标签。
|
||||
void detected.ocrTask
|
||||
.then(async paneOcr => {
|
||||
if (!paneOcr || win.isDestroyed()) return;
|
||||
const states = assignPaneOcr(paneOcr.lines, candidates);
|
||||
for (const candidate of candidates) {
|
||||
const state = states[candidate.candidateId];
|
||||
if (!state) continue;
|
||||
candidate.ocrText = state.ocrText;
|
||||
candidate.ocrState = state.ocrState;
|
||||
candidate.ocrConfidence = state.ocrConfidence;
|
||||
if (state.ocrText) candidate.name = state.ocrText;
|
||||
}
|
||||
console.log("[pane-ai] pane ocr", paneOcr.lines.length, paneOcr.elapsedMs + "ms", paneOcr.cacheHit ? "cache" : "inference");
|
||||
const ocrPayload = JSON.stringify(candidates.map(candidate => ({
|
||||
candidateId: candidate.candidateId,
|
||||
ocrText: candidate.ocrText,
|
||||
ocrState: candidate.ocrState,
|
||||
ocrConfidence: candidate.ocrConfidence
|
||||
})));
|
||||
await win.webContents.executeJavaScript("applyOcrUpdates(" + ocrPayload + ");", true);
|
||||
await win.webContents.executeJavaScript(
|
||||
"if(!window.__done)showStatus('OCR 完成：移动高亮；单击选中，再次单击或空格回填；Esc/F9 取消');",
|
||||
true
|
||||
);
|
||||
})
|
||||
.catch(error => {
|
||||
console.warn("[pane-ai] ocr update failed", error);
|
||||
if (win.isDestroyed()) return;
|
||||
void win.webContents.executeJavaScript(
|
||||
"applyOcrUpdates(" + JSON.stringify(candidates.map(candidate => ({ candidateId: candidate.candidateId, ocrText: "", ocrState: "error" }))) + ");",
|
||||
true
|
||||
).catch(() => {});
|
||||
});
|
||||
|
||||
const deadline = Date.now() + 180000;
|
||||
let selectedId: string | null = null;
|
||||
while (Date.now() < deadline) {
|
||||
await new Promise(resolve => setTimeout(resolve, 80));
|
||||
if (win.isDestroyed()) throw new RuntimeError("pick_cancelled", "已取消 AI 拾取");
|
||||
const state = await win.webContents.executeJavaScript(
|
||||
"JSON.stringify({done:window.__done===true,cancelled:window.__cancelled===true,selectedCid:window.__selectedCid||null})",
|
||||
true
|
||||
@ -450,20 +539,25 @@ async function runPaneAiPick(context: UiaPickOutput): Promise<DesktopPickResult>
|
||||
}
|
||||
if (!selectedId) throw new RuntimeError("pick_cancelled", "AI 拾取超时");
|
||||
|
||||
// 用户确认时 OCR 可能还差几百毫秒。等本轮任务收敛，避免回填名和最终 selector 不一致。
|
||||
await detected.ocrTask;
|
||||
|
||||
const box = candidates.find(item => item.candidateId === selectedId)!;
|
||||
const x = detected.capture.cropX + box.x;
|
||||
const y = detected.capture.cropY + box.y;
|
||||
const selectorParts = [
|
||||
"screen=" + [x, y, box.width, box.height].map(Math.round).join(","),
|
||||
context.window ? "window=" + safeSelectorText(context.window) : "",
|
||||
"name=" + safeSelectorText(box.name),
|
||||
"name=" + safeSelectorText(box.name || box.ocrText || box.controlType),
|
||||
"type=" + safeSelectorText(box.controlType),
|
||||
"aiType=" + safeSelectorText(box.controlType),
|
||||
box.ocrText ? "aiText=" + safeSelectorText(box.ocrText) : "",
|
||||
"ai=1"
|
||||
].filter(Boolean);
|
||||
return {
|
||||
source: "ai",
|
||||
window: context.window,
|
||||
name: box.name,
|
||||
name: box.name || box.ocrText || box.controlType,
|
||||
controlType: box.controlType,
|
||||
x, y, w: box.width, h: box.height,
|
||||
confidence: box.confidence,
|
||||
@ -471,7 +565,10 @@ async function runPaneAiPick(context: UiaPickOutput): Promise<DesktopPickResult>
|
||||
selector: selectorParts.join(";")
|
||||
};
|
||||
} finally {
|
||||
if (debugHotkeyRegistered) globalShortcut.unregister("F8");
|
||||
// 先注册的后注册的都要清理；Esc 失败时 F9 仍会兜底。
|
||||
for (const accelerator of [...registeredHotkeys].reverse()) {
|
||||
try { globalShortcut.unregister(accelerator); } catch {}
|
||||
}
|
||||
if (!win.isDestroyed()) win.destroy();
|
||||
}
|
||||
}
|
||||
|
||||
33
src/main/local-vision/pane-ocr-service.test.ts
Normal file
33
src/main/local-vision/pane-ocr-service.test.ts
Normal file
@ -0,0 +1,33 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { assignPaneOcr, type PaneOcrLine } from "./pane-ocr-service";
|
||||
|
||||
describe("pane OCR assignment", () => {
|
||||
it("assigns OCR lines to the boxes containing their center", () => {
|
||||
const lines: PaneOcrLine[] = [
|
||||
{ text: "登录", confidence: 0.9, x: 12, y: 10, width: 24, height: 12 }
|
||||
];
|
||||
const boxes = [
|
||||
{ candidateId: "button", x: 8, y: 6, width: 48, height: 24 },
|
||||
{ candidateId: "container", x: 0, y: 0, width: 120, height: 80 }
|
||||
];
|
||||
expect(assignPaneOcr(lines, boxes)).toEqual({
|
||||
button: { ocrText: "登录", ocrState: "done", ocrConfidence: 0.9 },
|
||||
container: { ocrText: "登录", ocrState: "done", ocrConfidence: 0.9 }
|
||||
});
|
||||
});
|
||||
|
||||
it("uses a strongly overlapping line even when its center falls outside", () => {
|
||||
const lines: PaneOcrLine[] = [
|
||||
{ text: "保存", confidence: 0.8, x: 0, y: 10, width: 30, height: 12 }
|
||||
];
|
||||
const boxes = [{ candidateId: "button", x: 8, y: 8, width: 30, height: 16 }];
|
||||
const result = assignPaneOcr(lines, boxes);
|
||||
expect(result.button.ocrText).toBe("保存");
|
||||
expect(result.button.ocrState).toBe("done");
|
||||
});
|
||||
|
||||
it("marks boxes without text as empty", () => {
|
||||
const result = assignPaneOcr([], [{ candidateId: "image", x: 0, y: 0, width: 20, height: 20 }]);
|
||||
expect(result.image).toEqual({ ocrText: "", ocrState: "empty" });
|
||||
});
|
||||
});
|
||||
177
src/main/local-vision/pane-ocr-service.ts
Normal file
177
src/main/local-vision/pane-ocr-service.ts
Normal file
@ -0,0 +1,177 @@
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
export interface PaneOcrLine {
|
||||
text: string;
|
||||
confidence: number;
|
||||
x: number;
|
||||
y: number;
|
||||
width: number;
|
||||
height: number;
|
||||
}
|
||||
|
||||
export interface PaneOcrResult {
|
||||
lines: PaneOcrLine[];
|
||||
elapsedMs: number;
|
||||
cacheHit: boolean;
|
||||
}
|
||||
|
||||
export interface CandidateOcrState {
|
||||
ocrText: string;
|
||||
ocrState: "pending" | "done" | "empty" | "error";
|
||||
ocrConfidence?: number;
|
||||
}
|
||||
|
||||
interface GutenLine {
|
||||
text?: unknown;
|
||||
mean?: unknown;
|
||||
score?: unknown;
|
||||
box?: unknown;
|
||||
}
|
||||
|
||||
interface GutenOcr {
|
||||
detect: (image: string) => Promise<{ texts?: GutenLine[] }>;
|
||||
}
|
||||
|
||||
type GutenOcrModule = {
|
||||
default: {
|
||||
create: (options?: unknown) => Promise<GutenOcr>;
|
||||
};
|
||||
};
|
||||
|
||||
let ocrPromise: Promise<GutenOcr> | null = null;
|
||||
const paneCache = new Map<string, Promise<PaneOcrResult>>();
|
||||
const PANE_CACHE_LIMIT = 32;
|
||||
|
||||
async function getOcr(): Promise<GutenOcr> {
|
||||
if (!ocrPromise) {
|
||||
ocrPromise = import("@gutenye/ocr-node")
|
||||
.then(async mod => await (mod as unknown as GutenOcrModule).default.create())
|
||||
.catch(error => {
|
||||
ocrPromise = null;
|
||||
throw error;
|
||||
});
|
||||
}
|
||||
return await ocrPromise;
|
||||
}
|
||||
|
||||
function clamp01(value: unknown): number {
|
||||
const next = Number(value);
|
||||
return Number.isFinite(next) ? Math.max(0, Math.min(1, next)) : 0;
|
||||
}
|
||||
|
||||
function lineRect(line: GutenLine): PaneOcrLine {
|
||||
const points = Array.isArray(line.box) ? line.box : [];
|
||||
const xs = points
|
||||
.map(point => Array.isArray(point) ? Number(point[0]) : Number.NaN)
|
||||
.filter(value => Number.isFinite(value));
|
||||
const ys = points
|
||||
.map(point => Array.isArray(point) ? Number(point[1]) : Number.NaN)
|
||||
.filter(value => Number.isFinite(value));
|
||||
const x = xs.length ? Math.min(...xs) : 0;
|
||||
const y = ys.length ? Math.min(...ys) : 0;
|
||||
const maxX = xs.length ? Math.max(...xs) : x;
|
||||
const maxY = ys.length ? Math.max(...ys) : y;
|
||||
return {
|
||||
text: String(line.text || "").replace(/\s+/g, " ").trim(),
|
||||
confidence: clamp01(line.mean ?? line.score ?? 0),
|
||||
x,
|
||||
y,
|
||||
width: Math.max(0, maxX - x),
|
||||
height: Math.max(0, maxY - y)
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* 每张 Pane 截图只做一次整图 OCR。候选框只是画面上的矩形，
|
||||
* 整图结果按几何关系分配给候选框，天然形成缓存并避免每个框重复推理。
|
||||
*/
|
||||
export async function recognizePane(input: {
|
||||
imagePath: string;
|
||||
image: Buffer;
|
||||
}): Promise<PaneOcrResult> {
|
||||
const imageHash = createHash("sha256").update(input.image).digest("hex");
|
||||
const cached = paneCache.get(imageHash);
|
||||
if (cached) {
|
||||
const result = await cached;
|
||||
return { ...result, cacheHit: true };
|
||||
}
|
||||
|
||||
const task = (async (): Promise<PaneOcrResult> => {
|
||||
const started = Date.now();
|
||||
try {
|
||||
const ocr = await getOcr();
|
||||
const response = await ocr.detect(input.imagePath);
|
||||
const lines = (response.texts || [])
|
||||
.map(lineRect)
|
||||
.filter(line => line.text.length > 0);
|
||||
return { lines, elapsedMs: Date.now() - started, cacheHit: false };
|
||||
} catch (error) {
|
||||
// 初始化或识别失败时清掉缓存任务，下一次悬浮仍有机会恢复。
|
||||
paneCache.delete(imageHash);
|
||||
throw error;
|
||||
}
|
||||
})();
|
||||
|
||||
paneCache.set(imageHash, task);
|
||||
while (paneCache.size > PANE_CACHE_LIMIT) {
|
||||
const oldest = paneCache.keys().next().value;
|
||||
if (!oldest) break;
|
||||
paneCache.delete(oldest);
|
||||
}
|
||||
return await task;
|
||||
}
|
||||
|
||||
export function clearPaneOcrCache(): void {
|
||||
paneCache.clear();
|
||||
}
|
||||
|
||||
interface AssignableBox {
|
||||
candidateId: string;
|
||||
x: number;
|
||||
y: number;
|
||||
width: number;
|
||||
height: number;
|
||||
}
|
||||
|
||||
function centerInsideBox(line: PaneOcrLine, box: AssignableBox): boolean {
|
||||
const cx = line.x + line.width / 2;
|
||||
const cy = line.y + line.height / 2;
|
||||
return cx >= box.x && cx <= box.x + box.width && cy >= box.y && cy <= box.y + box.height;
|
||||
}
|
||||
|
||||
function overlapRatio(line: PaneOcrLine, box: AssignableBox): number {
|
||||
const overlapX = Math.max(0, Math.min(line.x + line.width, box.x + box.width) - Math.max(line.x, box.x));
|
||||
const overlapY = Math.max(0, Math.min(line.y + line.height, box.y + box.height) - Math.max(line.y, box.y));
|
||||
const lineArea = Math.max(1, line.width * line.height);
|
||||
return (overlapX * overlapY) / lineArea;
|
||||
}
|
||||
|
||||
export function assignPaneOcr(
|
||||
lines: PaneOcrLine[],
|
||||
boxes: AssignableBox[]
|
||||
): Record<string, CandidateOcrState> {
|
||||
const result: Record<string, CandidateOcrState> = {};
|
||||
for (const box of boxes) {
|
||||
const matched = lines
|
||||
.map(line => ({
|
||||
line,
|
||||
inside: centerInsideBox(line, box),
|
||||
ratio: overlapRatio(line, box)
|
||||
}))
|
||||
.filter(item => item.inside || item.ratio >= 0.55)
|
||||
.sort((a, b) => Number(b.inside) - Number(a.inside) || b.ratio - a.ratio || a.line.y - b.line.y || a.line.x - b.line.x);
|
||||
|
||||
if (!matched.length) {
|
||||
result[box.candidateId] = { ocrText: "", ocrState: "empty" };
|
||||
continue;
|
||||
}
|
||||
const text = matched.map(item => item.line.text).join(" ").replace(/\s+/g, " ").trim();
|
||||
const confidence = matched.reduce((sum, item) => sum + item.line.confidence, 0) / matched.length;
|
||||
result[box.candidateId] = {
|
||||
ocrText: text,
|
||||
ocrState: text ? "done" : "empty",
|
||||
ocrConfidence: Math.round(confidence * 1000) / 1000
|
||||
};
|
||||
}
|
||||
return result;
|
||||
}
|
||||
@ -36,19 +36,28 @@ let nextRequestId = 1;
|
||||
const MAX_LOADED_WORKERS = 4;
|
||||
|
||||
export function localUiModelPath(): string {
|
||||
const candidates = [
|
||||
path.join(process.resourcesPath || "", "models", "ui-icon-detect", "model.onnx"),
|
||||
path.join(app.getAppPath(), "resources", "models", "ui-icon-detect", "model.onnx")
|
||||
const rootCandidates = [
|
||||
process.resourcesPath,
|
||||
app.getAppPath() ? path.join(app.getAppPath(), "resources") : "",
|
||||
path.join(app.getAppPath(), "temp", "models")
|
||||
].filter(Boolean);
|
||||
const found = candidates.find(candidate => fs.existsSync(candidate));
|
||||
// webpage-element 是实测后的首选模型；保留旧模型作为老安装包的兼容回退。
|
||||
const fileCandidates = rootCandidates.flatMap(root => [
|
||||
path.join(root, "models", "ui-element-detect", "webpage-element.onnx"),
|
||||
path.join(root, "webpage-element.onnx"),
|
||||
path.join(root, "models", "ui-icon-detect", "model.onnx")
|
||||
]);
|
||||
const found = fileCandidates.find(candidate => {
|
||||
try { return fs.existsSync(candidate) && fs.statSync(candidate).isFile(); }
|
||||
catch { return false; }
|
||||
});
|
||||
if (!found) {
|
||||
throw new RuntimeError("local_vision_model_missing", "本地 UI 检测模型缺失", {
|
||||
recommendation: "重新安装应用，或补齐 resources/models/ui-icon-detect/model.onnx"
|
||||
recommendation: "重新安装应用，或补齐 resources/models/ui-element-detect/webpage-element.onnx"
|
||||
});
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
export function defaultUiModelPath(): string {
|
||||
return localUiModelPath();
|
||||
}
|
||||
@ -142,4 +151,3 @@ export function closeLocalVisionWorkers(): void {
|
||||
}
|
||||
pending.clear();
|
||||
}
|
||||
|
||||
|
||||
@ -1,3 +1,4 @@
|
||||
import fs from "node:fs";
|
||||
import { parentPort, workerData } from "node:worker_threads";
|
||||
import { InferenceSession, Tensor } from "onnxruntime-node";
|
||||
import { createCanvas, loadImage } from "@napi-rs/canvas";
|
||||
@ -37,6 +38,9 @@ const defaultScoreThreshold = 0.25;
|
||||
const defaultIouThreshold = 0.42;
|
||||
const maxBoxes = 160;
|
||||
|
||||
const startupModelPath = String((workerData as { modelPath?: string }).modelPath || "");
|
||||
const modelLabels = readModelLabels(startupModelPath);
|
||||
|
||||
let sessionPromise: Promise<InferenceSession> | null = null;
|
||||
|
||||
function loadSession(): Promise<InferenceSession> {
|
||||
@ -76,6 +80,24 @@ function tensorAccessor(output: TensorLike, anchorCount: number, channelCount: n
|
||||
: output.data[anchor * channelCount + channel];
|
||||
}
|
||||
|
||||
// YOLO 导出的 ONNX 通常不含标签元数据。约定同名 *.labels.json 保存类别：
|
||||
// 1) ["button", "input"] 2) { "labels": ["button", "input"] }
|
||||
function readModelLabels(modelPath: string): string[] | undefined {
|
||||
const sidecarPath = modelPath.replace(/\.onnx$/i, ".labels.json");
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(fs.readFileSync(sidecarPath, "utf8"));
|
||||
const values = Array.isArray(parsed) ? parsed : (parsed as { labels?: unknown })?.labels;
|
||||
if (!Array.isArray(values)) return undefined;
|
||||
const labels = values
|
||||
.filter((item): item is string | number => typeof item === "string" || typeof item === "number")
|
||||
.map(item => String(item))
|
||||
.filter(Boolean);
|
||||
return labels.length ? labels : undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function extractBoxes(
|
||||
output: TensorLike,
|
||||
sourceWidth: number,
|
||||
@ -100,6 +122,7 @@ function extractBoxes(
|
||||
const offsetX = (inputSize - drawnWidth) / 2;
|
||||
const offsetY = (inputSize - drawnHeight) / 2;
|
||||
const minSide = Math.max(8, Math.min(sourceWidth, sourceHeight) * 0.006);
|
||||
const labels = modelLabels;
|
||||
const raw: LocalUiBox[] = [];
|
||||
|
||||
for (let index = 0; index < anchorCount; index++) {
|
||||
@ -137,7 +160,8 @@ function extractBoxes(
|
||||
y: Math.round(y),
|
||||
width: Math.round(boxWidth),
|
||||
height: Math.round(boxHeight),
|
||||
category: classCount <= 1 ? "控件" : `类别${classIndex + 1}`,
|
||||
category: labels?.[classIndex]
|
||||
?? (classCount <= 1 ? "控件" : `类别${classIndex + 1}`),
|
||||
confidence: Math.round(Math.max(0, Math.min(1, confidence)) * 1000) / 1000
|
||||
});
|
||||
}
|
||||
|
||||
Loading…
Reference in New Issue
Block a user