feat: 添加算术验证码与字符验证码处理能力
新增算术验证码解析、字符清洗能力及测试文件,注册OCR_CalculationCode和OCR_CommonEnglishNumbersCode两个新能力,补充验证码处理逻辑
This commit is contained in:
parent
902ee0cec4
commit
5ef797dcc0
179
docs/rpa-core-migration-guide.md
Normal file
179
docs/rpa-core-migration-guide.md
Normal file
@ -0,0 +1,179 @@
|
||||
# rpa-core 迁入 Agent-Employee 指南
|
||||
|
||||
> 本文是 `@rpa/core` 包从 demo(本仓库)迁入 `D:\code\Agent-Employee` 的完整操作指南。
|
||||
> 抽离改造已于 2026-11-05 完成(阶段 0→5,分支 refactor/rpa-core 已合入 main,
|
||||
> commit `4139400..a8084dc`),本指南对应任务清单中的「阶段 6 — 迁入 Agent-Employee」。
|
||||
> 配套文档:`docs/rpa-core-refactor-tasks.md`(任务与风险清单)、`docs/dev-conventions.md`(开发规范)、`packages/rpa-core/README.md`(包边界与契约)。
|
||||
|
||||
---
|
||||
|
||||
## 1. 迁入对象的当前形态
|
||||
|
||||
| 项 | 现状 |
|
||||
|---|---|
|
||||
| 包位置 | `packages/rpa-core/`(npm workspace 成员,`@rpa/core`,private) |
|
||||
| 依赖约束 | 零 Electron、零宿主引用(`npm run check:deps` 护栏强制) |
|
||||
| 导出入口 | `.`(engine/driver/browser/types)+ `./capabilities`(副作用注册全量能力) |
|
||||
| 运行时依赖 | exceljs / pdf-lib / nodemailer / imapflow / mailparser / mssql / mysql2 / pg / basic-ftp / adm-zip / pngjs / sm-crypto / ws / @gutenye/ocr-node |
|
||||
| 宿主耦合点 | 全部经 `HostBridge` / `SubflowSource` / `BrowserLike` 三个接口注入,无反向依赖 |
|
||||
| 验证基线 | 包内注册 719 个能力 id;demo 全量快照 909 id(含宿主浏览器能力);`npm run rpa-cli` 零 Electron 跑通 5 个代表能力 |
|
||||
| 测试 | 包内独立 vitest(串行模式,防桌面自动化竞态);部分冒烟测试需真实 Windows 桌面 |
|
||||
|
||||
## 2. 迁入前置项(在 demo 侧已延后,迁入时必须补做)
|
||||
|
||||
- [ ] **5.2 Bun 冒烟**(决策点 10 延期项,迁入第一件事)
|
||||
在 Agent-Employee 环境用 `bun test` 跑包内**纯逻辑测试**,逐项验证:
|
||||
- exceljs / pdf-lib / adm-zip / mailparser 等依赖在 Bun 下读写正常;
|
||||
- `driver/windows/powershell.ts` 的 `execFile("powershell.exe")` + `-EncodedCommand` 在 Bun 子进程下编码行为一致(GB18030 回退、CLIXML 解析);
|
||||
- Bun 的 `fetch` 与 Node 行为差异对 Http 能力无影响。
|
||||
- `@gutenye/ocr-node` / onnxruntime 原生依赖**不在本次范围**,单独立项验证(见 §6 留守清单)。
|
||||
- [ ] **catalog platformTag 数据重生成**
|
||||
`scripts/build-catalog.mjs` 的 `platformTag: "win32"` 填充逻辑已就绪(desktop/app/office/vision 组),
|
||||
因外部源数据(Agent-Employee 仓库)此前不在本机未生效。迁入后跑一次 `npm run catalog` 并核对。
|
||||
- [ ] **注册入口切换评估**
|
||||
双入口现状:旧 `import "@rpa/core/capabilities"`(副作用注册)+ 新 `await registerAll(engine, groups?)`(显式注册,可按域子集)。
|
||||
Agent-Employee 建议直接用 `registerAll`;确认稳定后可在包内废弃副作用入口(单独决策,不阻塞迁移)。
|
||||
|
||||
## 3. 迁入步骤(阶段 6.1 → 6.4)
|
||||
|
||||
### 6.1 ADR 评审材料
|
||||
|
||||
按 Agent-Employee 流程提交架构决策记录,三份材料可直接取自:
|
||||
|
||||
1. **包边界**:`packages/rpa-core/README.md`「包边界」表(engine / driver / capabilities / browser 四模块职责)。
|
||||
2. **HostBridge 契约**:README「HostBridge 契约」章节 + 包内 `src/engine/host-bridge.ts`(接口定义即契约,共 4 个方法 + SubflowSource 2 个方法,见 §4.1)。
|
||||
3. **driver 生命周期**:`RpaEngine.driver` 默认 `new WindowsDriver()`,`createEngine({ driver })` 可注入自定义实现;驱动只承载原始注入原语,拟真/换算/组合策略在能力层。element 域实现由 `capabilities/desktop-uia.ts` 经 `setElementDriverImplementation` 反向注册(避免 driver→capabilities 依赖)。
|
||||
|
||||
### 6.2 包迁入 `packages/`
|
||||
|
||||
1. **保留 git 历史**:用 `git subtree` 或先在 demo 仓库完成搬迁 commit 再整体 fetch,禁止复制目录后重开历史(总原则 1:单一源,杜绝代码漂移 R3)。
|
||||
2. **bun workspace 接入**:
|
||||
- Agent-Employee 根 `package.json` 的 `workspaces` 加 `packages/*`;
|
||||
- 包 `package.json` 的 `exports` 指向 `./src/index.ts` 源码直出(现状),若 Agent-Employee 构建链需要产物,再补 `build` 脚本与 `exports` 产物路径——此调整属于宿主侧决策,包源码不动;
|
||||
- 根 `tsconfig` paths 增加 `"@rpa/core": ["packages/rpa-core/src/index.ts"]` 与 `"@rpa/core/capabilities"`(参考 demo 根 `tsconfig.json` 现有写法);electron.vite/构建器的对应 alias 同步。
|
||||
3. **demo 侧解除引用**:迁入后 demo 改为只读引用该仓库(git dependency / npm pin),或明确解除引用并确认 demo 归档范围。两边同时活跃开发即违反单一源原则。
|
||||
4. **快照基线随包走**:`resources/capability-snapshot.json` 的生成逻辑在 demo 侧 `scripts/capability-snapshot.test.ts`;迁入后在 Agent-Employee 重建等价快照测试,作为能力集回归基准。
|
||||
|
||||
### 6.3 宿主适配层(product-shell 侧工作)
|
||||
|
||||
包要求的全部宿主注入点如下,迁入 = 逐项提供实现:
|
||||
|
||||
**① HostBridge(必做,4 个方法)**——demo 参考实现在 `src/main/index.ts` L28-50:
|
||||
|
||||
```ts
|
||||
import { setHostBridge, type HostDialogWindow } from "@rpa/core";
|
||||
|
||||
setHostBridge({
|
||||
// human-dialog 能力创建人机交互表单窗口;BrowserWindow 适配为 4 方法最小面
|
||||
createFormWindow({ title }) {
|
||||
const win = new BrowserWindow({ width: 480, useContentSize: true, alwaysOnTop: true,
|
||||
webPreferences: { contextIsolation: true, sandbox: true } });
|
||||
const dialogWindow: HostDialogWindow = {
|
||||
isDestroyed: () => win.isDestroyed(),
|
||||
destroy: () => win.destroy(),
|
||||
loadURL: (url) => win.loadURL(url),
|
||||
on: (event, listener) => win.on(event, listener),
|
||||
executeJavaScript: (script) => win.webContents.executeJavaScript(script, true)
|
||||
};
|
||||
return dialogWindow;
|
||||
},
|
||||
// window-extended 鼠标/拖拽坐标换算;未注入时包内回退恒等(测试模式)
|
||||
dipToScreenPoint: (p) => screen.dipToScreenPoint(p),
|
||||
screenToDipPoint: (p) => screen.screenToDipPoint(p),
|
||||
// ai-service 配置读取;未注入返回空配置,requireAiConfig 抛 ai_not_configured
|
||||
loadAiConfig: () => ({ baseUrl, apiKey, model, visionModel }) // 来自宿主 settings
|
||||
});
|
||||
```
|
||||
|
||||
未注入时的降级行为(包内已定义,宿主测试环境可依赖):DPI 换算恒等、AI 空配置、human-dialog 抛 `host_unavailable`。
|
||||
|
||||
**② SubflowSource(必做,2 个方法)**:`setSubflowSource({ listFlows, getFlow })`,子流程仓库在宿主侧(demo 对应 `flows.ts`)。`getFlow` 返回包内 `FlowDefinition`(注意:是包内领域类型,非宿主画布模型,需要做映射)。
|
||||
|
||||
**③ BrowserLike(内置浏览器路径)**:包内 `BrowserLike` 接口(`src/engine/types.ts`)是浏览器能力的抽象面。Agent-Employee 若沿用 Electron product-shell,可直接移植 demo 的 `src/main/browser/session.ts` + `capabilities/browser*.ts`;若不是 Electron,需按 BrowserLike 契约重新实现(这是迁入工作量的最大变数,先评审 Agent-Employee 的浏览器方案再估工作量)。
|
||||
|
||||
**④ ExternalTransport / EmbeddedTransport**:外部浏览器 CDP 走包内 `ExternalTransport`(零改动);内置浏览器 CDP 的 `EmbeddedTransport`(`webContents.debugger` 适配)留在宿主,随 ③ 一起移植。
|
||||
|
||||
**⑤ AI 编排能力**:`src/main/capabilities/ai-caps.ts` 消费包内 `chatCompletion`/`requireAiConfig`,配置经 `loadAiConfig` 注入。宿主提供等价 settings 即可,README 标注「可随后续阶段评估迁入包」。
|
||||
|
||||
### 6.4 权限与治理对接(R10)
|
||||
|
||||
- 危险能力清单(候选,迁入时与 Agent-Employee 权限模型逐一对齐):`System_ExecuteCommand`、`App_Kill`、键鼠注入家族(Mouse_*/Keyboard_*)、`Clipboard_*`、文件删除/移动家族。
|
||||
- 对接方式:在宿主的 RunContext 组装层做能力级 allowlist 拦截(能力 handler 内部无权限概念,这是刻意的——权限属宿主治理职责);`registerAll(engine, groups?)` 的按域注册可作为粗粒度开关。
|
||||
- 能力目录(catalog.json)与 Agent-Employee 插件体系对齐:`platformTag` 字段已就绪,能力 meta(`getCapabilityMeta(id)`,含参数 schema 与 errorCodes)可供其面板/Agent 消费。
|
||||
|
||||
## 4. 不迁入部分(宿主职责,迁入时明确归属)
|
||||
|
||||
| demo 侧模块 | 迁入时的处理 |
|
||||
|---|---|
|
||||
| `src/renderer/` | 不迁(Agent-Employee 用 SolidJS) |
|
||||
| `src/main/overlay.ts` / `desktop-pick.ts` / `vision-lab.ts` / `fixture/` | Electron 窗口强耦合,按 Agent-Employee 等价功能重写或放弃 |
|
||||
| `src/main/local-vision/` | onnxruntime-node 原生依赖,Bun 兼容性单独立项,**不阻塞迁移** |
|
||||
| `src/main/flows.ts` / `run/` / `settings.ts` | 应用级编排与持久化,宿主职责(包通过 SubflowSource/HostBridge 取用) |
|
||||
| `src/main/browser/session.ts` + `transport.ts` + browser* 能力 | 随宿主浏览器方案决策(§6.3 ③④) |
|
||||
|
||||
## 5. 迁入验收标准
|
||||
|
||||
1. `bun test`(包内纯逻辑测试)全绿 —— 补做 5.2 的结论。
|
||||
2. `bun run` 等价于 `npm run rpa-cli` 的 CLI host 跑通 5 个代表能力(桌面 / Excel / HTTP / 文件 / 屏幕),证明零宿主可运行。
|
||||
3. 能力快照零 diff(719 包内 id 对齐;宿主侧重建快照后 909 对齐)。
|
||||
4. `check:deps` 通过(保持零 Electron 零宿主引用)。
|
||||
5. HostBridge 注入后,human-dialog / window-extended DPI / ai-service 三条注入路径有真实交互验证(弹窗、高分屏拖拽、AI 调用各至少 1 例)。
|
||||
6. demo 侧(若保留):test + typecheck + catalog 快照三项全绿不回归。
|
||||
|
||||
## 6. 关键决策记录(迁入时不再重新讨论)
|
||||
|
||||
| # | 决策 |
|
||||
|---|---|
|
||||
| 1 | 物理搬迁 `git mv`,非引用保留 |
|
||||
| 2 | 保留全局注册表 + 副作用注册;`createEngine()` 工厂收拢进程级状态 |
|
||||
| 3 | 集中突击式迁移,不长期双轨 |
|
||||
| 4 | HostBridge 最小面:createFormWindow / dipToScreenPoint / screenToDipPoint / loadAiConfig |
|
||||
| 5 | 浏览器 CDP:外部路径入包,Embedded 留宿主 |
|
||||
| 6 | 五域 AutomationDriver 接口 + WindowsDriver 机械包一层(行为不变) |
|
||||
| 7 | catalog 只填 platformTag 数据,渲染层置灰延后 |
|
||||
| 8 | CapabilityMeta 可选 + 新能力强制 |
|
||||
| 9 | 注册双入口过渡(副作用 + registerAll) |
|
||||
| 10 | 5.2 Bun 冒烟延期到迁入时执行(本文 §2 第一项) |
|
||||
|
||||
风险 R1-R10 → 任务映射见 `docs/rpa-core-refactor-tasks.md` 末尾表格;迁移后的新规范增量见 `docs/dev-conventions.md`(结构化 meta 强制、错误码登记、powershell 助手唯一出口)。
|
||||
|
||||
---
|
||||
|
||||
## 7. 三端支持(Windows / macOS / Linux)
|
||||
|
||||
目标:项目最终要在三个桌面平台运行。架构钩子已预埋,工作分「迁入前小改」(demo 侧,避免跨仓库返工)与「迁入后实现」(Agent-Employee 侧,按域补 driver)两批。
|
||||
|
||||
### 7.1 已就位的三端基础(无需改动)
|
||||
|
||||
- **五域 AutomationDriver 接口**:能力层只面向 `driver.mouse/keyboard/window/screen/element`,Windows 实现整体可替换;
|
||||
- **`createEngine({ driver })` 注入** + **`registerAll(engine, groups?)` 按域注册**(可只注册当前平台支持的能力组);
|
||||
- **meta.platforms 字段**(`src/engine/registry.ts` 已定义,字面量待扩展);
|
||||
- **HostBridge 的 DPI 接口**走 Electron,mac Retina / Linux 由 Electron 屏蔽,包内无感知。
|
||||
|
||||
### 7.2 迁入前改(demo 侧,成本低的准备工作)
|
||||
|
||||
| # | 改什么 | 为什么现在做 |
|
||||
|---|---|---|
|
||||
| 1 | `platforms` 类型扩成 `"win32" \| "darwin" \| "linux"`,能力 meta 填 platforms 数据(catalog 从「只标 win32 组」改为逐能力标注) | 纯类型+数据;拖到 Agent-Employee 侧要跨仓库改 |
|
||||
| 2 | 引擎驱动工厂化:`engine.ts` 硬编码 `new WindowsDriver()`,改为按 `process.platform` 选择,非 win 平台明确抛 `platform_unsupported`;RUNTIME_ERROR_CODES 同步加该错误码 | 改动 10 行内 |
|
||||
| 3 | 注册组与平台边界对齐检查:excel-com / word-extended(COM)/ desktop-uia / desktop 等 Windows 专属能力确保归入可按平台过滤的组 | registerAll 的 groups 就是三端注册开关,组错位到时要重构 |
|
||||
| 4 | 测试 skipIf 守卫补齐:目前仅 4 个测试文件有 `process.platform !== "win32"` 守卫,键盘/鼠标/窗口冒烟都没有 | 补齐后纯逻辑测试能在 mac/linux CI 跑 |
|
||||
| 5 | dev-conventions 加约束:能力文件禁止散落 `process.platform` 分支(utility-extended.ts 现有 2 处,收口到明确清单) | 防止平台判断扩散,三端化时找不到边界 |
|
||||
|
||||
### 7.3 迁入后实现(Agent-Employee 侧,独立项目)
|
||||
|
||||
1. **LinuxDriver / MacDriver 实现**(接口已就位,按域补齐):
|
||||
- Linux:鼠标/键盘 xdotool(X11)、ydotool+uinput(Wayland,全局注入受限,是三端中最大技术坑)、element 走 AT-SPI、screen 走 xrandr;
|
||||
- macOS:鼠标/键盘 cliclick 或 CGEvent、window 走 AppleScript / System Events、element 走 AX Accessibility API;
|
||||
- **macOS TCC 权限**(辅助功能 / 屏幕录制 / 自动化授权)是 mac 端最大部署摩擦,迁入前先做可行性评估。
|
||||
2. **Excel/Word COM 替代**:mac 走 AppleScript + Office 自动化,linux 走 LibreOffice UNO;或第一版直接标 `platforms: ["win32"]` 不支持。
|
||||
3. **external.ts 浏览器路径表平台化**:当前硬编码 Windows 可执行路径 + 注册表查询 + powershell 进程枚举,需补 mac(`/Applications/Google Chrome.app/...` 等)与 linux(`which google-chrome`)路径表和对应进程发现分支。
|
||||
4. **剪贴板/桌面助手替换**:powershell → `pbpaste`/`pbcopy`(mac)、`xclip`/`wl-copy`(linux);人机对话框走 HostBridge 由宿主 GUI 方案实现,包内不感知。
|
||||
5. **onnxruntime/OCR 三端二进制**:原生依赖单独立项验证,不阻塞主线。
|
||||
6. **CI 三端矩阵**;catalog 渲染层按 platforms 置灰(决策点 7 的延后项此时落地)。
|
||||
|
||||
### 7.4 三端化原则
|
||||
|
||||
- 平台差异只存在于 **driver 实现**与**极少数能力文件的明确清单**内;能力 handler 层、引擎、注册表、HostBridge 契约保持平台无关。
|
||||
- 能力不支持当前平台时的统一行为:注册期按 platforms 过滤(面板不出现),运行期兜底抛 `platform_unavailable` / `platform_unsupported`。
|
||||
- 新增平台专属脚本助手(如 osascript / xdotool 包装)与 `powershell.ts` 同级同规:放 `driver/<platform>/`,经 check:deps 与 review 把关,禁止能力文件自行 spawn。
|
||||
56
packages/rpa-core/src/capabilities/captcha-caps.test.ts
Normal file
56
packages/rpa-core/src/capabilities/captcha-caps.test.ts
Normal file
@ -0,0 +1,56 @@
|
||||
import { existsSync } from "node:fs";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { getHandler } from "../engine/registry";
|
||||
import type { CatalogComponent, FlowNode } from "../types";
|
||||
import type { RunContext } from "../engine/types";
|
||||
import "./remaining-core";
|
||||
import "./captcha-caps"; // last-wins:覆盖 remaining-core 的通用实现
|
||||
import { cleanCodeText, parseCalculation } from "./captcha-caps";
|
||||
|
||||
const ctx = { signal: new AbortController().signal, log: () => {}, vars: {}, evidenceDir: "D:/office-test2/ocr-test" } as unknown as RunContext;
|
||||
const node: FlowNode = { id: "smoke", componentId: "smoke", position: { x: 0, y: 0 }, values: {}, options: {} };
|
||||
|
||||
describe("parseCalculation 解析层(纯逻辑)", () => {
|
||||
it("基础四则与等号问号", () => {
|
||||
expect(parseCalculation("3+5=?")).toEqual({ expression: "3+5", result: 8 });
|
||||
});
|
||||
it("x/X/* 归一为乘", () => {
|
||||
expect(parseCalculation("6x4=?")!.result).toBe(24);
|
||||
expect(parseCalculation("9X9=?")!.result).toBe(81);
|
||||
});
|
||||
it("全角与杂字符", () => {
|
||||
expect(parseCalculation("12+7=?")!.result).toBe(19);
|
||||
});
|
||||
it("长破折号/Unicode 减号", () => {
|
||||
expect(parseCalculation("12—7")!.result).toBe(5);
|
||||
});
|
||||
it("除法整除与小数", () => {
|
||||
expect(parseCalculation("8/2")!.result).toBe(4);
|
||||
expect(parseCalculation("7/2")!.result).toBe(3.5);
|
||||
});
|
||||
it("无算式返回 null", () => {
|
||||
expect(parseCalculation("ab12cd")).toBeNull();
|
||||
});
|
||||
it("字符验证码清洗", () => {
|
||||
expect(cleanCodeText("a b 7 X ? = ")).toBe("ab7X");
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(!existsSync("D:/office-test2/ocr-test/captcha_0_3p5_q.png"))("OCR_CalculationCode 端到端(真实模型)", () => {
|
||||
const cases: Array<[string, string]> = [
|
||||
["captcha_0_3p5_q", "8"],
|
||||
["captcha_1_12m7_q", "5"],
|
||||
["captcha_2_6t4_q", "24"],
|
||||
["captcha_3_15p28_q", "43"],
|
||||
["captcha_4_9t9_q", "81"],
|
||||
];
|
||||
for (const [file, expected] of cases) {
|
||||
it(`${file} -> ${expected}`, async () => {
|
||||
const handler = getHandler("OCR_CalculationCode");
|
||||
expect(handler).toBeTruthy();
|
||||
const out = await handler!(ctx, node, { path: `D:/office-test2/ocr-test/${file}.png` }, {} as CatalogComponent);
|
||||
console.log(`[${file}]`, String(out));
|
||||
expect(String(out)).toBe(expected);
|
||||
}, 300000);
|
||||
}
|
||||
});
|
||||
82
packages/rpa-core/src/capabilities/captcha-caps.ts
Normal file
82
packages/rpa-core/src/capabilities/captcha-caps.ts
Normal file
@ -0,0 +1,82 @@
|
||||
/**
|
||||
* 验证码类 OCR 专门实现。
|
||||
*
|
||||
* 此前 OCR_CalculationCode / OCR_CommonEnglishNumbersCode 与其他 30 个 OCR
|
||||
* 组件共用通用实现(remaining-core 的 ocrImage),只返回整图 OCR 原文:
|
||||
* - 算术验证码返回的是 "3+5=?" 这样的算式原文,没有计算结果,流程无法直接填入答案;
|
||||
* - 真实验证码的干扰线/噪点会让原文混入杂字符,符号常被认成 x/X/*、全角等。
|
||||
*
|
||||
* 这里提供专门处理:OCR 原文 -> 字符归一化 -> 算式解析 -> 计算结果。
|
||||
* 解析失败时返回清洗后的原文并记 warning,不让节点直接失败(调用方可自行判断)。
|
||||
*/
|
||||
import { getHandler, register } from "../engine/registry";
|
||||
import { RuntimeError, type RunContext } from "../engine/types";
|
||||
import type { CatalogComponent, FlowNode } from "../types";
|
||||
|
||||
interface CalculationParse {
|
||||
expression: string;
|
||||
result: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* 从 OCR 原文中解析算术算式。
|
||||
* 归一化范围:全角数字/符号、常见运算符误认(x/X/* -> 乘,全角+-×÷,长破折号 -> 减)。
|
||||
*/
|
||||
export function parseCalculation(raw: string): CalculationParse | null {
|
||||
const s = String(raw ?? "")
|
||||
.replace(/[0-9]/g, (c) => String.fromCharCode(c.charCodeAt(0) - 0xfee0))
|
||||
.replace(/+/g, "+")
|
||||
.replace(/[-―—–−]/g, "-")
|
||||
.replace(/[×✕✖xX*X*]/g, "*")
|
||||
.replace(/[÷/]/g, "/")
|
||||
.replace(/[==??。,,、\s]/g, "");
|
||||
const match = s.match(/(\d+)([+\-*/])(\d+)/);
|
||||
if (!match) return null;
|
||||
const a = Number(match[1]);
|
||||
const b = Number(match[3]);
|
||||
const op = match[2];
|
||||
let result: number;
|
||||
if (op === "+") result = a + b;
|
||||
else if (op === "-") result = a - b;
|
||||
else if (op === "*") result = a * b;
|
||||
else {
|
||||
if (b === 0) return null;
|
||||
result = a / b;
|
||||
}
|
||||
if (!Number.isFinite(result)) return null;
|
||||
result = Math.round(result * 100) / 100;
|
||||
const prettyOp = op === "*" ? "×" : op === "/" ? "÷" : op;
|
||||
return { expression: `${a}${prettyOp}${b}`, result };
|
||||
}
|
||||
|
||||
/** 普通字符验证码:去除空白与常见杂字符,保留字母数字 */
|
||||
export function cleanCodeText(raw: string): string {
|
||||
return String(raw ?? "")
|
||||
.replace(/\s+/g, "")
|
||||
.replace(/[==??.,,、;;:"'()()\[\]{}]/g, "");
|
||||
}
|
||||
|
||||
async function ocrRaw(args: Record<string, unknown>, node: FlowNode, ctx: RunContext): Promise<string> {
|
||||
// 复用通用 OCR(remaining-core 已注册的 OCR_GeneralTextV2)拿原文,避免重复加载模型
|
||||
const base = getHandler("OCR_GeneralTextV2");
|
||||
if (!base) throw new RuntimeError("capability_not_implemented", "通用 OCR 能力不可用", { nodeId: node.id, componentId: node.componentId });
|
||||
return String(await base(ctx, node, args, {} as CatalogComponent));
|
||||
}
|
||||
|
||||
register("OCR_CalculationCode", async (ctx, node, args) => {
|
||||
const raw = await ocrRaw(args, node, ctx);
|
||||
const parsed = parseCalculation(raw);
|
||||
if (!parsed) {
|
||||
await ctx.log(`算术验证码未能解析出算式(原文:${raw}),已返回清洗原文`, "warn", node.id);
|
||||
return raw.replace(/\s+/g, "");
|
||||
}
|
||||
await ctx.log(`算术验证码:${parsed.expression} = ${parsed.result}(OCR 原文:${raw})`, "info", node.id);
|
||||
return String(parsed.result);
|
||||
});
|
||||
|
||||
register("OCR_CommonEnglishNumbersCode", async (ctx, node, args) => {
|
||||
const raw = await ocrRaw(args, node, ctx);
|
||||
const cleaned = cleanCodeText(raw);
|
||||
await ctx.log(`字符验证码:${cleaned}(OCR 原文:${raw})`, "info", node.id);
|
||||
return cleaned;
|
||||
});
|
||||
20
packages/rpa-core/src/capabilities/captcha-recon.test.ts
Normal file
20
packages/rpa-core/src/capabilities/captcha-recon.test.ts
Normal file
@ -0,0 +1,20 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { getHandler } from "../engine/registry";
|
||||
import type { CatalogComponent, FlowNode } from "../types";
|
||||
import type { RunContext } from "../engine/types";
|
||||
import "./remaining-core";
|
||||
|
||||
const ctx = { signal: new AbortController().signal, log: () => {}, vars: {}, evidenceDir: "D:/office-test2/ocr-test" } as unknown as RunContext;
|
||||
const node: FlowNode = { id: "smoke", componentId: "smoke", position: { x: 0, y: 0 }, values: {}, options: {} };
|
||||
|
||||
describe("算术验证码现状侦察", () => {
|
||||
const files = ["captcha_0_3p5_q", "captcha_1_12m7_q", "captcha_2_6t4_q", "captcha_3_15p28_q", "captcha_4_9t9_q"];
|
||||
it("当前 OCR 原始输出", async () => {
|
||||
for (const f of files) {
|
||||
const handler = getHandler("OCR_CalculationCode");
|
||||
const out = await handler!(ctx, node, { path: `D:/office-test2/ocr-test/${f}.png` }, {} as CatalogComponent);
|
||||
console.log(`[${f}]`, JSON.stringify(out));
|
||||
}
|
||||
expect(true).toBe(true);
|
||||
}, 600000);
|
||||
});
|
||||
@ -36,6 +36,7 @@ import "./data-table-common";
|
||||
import "./basic-encryption";
|
||||
import "./mail-remaining";
|
||||
import "./remaining-core";
|
||||
import "./captcha-caps";
|
||||
import "./common-final";
|
||||
import "./human-dialog";
|
||||
import "./ai-service";
|
||||
|
||||
@ -101511,99 +101511,7 @@
|
||||
"platformTag": null,
|
||||
"icon": "app",
|
||||
"risk": "high",
|
||||
"fields": [
|
||||
{
|
||||
"key": "username",
|
||||
"label": "用户名",
|
||||
"widget": "text",
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": true,
|
||||
"defaultValue": "",
|
||||
"placeholder": "请设置锁屏后再次开启时显示的用户名",
|
||||
"description": "",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_str"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "pwd",
|
||||
"label": "密码",
|
||||
"widget": "secret",
|
||||
"type": "string",
|
||||
"displayType": "encryption",
|
||||
"required": true,
|
||||
"defaultValue": "",
|
||||
"placeholder": "请输入锁屏后再次开启需要输入的密码",
|
||||
"description": "",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_password",
|
||||
"rpa_str"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "width",
|
||||
"label": "分辨率(宽)",
|
||||
"widget": "text",
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": true,
|
||||
"defaultValue": "",
|
||||
"placeholder": "",
|
||||
"description": "",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_int"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "height",
|
||||
"label": "分辨率(高)",
|
||||
"widget": "text",
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": true,
|
||||
"defaultValue": "",
|
||||
"placeholder": "",
|
||||
"description": "",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_int"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "scale_factor",
|
||||
"label": "缩放比",
|
||||
"widget": "text",
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": true,
|
||||
"defaultValue": "",
|
||||
"placeholder": "",
|
||||
"description": "缩放比范围100-500,默认值为100",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_int"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "port",
|
||||
"label": "端口号",
|
||||
"widget": "text",
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": false,
|
||||
"defaultValue": "",
|
||||
"placeholder": "请输入端口号,示例: 8008",
|
||||
"description": "指定端口号,如果为空,将自动读取系统当前设置。",
|
||||
"advanced": true,
|
||||
"valueTypes": [
|
||||
"rpa_str"
|
||||
]
|
||||
}
|
||||
],
|
||||
"fields": [],
|
||||
"output": {
|
||||
"label": "输出到变量",
|
||||
"key": "var_name",
|
||||
@ -101622,7 +101530,8 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
"slots": []
|
||||
"slots": [],
|
||||
"label": "锁屏"
|
||||
},
|
||||
{
|
||||
"id": "ScreenControl_ScreenUnlock",
|
||||
@ -101675,7 +101584,8 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
"slots": []
|
||||
"slots": [],
|
||||
"label": "解锁屏幕(仅唤醒,登录锁屏需手动输入密码)"
|
||||
},
|
||||
{
|
||||
"id": "StrUtil_InStr",
|
||||
@ -125574,4 +125484,4 @@
|
||||
"slots": []
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user