fix: 图像匹配改相似度语义并修三处准确度缺陷(尺度对齐/早退阈值/噪声死区)
This commit is contained in:
yu
parent
38af534f70
commit
9868f5d51c
@ -66782,9 +66782,9 @@
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": true,
|
||||
"defaultValue": "0.2",
|
||||
"defaultValue": "0.8",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "即需要查找的图片与模板图片总体像素点误差差异判断相似度的阈值指标，与特征相似度共同决定组件运行结果。用户可自定义配置一个0~1区间的相似度取值（如0.55即55%相似度），0为完全不相似、1为完全一致，越大则命中阈值越高。（如无特殊需求，建议默认0.2） ",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -67016,7 +67016,7 @@
|
||||
"required": true,
|
||||
"defaultValue": "0.95",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "查找当前界面与截图图片 视觉特征差异相似程度的匹配阈值。当相似度100%时，即代表当前界面存在与截图图片完全一致的图像才能执行成功。",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -67359,9 +67359,9 @@
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": true,
|
||||
"defaultValue": "0.2",
|
||||
"defaultValue": "0.8",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "即需要查找的图片与模板图片总体像素点误差差异判断相似度的阈值指标，与特征相似度共同决定组件运行结果。用户可自定义配置一个0~1区间的相似度取值（如0.55即55%相似度），0为完全不相似、1为完全一致，越大则命中阈值越高。（如无特殊需求，建议默认0.2） ",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -67612,7 +67612,7 @@
|
||||
"required": true,
|
||||
"defaultValue": "0.95",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "查找当前界面与截图图片 视觉特征差异相似程度的匹配阈值。当相似度100%时，即代表当前界面存在与截图图片完全一致的图像才能执行成功。",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -67708,9 +67708,9 @@
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": false,
|
||||
"defaultValue": "0.1",
|
||||
"defaultValue": "0.8",
|
||||
"placeholder": "输入内容或 ${变量}",
|
||||
"description": "填写参数值，支持 ${变量} 引用上游输出",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": []
|
||||
},
|
||||
@ -67800,9 +67800,9 @@
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": true,
|
||||
"defaultValue": "0.2",
|
||||
"defaultValue": "0.8",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "即需要查找的图片与模板图片总体像素点误差差异判断相似度的阈值指标，与特征相似度共同决定组件运行结果。用户可自定义配置一个0~1区间的相似度取值（如0.55即55%相似度），0为完全不相似、1为完全一致，越大则命中阈值越高。（如无特殊需求，建议默认0.2） ",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -67982,7 +67982,7 @@
|
||||
"required": true,
|
||||
"defaultValue": "0.95",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "查找当前界面与截图图片 视觉特征差异相似程度的匹配阈值。当相似度100%时，即代表当前界面存在与截图图片完全一致的图像才能执行成功。",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -68138,7 +68138,7 @@
|
||||
"required": true,
|
||||
"defaultValue": "0.95",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "查找 当前界面 与 截图图片 视觉特征差异相似程度的匹配阈值。当相似度100%时，即代表当前界面存在与截图图片完全一致的图像才能执行成功。",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -68220,7 +68220,7 @@
|
||||
"required": false,
|
||||
"defaultValue": "0.8",
|
||||
"placeholder": "输入内容或 ${变量}",
|
||||
"description": "填写参数值，支持 ${变量} 引用上游输出",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": []
|
||||
}
|
||||
@ -68484,9 +68484,9 @@
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": true,
|
||||
"defaultValue": "0.2",
|
||||
"defaultValue": "0.8",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "即需要查找的图片与模板图片总体像素点误差差异判断相似度的阈值指标，与特征相似度共同决定组件运行结果。用户可自定义配置一个0~1区间的相似度取值（如0.55即55%相似度），0为完全不相似、1为完全一致，越大则命中阈值越高。（如无特殊需求，建议默认0.2） ",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -68754,7 +68754,7 @@
|
||||
"required": true,
|
||||
"defaultValue": "0.95",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "查找当前界面与截图图片 视觉特征差异相似程度的匹配阈值。当相似度100%时，即代表当前界面存在与截图图片完全一致的图像才能执行成功。",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -68962,7 +68962,7 @@
|
||||
"required": true,
|
||||
"defaultValue": "0.95",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "查找 当前界面 与 截图图片 视觉特征差异相似程度的匹配阈值。当相似度100%时，即代表当前界面存在与截图图片完全一致的图像才能执行成功。",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -69049,9 +69049,9 @@
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": true,
|
||||
"defaultValue": "0.2",
|
||||
"defaultValue": "0.8",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "即需要查找的图片与模板图片总体像素点误差差异判断相似度的阈值指标，与特征相似度共同决定组件运行结果。用户可自定义配置一个0~1区间的相似度取值（如0.55即55%相似度），0为完全不相似、1为完全一致，越大则命中阈值越高。（如无特殊需求，建议默认0.2） ",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -69202,9 +69202,9 @@
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": true,
|
||||
"defaultValue": "0.2",
|
||||
"defaultValue": "0.8",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "即需要查找的图片与模板图片总体像素点误差差异判断相似度的阈值指标，与特征相似度共同决定组件运行结果。用户可自定义配置一个0~1区间的相似度取值（如0.55即55%相似度），0为完全不相似、1为完全一致，越大则命中阈值越高。（如无特殊需求，建议默认0.2） ",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -69355,9 +69355,9 @@
|
||||
"type": "string",
|
||||
"displayType": "text-expression",
|
||||
"required": true,
|
||||
"defaultValue": "0.2",
|
||||
"defaultValue": "0.8",
|
||||
"placeholder": "请输入相似度阈值",
|
||||
"description": "即需要查找的图片与模板图片总体像素点误差差异判断相似度的阈值指标，与特征相似度共同决定组件运行结果。用户可自定义配置一个0~1区间的相似度取值（如0.55即55%相似度），0为完全不相似、1为完全一致，越大则命中阈值越高。（如无特殊需求，建议默认0.2） ",
|
||||
"description": "模板相似度阈值（0~1，越大越严格）：0.9 表示屏幕上至少 90% 像素与模板一致才算找到。框选的模板与实际界面光照/渲染略有差异时可适当调低，如无特殊需求建议 0.8。",
|
||||
"advanced": false,
|
||||
"valueTypes": [
|
||||
"rpa_num"
|
||||
@ -125582,4 +125582,4 @@
|
||||
"slots": []
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
83
src/main/capabilities/vision-match.test.ts
Normal file
83
src/main/capabilities/vision-match.test.ts
Normal file
@ -0,0 +1,83 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { PNG } from "pngjs";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { findImage } from "./vision";
|
||||
|
||||
function makeScreen(width: number, height: number, patch: { x: number; y: number; w: number; h: number }): PNG {
|
||||
const png = new PNG({ width, height });
|
||||
for (let y = 0; y < height; y++) {
|
||||
for (let x = 0; x < width; x++) {
|
||||
const i = (y * width + x) * 4;
|
||||
const inPatch = x >= patch.x && x < patch.x + patch.w && y >= patch.y && y < patch.y + patch.h;
|
||||
// 背景: 平滑斜向渐变(贴近真实界面); 补丁: 纯红色
|
||||
png.data[i] = inPatch ? 220 : (x + y) % 90;
|
||||
png.data[i + 1] = inPatch ? 30 : (x + y + 30) % 90;
|
||||
png.data[i + 2] = inPatch ? 40 : (x + y + 60) % 90;
|
||||
png.data[i + 3] = 255;
|
||||
}
|
||||
}
|
||||
return png;
|
||||
}
|
||||
|
||||
function crop(source: PNG, x: number, y: number, w: number, h: number): PNG {
|
||||
const out = new PNG({ width: w, height: h });
|
||||
PNG.bitblt(source, out, x, y, w, h, 0, 0);
|
||||
return out;
|
||||
}
|
||||
|
||||
describe("findImage 匹配质量", () => {
|
||||
it("相似度0.9时精确命中模板位置且不误报", async () => {
|
||||
const patch = { x: 123, y: 77, w: 48, h: 36 };
|
||||
const screen = makeScreen(400, 300, patch);
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "vision-"));
|
||||
const screenPath = path.join(dir, "screen.png");
|
||||
const templatePath = path.join(dir, "template.png");
|
||||
fs.writeFileSync(screenPath, PNG.sync.write(screen));
|
||||
fs.writeFileSync(templatePath, PNG.sync.write(crop(screen, patch.x, patch.y, patch.w, patch.h)));
|
||||
// 相似度 0.9 → 最大差异 0.1
|
||||
const result = await findImage(templatePath, screenPath, 0.1);
|
||||
expect(result).not.toBeNull();
|
||||
// 采样步长落在整数边界时, 相邻1px偏移可能同样满分, 允许±1px
|
||||
expect(Math.abs(result!.x - patch.x)).toBeLessThanOrEqual(1);
|
||||
expect(Math.abs(result!.y - patch.y)).toBeLessThanOrEqual(1);
|
||||
expect(result!.score).toBeGreaterThanOrEqual(0.9);
|
||||
// 无该补丁的屏幕应返回 null 而非乱命中
|
||||
const other = makeScreen(400, 300, { x: 0, y: 0, w: 1, h: 1 });
|
||||
const otherPath = path.join(dir, "other.png");
|
||||
fs.writeFileSync(otherPath, PNG.sync.write(other));
|
||||
const miss = await findImage(templatePath, otherPath, 0.1);
|
||||
expect(miss).toBeNull();
|
||||
});
|
||||
|
||||
it("模板被缩放(1.25倍)时按尺度兜底命中", async () => {
|
||||
const patch = { x: 200, y: 120, w: 60, h: 40 };
|
||||
const screen = makeScreen(500, 350, patch);
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "vision-"));
|
||||
const screenPath = path.join(dir, "screen.png");
|
||||
fs.writeFileSync(screenPath, PNG.sync.write(screen));
|
||||
// 模板取自 DIP 截图(比物理屏小1.25倍): 把补丁缩小1/1.25后存为模板
|
||||
const { createCanvas, loadImage } = await import("@napi-rs/canvas");
|
||||
const small = crop(screen, patch.x, patch.y, patch.w, patch.h);
|
||||
const srcCanvas = createCanvas(small.width, small.height);
|
||||
const srcCtx = srcCanvas.getContext("2d");
|
||||
const imgData = srcCtx.createImageData(small.width, small.height);
|
||||
imgData.data.set(small.data);
|
||||
srcCtx.putImageData(imgData, 0, 0);
|
||||
const dstCanvas = createCanvas(Math.round(small.width / 1.25), Math.round(small.height / 1.25));
|
||||
const dstCtx = dstCanvas.getContext("2d");
|
||||
dstCtx.drawImage(srcCanvas, 0, 0, dstCanvas.width, dstCanvas.height);
|
||||
// canvas → pngjs PNG（PNG.sync.write 只认 pngjs 对象）
|
||||
const scaledData = dstCtx.getImageData(0, 0, dstCanvas.width, dstCanvas.height);
|
||||
const templatePng = new PNG({ width: dstCanvas.width, height: dstCanvas.height });
|
||||
templatePng.data = Buffer.from(scaledData.data);
|
||||
const templatePath = path.join(dir, "template-small.png");
|
||||
fs.writeFileSync(templatePath, PNG.sync.write(templatePng));
|
||||
const result = await findImage(templatePath, screenPath, 0.1, [1.25, 1 / 1.25]);
|
||||
expect(result).not.toBeNull();
|
||||
expect(Math.abs(result!.x - patch.x)).toBeLessThanOrEqual(3);
|
||||
expect(Math.abs(result!.y - patch.y)).toBeLessThanOrEqual(3);
|
||||
void loadImage;
|
||||
});
|
||||
});
|
||||
@ -10,18 +10,41 @@ function text(args: Record<string, unknown>, keys: string[], fallback = ""): str
|
||||
return fallback;
|
||||
}
|
||||
|
||||
/** 主屏物理分辨率截图。desktopCapturer thumbnailSize 乘 scaleFactor，
|
||||
* 与「屏幕框选」生成的模板（物理像素裁剪）尺度一致——
|
||||
* 若用 DIP 逻辑分辨率, 缩放≠100% 的显示器上模板比截图里的同款 UI 大 scaleFactor 倍, 像素匹配永远对不齐 */
|
||||
async function capturePrimaryScreenPng(): Promise<PNG> {
|
||||
const { desktopCapturer, screen } = await import("electron");
|
||||
const primary = screen.getPrimaryDisplay();
|
||||
const scale = primary.scaleFactor || 1;
|
||||
const sources = await desktopCapturer.getSources({
|
||||
types: ["screen"],
|
||||
thumbnailSize: { width: primary.size.width, height: primary.size.height }
|
||||
thumbnailSize: {
|
||||
width: Math.round(primary.size.width * scale),
|
||||
height: Math.round(primary.size.height * scale)
|
||||
}
|
||||
});
|
||||
const source = sources[0];
|
||||
if (!source) throw new RuntimeError("desktop_error", "无法获取屏幕截图");
|
||||
return PNG.sync.read(source.thumbnail.toPNG());
|
||||
}
|
||||
|
||||
/** 主屏缩放比：区域 JSON 等参数按 DIP 语义存储，裁剪物理分辨率截图前需换算 */
|
||||
async function primaryScale(): Promise<number> {
|
||||
const { screen } = await import("electron");
|
||||
return screen.getPrimaryDisplay().scaleFactor || 1;
|
||||
}
|
||||
|
||||
/** DIP 区域 → 物理像素区域（截图已改为物理分辨率，坐标需同步放大） */
|
||||
function scaleRegion(region: { x: number; y: number; width: number; height: number }, scale: number) {
|
||||
return {
|
||||
x: Math.round(region.x * scale),
|
||||
y: Math.round(region.y * scale),
|
||||
width: Math.max(1, Math.round(region.width * scale)),
|
||||
height: Math.max(1, Math.round(region.height * scale))
|
||||
};
|
||||
}
|
||||
|
||||
/** 按魔数判断是否 PNG（89 50 4E 47） */
|
||||
function isPngBuffer(buffer: Buffer): boolean {
|
||||
return buffer.length > 8 && buffer[0] === 0x89 && buffer[1] === 0x50 && buffer[2] === 0x4e && buffer[3] === 0x47;
|
||||
@ -156,7 +179,9 @@ register("Image_Screenshot", async (ctx, node, args) => {
|
||||
recommendation: '在 position 填写 {"x":0,"y":0,"width":100,"height":80} 或 [x, y, width, height]，或将截图范围改回「整个屏幕」'
|
||||
});
|
||||
}
|
||||
image = cropPng(image, parseRegion(regionRaw, image));
|
||||
// 区域 JSON 是 DIP 坐标，截图是物理分辨率，裁剪前换算
|
||||
const scale = await primaryScale();
|
||||
image = cropPng(image, clampRegion(scaleRegion(parseRegion(regionRaw, image), scale), image));
|
||||
}
|
||||
|
||||
await fs.mkdir(path.dirname(output), { recursive: true });
|
||||
@ -208,6 +233,9 @@ Write-Output ("{0},{1},{2},{3}" -f $r.Left, $r.Top, $r.Right, $r.Bottom)`);
|
||||
return { x: left, y: top, width: Math.max(1, right - left), height: Math.max(1, bottom - top) };
|
||||
}
|
||||
|
||||
/** 单通道噪声死区：JPG 压缩伪影/抗锯齿带来 ±10 级内的像素波动，不应计入差异 */
|
||||
const NOISE_FLOOR = 10;
|
||||
|
||||
function pixelDifference(screen: PNG, template: PNG, offsetX: number, offsetY: number, sampleStep: number, cutoff: number): number {
|
||||
let difference = 0;
|
||||
let samples = 0;
|
||||
@ -215,27 +243,21 @@ function pixelDifference(screen: PNG, template: PNG, offsetX: number, offsetY: n
|
||||
for (let x = 0; x < template.width; x += sampleStep) {
|
||||
const sourceIndex = ((offsetY + y) * screen.width + offsetX + x) * 4;
|
||||
const templateIndex = (y * template.width + x) * 4;
|
||||
difference += Math.abs(screen.data[sourceIndex] - template.data[templateIndex]);
|
||||
difference += Math.abs(screen.data[sourceIndex + 1] - template.data[templateIndex + 1]);
|
||||
difference += Math.abs(screen.data[sourceIndex + 2] - template.data[templateIndex + 2]);
|
||||
for (let c = 0; c < 3; c++) {
|
||||
const d = Math.abs(screen.data[sourceIndex + c] - template.data[templateIndex + c]);
|
||||
difference += d <= NOISE_FLOOR ? 0 : d - NOISE_FLOOR;
|
||||
}
|
||||
samples += 3;
|
||||
if (difference > cutoff * samples) return difference / (samples * 255);
|
||||
// cutoff 是 0~1 的归一化差异上限; difference 是通道差累加值(0~255 每通道),
|
||||
// 提前退出必须按 255 折算——之前漏乘导致几乎所有位置在第一个像素就被剪掉, 得分全乱
|
||||
if (difference > cutoff * samples * 255) return difference / (samples * 255);
|
||||
}
|
||||
}
|
||||
return samples ? difference / (samples * 255) : 1;
|
||||
}
|
||||
|
||||
export async function findImage(templatePath: string, screenshotPath: string, accuracy: number): Promise<MatchResult | null> {
|
||||
const [screenBuffer, templateBuffer] = await Promise.all([fs.readFile(screenshotPath), fs.readFile(templatePath)]);
|
||||
const screen = PNG.sync.read(screenBuffer);
|
||||
// 模板图可能是 JPG 等非 PNG 格式, 按魔数选择解码器
|
||||
const template = isPngBuffer(templateBuffer) ? PNG.sync.read(templateBuffer) : await decodeImageFile(templatePath);
|
||||
if (template.width > screen.width || template.height > screen.height) {
|
||||
throw new RuntimeError("validation_error", `模板图片（${template.width}×${template.height}）比截图画面（${screen.width}×${screen.height}）还大，无法查找`, {
|
||||
recommendation: "模板应是屏幕上要寻找的小图案（按钮、图标、文字块），不要用整屏大图"
|
||||
});
|
||||
}
|
||||
const maximumDifference = Math.max(0.01, Math.min(0.95, accuracy || 0.2));
|
||||
/** 在屏幕图里扫描模板的最佳位置（粗扫网格 + 精化） */
|
||||
function scanBest(screen: PNG, template: PNG, maximumDifference: number): MatchResult | null {
|
||||
const sampleStep = Math.max(1, Math.floor(Math.min(template.width, template.height) / 18));
|
||||
const scanStep = Math.max(1, Math.floor(Math.min(template.width, template.height) / 12));
|
||||
let best: MatchResult | null = null;
|
||||
@ -262,6 +284,57 @@ export async function findImage(templatePath: string, screenshotPath: string, ac
|
||||
return best;
|
||||
}
|
||||
|
||||
/** 用 canvas 双线性缩放模板（匹配尺度兜底用） */
|
||||
async function resizeTemplate(template: PNG, scale: number): Promise<PNG> {
|
||||
const { createCanvas } = await import("@napi-rs/canvas");
|
||||
const width = Math.max(1, Math.round(template.width * scale));
|
||||
const height = Math.max(1, Math.round(template.height * scale));
|
||||
const source = createCanvas(template.width, template.height);
|
||||
const sourceContext = source.getContext("2d");
|
||||
const imageData = sourceContext.createImageData(template.width, template.height);
|
||||
imageData.data.set(template.data);
|
||||
sourceContext.putImageData(imageData, 0, 0);
|
||||
const target = createCanvas(width, height);
|
||||
const targetContext = target.getContext("2d");
|
||||
targetContext.imageSmoothingEnabled = true;
|
||||
targetContext.imageSmoothingQuality = "high";
|
||||
targetContext.drawImage(source, 0, 0, width, height);
|
||||
const scaled = targetContext.getImageData(0, 0, width, height);
|
||||
const png = new PNG({ width, height });
|
||||
png.data = Buffer.from(scaled.data);
|
||||
return png;
|
||||
}
|
||||
|
||||
/**
|
||||
* 模板匹配。extraScales 提供尺度兜底：截图与模板的获取渠道分辨率不一致时
|
||||
* （如旧版 DIP 截图存下的模板 vs 现在的物理分辨率截图），按给定缩放比重试。
|
||||
*/
|
||||
export async function findImage(templatePath: string, screenshotPath: string, accuracy: number, extraScales: number[] = []): Promise<MatchResult | null> {
|
||||
const [screenBuffer, templateBuffer] = await Promise.all([fs.readFile(screenshotPath), fs.readFile(templatePath)]);
|
||||
const screen = PNG.sync.read(screenBuffer);
|
||||
// 模板图可能是 JPG 等非 PNG 格式, 按魔数选择解码器
|
||||
const template = isPngBuffer(templateBuffer) ? PNG.sync.read(templateBuffer) : await decodeImageFile(templatePath);
|
||||
if (template.width > screen.width || template.height > screen.height) {
|
||||
throw new RuntimeError("validation_error", `模板图片（${template.width}×${template.height}）比截图画面（${screen.width}×${screen.height}）还大，无法查找`, {
|
||||
recommendation: "模板应是屏幕上要寻找的小图案（按钮、图标、文字块），不要用整屏大图"
|
||||
});
|
||||
}
|
||||
const maximumDifference = Math.max(0.01, Math.min(0.95, accuracy || 0.2));
|
||||
const best = scanBest(screen, template, maximumDifference);
|
||||
if (best) return best;
|
||||
for (const scale of extraScales) {
|
||||
if (!Number.isFinite(scale) || scale === 1) continue;
|
||||
const scaled = await resizeTemplate(template, scale);
|
||||
if (scaled.width > screen.width || scaled.height > screen.height) continue;
|
||||
const scaledBest = scanBest(screen, scaled, maximumDifference);
|
||||
if (scaledBest) {
|
||||
// 结果框用原模板尺寸描述（宽高仅供点击锚点计算，位置才是关键）
|
||||
return { ...scaledBest, width: template.width, height: template.height };
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function locate(args: Record<string, unknown>, evidenceDir: string, nodeId: string): Promise<MatchResult> {
|
||||
const rawTemplate = text(args, ["image", "image_path", "template"]);
|
||||
if (!rawTemplate.trim()) throw new RuntimeError("validation_error", "模板图片路径为空", { nodeId });
|
||||
@ -283,8 +356,10 @@ async function locate(args: Record<string, unknown>, evidenceDir: string, nodeId
|
||||
} else if (rangeMode === "3") {
|
||||
const regionRaw = text(args, ["shot_position", "action", "position"]);
|
||||
if (!regionRaw.trim()) throw new RuntimeError("validation_error", "查找范围为「指定区域」时，需要填写区域 JSON", { nodeId, recommendation: '在 shot_position 填写 {"x":0,"y":0,"width":100,"height":80} 或 [x, y, width, height]' });
|
||||
const scale = await primaryScale();
|
||||
const screen = await capturePrimaryScreenPng();
|
||||
const region = clampRegion(parseRegion(regionRaw, screen), screen);
|
||||
// 区域 JSON 是 DIP 坐标（框选工具按 DIP 返回），截图是物理分辨率，裁剪前放大
|
||||
const region = clampRegion(scaleRegion(parseRegion(regionRaw, screen), scale), screen);
|
||||
await fs.writeFile(screenshot, PNG.sync.write(cropPng(screen, region)));
|
||||
offsetX = region.x;
|
||||
offsetY = region.y;
|
||||
@ -292,8 +367,16 @@ async function locate(args: Record<string, unknown>, evidenceDir: string, nodeId
|
||||
await captureScreen(screenshot);
|
||||
}
|
||||
|
||||
const result = await findImage(template, screenshot, Number(text(args, ["accuracy"], "0.2")));
|
||||
if (!result) throw new RuntimeError("not_found", "屏幕中未找到目标图像", { nodeId, recommendation: "重新截取模板，或调低匹配精度" });
|
||||
// accuracy 字段语义为「相似度」(0~1, 越大越严格, 0.9=至少90%像素一致)——与界面文案一致。
|
||||
// 引擎内部用最大允许差异 = 1 - 相似度。此前引擎直接把该值当差异阈值,
|
||||
// 用户填 0.9 反而放宽到"接受90%像素差异", 导致到处误命中。
|
||||
const rawAccuracy = Number(text(args, ["accuracy"], "0.8"));
|
||||
const similarity = Math.max(0.05, Math.min(1, Number.isFinite(rawAccuracy) && rawAccuracy > 0 ? rawAccuracy : 0.8));
|
||||
// 原生 1:1 匹配失败时, 按主屏缩放比对模板放大/缩小重试——
|
||||
// 兼容旧版 DIP 截图存的模板与任意来源的图片
|
||||
const scale = await primaryScale();
|
||||
const result = await findImage(template, screenshot, 1 - similarity, scale !== 1 ? [scale, 1 / scale] : []);
|
||||
if (!result) throw new RuntimeError("not_found", `屏幕中未找到目标图像（要求相似度 ${(similarity * 100).toFixed(0)}%，未达到阈值）`, { nodeId, recommendation: "重新框选模板（框小一点、只框目标图案），或适当调低相似度（如 0.8）" });
|
||||
return { ...result, x: result.x + offsetX, y: result.y + offsetY };
|
||||
}
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user