wangyukun/remote
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: wangyukun/remote:backup/agent-linux-0f3f1cf
wangyukun/remote:main wangyukun/remote:feature/android-viewer-core wangyukun/remote:feature/proto-versioning wangyukun/remote:feature/common-punch-signaling wangyukun/remote:feature/agent-linux wangyukun/remote:backup/agent-linux-0f3f1cf
...
compare: wangyukun/remote:8993a386584c159d9578143418c3db53c15edcea
wangyukun/remote:feature/android-viewer-core wangyukun/remote:feature/proto-versioning wangyukun/remote:feature/common-punch-signaling wangyukun/remote:feature/agent-linux wangyukun/remote:main wangyukun/remote:backup/agent-linux-0f3f1cf
55 Commits
backup/agent-linux-0f3f1cf ... 8993a38658
Author SHA1 Message Date
郑博文 8993a38658 fix(android): BACK 键改用自测键盘可见性——isAcceptingText 会让会话永远断不掉 
真机实测（vivo S6 + 讯飞，2026-10-30）：软键盘已隐藏、mInputShown=false，
但 imm.isAcceptingText 仍返回 true——只要 InputConnection 还绑着它就是 true。
原先用它判断「先关键盘还是先断会话」，后果是第一次 BACK 空转，第二次也进不来
（我们已 return true），会话再也断不掉。这不是推测，是完整链路 proof 跑到
step 5 时实测失败的。

改为自测：比较根窗口可见区域与窗口高度，被 IME 吃掉超过 15% 视为可见；
用 OnGlobalLayoutListener 被动跟踪（没有公开 API 能直接查询，API 34 上
getInputMethodWindowVisibleHeight 也不是公开接口）。阈值用比例不用固定像素。

验证：diag 依次显示「键盘可见」→「键盘隐藏」→ 会话断开回表单；
随后完整链路 proof 与中文 proof 均复跑通过。

同时修正 phone_chain_proof 的 V4 断言：uiautomator 属性顺序是 text= 在
class= 之前，原正则要求反序，导致打印空表单行仍判 PASS（假绿）。
空 EditText 的 text 会回显 hint，所以判据改为比对实际输入值。
 2026-10-30 13:31:28 +08:00
郑博文 9792662181 test(android-2b): 真机中文候选词 proof——拼音组词经 commitText 落到 Windows 
V6 的判据是「手机中文输入法能把汉字送进远端」，此前只能靠人手点候选词。
实测发现两条关键事实，决定了 proof 的可行形态：
1. adb keyevent 相当于物理键盘输入，中文 IME **不组词**，字母原样透传到
   Activity.onKeyDown——所以驱动候选词必须用 adb input text（走 IME）。
2. IME 自己的按键视图对 uiautomator 不可见（dump 里只有我们 App 的 15 个节点），
   中/英切换按钮读不到坐标，只能按截图标定一次点击位；且该按钮**有状态**，
   于是写成幂等重试：先试，只收到字母才翻转模式再来一次。

跑通结果：讯飞拼音 nihao + 空格 -> commitText 2u -> 靶窗收到 U+4F60 U+597D。
这补上了方案风险 R1（vivo ROM + 第三方中文 IME 行为未观测）的核心一环。
 2026-10-30 13:15:27 +08:00
郑博文 0089ae86e1 test(android-2b): 真机 IME 提交 proof 与字节安全证据采集 
ime_commit_proof 补上 phone_chain_proof 没覆盖的那半段：IME **显示**状态下
按键被输入法消费，只有 RemoteInputConnection.commitText 能到达我们——这正是
方案风险 R1（vivo ROM + 第三方中文 IME 行为未观测）此前唯一没有证据的点。
真机实测：mInputShown=true、mHaveConnection=true、状态栏 commit 2u、
Windows 靶窗收到 Hi（讯飞把首字母自动大写，属输入法行为，不是链路缺陷）。
back 键第一次只关键盘不断会话，也在这支脚本里断言。

shot.ps1 / capture_evidence.ps1：PS 5.1 的 '>' 以 UTF-16 写文件，
'adb exec-out screencap -p > x.png' 产出的 9 张证据图首字节全是 FF FE，
全部不可用；改为设备内 screencap 落地再 pull，并校验 PNG 签名。

V5 等待预算 30s→50s：不可达 IP 实测 32s 才回表单报 timed out，
预算过小会把「仍在超时中」误判成验收失败。
 2026-10-30 13:06:08 +08:00
郑博文 9f0b0742a8 feat(android): 键盘事件双通道 + 手机到 Windows 全链路 proof 
真机暴露出 IME/按键通道的缺口，一并补上并证明：
1. RemoteInputConnection.sendKeyEvent 与 Activity.onKeyDown 都先处理**可打印**
   unicodeChar（走 v3 Text 通道），再退到扫描码查表。原先只查表：软键盘发的是
   keyCode=UNKNOWN + unicodeChar，查不到表就被当未映射键吞掉，字整个丢了。
   门槛排除控制值（Enter/Tab）——远端应用要的是按键事件而非 WM_CHAR。
2. onKeyDown/onKeyUp 补齐硬键与物理键盘路径（阶段 2 验收里的软键盘按键），
   repeatCount>0 只发一次 down，避免长按把远端消息队列刷爆。
   isAutorepeat() 是 @hide，改用公开的 getRepeatCount()（javap 核实）。
3. getUnicodeChar() 返回的是单个 UTF-16 code unit 而非码点，注释按此改正：
   BMP 外字符由 IME commitText 成对送达，不经这条路。

tools/phone_chain_proof.ps1：手机按键 -> v3 Text -> agent SendInput -> 自有
Windows 靶窗自报内容，全链路一次跑通（真机 vivo S6 实测 abc 到位、ENTER 走
scan 28、BACK 断开回表单、表单 UI 重连成功）。要点：
- 靶窗必须自报窗口句柄（pid+hwnd 两行），Get-Process 取 MainWindowHandle 有竞态；
- 不能用 keybd_event(VK_MENU) 破前台锁：ALT 之后的首个字符被当菜单助记键吞掉，
  实测三个字符只剩 b；改用 AttachThreadInput + SetForegroundWindow，不合成按键；
- 退役遗留靶窗只能 touch 已存在的 stop.flag：新建会命中即将启动那次的停止路径，
  靶窗刚起就自杀（实测 pid 报了又消失）；
- adb 进度信息走 stderr，PS 5.1 + ErrorActionPreference=Stop 会当终止错误，
  按 build-android.ps1 同款办法临时放宽并只看退出码。
- 表单填字用 keyevent 而非 adb input text：设备只有第三方中文 IME，text 注入会被
  组词（ASCII 句点变 U+3002）。跑前记录并禁用原 IME，finally 精确还原。
 2026-10-30 12:58:36 +08:00
郑博文 a93430dd3f chore(tools): 2b 注入 proof 路径随仓库位置推导，缺失参数报错文案改正 
proof 脚本原先硬编码 D 盘绝对路径，换机器就失效；仓库内已有同约定
（build-android.ps1 用 \ 推导），此处对齐。
text_check 的报错文案说「为空」，实际触发条件是 --units 缺失，改正。
 2026-10-30 12:21:19 +08:00
郑博文 79e3e77c97 fix(android): windowSoftInputMode=adjustPan——软键盘弹出不再踢掉会话 
默认 adjustResize 会缩小窗口高度，SurfaceView 随之收到 surfaceDestroyed，
而现有回调把 surfaceDestroyed 当作「断开」处理——等于用户一点键盘输入，
2a 验收过的会话就自己断了。真机首验前先做这道防御；adjustPan 只平移窗口，
不改变 surface 尺寸。合并后的 manifest 已核对（android:windowSoftInputMode 存在）。
 2026-10-30 12:17:20 +08:00
郑博文 8ffe19cd5f test(agent-core): v3 分发路径单测——默认后端不 panic、截断契约、空输入 
新功能必须带测试，agent-core 此前无测试模块，v3 新增的 text 默认实现与
截断分支正好落在盲区里：

- 后端不实现 text() 时（Linux/X11 的真实形态）必须被正常调用且不 panic；
- 超 MAX_TEXT_UNITS 必须以截断后的长度到注入侧，未超限原样；
- 空 Text 仍会下发到后端——被控端不假设对端守规矩，线格式输入按不可信处理。
 2026-10-30 12:11:28 +08:00
郑博文 936986a940 test(android-2b): 文本注入真边界工具——viewer-core 直发 Text，靶窗自报收到内容 
text_check 走的是产品同一条路径（Session::send_text → v3 线格式 → agent
SendInput KEYEVENTF_UNICODE），手机掉线也能先把 Windows 半段验实。
参数是十进制 code unit 列表而不是自然语言字符串：命令行到 Rust 的编码往返
会引入与本轮无关的变量，由 PowerShell 从 .NET 字符串取 char[] 更忠实。

proof 脚本刻意用自有 WinForms 靶窗而非记事本：Win11 记事本所有标签页同进程，
本机当时开着用户文档，往里注入等于改用户文件。靶窗自报文本（写文件），
判据不依赖截图和 OCR。注入前后各断言一次前台窗口——SendInput 打的是前台，
靶窗不在前台时的注入会进别的应用，宁可失败也不误注入。
 2026-10-30 12:09:48 +08:00
郑博文 6c28a3f7e3 feat(android): 阶段 2b——连接界面 + IME 文本通道 + 控制键扫描码 
两态 UI：连接态是 host/port/code 表单（原生 View 零依赖，intent extras 保留且
优先，构建脚本不受影响），成功后切远控态。只存 host/port；验证码不落盘（用户裁决）。
返回键分态：IME 显示时先收键盘，否则断开回表单——否则一次返回关键盘又杀会话。

IME 走 P1：onCheckIsTextEditor + BaseInputConnection 子类。commitText 整段
经 v3 Text 通道上行（char[]→GetCharArrayRegion，UTF-16 无损）；控制键映射
Set-1 扫描码（值与 viewer/keymap.rs 同源，右 Shift 按官方文档不置 extended）。
composing 不上远端（U4），getText* 回空串——本地没有也不伪造远端缓冲区。

键盘按钮显式 showSoftInput，不自动弹（vivo 行为不一致 + 挡画面）。
RemoteSurfaceView 补 isFocusableInTouchMode：没有它触摸态拿不到焦点，
InputConnection 根本不会建立（2a 弹不出键盘的根因之一）。
 2026-10-30 11:51:01 +08:00
郑博文 55cc8edbc1 fix(agent): 文本注入改为单次 SendInput 批量——按官方文档消除事件被打散的风险 
实读 Keyboard Input Overview 后修正两处凭记忆的实现：
1. 原文承诺「单次调用的事件不会被其他输入插入」，此前拆成 down 批 + up 批
   两次调用，两次之间可能夹进真实键鼠事件，代理对存在被拆开的风险。合并为
   一次提交（全部 down 紧接全部 up）。
2. KEYBDINPUT 文档「只能与 KEYEVENTF_KEYUP 组合」那句按推断写成结论；
   改为如实标注为推断，并写明真机失败时的退路（只发 UNICODE|KEYUP 单事件）。
 2026-10-30 11:41:54 +08:00
郑博文 174a3f8ac0 feat(viewer-core,android-ffi): 主控端文本上行与 JNI 入口 
Session::send_text 收 &[u16] 而不是 &str：整条链（Java 字符串 → JNI → 线格式 →
Windows 注入）的原生表示都是 UTF-16，任何一次 UTF-8 往返都有损，故中间层不做转换。
空输入与超限在本层拦掉，超限走 clamp_text_units 并打日志。

nativeSendText 的参数用 char[]（JObject 收再转 JCharArray），经 GetCharArrayRegion
按 u16 直接拷贝。不用 JString + get_string：jni 0.21 那条路走 Modified-UTF-8，
IME 提交边界的落单代理项会在往返里丢。JNI 取数失败必须 eprintln，不能静默——
用户对着远端打字最坏的情况是「看起来发送了」。
 2026-10-30 11:41:53 +08:00
郑博文 5075888861 feat(agent): v3 文本注入用 KEYEVENTF_UNICODE，一次批量提交成对代理项 
Windows 侧按官方 KEYBDINPUT 语义实现：置 KEYEVENTF_UNICODE 时 wVk 必须为 0、
wScan 即要送出的字符，系统合成 VK_PACKET 经 TranslateMessage 产出 WM_CHAR。
down 批与 up 批分两次 SendInput，且让同一段文本的 code unit 在输入队列里相邻，
使代理对能被远端合成一个字符（最终由真机 emoji 验收定论）。

agent-core 的 InputInjector::text 用默认实现记日志而非静默忽略：Linux 后端
本轮不做文本注入（X11 需 keysym/XKB 映射，另立循环），但用户对着 Linux
被控端打字不能「什么都没发生还没痕迹」。Windows 端走自己的 handle()，
不经 agent-core，故超限截断日志在两侧同源补齐。
日志只打长度，不打用户输入内容。
 2026-10-30 11:29:25 +08:00
郑博文 38df106fb6 feat(common): 协议 v3——新增 InputEvent::Text 文本注入通道 
安卓软键盘的中文候选词结果不是扫描码，扫描码通道对中文天然不通，必须
另开一条按字符注入的通道。载体选 Vec<u16>（UTF-16 code unit）而非 String：
Windows 注入侧与 Java 字符串原生都是 UTF-16，中间任何一次 UTF-8 往返都会
把 IME 提交边界的落单代理项变成替换字符。

InputEvent 因携带 Vec 不再是 Copy，调用方改用 clone；三端必须同步升级，
旧端由 Auth 握手显式拒绝（这正是 PROTO_VERSION 递增的目的）。
新增 6 个单测覆盖 roundtrip、落单代理项、上限边界与 1MiB 长度闸共存。
 2026-10-30 11:29:13 +08:00
郑博文 5079d33b0a fix(agent): 显式 max_idle_timeout=10s——强杀主控端后 10 秒内可重连 
问题（实测复现）
串行 accept 下，主控端被强杀（force-stop，非正常 close）时 QUIC 连接
不会立即消亡：agent 还在向死连接发帧，新连接只能排队，要等 quinn 默认
idle 超时（约 30s）才能建立。表现为「杀掉 App 重开，一直连不上」。

修法
TransportConfig 显式设 max_idle_timeout=10s。正常会话每 500ms 一个 Ping
永不触发该超时，只回收死连接。10s = 20 个 Ping 周期，抗抖动余量足够。

验证（vivo S6 实测）
- 会话1 auth ok -> force-stop App -> 立即重连
- 10.2s 后 peer connected + auth ok（修复前 ~30s）
 2026-10-30 10:38:04 +08:00
郑博文 fe558a1c1e feat(android): 状态栏触摸诊断通道（vivo logcat 被限流的对策） 
本机（vivo S6 + 办公 Windows）logcat 只能抓到 4 行，App 日志完全不可靠，
排查触摸/坐标问题时没有任何可用通道。

- RemoteSurfaceView 加 onDebug 回调：ACTION_DOWN/MOVE/UP 的视图坐标与
  发出的 norm 归一化坐标实时上报
- MainActivity 统一 refreshStatus()：连接态 + 帧数/RTT + 最近触摸合并成
  一处刷新（原来 8 处各自 statusView.text 互相覆盖，且引用过已删的 showHint）
- 删掉临时步骤标记与 lastMeasureInfo（诊断价值已被 onDebug 覆盖）

实测：状态栏正确显示 UP view=(643,222) norm=(39017,23968) 1080x607，
据此确认了 letterbox 视图尺寸与归一化换算全对，把排查焦点正确引向
agent 注入侧（VIRTUALDESK，已另行修复）。
 2026-10-30 10:35:21 +08:00
郑博文 6a6f989595 fix(agent): 绝对坐标注入去掉 VIRTUALDESK——注入空间与采集空间对齐 
根因（实测确认）
capture 用 EnumOutputs(0) 只采**主屏**，AuthOk 报告的 width/height 也是主屏；
但 input 注入带 MOUSEEVENTF_VIRTUALDESK，归一化 0..=65535 被映射到**整个虚拟
桌面**。双显示器机器上虚拟桌面（主屏 2560x1440 + 副屏 1920x1080 在左）比主屏
宽得多，手机画面与指针位置系统性错位——点主屏任务栏，指针落到别处。
单显示器时两个空间重合，所以桌面 viewer 联调从未暴露。

修法
MOUSEEVENTF_ABSOLUTE 不带 VIRTUALDESK 时，0..=65535 恰好映射到主显示器，
与采集空间一致。协议注释「鼠标坐标为虚拟桌面归一化」的语义由「整个虚拟桌面」
修正为「被采集的输出」——桌面 viewer 是绝对定位透传，行为不变。

验证（vivo S6 -> 本机双屏，修复后）
- 手机拖微信标题栏：窗口跟随手指移动（drag-before/after 截图对比）
- App 状态栏 norm 坐标与视图坐标换算一致（view=(643,222) norm=(39017,23968)）
 2026-10-30 10:30:53 +08:00
郑博文 5047618c01 feat(android): 触摸映射 + letterbox 显示（阶段 2a 代码部分） 
问题：阶段 1 的 SurfaceView 用 MATCH_PARENT，2560x1440 的视频被拉进
1080x2400 竖屏，画面变形。变形状态下手指位置与远端指针不再一一对应，
触摸映射无从谈起。所以两者必须一起做。

新增 RemoteSurfaceView
- onMeasure 按视频宽高比 letterbox：视图尺寸 = 画面尺寸，黑边不属于视图，
  触摸天然只在画面内产生事件，坐标归一化到 0..=65535 才准确
- 手势：单指轻点=左键、单指拖动=左键拖拽、双指轻点=右键、双指滑动=滚轮
- 刻意不做「手指即光标」的绝对跟随：手指会完全挡住目标，精细操作不可用
- 边界处理：进入拖拽后落下第二指，先抬起左键再切双指手势，
  否则远端会留下一个被按住未释放的按键

协议侧（不改线格式，只加只读查询）
- viewer-core::Session 记录 AuthOk 的桌面尺寸，新增 desktop_size()
- android-ffi 新增 nativeDesktopSize(handle) -> i64（高32=宽，低32=高）
- RustBridge.desktopSize() 拆包
- 尺寸必须在建解码器之前拿到：等首帧再改视图尺寸会触发重新测量，
  把刚创建的解码器 surface 打掉

MainActivity
- 接入 RemoteSurfaceView.Listener；连接成功后先取尺寸设宽高比，再起解码线程
- 返回键 = 断开（不退出 App，便于重连）；去掉阶段 1 的双击断开
  （双击现在语义上是「两次左键点击」，会和远端交互冲突）

验证：cargo build --workspace 通过；cargo test 全绿（common 20 + viewer-core 3）。
真机手势验收待手机重新连接后补。
 2026-10-30 09:51:38 +08:00
郑博文 f119300a57 feat(agent): 串行多会话——会话结束不再退出进程 
问题（阶段 2 的硬阻塞）
accept() 只接一次，会话结束进程退出。阶段 1 一次验收看不出来，
但阶段 2 要反复测试输入，每测一次都要手动重启被控端。
（即公共阻碍清单里的 P5，此前判断为「只影响调试效率」，
 现在它阻塞迭代，提前做掉。）

改动
1. main：会话体抽成 serve_session()，外层 accept 循环。
   会话失败（含认证失败）只记日志不退出进程。
   明确不做并发多会话：DXGI Desktop Duplication 每个输出仅允许一个
   活动实例，第二个 DuplicateOutput 会让第一个收到 ACCESS_LOST。
2. 认证失败路径补 conn.closed() 等待（agent-core 已有同款修复，
   此处同源补齐 = 手册 §6.1 第 1 条）：否则客户端只看到 connection
   lost 而不是 AuthFail。
3. capture 循环加 tx.receiver_count()==0 守卫。
   只靠「send 失败退出」不够——桌面完全静止时那个分支根本不发送，
   线程会永远持有 duplication，下一个会话的 DuplicateOutput 直接失败
   （表现为新会话黑屏）。agent-core 同源修复。
4. 会话收尾：finish 流 + drop 接收端 + 等采集线程退出（2s 超时）后才
   返回，调用方才 accept 下一个连接。

验证（连续两个会话，中间不重启 agent）
  会话1：收到帧 40/40，结论 ✅
  agent 存活：✅
  会话2：收到帧 40/40，结论 ✅
  日志：session closed -> waiting for next connection -> peer connected
        -> auth ok；stderr 有 [capture] no receiver, stop 证明线程真退出
 2026-10-30 09:31:44 +08:00
郑博文 58bb66f4c2 feat(android): 阶段 1 安卓主控 App 工程（已真机验收出画面） 
Kotlin 层三个文件（刻意零第三方依赖，纯 View 布局——阶段 1 只验证链路）：
- RustBridge.kt：android-ffi 的 JNI 声明 + 帧头拆解（25 字节定长头 + Annex-B 载荷）
- H264Decoder.kt：MediaCodec 同步模式 + Surface 直出；内置两条实测教训——
  只用 getInputBuffer(i)（Exynos Quirk）、awaitingIdr 状态机（R-3 跳帧断参考链）
- MainActivity.kt：连接/解码循环/触摸绝对坐标映射；断开顺序遵守
  「先停解码线程 join 后 destroy」的 FFI 纪律

构建链（build-android.ps1，一条命令：cargo ndk -> .so -> gradle assembleDebug -> 安装启动）：
- 路径自适应（ 定位仓库根），local.properties 自动生成
- gradle.properties 固定 JDK 17，不动全局 JAVA_HOME

真机验收（vivo S6 / 192.168.1.163 -> PC / 192.168.1.180:47800）：
- agent 日志：peer connected + auth ok, desktop 2560x1440
- 手机截图：Windows 桌面（WPS 文件列表）已渲染在 SurfaceView 上
  证据：evidence/screen-connected.png（本地，不入库）

已知环境限制（记录在个人层 baseline）：本机 Windows 防火墙的 agent.exe
程序级放行规则对 UDP 入站不生效（临时关防火墙即通），QQPC 驱动级防护
静默拦 TCP 入站（管理员无法停止/杀死）。阶段 2 前需落定长期方案：
端口级防火墙规则或换开发机。
 2026-10-29 19:59:43 +08:00
郑博文 64d497e577 test(viewer-core): headless_check 增加 --host，可指定被控端地址 
默认仍 127.0.0.1。加 --host 后能用局域网 IP 跑同一验收，
便于排查「桌面通、手机不通」这类只影响入站路径的问题
（注：本机回环连接不经过防火墙，只有跨主机连接才验证防火墙路径）。
 2026-10-29 19:18:11 +08:00
郑博文 44767ed370 feat(android-ffi): viewer-core 的 JNI 薄层，arm64 交叉编译通过（QUIC 栈上手机的第一道关） 
方案变更说明（A2 落地修正）：原裁决选 UniFFI，但实查 Maven Central 与
maven.mozilla.org 均无 uniffi Kotlin 运行时构件（404），自编译发布成本超过收益。
改为裸 JNI：阶段 0（真机）已验证该链路，接口刻意保持窄（连接/取帧/发输入/销毁），
将来若引入 UniFFI 只替换本层。

线程与锁纪律（决定接口形态）：
- nativeNextFrame 等帧期间不持会话锁（clone Arc<Session> 后即释放），
  桌面静止 500ms 帧间隔不会卡住 UI 线程的输入调用
- destroy 前必须先停解码线程，Kotlin 侧顺序纪律保证（已登记风险清单）
- 句柄 = Box 裸指针（永不为 0），destroy 后置 0

帧返回约定：byte[] = [keyframe 标志 1][seq 8][ts 8][w 4][h 4][Annex-B payload]，
定长 25 字节头，Kotlin 切片零 JNI 往返。

viewer-core 适配：Session.frames 改 Arc<tokio Mutex>，next_frame 取 &self，
使解码线程与输入线程可并发借用同一 Session。

验证：
- 桌面 cargo build -p android-ffi 零警告
- cargo ndk -t arm64-v8a --release：quinn/ring/rustls 交叉编译全部通过
  （消除阶段 1 最大未知：QUIC 栈能否编到 Android）
- libandroidffi.so 4.2MB，10 个 Java_com_verify_remote_RustBridge_* 符号导出正确
 2026-10-29 18:27:34 +08:00
郑博文 34df7b070b feat(viewer-core): 新增安卓主控的连接/协议/收帧层（阶段 1 桌面验证通过） 
定位（2026-10-29 方案裁决 A1-A4）：
纯 Rust、平台无关，只做「怎么连 + 收发什么」——QUIC 会话、协议 v2 认证、
视频帧流、输入上行、Ping/Pong、KeyFrameReq。解码/渲染/触摸全在 Kotlin 侧，
本 crate 不碰像素、不依赖 openh264（dev-dependency 仅验证工具用）。

设计要点
- PeerTarget::Direct 是唯一的连接形态；不做中继、不定义 Relay 变体
  （《多平台架构设计》§6.1 裁决）。将来设备码+打洞只换 connect() 内部。
- 帧经容量 2 的 mpsc 交付，**不丢帧**：跳帧决策属于看得到解码器状态的消费方
  （Kotlin），Rust 层丢帧会重演 R-3 教训（跳帧断参考链）。
- insecure_skip_verify=false 时明确报错而非静默放行——证书钉扎登记在方案
  阶段 4，安全红线不允许默默降级。
- 双流结构与桌面 viewer 一致（流1 认证+视频，流2 输入+控制），入会即请求 IDR。

验证（headless，桌面→桌面 127.0.0.1）
- 单测 3 passed（帧契约、默认配置、strict-verify 显式报错）
- examples/headless_check：300 帧，seq 违例 0、keyframe 标志不符 0、
  解码成功 300/300、RTT 路径工作（RTT=1ms 出现）
- 全 workspace 回归：common 20 passed，构建全绿
 2026-10-29 18:14:27 +08:00
郑博文 1837658c85 test(agent): dump_stream 增加 --save，可导出裸 H.264 码流供外部解码器验证 
目的
安卓主控方案的唯一高风险未知项是「agent 发出的码流能否被 MediaCodec 解码」。
要验证它，需要一份脱离 QUIC/自定义帧头的**裸 Annex-B 码流文件**，供手机端
或任何标准解码器直接读取。

改动
- 新增 --save <路径>：把各帧载荷按到达顺序拼接成 Annex-B 文件
- 存的是**载荷**而非完整帧：外部解码器只认标准 Annex-B，不认识本项目的
  长度前缀与 bincode 帧头，故不能存 encode_video_frame 的输出
- 保存时打印字节数与 IDR 数量，便于快速判断样本是否可用

不做的事
不引入独立的码流解析库，IDR 统计复用文件内已有的起始码扫描逻辑，
避免为一次性诊断工具增加依赖。

验证
cargo build --workspace 退出码 0；cargo test --workspace --exclude linux-agent 全绿。
实际使用：导出 90 帧、437823 字节、含 2 个 IDR，该文件已用于安卓 MediaCodec
兼容性验证（vivo S6 / Exynos 硬解，87/89 帧解出）。
 2026-10-29 16:13:58 +08:00
郑博文 6e83bda8b8 perf(agent): 关键帧周期从 4s/10s 统一到 1s，缩短丢帧后的兜底恢复窗口 
背景
协议 v2 与 vendor nvenc 补丁之后，KeyFrameReq 已成为**真正生效**的恢复主路径
（此前 NVENC 缺 FORCEIDR 标志，请求被静默忽略）。因此周期 IDR 的角色变了：
从「唯一恢复手段」降级为「兜底手段」，只用于防「请求连续丢失」这种极端情况。
既然是兜底，就没必要配得那么长。

问题：三处周期不一致，且都过长
- agent/src/capture.rs NVENC：gop_len = 240 帧 @60fps = 4 秒
- agent/src/capture.rs 软编：    intra_period = 300 帧 @60fps = 5 秒
- agent-core/src/lib.rs Linux：  intra_period = 300 帧 @30fps = **10 秒**

同一产品在三种编码后端上的恢复等待时间差 2.5 倍，且 Linux 端最差。

改动
- 新增 agent/src/capture.rs::INTRA_PERIOD_FRAMES = 60（Windows 两个编码器共用）
- agent-core 的 EncoderSettings::default() 改为按 max_fps 推导（1 秒对应帧数），
  避免以后调帧率时周期又漂移

代价
IDR 是整帧内编码，实测一次约 95-120KB，而 P 帧约 3-10KB。1 秒一次 IDR 约
增加 0.8Mbps 开销，在 5Mbps 码率下可接受。

验证（dump_stream 观察 150 帧）
  [  0] seq=0   IDR   ← 首帧
  [ 60] seq=60  IDR   ← 间隔精确 60 帧
  [120] seq=120 IDR
@60fps 即每 1 秒一个自动 IDR，与 gop_len=60 配置一致。
 2026-10-28 16:35:44 +08:00
郑博文 8410a8de2a docs: 记录「不做中继」的最终裁决（2026-10-28） 
背景：公司决定不提供中继服务器，中继方案从「暂缓」变为「排除」。

《多平台架构设计.md》
- §6 从「⚠️ 必须现在拍板」改为「✅ 已裁决」，保留原始分析但标注结论
- 新增 §6.1 裁决内容：不做中继、不预留中继接口、IPv6 一等公民、
  连接层边界仍要做
- 新增 §6.2 澄清一件常被混淆的事：**连接层边界 ≠ 中继接口**
  - 连接层边界（把「怎么连」与「连上干什么」分开）：要做，与中继无关，
    因为连接方式确定会变（IP→设备码）、安卓端要复用、解码逻辑要可单测
  - 中继接口（Path::Relay 这类类型预留）：不做。中继已排除，这些变体
    永远不会被构造，只会产生 dead_code 警告——零收益的预留就是纯负担
- 新增 §6.3 对产品的影响：明确哪些网络环境连不上，并说明这不是 bug
  而是已知边界，UI 必须给出可执行提示
- §10 第 1 条标记为已裁决

《打洞与信令设计.md》
- 状态更新为「已合入 main（703eb64）」
- 补充裁决确认，指向 §6.1/§6.2

补充事实：此前代码中并不存在 PathProvider / Path::Relay，唯一提到 relay 的
signaling.rs:7 是在声明「本协议不含中继」。因此本次是「确认不留」而非
「拆除预留」。
 2026-10-28 16:35:32 +08:00
郑博文 192513dd2a fix(viewer): 跳帧后进入「等待 IDR」状态，不再把必然失败的 P 帧喂给解码器 
问题
H.264 的 P 帧依赖前一帧的参考图像。主控端的积压跳帧逻辑（读出但不解码）
会切断参考链，此后所有 P 帧都解不出来，且不会自行恢复——必须收到 IDR。
旧实现跳帧后只是 continue，后续 P 帧仍被送进 decoder，导致：

- 每帧解码失败一次 → stderr 刷屏「decode failed」
- 每次失败都触发一次 KeyFrameReq → 请求风暴
- 解码器反复在错误状态里挣扎

实测（修复前，10 秒会话）：decode failed 刷屏数十条。

修复
引入 awaiting_idr 状态，语义是「参考链已断，等 IDR 重新同步」：

- 跳帧时置位，并请求关键帧
- 置位期间跳过所有 P 帧（这是正确性要求，不是优化）
- 收到 hdr.keyframe=true 的帧才清位并恢复解码
- 解码出错时也置位
- 初始值为 true：入会时已发 KeyFrameReq，拿到第一个 IDR 前不该硬解

该实现依赖协议 v2 新增的 hdr.keyframe 字段（从实际码流 NAL 类型判定），
因此必须在协议改动之后合入。

HUD 新增 STATE 行显示 OK / RECOVER。RECOVER 期间画面冻结是**预期行为**，
没有这个指示用户会误以为软件卡死。

验证（同机 agent+viewer，25 秒）
- decode failed：0 次（修复前刷屏）
- stderr 总行数：3（仅首帧日志 + 2 条积压提示）
- 窗口稳定无假死

已知局限（下一步）
本修复解决的是「跳帧后不再做无用功」，但跳帧本身仍会造成画面冻结直到
IDR 到达。NVENC 的 gop_len=240、@60fps 即 4 秒才有一个自动 IDR，恢复窗口
偏长。缩短 GOP 或降低跳帧触发频率是后续优化项。
 2026-10-28 16:10:40 +08:00
郑博文 e778ff73cc test(agent): 新增 dump_stream 码流诊断工具 
用途：不依赖 viewer 的 Minifb 窗口，直接抓 agent 下发的码流并逐帧打印
NAL 结构与解码结果，用于定位解码恢复类问题。

能力
- 打印每帧的 seq / keyframe / 字节数 / NAL 类型（SPS PPS IDR P）
- --skip-at N：模拟主控端「积压跳帧」——读出但不解码，用于验证跳帧是否
  切断参考链（实测确认：跳掉一帧后所有 P 帧持续解码失败，直到收到 IDR）
- 交叉校验帧头 keyframe 标志与码流实际 NAL 类型是否一致
- 末尾补发 KeyFrameReq 并观察 IDR 是否真的回来

用法
  cargo run --release -p agent --bin dump_stream -- --port 47800 --code 123456
  cargo run --release -p agent --bin dump_stream -- --port 47800 --code 123456 --skip-at 1

注意：KeyFrameReq 必须写在流 2（isend）——agent 的读循环只监听流 2，
写到流 1 会被静默忽略（本工具第一版踩过，误判「修复无效」）。

这个工具是发现 nvenc FORCEIDR 缺陷的直接手段，保留以备回归排查。
 2026-10-28 16:05:41 +08:00
郑博文 e814cb64e7 fix(agent): vendor nvenc 并补 FORCEIDR 标志——修复运行期关键帧请求被静默忽略 
根因（实测确认，非推测）
上游 nvenc 0.1.0 的 encode_picture 只设置 picture_type=NV_ENC_PIC_TYPE_IDR，
却从未设置 encode_pic_flags=NV_ENC_PIC_FLAG_FORCEIDR。NVENC 会据此忽略请求、
继续输出 P 帧。首帧之所以「看起来正常」，是因为 frame_idx==0 本来就强制 IDR，
与该字段无关——这也是该缺陷长期未被发现的原因。

证据（同一 frame_count=4，修复前后）
- 修复前：输出 NAL 首字节 0x61 → 0x61 & 0x1F = 1 = 普通 P 片
- 修复后：输出 NAL 首字节 0x65 → 0x65 & 0x1F = 5 = IDR 片

影响
这解释了构建运行手册 §6.1 记录的「静止桌面黑屏」：NVENC 路径的重编 IDR 从未
真正生效，KeyFrameReq 拿不到 IDR 就无法从解码错误中恢复。

改动
- 新增 vendor/nvenc（上游为 MIT 许可，可自由修改分发，符合商用要求）
- vendor/nvenc/src/safe/encoder.rs：picture_type=IDR 时置 ForceIDR；
  IntraRefresh 时置 ForceIntra。改动处有「本项目补丁」注释标记
- agent/Cargo.toml：nvenc 改为 path 依赖指向 vendor 版本

建议后续向上游提 PR（github.com/AlsoSylv/nvenc），以降低长期维护成本。

验证：跳帧后发 KeyFrameReq，第 3 帧即收到 IDR 并解码成功
 2026-10-28 16:05:41 +08:00
郑博文 9e50bc6237 feat(common): 协议 v2——版本号 + 帧头 seq/keyframe + 长度下界校验 
为接入安卓主控做准备，先把三端共享契约改到能承载多客户端。改动集中在
common/src/lib.rs，三端适配在同一提交内完成以保证可编译。

协议版本（PROTO_VERSION=2）
- ControlMsg::Auth/AuthOk 增加 proto_version 字段；认证顺序改为「先版本后验证码」：
  版本不一致时字段布局可能不同，验证码的解析结果不可信
- 任一端版本不符即 AuthFail 并明确报错，把「连上了但行为诡异」变成「连不上且提示升级」

视频帧头（VideoFrameHeader）
- 新增 seq: u64（每次实际发送递增，含静止桌面重发）与 keyframe: bool
- keyframe 由新增的 common::h264_contains_idr() 从实际 Annex-B 码流判定，
  不依赖编码器 API 推断——NVENC 与 openh264 规则不同，实际输出才是唯一可信来源
- 该字段是 MediaCodec 判断解码器重置时机的必要条件，避免客户端自己写 NAL 扫描

不可信输入防护
- read_video_frame 补齐长度下界：total 必须 >= 4，hlen 必须落在 [1, total-4]
- 旧实现只校验上界，构造 total<4 的帧会让接收端在 buf[0..4] 处 panic（进程级DoS）
- 测试从 0 增至 11 个，覆盖序列化往返与 6 类越界/损坏输入

顺带修复（与上述契约直接相关）
- 新增 common::split_video_frame()：静止桌面重发时复用载荷但重建帧头，
  否则 seq/ts_ms 会停在上次发送时刻，破坏 seq 单调递增契约
- agent-core 与 agent 两端的静止重发路径均改用该函数

验证：cargo test -p common → 20 passed（9 signaling + 11 新增）
 2026-10-28 16:05:41 +08:00
郑博文 703eb64fc0 merge: 合入 feature/common-punch-signaling（P2P 信令协议 + 打洞探针 + 4 份架构设计文档） 2026-10-28 15:14:56 +08:00
郑博文 4533968057 merge: 合入 feature/agent-linux（测试期望值修正 + 冒烟脚本纳入单测 + cargo 网络配置文档） 2026-10-28 15:14:51 +08:00
yu 18d12baba1 fix(tools): 探针区分信令回复与打洞包，修掉打洞假阳性 
问题：客户端反复注册引来的 PEER/WAIT 回复在 discover_peer 返回后才到达，
被接收线程当成「对方打进来了」，输出假的「✅ 打洞成功」。
实测形态：报告的对方地址实为信令服务器 47.98.245.66:47999，包长 22 字节
（真打洞包固定 16 字节 = PUNCHv1| + 8 字节时间戳）。

修复：
- 接收线程只认以 MAGIC 开头、且长度正确的包
- 显式排除信令服务器的 IP:端口
- 失败结论里报告被忽略的非打洞包数量，避免再次误读
- 验证记录补 4.5 节，写明 Rust 实现的对应判据
 2026-10-23 18:02:14 +08:00
yu 2487e616c2 docs: 打洞设计/多平台架构/借鉴映射/验证记录 
- 打洞与信令设计.md: 无中继方案的流程、成功率边界、Rust 实现约束
- 多平台架构设计.md: 6 层分层 + 平台矩阵 + 六项关键决策 + 分期路线
- 借鉴映射与落地规范.md: 每层看谁/抄什么/不抄什么 + 许可证红线（AGPL 只读）
- 打洞验证记录.md: POC 实测结论（锥形 NAT + 端口保留）与 4 个坑
- README: 文档索引补充
 2026-10-23 17:32:17 +08:00
yu d659a4e7f1 feat(tools): 打洞探针与极简 rendezvous 服务器（Python POC） 
- punch_probe.py: UDP 打洞探针，支持 STUN 查映射、--peer 手工配对、--server/--room 信令配对
- rendezvous.py: 极简 UDP 信令服务器，只做房间配对与地址回推，不承载媒体
- 已在 47.98.245.66 实测跨公网注册/配对可用
 2026-10-23 17:32:17 +08:00
yu bb392507bd feat(common): 新增 P2P 打洞信令协议（无中继） 
- signaling.rs: 信令消息契约 + NAT 画像/打洞前景诊断 + 证书指纹
- 明确不含 relay/TURN：打洞失败必须给出可执行原因（PunchFailReason::user_hint）
- 时间对齐不做时钟同步：服务器按两侧 RTT 下发不同 start_in_ms
- 保活按 binding_timeout_s 折半（keepalive_secs）
 2026-10-23 17:32:17 +08:00
yu bae871c78c fix(linux-agent): 修正坐标换算单测期望值 + 冒烟脚本纳入单测 
- scale(32768,1920) 期望值应为 959（32768×1919÷65535=959.49 向下取整），
  原写成 960 —— 是**测试期望值错**，实现本身正确（0→0、65535→extents-1 均通过）；
  集成测试因带 ±2 像素容差未能暴露，是跑单测才抓到的
- scripts/linux-smoke.sh 原先只跑 --test smoke，漏掉 --bins 单测；现两层都跑
- 手册 §8.4 说明补充单测这一层及其必要性

验证：bash scripts/linux-smoke.sh → 单测 4 passed + 端到端 2 passed
 2026-10-23 15:04:56 +08:00
yu 49f85db9f9 docs: 修正 cargo 网络配置建议（代理而非镜像源）+ 更新构建命令路径 
- 实测 USTC 镜像索引 cksum 与下发的 .crate 字节不一致（nvenc 0.1.0），
  源替换会触发 checksum 校验失败并改变 registry 身份；改为 [http] proxy 走官方源
- 构建命令里 D:\\wordkb\\... 的失效路径替换为可用写法
- 已实测：Windows cargo build --release 通过（agent.exe/viewer.exe/encbench.exe）
  且 cargo test -p linux-agent 在容器内 2 passed
 2026-10-23 14:10:20 +08:00
yu 9fda49d9c9 Merge branch 'main' of http://43.140.215.244:7040/wangyukun/remote into feature/agent-linux 2026-10-23 14:02:44 +08:00
dev-team ad022ffcf4 fix: v0.8.1 静止桌面黑屏 - 积压跳帧限定突发到达, 空闲重发帧不判陈旧不计E2E 2026-10-23 13:36:21 +08:00
yu 9c340813a8 feat(agent): 新增 agent-core 与 Linux 被控端（X11 + XTest，纯新增不动 agent/） 
本期只在新增文件里做，**未改动 Windows agent/ 一个字节**（与 NVENC 线并行，避免争抢）。

- 新增 agent-core：平台无关会话层 —— QUIC/TLS 握手、传输调优（沿用 512KB 发送窗口
  结论）、验证码认证、会话/select 循环、latest-wins 单槽、输入分发，以及软件编码
  视频管线（openh264）。同步跟进协议新增的 Pong{lost,sent} 下行丢包统计字段
- 新增 linux-agent：X11 GetImage 采集 + 整帧变更检测；XTest 注入 + 扫描码→X11
  keycode 映射（主键区 = scancode+8，E0 扩展键必须查表，含单元测试）
- 协议零改动：复用 Set-1 扫描码与虚拟桌面归一化坐标，viewer 无需任何修改
- 新增端到端冒烟测试 tests/smoke.rs 与 scripts/linux-smoke.sh（自建 Xvfb，无需
  图形桌面）：采集→编码→QUIC 解帧、认证（含错误码拒绝）、Ping/Pong（含丢包字段）、
  静止桌面关键帧恢复、输入注入实证（注入后从另一条 X11 连接读回指针真实坐标）
- 该关键帧用例做过变异测试：把修复退回会失败、恢复后通过，确认断言有效
- .gitattributes：shell 脚本强制 LF（Windows checkout 成 CRLF 会导致容器内无法执行）
- 手册：修正已失效的工具链路径；补充 cargo 镜像；新增第 8 节（Linux）与 6.1 节
  （三条待修问题，含两条 Windows 侧仍存在的关键帧缺陷）

验证：
- Linux(容器, USTC 源 + Xvfb): cargo test -p linux-agent 通过，2 passed
- Windows: 本期未改 agent/，default-members 未变，`cargo build --release` 行为不变
 2026-10-23 13:31:08 +08:00
dev-team 8dd18a06a1 fix: E2E 积压 - 陈旧帧跳过解码排空 QUIC 队列 + rgb 缓冲复用 + 版本号升 v0.8 2026-10-23 13:27:18 +08:00
dev-team 04c02d9ced fix: 启动黑屏 - 静止桌面响应KeyFrameReq + 首帧强制IDR + 入会主动请求关键帧 2026-10-23 13:12:27 +08:00
dev-team 68b2ddc59a merge: E2E 排队修复（64KB 窗口 + 拥塞跳帧） 2026-10-23 12:48:17 +08:00
dev-team 168f600fe1 perf(agent): E2E 排队修复——发送窗口 512KB→64KB + 拥塞反馈自动跳帧 
- 实测数据：FPS 61 / RTT 7ms / E2E 618ms → 延迟全在发送侧排队：
  512KB 窗口 ÷ WiFi 实际 ~3Mbps 吞吐 ≈ 1.3s 排队上限，latest-wins 管不到已进 QUIC 的字节
- send_window 64KB：排队硬上限 ~170ms@3Mbps；突发吞吐 64KB/7ms≈72Mbps 不受限；
  连接级窗口，流 2 输入数据量极小不受影响
- 拥塞反馈跳帧：send write 耗时 >20ms(单帧预算 1.2 倍) → 置位标志 → 采集线程
  跳过当前帧（NVENC 少输入一帧，参考链连续无花屏），注入率自适应匹配网络吞吐；
  capture 日志新增 skipped 计数
- viewer 标题版本 v0.7-LowLatency
- 回环实测：NVENC 4ms/帧、skipped 0（回环无拥塞符合预期）、零错误
 2026-10-23 12:48:16 +08:00
dev-team bd7ca29945 merge: HUD 常显修复 2026-10-23 10:45:47 +08:00
dev-team 00b318af8b fix(viewer): HUD 每秒强制呈现（不依赖视频帧）+ 标题栏版本号 
- HUD 此前仅在 dirty（收到新帧）时绘制，视频流异常/帧被丢弃时面板
  永不出现，且掩盖了排障信息；改为每秒统计刷新时强制 update_with_buffer
- 窗口标题加 v0.6-HUD 版本标记，用于识别误跑旧版 exe（排障关键）
- 回环实测：NVENC 4ms/帧、解码 1920x1080 匹配、14s 零错误
 2026-10-23 10:45:46 +08:00
dev-team e00175385d merge: 修 HUD 停刷与帧率不稳（单槽 latest-wins + GOP 自愈） 2026-10-23 10:31:31 +08:00
dev-team 9c72aecc3b fix(viewer): 帧管线 latest-wins 单槽（修 NVENC 60fps 下积压假死/HUD 停刷）+ GOP 周期自愈 
- viewer: 无界 mpsc → Mutex 单槽 latest-wins。根因：NVENC 60fps 到达率 > 渲染
  吞吐(~55fps)，无界队列无限积压 → 延迟线性增长 + 渲染 drain 阻塞消息泵 →
  窗口假死、HUD 停刷（软编 20fps 时代掩盖，NVENC 后暴露）
- viewer: 解码帧尺寸与窗口解耦（blit 逐行拷贝，防 NVENC 对齐尺寸不匹配）；
  首帧打印解码尺寸诊断
- agent: NVENC gop_len 0xffffffff→240（60fps 下 4s 周期 IDR 自愈恢复点，
  配合 KeyFrameReq 即时恢复，消除长时间冻结锯齿）
- 回环实测：NVENC 4ms/帧满速、解码 1920x1080 匹配、14s 零错误
 2026-10-23 10:31:30 +08:00
dev-team 82ec166f23 merge: NVENC 硬编（4ms/帧@1080p，编码瓶颈解除） 2026-10-23 10:18:29 +08:00
dev-team ddbb53d41a feat(agent): NVENC 硬编接入（GPU 零拷贝 ARGB 直通，软编自动兜底） 
- 编码器自动选择：NVENC（H264 High/CBR 5Mbps/UltraLowLatency/无B帧/按需IDR）→ openh264 兜底
- Duplication 增加 GPU 中间纹理（B8G8R8A8 DEFAULT），NVENC register_resource_dx11 直读，颜色转换完全省略
- KeyFrameReq → encode_picture(picture_type=IDR)，静止 500ms 重编码 P 帧
- windows 0.58→0.62（nvenc crate 依赖 0.62，运行时 dlopen nvEncodeAPI64.dll 零 SDK 依赖）
- 实测（GTX 1650 Ti 回环）：encode 41ms→4ms/帧（10x），convert 0ms，viewer 解码正常
 2026-10-23 10:18:29 +08:00
dev-team 94c45aff3f merge: FPS 调优（60fps上限/零拷贝/E2E 修复） 2026-10-23 10:02:42 +08:00
dev-team e7472fa12f perf(agent): 60fps上限/零拷贝转换/多线程编码/发送缓冲512KB + encbench基准 
- capture: min_frame 33→16ms，RC 60fps；with_mapped 映射窗口内直接 BGRA→YUV（行距连续零拷贝）
- capture: num_threads(4) + 每120帧输出 convert/encode 耗时（实测：convert 2ms，encode 41ms/帧真实工况）
- agent: send_window 2MB→512KB，E2E 排队天花板 ~0.15s（修复 E2E 633ms 红）
- encbench: 1080p 编码基准（合成 5ms/帧，真实工况 41ms/帧 → 软编 1080p 上限 ~24fps，60fps 需 NVENC）
- tools/bench.md
 2026-10-23 10:02:41 +08:00
dev-team 881341be3a merge: B1 联调基线（HUD 扩展 + bench 脚本） 2026-10-23 09:47:10 +08:00
dev-team 7dbb04da17 feat(hud): B1 基线指标——抖动/帧间隔p95/丢包率/解码失败计数 + 测试脚本 
- common: Pong 扩展 lost/sent（agent 发送方向 QUIC 计数，经交叉评审项）
- agent: Pong 附带 conn.stats().path 丢包统计
- viewer: Stats 加 GapRing(64)/jitter EMA/dec_fail；HUD 新增 JIT/P95/LOSS/DECFAIL 四行，% 字形
- tools/bench.md: 三场景基线测试脚本
 2026-10-23 09:47:09 +08:00
dev-team 95049b6642 docs: B侧设计文档（联调基线/码率自适应/打洞中继/多屏文件） 2026-10-23 09:34:21 +08:00
75 changed files with 12908 additions and 199 deletions
Expand all files Collapse all files

No files matched your search

.gitattributes
+15
View File
@@ -0,0 +1,15 @@
# 统一换行符，避免跨平台踩坑
# shell 脚本必须 LF：Windows 上若被 checkout 成 CRLF，容器/Linux 里执行会报
# "$'\r': command not found"，或被 sed/grep 当成带 \r 的内容
*.sh text eol=lf
# Windows 侧脚本保持 CRLF
*.ps1 text eol=crlf
*.bat text eol=crlf
# 二进制/生成物不参与换行转换
*.exe binary
*.dll binary
*.png binary
*.jpg binary
*.ico binary
B侧设计文档.md
+117
View File
@@ -0,0 +1,117 @@
# B 侧设计文档（主控端 + 协议 + 网络）
> 负责人：成员 B ｜ 主责 crate：`viewer/`、`common/` ｜ 配合：A（agent/）
> 按里程碑推进，每个里程碑独立 feature 分支，合入即打 tag。
---
## B1 · 联调基线（1-2 天，和 A 合作完成）
目标：拿到 H.264 版双机实测数据，作为一切优化的对照基线。
1. **HUD 扩展**：新增 丢包率(估算)/抖动(jitter)/帧间隔 p95 三项读数
- 抖动 = 相邻帧 E2E 差值的绝对值 EMA；帧间隔 p95 用环形缓冲取 95 分位；
- 码流异常计数（解码失败次数）也上 HUD，验证关键帧自愈是否生效。
2. **测试脚本** `tools/bench.md`：固定三个场景各测 60s —— 静止桌面 / 播放视频 / 拖动窗口，记录 HUD 截图。
3. **产出**：基线数据表填入本文件末尾。
## B2 · 码率自适应（3-4 天）
目标：弱网自动降码率保流畅，好网缓慢升码率提画质。
### 协议扩展（common/）
```rust
/// B2 新增控制消息（流 2）
RateReport {
loss_permille: u16, // 接收端估算丢包率 ‰
rtt_ms: u32, // 最近 RTT
e2e_ms: u32, // 平滑后端到端延迟
}
```
viewer 每 2s 汇总一次发送（流 2 复用，无新增握手）。
### 控制算法（先简后精，AIMD 风格）
| 网络状态（连续 3 个周期） | 动作 |
|---|---|
| 丢包 > 8% 或 E2E > 400ms | 码率 ×0.6（下限 1Mbps），同时通知编码器降帧率到 20fps |
| 丢包 3-8% | 码率 ×0.9 |
| 丢包 < 1% 且 E2E < 150ms 持续 10s | 码率 +500Kbps 步进上探（上限 15Mbps） |
### agent 侧配合点（给 A 的接口需求）
- 需要运行时改码率的入口：优先查 openh264 crate 是否暴露 `SetOption(RC_PARAM)`；
- 若 crate 不支持 → 方案 B：保留 encoder 实例但按新码率重建（重建成本 ~50ms，降码率不频繁，可接受）；
- 降级到极限（1Mbps 仍丢包 >15%）→ 主动降分辨率 1080p→720p（A 实现，协议传 `ResChange` 通知，viewer 重配解码器与窗口尺寸）。
## B3 · P2P 打洞 + 信令 + 中继（1.5-2 周，核心里程碑）
目标：跨公网可用。三层结构：
```
┌─────────┐ ①注册/交换地址(WebSocket) ┌──────────┐
│ viewer │ ───────────────────────────→ │ 信令服务器 │
└────┬────┘ └──────────┘
│ ②拿到对方候选地址(IP:port×N) ↑
│ ③a 直连(打洞) ──────────────────── agent 注册
│ ③b 失败 → 走 TURN 中继
```
### 信令服务器（B 主建，A 参与评审）
- 最小实现：单二进制 Rust + WebSocket（tokio-tungstenite），无数据库；
- 消息：`Register{agent_id}` / `Connect{agent_id}` / `Answer{candidates, auth_code_hint}`；
- agent 开机注册自身 ID（设备码），viewer 输对方 ID 发起；
- 中立服务器，不碰媒体数据，带宽成本 ≈ 0。
### 候选地址与打洞
1. agent 侧收集候选：本机局域网地址、路由器公网映射（STUN 探测，自建 coturn 的 stun 或公共 stun）；
2. 双方经信令交换候选后**同时**向对方发探测包（UDP 打洞，锥形 NAT 成功率高；对称 NAT 基本打不通）；
3. 打通 → 直连（QUIC 直接到该地址）；打不通 → TURN 中继兜底。
### 中继方案（务实路线）
QUIC 本身没有标准 TURN 集成，两条路：
- **路线一（推荐先做）**：自建中继服务 `relay` —— viewer 把 QUIC 包封装转发给 relay，relay 转给 agent（本质是应用层 TURN，复用信令服务器的部署）；
- **路线二（二期）**：coturn + 手写 ICE-lite，仅在路线一性能不足时投入。
### 安全（随 B3 一起做，替代现在的跳过校验）
- 预共享验证码升级为 **PAKE（SPAKE2）**：验证码不落服务器、不落网络，双方推导会话密钥；
- 信令只传地址候选，媒体端到端加密保持 QUIC TLS 不变。
## B4 · 多屏 + 文件传输（各 2-3 天，穿插做）
**多屏**：
- 协议：`ScreenList{screens: Vec<{id, w, h}>}`（AuthOk 里带）、`SelectScreen{id}`；
- agent 枚举 DXGI outputs（A 实现），viewer 加屏幕切换下拉；
- 第二期再做"多屏同显"。
**文件传输**：
- 首选走已有 QUIC 连接开独立双向流（不引入 aria2 依赖，部署简单）；
- 协议：`FileOffer{name, size}` → `FileAccept` → 原始分块流（1MB 块 + 校验）；
- viewer 做拖拽发送 + 保存对话框。
---
## 里程碑与分支规划
| 里程碑 | 分支 | 预估 | 验收标准 |
|---|---|---|---|
| B1 联调基线 | feature/hud-baseline | 1-2 天 | 三场景基线数据入库 |
| B2 码率自适应 | feature/rate-adapt | 3-4 天 | 模拟弱网（clumsy/丢包工具）下自动降码率，画面不断流 |
| B3 打洞+中继 | feature/p2p-relay | 1.5-2 周 | 两台跨公网机器通过 ID 互联；打洞失败自动走中继 |
| B4 多屏/文件 | feature/screens-files | 3-5 天 | 多屏切换、拖拽传文件可用 |
顺序建议：B1 → B2 → B3 → B4（B3 是价值最大项，B4 可穿插）。
## 基线数据表（B1 完成后填写）
| 场景 | FPS | RTT | E2E | 码率 | 丢包 | 抖动 |
|---|---|---|---|---|---|---|
| 静止桌面 | | | | | | |
| 视频播放 | | | | | | |
| 拖动窗口 | | | | | | |
Cargo.lock
Generated
+296 -32
View File
@@ -9,6 +9,7 @@ dependencies = [
"anyhow", "anyhow",
"common", "common",
"lz4_flex", "lz4_flex",
"nvenc",
"openh264", "openh264",
"quinn", "quinn",
"rcgen", "rcgen",
@@ -17,6 +18,30 @@ dependencies = [
"windows", "windows",
] ]
[[package]]
name = "agent-core"
version = "0.1.0"
dependencies = [
"anyhow",
"common",
"openh264",
"quinn",
"rcgen",
"rustls",
"tokio",
]
[[package]]
name = "android-ffi"
version = "0.1.0"
dependencies = [
"anyhow",
"common",
"jni 0.21.1",
"tokio",
"viewer-core",
]
[[package]] [[package]]
name = "anyhow" name = "anyhow"
version = "1.0.104" version = "1.0.104"
@@ -44,6 +69,27 @@ dependencies = [
"serde", "serde",
] ]
[[package]]
name = "bitfields"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef6e59298da389bc0649c7463856b34c6e17fe542f88939426ede4436c6b1195"
dependencies = [
"bitfields-impl",
]
[[package]]
name = "bitfields-impl"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2c044f98f86f15414668d6c8187c7e4fadab1ad2b31680f648703e0fe07c555"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"thiserror 2.0.21",
]
[[package]] [[package]]
name = "bitflags" name = "bitflags"
version = "1.3.2" version = "1.3.2"
@@ -86,6 +132,12 @@ dependencies = [
"shlex", "shlex",
] ]
[[package]]
name = "cesu8"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6d43a04d8753f35258c91f8ec639f792891f748a1edbd759cf1dcea3382ad83c"
[[package]] [[package]]
name = "cfg-if" name = "cfg-if"
version = "1.0.5" version = "1.0.5"
@@ -303,6 +355,16 @@ dependencies = [
"slab", "slab",
] ]
[[package]]
name = "gethostname"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1bd49230192a3797a9a4d6abe9b3eed6f7fa4c8a8a4947977c6f80025f92cbd8"
dependencies = [
"rustix",
"windows-link",
]
[[package]] [[package]]
name = "getrandom" name = "getrandom"
version = "0.2.17" version = "0.2.17"
@@ -342,6 +404,22 @@ dependencies = [
"web-sys", "web-sys",
] ]
[[package]]
name = "jni"
version = "0.21.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a87aa2bb7d2af34197c04845522473242e1aa17c12f4935d5856491a7fb8c97"
dependencies = [
"cesu8",
"cfg-if",
"combine",
"jni-sys 0.3.1",
"log",
"thiserror 1.0.69",
"walkdir",
"windows-sys 0.45.0",
]
[[package]] [[package]]
name = "jni" name = "jni"
version = "0.22.4" version = "0.22.4"
@@ -351,10 +429,10 @@ dependencies = [
"cfg-if", "cfg-if",
"combine", "combine",
"jni-macros", "jni-macros",
"jni-sys", "jni-sys 0.4.1",
"log", "log",
"simd_cesu8", "simd_cesu8",
"thiserror", "thiserror 2.0.21",
"walkdir", "walkdir",
"windows-link", "windows-link",
] ]
@@ -372,6 +450,15 @@ dependencies = [
"syn 2.0.119", "syn 2.0.119",
] ]
[[package]]
name = "jni-sys"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41a652e1f9b6e0275df1f15b32661cf0d4b78d4d87ddec5e0c3c20f097433258"
dependencies = [
"jni-sys 0.4.1",
]
[[package]] [[package]]
name = "jni-sys" name = "jni-sys"
version = "0.4.1" version = "0.4.1"
@@ -452,6 +539,19 @@ dependencies = [
"redox_syscall 0.9.5", "redox_syscall 0.9.5",
] ]
[[package]]
name = "linux-agent"
version = "0.1.0"
dependencies = [
"agent-core",
"anyhow",
"common",
"quinn",
"rustls",
"tokio",
"x11rb",
]
[[package]] [[package]]
name = "linux-raw-sys" name = "linux-raw-sys"
version = "0.12.1" version = "0.12.1"
@@ -567,6 +667,15 @@ version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441"
[[package]]
name = "nvenc"
version = "0.1.0"
dependencies = [
"bitfields",
"libloading",
"windows",
]
[[package]] [[package]]
name = "once_cell" name = "once_cell"
version = "1.21.4" version = "1.21.4"
@@ -697,7 +806,7 @@ dependencies = [
"rustc-hash", "rustc-hash",
"rustls", "rustls",
"socket2", "socket2",
"thiserror", "thiserror 2.0.21",
"tokio", "tokio",
"tracing", "tracing",
"web-time", "web-time",
@@ -721,7 +830,7 @@ dependencies = [
"rustls-pki-types", "rustls-pki-types",
"rustls-platform-verifier", "rustls-platform-verifier",
"slab", "slab",
"thiserror", "thiserror 2.0.21",
"tinyvec", "tinyvec",
"tracing", "tracing",
"web-time", "web-time",
@@ -906,7 +1015,7 @@ checksum = "1167586491e2b18b8bfbb293e8180ec17c201c4f076d7cb3070ca964e7598f98"
dependencies = [ dependencies = [
"core-foundation", "core-foundation",
"core-foundation-sys", "core-foundation-sys",
"jni", "jni 0.22.4",
"log", "log",
"once_cell", "once_cell",
"rustls", "rustls",
@@ -1164,13 +1273,33 @@ dependencies = [
"windows-sys 0.61.2", "windows-sys 0.61.2",
] ]
[[package]]
name = "thiserror"
version = "1.0.69"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52"
dependencies = [
"thiserror-impl 1.0.69",
]
[[package]] [[package]]
name = "thiserror" name = "thiserror"
version = "2.0.21" version = "2.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09e52cb86a36cede5cb101bf8908837b3e4c6e5e59fe7fd85c23fb56200d189e" checksum = "09e52cb86a36cede5cb101bf8908837b3e4c6e5e59fe7fd85c23fb56200d189e"
dependencies = [ dependencies = [
"thiserror-impl", "thiserror-impl 2.0.21",
]
[[package]]
name = "thiserror-impl"
version = "1.0.69"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
] ]
[[package]] [[package]]
@@ -1295,6 +1424,18 @@ dependencies = [
"tokio", "tokio",
] ]
[[package]]
name = "viewer-core"
version = "0.1.0"
dependencies = [
"anyhow",
"common",
"openh264",
"quinn",
"rustls",
"tokio",
]
[[package]] [[package]]
name = "walkdir" name = "walkdir"
version = "2.5.0" version = "2.5.0"
@@ -1512,32 +1653,54 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]] [[package]]
name = "windows" name = "windows"
version = "0.58.0" version = "0.62.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dd04d41d93c4992d421894c18c8b43496aa748dd4c081bac0dc93eb0489272b6" checksum = "527fadee13e0c05939a6a05d5bd6eec6cd2e3dbd648b9f8e447c6518133d8580"
dependencies = [
"windows-collections",
"windows-core",
"windows-future",
"windows-numerics",
]
[[package]]
name = "windows-collections"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "23b2d95af1a8a14a3c7367e1ed4fc9c20e0a26e79551b1454d72583c97cc6610"
dependencies = [ dependencies = [
"windows-core", "windows-core",
"windows-targets",
] ]
[[package]] [[package]]
name = "windows-core" name = "windows-core"
version = "0.58.0" version = "0.62.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ba6d44ec8c2591c134257ce647b7ea6b20335bf6379a27dac5f1641fcf59f99" checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb"
dependencies = [ dependencies = [
"windows-implement", "windows-implement",
"windows-interface", "windows-interface",
"windows-link",
"windows-result", "windows-result",
"windows-strings", "windows-strings",
"windows-targets", ]
[[package]]
name = "windows-future"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e1d6f90251fe18a279739e78025bd6ddc52a7e22f921070ccdc67dde84c605cb"
dependencies = [
"windows-core",
"windows-link",
"windows-threading",
] ]
[[package]] [[package]]
name = "windows-implement" name = "windows-implement"
version = "0.58.0" version = "0.60.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2bbd5b46c938e506ecbce286b6628a02171d56153ba733b6c741fc627ec9579b" checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf"
dependencies = [ dependencies = [
"proc-macro2", "proc-macro2",
"quote", "quote",
@@ -1546,9 +1709,9 @@ dependencies = [
[[package]] [[package]]
name = "windows-interface" name = "windows-interface"
version = "0.58.0" version = "0.59.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "053c4c462dc91d3b1504c6fe5a726dd15e216ba718e84a0e46a88fbe5ded3515" checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358"
dependencies = [ dependencies = [
"proc-macro2", "proc-macro2",
"quote", "quote",
@@ -1562,22 +1725,40 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]] [[package]]
name = "windows-result" name = "windows-numerics"
version = "0.2.0" version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d1043d8214f791817bab27572aaa8af63732e11bf84aa21a45a78d6c317ae0e" checksum = "6e2e40844ac143cdb44aead537bbf727de9b044e107a0f1220392177d15b0f26"
dependencies = [ dependencies = [
"windows-targets", "windows-core",
"windows-link",
]
[[package]]
name = "windows-result"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5"
dependencies = [
"windows-link",
] ]
[[package]] [[package]]
name = "windows-strings" name = "windows-strings"
version = "0.1.0" version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4cd9b125c486025df0eabcb585e62173c6c9eddcec5d117d3b6e8c30e2ee4d10" checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091"
dependencies = [ dependencies = [
"windows-result", "windows-link",
"windows-targets", ]
[[package]]
name = "windows-sys"
version = "0.45.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "75283be5efb2831d37ea142365f009c02ec203cd29a3ebecbc093d52315b66d0"
dependencies = [
"windows-targets 0.42.2",
] ]
[[package]] [[package]]
@@ -1586,7 +1767,7 @@ version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
dependencies = [ dependencies = [
"windows-targets", "windows-targets 0.52.6",
] ]
[[package]] [[package]]
@@ -1598,34 +1779,76 @@ dependencies = [
"windows-link", "windows-link",
] ]
[[package]]
name = "windows-targets"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e5180c00cd44c9b1c88adb3693291f1cd93605ded80c250a75d472756b4d071"
dependencies = [
"windows_aarch64_gnullvm 0.42.2",
"windows_aarch64_msvc 0.42.2",
"windows_i686_gnu 0.42.2",
"windows_i686_msvc 0.42.2",
"windows_x86_64_gnu 0.42.2",
"windows_x86_64_gnullvm 0.42.2",
"windows_x86_64_msvc 0.42.2",
]
[[package]] [[package]]
name = "windows-targets" name = "windows-targets"
version = "0.52.6" version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
dependencies = [ dependencies = [
"windows_aarch64_gnullvm", "windows_aarch64_gnullvm 0.52.6",
"windows_aarch64_msvc", "windows_aarch64_msvc 0.52.6",
"windows_i686_gnu", "windows_i686_gnu 0.52.6",
"windows_i686_gnullvm", "windows_i686_gnullvm",
"windows_i686_msvc", "windows_i686_msvc 0.52.6",
"windows_x86_64_gnu", "windows_x86_64_gnu 0.52.6",
"windows_x86_64_gnullvm", "windows_x86_64_gnullvm 0.52.6",
"windows_x86_64_msvc", "windows_x86_64_msvc 0.52.6",
] ]
[[package]]
name = "windows-threading"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3949bd5b99cafdf1c7ca86b43ca564028dfe27d66958f2470940f73d86d75b37"
dependencies = [
"windows-link",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "597a5118570b68bc08d8d59125332c54f1ba9d9adeedeef5b99b02ba2b0698f8"
[[package]] [[package]]
name = "windows_aarch64_gnullvm" name = "windows_aarch64_gnullvm"
version = "0.52.6" version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
[[package]]
name = "windows_aarch64_msvc"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e08e8864a60f06ef0d0ff4ba04124db8b0fb3be5776a5cd47641e942e58c4d43"
[[package]] [[package]]
name = "windows_aarch64_msvc" name = "windows_aarch64_msvc"
version = "0.52.6" version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
[[package]]
name = "windows_i686_gnu"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c61d927d8da41da96a81f029489353e68739737d3beca43145c8afec9a31a84f"
[[package]] [[package]]
name = "windows_i686_gnu" name = "windows_i686_gnu"
version = "0.52.6" version = "0.52.6"
@@ -1638,24 +1861,48 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
[[package]]
name = "windows_i686_msvc"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "44d840b6ec649f480a41c8d80f9c65108b92d89345dd94027bfe06ac444d1060"
[[package]] [[package]]
name = "windows_i686_msvc" name = "windows_i686_msvc"
version = "0.52.6" version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
[[package]]
name = "windows_x86_64_gnu"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8de912b8b8feb55c064867cf047dda097f92d51efad5b491dfb98f6bbb70cb36"
[[package]] [[package]]
name = "windows_x86_64_gnu" name = "windows_x86_64_gnu"
version = "0.52.6" version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "26d41b46a36d453748aedef1486d5c7a85db22e56aff34643984ea85514e94a3"
[[package]] [[package]]
name = "windows_x86_64_gnullvm" name = "windows_x86_64_gnullvm"
version = "0.52.6" version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
[[package]]
name = "windows_x86_64_msvc"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
[[package]] [[package]]
name = "windows_x86_64_msvc" name = "windows_x86_64_msvc"
version = "0.52.6" version = "0.52.6"
@@ -1673,6 +1920,23 @@ dependencies = [
"pkg-config", "pkg-config",
] ]
[[package]]
name = "x11rb"
version = "0.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9993aa5be5a26815fe2c3eacfc1fde061fc1a1f094bf1ad2a18bf9c495dd7414"
dependencies = [
"gethostname",
"rustix",
"x11rb-protocol",
]
[[package]]
name = "x11rb-protocol"
version = "0.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ea6fc2961e4ef194dcbfe56bb845534d0dc8098940c7e5c012a258bfec6701bd"
[[package]] [[package]]
name = "xcursor" name = "xcursor"
version = "0.3.11" version = "0.3.11"
Cargo.toml
+9 -2
View File
@@ -1,6 +1,13 @@
[workspace] [workspace]
resolver = "2" resolver = "2"
members = ["common", "agent", "viewer"] # agent-core：平台无关会话层 + 软件编码视频管线（当前仅 linux-agent 使用）
# linux-agent：Linux 被控端（X11 + XTest）
# viewer-core：安卓主控的连接/协议/收帧层（桌面 headless 可验证；Android 经 JNI 复用）
# android-ffi：viewer-core 的 JNI 薄层（cdylib，交叉编译成 libandroidffi.so）
members = ["common", "agent-core", "agent", "linux-agent", "viewer", "viewer-core", "android-ffi"]
# 默认成员保持 Windows 三件套，`cargo build --release` 行为与改动前完全一致；
# Linux 被控端显式构建：`cargo build --release -p linux-agent`
default-members = ["common", "agent", "viewer"]
[workspace.dependencies] [workspace.dependencies]
anyhow = "1" anyhow = "1"
@@ -11,5 +18,5 @@ quinn = "0.11"
rustls = { version = "0.23", default-features = false, features = ["ring", "std", "logging"] } rustls = { version = "0.23", default-features = false, features = ["ring", "std", "logging"] }
rcgen = { version = "0.13", default-features = false, features = ["ring", "pem"] } rcgen = { version = "0.13", default-features = false, features = ["ring", "pem"] }
lz4_flex = "0.11" lz4_flex = "0.11"
windows = "0.58" windows = "0.62"
minifb = "0.27" minifb = "0.27"
README.md
+1 -1
View File
@@ -27,4 +27,4 @@ agent.exe --port 47800 --code 123456
viewer.exe --addr <被控机IP> --port 47800 --code 123456 viewer.exe --addr <被控机IP> --port 47800 --code 123456
``` ```
详见《构建运行手册.md》与《Git协作规范.md》。 详见《构建运行手册.md》、《Git协作规范.md》与《打洞与信令设计.md》。
agent-core/Cargo.toml
+13
View File
@@ -0,0 +1,13 @@
[package]
name = "agent-core"
version = "0.1.0"
edition = "2021"
[dependencies]
common = { path = "../common" }
anyhow.workspace = true
tokio.workspace = true
quinn.workspace = true
rustls.workspace = true
rcgen.workspace = true
openh264 = { version = "0.9.9", features = ["source"] }
agent-core/src/lib.rs
+639
View File
@@ -0,0 +1,639 @@
//! 平台无关的被控端会话层：QUIC/TLS 握手、验证码认证、H.264 编码下行、输入分发。
//!
//! 从 `agent/`（Windows）抽出来，供 Windows 与 Linux 被控端共用，避免两端各写一份
//! QUIC 握手/认证/丢帧逻辑造成协议漂移。平台差异只通过两个 trait 注入：
//!
//! - [`ScreenCapturer`]：屏幕采集（Windows = DXGI Desktop Duplication，Linux = X11）
//! - [`InputInjector`]：输入注入（Windows = SendInput，Linux = XTest）
//!
//! 线格式完全复用 `common`，**不改协议**：键盘沿用 Set-1 扫描码 + extended 标志，
//! 鼠标沿用虚拟桌面归一化坐标 0..=65535。
use anyhow::Result;
use common::{
encode_video_frame, h264_contains_idr, read_control, write_control, ControlMsg, InputEvent,
MouseButton, VideoFrameHeader,
};
use openh264::encoder::{
BitRate, Encoder, EncoderConfig, FrameRate, IntraFramePeriod, RateControlMode, UsageType,
};
use openh264::formats::{BgraSliceU8, YUVBuffer};
use openh264::OpenH264API;
use rustls::pki_types::PrivatePkcs8KeyDer;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::sync::{mpsc, watch};
/// 屏幕采集后端。
///
/// 语义刻意对齐 DXGI Desktop Duplication，使 Linux 侧只需实现同一套模型：
/// `acquire()` 阻塞至多 `timeout_ms`，返回是否拿到**新**帧；`copy_frame()` 再取出像素。
pub trait ScreenCapturer: Send {
/// 桌面尺寸（物理像素）
fn size(&self) -> (u32, u32);
/// 阻塞至多 `timeout_ms`。
/// `Ok(true)` = 有新帧（已拷进内部缓冲，可由 `copy_frame` 取出）；
/// `Ok(false)` = 超时且画面未变（静止桌面）。
fn acquire(&mut self, timeout_ms: u32) -> Result<bool>;
/// 取出最近一帧，BGRA 紧凑排列（已去掉 row pitch），供 openh264 直接编码。
fn copy_frame(&mut self) -> Result<Vec<u8>>;
}
/// 输入注入后端。实现需可在异步运行时中被多任务共享，故要求 `Send + Sync`。
pub trait InputInjector: Send + Sync + 'static {
fn mouse_move(&self, x: u16, y: u16);
fn mouse_button(&self, b: MouseButton, down: bool);
fn wheel(&self, dy: i32);
fn key(&self, scancode: u16, extended: bool, down: bool);
/// 协议 v3 的文本注入（UTF-16 code unit）。
///
/// **默认实现刻意不注入任何东西，但必须留下可见痕迹**：Linux 后端本轮未实现
/// 文本注入（X11 需要 keysym/XKB 映射，与 Windows 的 `KEYEVENTF_UNICODE`
/// 不同构，另立循环）。全局规则要求异常路径有可见后果——用户对着 Linux 被控端
/// 打字时「什么都没发生」不能是静默的，日志必须说明是没实现。
///
/// 日志纪律：只打印长度，**不打印用户输入内容**。
fn text(&self, units: &[u16]) {
eprintln!(
"[agent-core] 文本注入未实现（忽略 {} 个 UTF-16 code unit）",
units.len()
);
}
}
/// 把一条协议输入事件分发到注入后端。
pub fn dispatch_input(inj: &dyn InputInjector, ev: InputEvent) {
match ev {
InputEvent::MouseMove { x, y } => inj.mouse_move(x, y),
InputEvent::MouseDown(b) => inj.mouse_button(b, true),
InputEvent::MouseUp(b) => inj.mouse_button(b, false),
InputEvent::Wheel { dy } => inj.wheel(dy),
InputEvent::KeyDown { scancode, extended } => inj.key(scancode, extended, true),
InputEvent::KeyUp { scancode, extended } => inj.key(scancode, extended, false),
InputEvent::Text { units } => {
// 上限在前端截断，截断必须可见（否则用户以为长文本已送达）。
let (units, truncated) = common::clamp_text_units(units);
if truncated {
eprintln!(
"[agent-core] 文本超过上限，已截断到 {} 个 code unit",
common::MAX_TEXT_UNITS
);
}
inj.text(&units);
}
}
}
/// 会话参数
#[derive(Debug, Clone)]
pub struct SessionConfig {
pub port: u16,
pub code: String,
}
/// 编码档位。默认值即 Windows 现有档位，改动需两端同步评估。
#[derive(Debug, Clone, Copy)]
pub struct EncoderSettings {
pub bitrate_bps: u32,
pub max_fps: f32,
pub intra_period: u32,
}
impl Default for EncoderSettings {
fn default() -> Self {
// 关键帧周期：以**时间**为准而非固定帧数，避免帧率变化后恢复窗口漂移。
// 1 秒 @30fps = 30 帧。主恢复路径是 KeyFrameReq（协议 v2 后已确认生效），
// 自动 IDR 只是兜底；此前 300 帧 @30fps 意味着兜底窗口长达 10 秒。
let max_fps = 30.0f32;
Self {
bitrate_bps: 5_000_000,
max_fps,
intra_period: (max_fps as u32).max(1),
}
}
}
/// 解析 `--port` / `--code`（两端共用，保持命令行一致）
pub fn parse_args() -> Result<(u16, String)> {
let mut port = 47800u16;
let mut code = "123456".to_string();
let a: Vec<String> = std::env::args().collect();
let mut i = 1;
while i + 1 < a.len() {
match a[i].as_str() {
"--port" => port = a[i + 1].parse()?,
"--code" => code = a[i + 1].clone(),
_ => {}
}
i += 2;
}
Ok((port, code))
}
fn make_encoder(enc: &EncoderSettings) -> Result<Encoder> {
Ok(Encoder::with_api_config(
OpenH264API::from_source(),
EncoderConfig::new()
.bitrate(BitRate::from_bps(enc.bitrate_bps))
.max_frame_rate(FrameRate::from_hz(enc.max_fps))
.rate_control_mode(RateControlMode::Bitrate)
.usage_type(UsageType::ScreenContentRealTime)
.intra_frame_period(IntraFramePeriod::from_num_frames(enc.intra_period)),
)?)
}
enum LoopEnd {
/// 下行通道关闭 —— 会话结束，采集线程退出
ChannelClosed,
/// 采集后端失效（Windows 的 ACCESS_LOST / X11 连接断开）—— 重建后继续
Rebuild,
}
/// 编码一帧并推入下行通道。返回 false 表示通道已关闭（会话结束）。
///
/// `seq` 由调用方持有并自增：帧序号必须覆盖**每一次实际发送**（含静止桌面重发），
/// 不能由编码器计数（编码器只知道编码了多少帧，不知道发了多少帧）。
fn encode_send(
encoder: &mut Encoder,
raw: &[u8],
(w, h): (u32, u32),
ts_ms: u64,
seq: u64,
tx: &watch::Sender<Option<Vec<u8>>>,
last_frame: &mut Option<Vec<u8>>,
) -> bool {
let yuv = YUVBuffer::from_bgra8_source(BgraSliceU8::new(raw, (w as usize, h as usize)));
match encoder.encode(&yuv) {
Ok(ebs) => {
let payload = ebs.to_vec();
let header = VideoFrameHeader {
width: w,
height: h,
ts_ms,
seq,
keyframe: h264_contains_idr(&payload),
};
let frame = encode_video_frame(&header, &payload);
if tx.send(Some(frame.clone())).is_err() {
return false;
}
*last_frame = Some(frame);
true
}
Err(e) => {
eprintln!("[capture] encode failed: {e}");
true
}
}
}
/// 单次采集会话：抓帧 → 编码 → 推入 watch 单槽。
fn capture_session(
cap: &mut dyn ScreenCapturer,
tx: &watch::Sender<Option<Vec<u8>>>,
keyreq: &AtomicBool,
enc: &EncoderSettings,
) -> LoopEnd {
let mut encoder = match make_encoder(enc) {
Ok(e) => e,
Err(e) => {
eprintln!("[capture] encoder init failed: {e}");
return LoopEnd::ChannelClosed;
}
};
let (w, h) = cap.size();
eprintln!("[capture] started {w}x{h}");
let start = Instant::now();
let min_frame = Duration::from_secs_f32(1.0 / enc.max_fps);
let mut last_frame: Option<Vec<u8>> = None;
// 保留最近一帧裸像素：静止桌面收到关键帧请求时，仍能重编一个真关键帧
let mut last_raw: Option<Vec<u8>> = None;
let mut last_sent = Instant::now() - Duration::from_secs(10);
// 帧序号：覆盖每一次实际发送（含静止桌面重发），供主控端做丢帧检测
let mut seq: u64 = 0;
loop {
// 会话结束守卫：发送端已 drop 接收器时立即退出。
// 与 agent/src/capture.rs 同源——静止桌面分支不发送，只靠 send 失败退出
// 会让线程在画面不动时永不退出（X11 无 DXGI 式独占，危害较小但同样泄漏线程）。
if tx.receiver_count() == 0 {
eprintln!("[capture] no receiver, stop");
return LoopEnd::ChannelClosed;
}
let iter_start = Instant::now();
match cap.acquire(80) {
Ok(true) => match cap.copy_frame() {
Ok(raw) => {
// 主控端请求关键帧（解码异常恢复）
if keyreq.swap(false, Ordering::Relaxed) {
encoder.force_intra_frame();
}
let ts = start.elapsed().as_millis() as u64;
if !encode_send(&mut encoder, &raw, (w, h), ts, seq, tx, &mut last_frame) {
return LoopEnd::ChannelClosed; // 会话结束
}
seq = seq.wrapping_add(1);
last_raw = Some(raw);
last_sent = Instant::now();
// 帧间隔节流到 max_fps
let el = iter_start.elapsed();
if el < min_frame {
std::thread::sleep(min_frame - el);
}
}
Err(e) => {
eprintln!("[capture] frame copy failed: {e}");
return LoopEnd::Rebuild;
}
},
Ok(false) => {
// 静止桌面：先看关键帧请求，再决定是否走 500ms 保活节流。
//
// ⚠️ 这里必须保证「消费了标志就一定发送」：先前写法是
// `if force && last_raw.is_some() { ... }`，当 force 为真但
// last_raw 还是 None（刚建会话、尚未抓到任何一帧）时，标志被
// swap 消费掉却没有发送，解码端的这次请求就永久丢失。
let force = keyreq.swap(false, Ordering::Relaxed);
let can_key = last_raw.is_some();
if force && !can_key {
// 还没有裸帧可重编：把请求还回去，让下一轮（拿到首帧后）继续生效
keyreq.store(true, Ordering::Relaxed);
eprintln!("[capture] KeyFrameReq 暂存：尚无裸帧可重编");
std::thread::sleep(Duration::from_millis(8));
} else if force {
// 静止桌面下的关键帧恢复：画面没变不代表可以不给关键帧——
// 解码端一旦出错，若只会收到旧 P 帧就永远恢复不了。
// 关键帧请求绕过 500ms 节流：它是错误恢复路径，延迟一次 IDR
// 的代价是持续黑屏，远高于多发一帧的带宽成本。
encoder.force_intra_frame();
let raw = last_raw.as_ref().expect("can_key 已保证");
let ts = start.elapsed().as_millis() as u64;
if !encode_send(&mut encoder, raw, (w, h), ts, seq, tx, &mut last_frame) {
return LoopEnd::ChannelClosed;
}
seq = seq.wrapping_add(1);
last_sent = Instant::now();
} else if last_sent.elapsed() > Duration::from_millis(500) && last_frame.is_some() {
// 桌面静止：低频重发最后一帧（重复 P 帧，解码端画面不变）。
// 帧头必须重建：直接 clone 旧帧会让 seq/ts_ms 停在上次发送时刻，
// 破坏 seq 单调递增契约。
let f = last_frame.as_ref().expect("上面已判断 is_some");
match common::split_video_frame(f) {
Ok((mut hdr, payload)) => {
hdr.seq = seq;
hdr.ts_ms = start.elapsed().as_millis() as u64;
let frame = encode_video_frame(&hdr, payload);
if tx.send(Some(frame)).is_err() {
return LoopEnd::ChannelClosed;
}
seq = seq.wrapping_add(1);
last_sent = Instant::now();
}
Err(e) => {
// 自己产出的帧解析失败 = 编码/封装路径有 bug。
// 丢弃缓存而不是每轮重复报错：静止桌面下这里会以 ~125Hz 触发。
eprintln!("[capture] 静止重发时帧格式异常，丢弃缓存: {e}");
last_frame = None;
std::thread::sleep(Duration::from_millis(8));
}
}
} else {
std::thread::sleep(Duration::from_millis(8));
}
}
Err(e) => {
eprintln!("[capture] acquire failed: {e}");
return LoopEnd::Rebuild;
}
}
}
}
/// 采集主循环：独立线程运行；后端失效时经 `factory` 重建，通道关闭时退出。
/// 通道为 watch 单槽（latest-wins）：发送即覆盖，网络慢时不产生积压。
pub fn run_capture_loop<F>(
tx: watch::Sender<Option<Vec<u8>>>,
keyreq: Arc<AtomicBool>,
mut cap: Box<dyn ScreenCapturer>,
mut factory: F,
enc: EncoderSettings,
) where
F: FnMut() -> Result<Box<dyn ScreenCapturer>>,
{
loop {
match capture_session(cap.as_mut(), &tx, keyreq.as_ref(), &enc) {
LoopEnd::ChannelClosed => return,
LoopEnd::Rebuild => {}
}
std::thread::sleep(Duration::from_millis(300));
loop {
match factory() {
Ok(c) => {
cap = c;
break;
}
Err(e) => {
eprintln!("[capture] init failed: {e}, retry in 2s");
std::thread::sleep(Duration::from_secs(2));
}
}
}
}
}
/// 跑一次完整的被控端会话：监听 → 认证 → 视频下行 + 输入上行。
///
/// `capture_factory` 可被多次调用（采集后端失效时重建），故需为 `FnMut`。
pub async fn serve<F>(
cfg: SessionConfig,
enc: EncoderSettings,
mut capture_factory: F,
injector: Arc<dyn InputInjector>,
) -> Result<()>
where
F: FnMut() -> Result<Box<dyn ScreenCapturer>> + Send + 'static,
{
rustls::crypto::ring::default_provider()
.install_default()
.ok();
// 自签名证书（MVP：客户端跳过校验；正式版改为固定公钥/账号体系）
let ck = rcgen::generate_simple_self_signed(vec!["localhost".into()])?;
let cert = ck.cert.der().clone();
let key = PrivatePkcs8KeyDer::from(ck.key_pair.serialize_der());
// 传输参数调优：
// - 收发窗口放大，避免高 RTT 下吞吐被 窗口/RTT 死锁（默认 512KB 流窗口）
// - 发送缓冲上限 512KB：QUIC 内部排队 ≈ 3 帧 @5Mbps，E2E 延迟天花板 ~150ms；
// 配合上游 latest-wins 丢帧，超量部分在进 QUIC 前就被丢弃
// （与 Windows 端 agent/src/main.rs 保持一致，沿用同一次调优结论）
let mut transport = quinn::TransportConfig::default();
transport.receive_window(quinn::VarInt::from_u32(16 * 1024 * 1024));
transport.stream_receive_window(quinn::VarInt::from_u32(8 * 1024 * 1024));
transport.send_window(512 * 1024);
let mut server_config = quinn::ServerConfig::with_single_cert(vec![cert], key.into())?;
server_config.transport_config(Arc::new(transport));
let endpoint = quinn::Endpoint::server(server_config, format!("0.0.0.0:{}", cfg.port).parse()?)?;
println!(
"[agent] listening 0.0.0.0:{}, verify code = {}",
cfg.port, cfg.code
);
let conn = endpoint
.accept()
.await
.ok_or_else(|| anyhow::anyhow!("endpoint closed"))?
.await?;
println!("[agent] peer connected: {}", conn.remote_address());
// 流 1：认证 + 视频下行
let (mut send, mut recv) = conn.accept_bi().await?;
let first = read_control(&mut recv).await?;
// 先校验版本，再校验验证码：版本不一致时字段布局可能不同，验证码的解析结果不可信
let auth_err: Option<String> = match &first {
ControlMsg::Auth {
code: c,
proto_version,
} => {
if *proto_version != common::PROTO_VERSION {
Some(format!(
"protocol version mismatch: client={proto_version}, server={}",
common::PROTO_VERSION
))
} else if *c != cfg.code {
Some("bad code".into())
} else {
None
}
}
other => Some(format!("expected Auth, got {other:?}")),
};
if let Some(msg) = auth_err {
write_control(&mut send, &ControlMsg::AuthFail(msg.clone())).await?;
send.finish()?;
// 必须等对端读到 AuthFail 并挂断再退出：直接 return 会让进程带走 QUIC
// 缓冲区里尚未发出的数据，客户端只会看到 "connection lost" 而不是 AuthFail
let _ = tokio::time::timeout(Duration::from_secs(5), conn.closed()).await;
anyhow::bail!(msg);
}
// 采集后端先建一次：既拿桌面尺寸回给主控端，也避免白建一个 duplication 只为读尺寸
let first_cap = capture_factory()?;
let (w, h) = first_cap.size();
write_control(
&mut send,
&ControlMsg::AuthOk {
width: w,
height: h,
proto_version: common::PROTO_VERSION,
},
)
.await?;
println!("[agent] auth ok, desktop {w}x{h}");
// 流 2：键鼠输入 + 控制上行（与视频分离，避免被视频积压阻塞）
let (mut isend, mut irecv) = conn.accept_bi().await?;
// 采集线程 → 视频帧（watch 单槽：新帧覆盖旧帧，latest-wins）
let (tx, mut frames_rx) = watch::channel::<Option<Vec<u8>>>(None);
let keyreq = Arc::new(AtomicBool::new(false));
let keyreq_capture = keyreq.clone();
let enc_settings = enc;
std::thread::spawn(move || {
run_capture_loop(tx, keyreq_capture, first_cap, capture_factory, enc_settings)
});
// 上行控制消息（Pong）经通道交给主循环写流 2，避免多任务争抢
let (ctl_tx, mut ctl_rx) = mpsc::unbounded_channel::<ControlMsg>();
let t0 = Instant::now();
// 读循环（流 2）：输入事件 + Ping；Pong 附带**视频下行方向**的 QUIC 丢包统计
// （conn.stats().path 统计的是本端发送方向，被控端的发送方向正是视频下行）
let keyreq_reader = keyreq.clone();
let conn_stats = conn.clone();
let reader = tokio::spawn(async move {
loop {
match read_control(&mut irecv).await {
Ok(ControlMsg::Input(ev)) => dispatch_input(injector.as_ref(), ev),
Ok(ControlMsg::KeyFrameReq) => {
keyreq_reader.store(true, Ordering::Relaxed);
}
Ok(ControlMsg::Ping(t)) => {
let agent_ts = t0.elapsed().as_millis() as u64;
let p = conn_stats.stats().path;
let _ = ctl_tx.send(ControlMsg::Pong {
t,
agent_ts,
lost: p.lost_packets,
sent: p.sent_packets,
});
}
Ok(_) => {}
Err(_) => break,
}
}
});
loop {
tokio::select! {
changed = frames_rx.changed() => {
if changed.is_err() {
break; // 采集线程退出
}
if let Some(data) = frames_rx.borrow().clone() {
if send.write_all(&data).await.is_err() {
break;
}
}
}
ctl = ctl_rx.recv() => {
match ctl {
Some(m) => {
if write_control(&mut isend, &m).await.is_err() {
break;
}
}
None => break,
}
}
}
}
reader.abort();
println!("[agent] session closed");
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Mutex;
/// 记录调用轨迹的注入后端：验证分发与截断，不碰任何真实输入。
#[derive(Default)]
struct MockInjector {
calls: Mutex<Vec<String>>,
}
impl MockInjector {
fn take(&self) -> Vec<String> {
let mut g = self.calls.lock().unwrap();
std::mem::take(&mut g)
}
}
impl InputInjector for MockInjector {
fn mouse_move(&self, x: u16, y: u16) {
self.calls.lock().unwrap().push(format!("move {x},{y}"));
}
fn mouse_button(&self, b: MouseButton, down: bool) {
self.calls.lock().unwrap().push(format!("btn {b:?} {down}"));
}
fn wheel(&self, dy: i32) {
self.calls.lock().unwrap().push(format!("wheel {dy}"));
}
fn key(&self, scancode: u16, extended: bool, down: bool) {
self.calls
.lock()
.unwrap()
.push(format!("key {scancode:#x} ext={extended} {down}"));
}
// 刻意**不**实现 text()：让默认实现进入测试面，验证「后端没做文本注入
// 时也必须被 dispatch_input 正常调用、且不 panic」——Linux 端正是这个形态。
}
/// 实现了 text() 的后端，用来验证截断后的长度真的传给了注入侧。
struct TextCapableInjector {
received: Mutex<Vec<usize>>,
}
impl InputInjector for TextCapableInjector {
fn mouse_move(&self, _x: u16, _y: u16) {}
fn mouse_button(&self, _b: MouseButton, _down: bool) {}
fn wheel(&self, _dy: i32) {}
fn key(&self, _scancode: u16, _extended: bool, _down: bool) {}
fn text(&self, units: &[u16]) {
self.received.lock().unwrap().push(units.len());
}
}
#[test]
fn dispatch_routes_each_event_kind() {
let inj = MockInjector::default();
dispatch_input(&inj, InputEvent::MouseMove { x: 1, y: 2 });
dispatch_input(&inj, InputEvent::MouseDown(MouseButton::Right));
dispatch_input(&inj, InputEvent::Wheel { dy: -120 });
dispatch_input(
&inj,
InputEvent::KeyDown {
scancode: 0x48,
extended: true,
},
);
let calls = inj.take();
assert_eq!(
calls,
vec![
"move 1,2",
"btn Right true",
"wheel -120",
"key 0x48 ext=true true",
]
);
}
/// v3 关键回归：`Text` 必须被分发，且**后端没有实现 text() 时不能 panic**。
/// 这是 Linux/X11 后端的真实形态——用户对着 Linux 被控端打字会走到这里。
#[test]
fn text_event_reaches_default_backend_without_panic() {
let inj = MockInjector::default();
dispatch_input(
&inj,
InputEvent::Text {
units: vec![0x4F60, 0x597D],
},
);
// 默认实现只记日志，不产生调用记录；此处断言"没炸"即为通过。
assert!(inj.take().is_empty());
}
/// 截断契约：超过 `MAX_TEXT_UNITS` 的输入必须以截断后的长度到达注入侧。
/// 不截断会让长粘贴把远端注入队列刷爆；截断不告知则是静默丢数据。
#[test]
fn text_is_clamped_to_protocol_cap() {
let inj = TextCapableInjector {
received: Mutex::new(vec![]),
};
dispatch_input(
&inj,
InputEvent::Text {
units: vec![0x61; common::MAX_TEXT_UNITS + 500],
},
);
dispatch_input(
&inj,
InputEvent::Text {
units: vec![0x62; 10],
},
);
let got = inj.received.lock().unwrap().clone();
assert_eq!(got, vec![common::MAX_TEXT_UNITS, 10], "超限截断、未超限原样");
}
/// 空 Text 仍会调用后端（是否忽略由后端决定）：主控侧 `send_text` 已拦掉空输入，
/// 但被控端不能假设对端守规矩——线格式输入一律按不可信处理。
#[test]
fn empty_text_still_dispatches() {
let inj = TextCapableInjector {
received: Mutex::new(vec![]),
};
dispatch_input(&inj, InputEvent::Text { units: vec![] });
assert_eq!(inj.received.lock().unwrap().clone(), vec![0]);
}
}
agent/Cargo.toml
+4
View File
@@ -12,6 +12,10 @@ rustls.workspace = true
rcgen.workspace = true rcgen.workspace = true
lz4_flex.workspace = true lz4_flex.workspace = true
openh264 = { version = "0.9.9", features = ["source"] } openh264 = { version = "0.9.9", features = ["source"] }
# 上游 nvenc 0.1.0 的 encode_picture 只设 picture_type=IDR、不置 FORCEIDR 标志，
# 导致运行期 KeyFrameReq 被编码器静默忽略（实测输出 NAL type=1 而非 5）。
# vendor 版本补上了该标志，见 vendor/nvenc/src/safe/encoder.rs 的「本项目补丁」注释。
nvenc = { path = "../vendor/nvenc" }
windows = { workspace = true, features = [ windows = { workspace = true, features = [
"Win32_Foundation", "Win32_Foundation",
"Win32_Graphics_Direct3D", "Win32_Graphics_Direct3D",
agent/src/bin/dump_stream.rs
+316
View File
@@ -0,0 +1,316 @@
//! 临时诊断工具：抓 agent 下发的码流，逐帧打印 NAL 结构并试解码。
//!
//! 目的：定位「首帧能解、之后连续 decode failed」的根因，不靠猜。
//!
//! 用法：
//! cargo run --release -p agent --bin dump_stream -- --port 47805 --code t
//! cargo run --release -p agent --bin dump_stream -- --port 47805 --code t --skip-at 2
//!
//! `--skip-at N`：模拟主控端的「积压跳帧」行为——第 N 帧读出但不喂给解码器，
//! 用来验证「跳帧是否直接导致后续所有 P 帧解不出来」。
use anyhow::{bail, Result};
use common::{read_control, read_video_frame, write_control, ControlMsg};
use openh264::decoder::Decoder;
use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier};
use rustls::pki_types::{CertificateDer, ServerName, UnixTime};
use rustls::{DigitallySignedStruct, SignatureScheme};
use std::net::ToSocketAddrs;
use std::sync::Arc;
use std::time::Duration;
#[derive(Debug)]
struct SkipVerify;
impl ServerCertVerifier for SkipVerify {
fn verify_server_cert(
&self,
_e: &CertificateDer<'_>,
_i: &[CertificateDer<'_>],
_s: &ServerName<'_>,
_o: &[u8],
_n: UnixTime,
) -> Result<ServerCertVerified, rustls::Error> {
Ok(ServerCertVerified::assertion())
}
fn verify_tls12_signature(
&self,
_m: &[u8],
_c: &CertificateDer<'_>,
_d: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
Ok(HandshakeSignatureValid::assertion())
}
fn verify_tls13_signature(
&self,
_m: &[u8],
_c: &CertificateDer<'_>,
_d: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
Ok(HandshakeSignatureValid::assertion())
}
fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
vec![
SignatureScheme::RSA_PKCS1_SHA256,
SignatureScheme::ECDSA_NISTP256_SHA256,
SignatureScheme::ED25519,
SignatureScheme::RSA_PSS_SHA256,
]
}
}
/// 扫 Annex-B 起始码，返回 NAL 类型列表（如 [7, 8, 5]）
fn nal_types(payload: &[u8]) -> Vec<u8> {
let mut out = Vec::new();
let mut i = 0usize;
while i + 3 < payload.len() {
let sc = if payload[i..].starts_with(&[0, 0, 0, 1]) {
4
} else if payload[i..].starts_with(&[0, 0, 1]) {
3
} else {
i += 1;
continue;
};
match payload.get(i + sc) {
Some(&b) => {
out.push(b & 0x1F);
i += sc + 1;
}
None => break,
}
}
out
}
fn describe(types: &[u8]) -> String {
let has = |t: u8| types.contains(&t);
let mut s = String::new();
if has(7) {
s.push_str("SPS ");
}
if has(8) {
s.push_str("PPS ");
}
if has(5) {
s.push_str("IDR ");
}
if has(1) {
s.push_str("P ");
}
if has(6) {
s.push_str("SEI ");
}
if s.is_empty() {
s.push_str("(未知)");
}
format!("{}[{}]", s.trim(), types.len())
}
fn arg(name: &str) -> Option<String> {
let a: Vec<String> = std::env::args().collect();
a.iter()
.position(|x| x == name)
.and_then(|i| a.get(i + 1))
.cloned()
}
#[tokio::main]
async fn main() -> Result<()> {
rustls::crypto::ring::default_provider()
.install_default()
.ok();
let port: u16 = arg("--port")
.map(|s| s.parse())
.transpose()?
.unwrap_or(47800);
let code = arg("--code").unwrap_or_else(|| "123456".into());
let skip_at: Option<usize> = arg("--skip-at").map(|s| s.parse()).transpose()?;
let total: usize = arg("--frames")
.map(|s| s.parse())
.transpose()?
.unwrap_or(30);
// --save <路径>：把**裸 H.264 码流**（各帧载荷按到达顺序直接拼接）
// 写成 Annex-B 文件，供安卓 MediaCodec 等外部解码器验证兼容性。
// 注意存的是载荷而非完整帧（不含自定义帧头），因为外部解码器只认标准 Annex-B。
let save_path = arg("--save");
let tls = rustls::ClientConfig::builder()
.dangerous()
.with_custom_certificate_verifier(Arc::new(SkipVerify))
.with_no_client_auth();
let qc: quinn::crypto::rustls::QuicClientConfig = Arc::new(tls).try_into()?;
let mut transport = quinn::TransportConfig::default();
transport.receive_window(quinn::VarInt::from_u32(16 * 1024 * 1024));
transport.stream_receive_window(quinn::VarInt::from_u32(8 * 1024 * 1024));
let mut client_config = quinn::ClientConfig::new(Arc::new(qc));
client_config.transport_config(Arc::new(transport));
let mut endpoint = quinn::Endpoint::client("0.0.0.0:0".parse()?)?;
endpoint.set_default_client_config(client_config);
let addr = ("127.0.0.1", port)
.to_socket_addrs()?
.next()
.ok_or_else(|| anyhow::anyhow!("解析失败"))?;
let conn = endpoint.connect(addr, "localhost")?.await?;
println!("[dump] connected");
let (mut send, mut recv) = conn.open_bi().await?;
write_control(
&mut send,
&ControlMsg::Auth {
code,
proto_version: common::PROTO_VERSION,
},
)
.await?;
match read_control(&mut recv).await? {
ControlMsg::AuthOk {
width,
height,
proto_version,
} => println!("[dump] auth ok {width}x{height} proto={proto_version}"),
other => bail!("认证失败: {other:?}"),
}
let (mut isend, _irecv) = conn.open_bi().await?;
// 入会先请求一次 IDR，和 viewer 行为一致。
// 注意：KeyFrameReq 必须走**流 2**（isend），agent 的读循环只监听流 2。
write_control(&mut isend, &ControlMsg::KeyFrameReq).await?;
let mut decoder = Decoder::new()?;
let mut sent_kfreq = false;
// 累积裸码流，用于 --save
let mut saved: Vec<u8> = Vec::new();
println!("\n序号 seq kf 载荷字节 NAL 结构 解码结果");
println!("{}", "-".repeat(72));
for i in 0..total {
let (hdr, payload) = match tokio::time::timeout(
Duration::from_secs(10),
read_video_frame(&mut recv),
)
.await
{
Ok(r) => r?,
Err(_) => {
println!("[{i}] 读帧超时（桌面可能完全静止）");
break;
}
};
let types = nal_types(&payload);
let is_idr = types.contains(&5);
// 需要在 save 里也保留这一帧时，先记录到缓冲
if save_path.is_some() {
saved.extend_from_slice(&payload);
}
// 交叉校验：帧头 keyframe 标志必须与码流实际 NAL 类型一致。
// 这条不一致意味着时间戳/序号契约或编码器标志有 bug，比画面花屏更难排查。
if hdr.keyframe != is_idr {
println!(
" ⚠️ 帧头 keyframe={} 但码流实际 IDR={}（seq={}）",
hdr.keyframe, is_idr, hdr.seq
);
}
// 模拟主控端积压跳帧：读出但不解码
if Some(i) == skip_at {
println!(
"[{i:2}] {:3} {:<5} {:8} {:<14} ⏭ 跳过解码（模拟积压跳帧）",
hdr.seq,
hdr.keyframe,
payload.len(),
describe(&types)
);
continue;
}
let res = match decoder.decode(&payload) {
Ok(Some(_)) => "✅ 解出画面".to_string(),
Ok(None) => "○ 无输出（缓冲中）".to_string(),
Err(e) => format!("❌ {e}"),
};
println!(
"[{i:2}] {:3} {:<5} {:8} {:<14} {}",
hdr.seq,
hdr.keyframe,
payload.len(),
describe(&types),
res
);
// 和 viewer 一样：解码失败就请求关键帧
if res.starts_with('❌') && !sent_kfreq {
sent_kfreq = true;
write_control(&mut isend, &ControlMsg::KeyFrameReq).await?;
println!(" ↑ 已发 KeyFrameReq");
}
}
// 诊断：静止桌面下再请求一次关键帧，看是否真能拿到 IDR
println!("\n[dump] 再发一次 KeyFrameReq，观察 10 帧…");
write_control(&mut isend, &ControlMsg::KeyFrameReq).await?;
for i in 0..10 {
match tokio::time::timeout(Duration::from_secs(5), read_video_frame(&mut recv)).await {
Ok(Ok((hdr, payload))) => {
let types = nal_types(&payload);
println!(
"[k{i:2}] seq={:3} kf={:<5} {:6}B {}",
hdr.seq,
hdr.keyframe,
payload.len(),
describe(&types)
);
if types.contains(&5) {
println!(" ↑ 收到 IDR");
}
}
_ => {
println!("[k{i:2}] 超时/流关闭");
break;
}
}
}
// 导出裸码流文件（Annex-B 拼接），供外部解码器验证
if let Some(p) = save_path.as_ref() {
std::fs::write(p, &saved)?;
let idr_cnt = {
// 粗略统计 IDR 数量：扫起始码后的 NAL 类型
let mut n = 0;
let mut i = 0usize;
while i + 3 < saved.len() {
let sc = if saved[i..].starts_with(&[0, 0, 0, 1]) {
4
} else if saved[i..].starts_with(&[0, 0, 1]) {
3
} else {
i += 1;
continue;
};
match saved.get(i + sc) {
Some(&b) if b & 0x1F == 5 => {
n += 1;
i += sc + 1;
}
Some(_) => i += sc + 1,
None => break,
}
}
n
};
println!(
"\n[dump] 已保存裸码流: {p}（{} 字节，含 {idr_cnt} 个 IDR）",
saved.len()
);
}
let _ = conn.close(0u32.into(), b"done");
Ok(())
}
agent/src/bin/encbench.rs
+59
View File
@@ -0,0 +1,59 @@
//! 编码基准：1080p 噪声帧（最恶劣压缩工况）对比单线程与 4 线程 openh264。
//! 运行：cargo run --release -p agent --bin encbench
use openh264::encoder::{
BitRate, Encoder, EncoderConfig, FrameRate, IntraFramePeriod, RateControlMode, UsageType,
};
use openh264::formats::{BgraSliceU8, YUVBuffer};
use openh264::OpenH264API;
use std::time::Instant;
fn bench(threads: u16, frames: usize) -> f64 {
let mut enc = Encoder::with_api_config(
OpenH264API::from_source(),
EncoderConfig::new()
.bitrate(BitRate::from_bps(5_000_000))
.max_frame_rate(FrameRate::from_hz(30.0))
.rate_control_mode(RateControlMode::Bitrate)
.usage_type(UsageType::ScreenContentRealTime)
.intra_frame_period(IntraFramePeriod::from_num_frames(300))
.num_threads(threads),
)
.expect("encoder");
// 预生成 10 张不同噪声帧轮换（保证每帧 P 帧都真实编码，避免运动估计空转）
let mut frames_in = Vec::with_capacity(10);
for f in 0..10u32 {
let mut frame = vec![0u8; 1920 * 1080 * 4];
let mut seed = 12345u32.wrapping_add(f * 7919);
for px in frame.chunks_exact_mut(4) {
seed = seed.wrapping_mul(1103515245).wrapping_add(12345);
px.copy_from_slice(&seed.to_le_bytes());
}
frames_in.push(YUVBuffer::from_bgra8_source(BgraSliceU8::new(&frame, (1920, 1080))));
}
// 预热 5 帧（首帧 IDR、线程池初始化）
for (i, y) in frames_in.iter().enumerate().take(5) {
let _ = enc.encode(y).unwrap_or_else(|_| panic!("frame {i}"));
}
let t = Instant::now();
for (i, y) in frames_in.iter().cycle().take(frames).enumerate() {
let _ = enc.encode(y).unwrap_or_else(|_| panic!("frame {i}"));
}
t.elapsed().as_millis() as f64 / frames as f64
}
fn main() {
println!("1080p 噪声帧编码基准（含 BGRA→YUV 转换）");
let t = Instant::now();
let _ = YUVBuffer::from_bgra8_source(BgraSliceU8::new(
&vec![7u8; 1920 * 1080 * 4],
(1920, 1080),
));
println!("BGRA→YUV 转换: {}ms/帧", t.elapsed().as_millis());
let s = bench(1, 20);
println!("单线程: {s:.1}ms/帧 → {}fps", (1000.0 / s) as u32);
let m = bench(4, 20);
println!("4 线程: {m:.1}ms/帧 → {}fps", (1000.0 / m) as u32);
}
agent/src/capture.rs
+406 -75
View File
@@ -1,9 +1,17 @@
//! DXGI Desktop Duplication 采集（Windows 8+）。 //! DXGI Desktop Duplication 采集（Windows 8+）。
//! 帧处理：BGRA → openh264 H.264 编码（屏幕内容实时档，5Mbps）→ 线格式。 //! 编码器自动选择：NVENC 硬编（N 卡：GPU 内 BGRA→H.264 零转换零拷贝直通）
//! → openh264 软编兜底（无 N 卡/驱动异常时自动启用）。
//! 支持主控端 KeyFrameReq 强制关键帧（解码异常恢复）。 //! 支持主控端 KeyFrameReq 强制关键帧（解码异常恢复）。
use anyhow::{anyhow, Result}; use anyhow::{anyhow, Result};
use common::{encode_video_frame, VideoFrameHeader}; use common::{encode_video_frame, h264_contains_idr, VideoFrameHeader};
use nvenc::bitstream::BitStream;
use nvenc::encoder::RegisteredResource;
use nvenc::session::{InitParams, Session};
use nvenc::sys::enums::{
NVencBufferFormat, NVencPicStruct, NVencPicType, NVencParamsRcMode, NVencTuningInfo,
};
use nvenc::sys::guids::{NV_ENC_CODEC_H264_GUID, NV_ENC_PRESET_P3_GUID};
use openh264::encoder::{ use openh264::encoder::{
BitRate, Encoder, EncoderConfig, FrameRate, IntraFramePeriod, RateControlMode, UsageType, BitRate, Encoder, EncoderConfig, FrameRate, IntraFramePeriod, RateControlMode, UsageType,
}; };
@@ -19,8 +27,9 @@ use windows::Win32::Graphics::Direct3D::{
}; };
use windows::Win32::Graphics::Direct3D11::{ use windows::Win32::Graphics::Direct3D11::{
D3D11CreateDevice, ID3D11Device, ID3D11DeviceContext, ID3D11Resource, ID3D11Texture2D, D3D11CreateDevice, ID3D11Device, ID3D11DeviceContext, ID3D11Resource, ID3D11Texture2D,
D3D11_CREATE_DEVICE_FLAG, D3D11_CPU_ACCESS_READ, D3D11_MAP, D3D11_MAP_READ, D3D11_BIND_RENDER_TARGET, D3D11_BIND_SHADER_RESOURCE, D3D11_CREATE_DEVICE_FLAG,
D3D11_MAPPED_SUBRESOURCE, D3D11_SDK_VERSION, D3D11_TEXTURE2D_DESC, D3D11_USAGE_STAGING, D3D11_CPU_ACCESS_READ, D3D11_MAP_READ, D3D11_MAPPED_SUBRESOURCE,
D3D11_SDK_VERSION, D3D11_TEXTURE2D_DESC, D3D11_USAGE_DEFAULT, D3D11_USAGE_STAGING,
}; };
use windows::Win32::Graphics::Dxgi::Common::{DXGI_FORMAT_B8G8R8A8_UNORM, DXGI_SAMPLE_DESC}; use windows::Win32::Graphics::Dxgi::Common::{DXGI_FORMAT_B8G8R8A8_UNORM, DXGI_SAMPLE_DESC};
use windows::Win32::Graphics::Dxgi::{ use windows::Win32::Graphics::Dxgi::{
@@ -30,10 +39,24 @@ use windows::Win32::Graphics::Dxgi::{
use std::time::{Duration, Instant}; use std::time::{Duration, Instant};
/// 周期 IDR 的帧间隔（兜底恢复点）。
///
/// 两个编码器共用同一常量，避免 NVENC 与软编的恢复窗口不一致——此前分别是
/// 240 帧@60fps=4s 与 300 帧@60fps=5s，而 Linux 端又是 300 帧@30fps=10s。
///
/// 取值理由：**主恢复路径是 KeyFrameReq，不是自动 IDR**（协议 v2 后已确认请求
/// 真正生效，见 vendor/nvenc 的 FORCEIDR 补丁）。自动 IDR 只用于兜底
/// 「请求连续丢失」这种极端情况。60 帧 @60fps = 1 秒，既足够兜底，
/// 又不会因过密的整帧内编码抬高码率。
pub const INTRA_PERIOD_FRAMES: u32 = 60;
pub struct Duplication { pub struct Duplication {
_device: ID3D11Device, pub device: ID3D11Device,
context: ID3D11DeviceContext, context: ID3D11DeviceContext,
staging_res: ID3D11Resource, staging_res: ID3D11Resource,
/// GPU 内中间纹理（NVENC 注册目标；DEFAULT usage，BGRA 与采集一致）
gpu_res: ID3D11Resource,
gpu_tex: ID3D11Texture2D,
dup: IDXGIOutputDuplication, dup: IDXGIOutputDuplication,
pub width: u32, pub width: u32,
pub height: u32, pub height: u32,
@@ -47,13 +70,13 @@ impl Duplication {
D3D11CreateDevice( D3D11CreateDevice(
None, None,
D3D_DRIVER_TYPE_HARDWARE, D3D_DRIVER_TYPE_HARDWARE,
HMODULE::default(), HMODULE(std::ptr::null_mut()),
D3D11_CREATE_DEVICE_FLAG(0), D3D11_CREATE_DEVICE_FLAG(0),
Some(&[D3D_FEATURE_LEVEL_11_0]), Some(&[D3D_FEATURE_LEVEL_11_0]),
D3D11_SDK_VERSION, D3D11_SDK_VERSION,
Some(&mut device), Some(&raw mut device),
None, None,
Some(&mut context), Some(&raw mut context),
) )
.map_err(|e| anyhow!("D3D11CreateDevice failed: {e}"))?; .map_err(|e| anyhow!("D3D11CreateDevice failed: {e}"))?;
let device = device.ok_or_else(|| anyhow!("no d3d device"))?; let device = device.ok_or_else(|| anyhow!("no d3d device"))?;
@@ -73,6 +96,7 @@ impl Duplication {
let (width, height) = (desc.ModeDesc.Width, desc.ModeDesc.Height); let (width, height) = (desc.ModeDesc.Width, desc.ModeDesc.Height);
anyhow::ensure!(width > 0 && height > 0, "bad desktop size"); anyhow::ensure!(width > 0 && height > 0, "bad desktop size");
// staging：软编路径 Map 读内存
let sdesc = D3D11_TEXTURE2D_DESC { let sdesc = D3D11_TEXTURE2D_DESC {
Width: width, Width: width,
Height: height, Height: height,
@@ -95,10 +119,26 @@ impl Duplication {
let staging = staging_opt.ok_or_else(|| anyhow!("no staging texture"))?; let staging = staging_opt.ok_or_else(|| anyhow!("no staging texture"))?;
let staging_res: ID3D11Resource = staging.cast()?; let staging_res: ID3D11Resource = staging.cast()?;
// GPU 中间纹理：NVENC 路径的注册目标（DEFAULT usage，GPU 可读）
let gdesc = D3D11_TEXTURE2D_DESC {
Usage: D3D11_USAGE_DEFAULT,
BindFlags: (D3D11_BIND_RENDER_TARGET.0 | D3D11_BIND_SHADER_RESOURCE.0) as u32,
CPUAccessFlags: 0,
..sdesc
};
let mut gpu_opt: Option<ID3D11Texture2D> = None;
device
.CreateTexture2D(&gdesc, None, Some(&mut gpu_opt))
.map_err(|e| anyhow!("CreateTexture2D(gpu) failed: {e}"))?;
let gpu_tex = gpu_opt.ok_or_else(|| anyhow!("no gpu texture"))?;
let gpu_res: ID3D11Resource = gpu_tex.cast()?;
Ok(Self { Ok(Self {
_device: device, device,
context, context,
staging_res, staging_res,
gpu_res,
gpu_tex,
dup, dup,
width, width,
height, height,
@@ -106,7 +146,8 @@ impl Duplication {
} }
} }
/// true=拿到新帧并已复制到 staging；false=超时无更新 /// true=拿到新帧并已复制；false=超时无更新。
/// 同时复制到 staging（软编读内存）与 GPU 中间纹理（NVENC 注册输入）。
pub fn acquire(&mut self, timeout_ms: u32) -> Result<bool> { pub fn acquire(&mut self, timeout_ms: u32) -> Result<bool> {
unsafe { unsafe {
let mut info = DXGI_OUTDUPL_FRAME_INFO::default(); let mut info = DXGI_OUTDUPL_FRAME_INFO::default();
@@ -116,6 +157,7 @@ impl Duplication {
let res = resource.ok_or_else(|| anyhow!("no desktop resource"))?; let res = resource.ok_or_else(|| anyhow!("no desktop resource"))?;
let src: ID3D11Resource = res.cast()?; let src: ID3D11Resource = res.cast()?;
self.context.CopyResource(&self.staging_res, &src); self.context.CopyResource(&self.staging_res, &src);
self.context.CopyResource(&self.gpu_res, &src);
let _ = self.dup.ReleaseFrame(); let _ = self.dup.ReleaseFrame();
Ok(true) Ok(true)
} }
@@ -125,8 +167,8 @@ impl Duplication {
} }
} }
/// 把 staging 纹理映射到内存并去掉行间距 /// 在 staging 映射窗口内执行回调（软编路径）。回调参数 = (像素数据, 行距)。
pub fn copy_frame(&mut self) -> Result<Vec<u8>> { pub fn with_mapped<R>(&mut self, f: impl FnOnce(&[u8], usize) -> R) -> Result<R> {
unsafe { unsafe {
let mut mapped = D3D11_MAPPED_SUBRESOURCE::default(); let mut mapped = D3D11_MAPPED_SUBRESOURCE::default();
self.context self.context
@@ -135,16 +177,115 @@ impl Duplication {
let (w, h) = (self.width as usize, self.height as usize); let (w, h) = (self.width as usize, self.height as usize);
let pitch = mapped.RowPitch as usize; let pitch = mapped.RowPitch as usize;
let src = std::slice::from_raw_parts(mapped.pData as *const u8, pitch * h); let src = std::slice::from_raw_parts(mapped.pData as *const u8, pitch * h);
let mut out = Vec::with_capacity(w * h * 4); let r = f(src, pitch);
for row in 0..h {
out.extend_from_slice(&src[row * pitch..row * pitch + w * 4]);
}
self.context.Unmap(&self.staging_res, 0); self.context.Unmap(&self.staging_res, 0);
Ok(out) Ok(r)
} }
} }
} }
/// NVENC 硬编封装：H.264 High、CBR 码率控制、UltraLowLatency 调优、无 B 帧。
/// 输入 = 注册的 GPU 纹理（BGRA 与 DXGI 采集字节序一致，零转换）。
struct NvencEncoder {
encoder: nvenc::encoder::Encoder,
registered: RegisteredResource,
bitstream: BitStream,
frame_count: usize,
}
impl NvencEncoder {
fn new(dup: &mut Duplication, bitrate_bps: u32) -> Result<Self> {
// 打开会话（无 N 卡 / 驱动不含编码库时在此失败）
let session: Session<_> = Session::open_dx(&dup.device)
.map_err(|e| anyhow!("open_dx: {e:?}"))?;
let codecs = session
.get_encode_codecs()
.map_err(|e| anyhow!("codecs: {e:?}"))?;
anyhow::ensure!(codecs.contains(&NV_ENC_CODEC_H264_GUID), "NVENC no H264 codec");
let (session, mut config) = session
.get_encode_preset_config_ex(
NV_ENC_CODEC_H264_GUID,
NV_ENC_PRESET_P3_GUID,
NVencTuningInfo::UltraLowLatency,
)
.map_err(|e| anyhow!("preset_config: {e:?}"))?;
// CBR 桌面实时档：码率恒定，延迟可预测（对标 UU 3.2Mbps@60fps）
config.preset_cfg.rc_params.rate_control_mode = NVencParamsRcMode::CBR;
config.preset_cfg.rc_params.average_bit_rate = bitrate_bps;
// 周期 IDR 是**兜底恢复点**，不是主恢复路径——主路径是 KeyFrameReq
// （协议 v2 后已确认真正生效，见 vendor/nvenc 的 FORCEIDR 补丁）。
//
// 取值权衡：GOP 越短，丢帧后自动恢复越快，但每次 IDR 都是整帧内编码，
// 码率开销显著（1080p 桌面 IDR 通常数倍于 P 帧）。
// 60 帧 @60fps = 1s：既覆盖「KeyFrameReq 连续丢失」这种极端情况，
// 又把兜底窗口压到 1 秒以内。原先 240 帧 = 4s，在请求已可靠的前提下
// 只会让受害时长无谓地长 4 倍。
config.preset_cfg.gop_len = INTRA_PERIOD_FRAMES;
config.preset_cfg.frame_interval_p = 1; // 无 B 帧
let init = InitParams {
encode_guid: NV_ENC_CODEC_H264_GUID,
preset_guid: NV_ENC_PRESET_P3_GUID,
resolution: [dup.width, dup.height],
aspect_ratio: [dup.width, dup.height],
frame_rate: [60, 1],
tuning_info: NVencTuningInfo::UltraLowLatency,
buffer_format: NVencBufferFormat::ARGB,
encode_config: &mut config.preset_cfg,
enable_ptd: true,
max_encoder_resolution: [dup.width, dup.height],
};
let encoder = session
.init_encoder(init)
.map_err(|e| anyhow!("init_encoder: {e:?}"))?;
let registered = encoder
.register_resource_dx11(&dup.gpu_tex, NVencBufferFormat::ARGB, 0)
.map_err(|e| anyhow!("register_resource: {e:?}"))?;
let bitstream = encoder
.create_bitstream_buffer()
.map_err(|e| anyhow!("bitstream: {e:?}"))?;
Ok(Self {
encoder,
registered,
bitstream,
frame_count: 0,
})
}
/// 编码当前 GPU 纹理内容，返回 Annex B H.264 码流。
fn encode(&mut self, force_idr: bool, ts_ms: u64) -> Result<Vec<u8>> {
// 首帧强制 IDR：解码端从任意时刻接入，没有 IDR 无法解出后续 P 帧
let ty = if force_idr || self.frame_count == 0 {
NVencPicType::IDR
} else {
NVencPicType::P
};
self.encoder.encode_picture(
&self.registered,
&self.bitstream,
self.frame_count,
ts_ms,
NVencBufferFormat::ARGB,
NVencPicStruct::Frame,
ty,
None,
)
.map_err(|e| anyhow!("encode_picture: {e:?}"))?;
self.frame_count += 1;
// 同步读出码流（NVENC 单帧编码 1-3ms，阻塞等待可接受）
let lock = self
.bitstream
.try_lock(true)
.map_err(|e| anyhow!("bitstream lock: {e:?}"))?;
let out = lock.as_slice().to_vec();
drop(lock);
if out.is_empty() {
anyhow::bail!("nvenc empty bitstream");
}
Ok(out)
}
}
#[allow(dead_code)] #[allow(dead_code)]
pub fn desktop_size() -> Result<(u32, u32)> { pub fn desktop_size() -> Result<(u32, u32)> {
let d = Duplication::new()?; let d = Duplication::new()?;
@@ -153,7 +294,8 @@ pub fn desktop_size() -> Result<(u32, u32)> {
/// 采集主循环：独立线程运行；连接断开（通道关闭）时退出。 /// 采集主循环：独立线程运行；连接断开（通道关闭）时退出。
/// 通道为 watch 单槽（latest-wins）：发送即覆盖，网络慢时不产生积压。 /// 通道为 watch 单槽（latest-wins）：发送即覆盖，网络慢时不产生积压。
pub fn run(tx: Sender<Option<Vec<u8>>>, keyreq: Arc<AtomicBool>) { /// congestion：发送侧 write 超时置位 → 跳过当前帧（不编码不发送），注入率自适应降速。
pub fn run(tx: Sender<Option<Vec<u8>>>, keyreq: Arc<AtomicBool>, congestion: Arc<AtomicBool>) {
let start = Instant::now(); let start = Instant::now();
loop { loop {
let mut dup = match Duplication::new() { let mut dup = match Duplication::new() {
@@ -169,78 +311,267 @@ pub fn run(tx: Sender<Option<Vec<u8>>>, keyreq: Arc<AtomicBool>) {
dup.width, dup.height dup.width, dup.height
); );
// H.264 编码器：屏幕内容实时档 + 码率控制（对标 UU：3.2Mbps@60fps） // ---- 编码器自动选择：NVENC 硬编优先，失败落回 openh264 软编 ----
let mut encoder = match Encoder::with_api_config( enum EncoderKind {
OpenH264API::from_source(), Nvenc(Box<NvencEncoder>),
EncoderConfig::new() Soft(Box<openh264::encoder::Encoder>),
.bitrate(BitRate::from_bps(5_000_000)) }
.max_frame_rate(FrameRate::from_hz(30.0)) let mut encoder: EncoderKind = match NvencEncoder::new(&mut dup, 5_000_000) {
.rate_control_mode(RateControlMode::Bitrate) Ok(nv) => {
.usage_type(UsageType::ScreenContentRealTime) eprintln!("[capture] encoder = NVENC hardware (CBR 5Mbps@60fps, ARGB direct)");
.intra_frame_period(IntraFramePeriod::from_num_frames(300)), EncoderKind::Nvenc(Box::new(nv))
) { }
Ok(e) => e,
Err(e) => { Err(e) => {
eprintln!("[capture] encoder init failed: {e}"); eprintln!("[capture] NVENC unavailable ({e}); fallback to openh264 software");
return; match Encoder::with_api_config(
OpenH264API::from_source(),
EncoderConfig::new()
.bitrate(BitRate::from_bps(5_000_000))
.max_frame_rate(FrameRate::from_hz(60.0))
.rate_control_mode(RateControlMode::Bitrate)
.usage_type(UsageType::ScreenContentRealTime)
.intra_frame_period(IntraFramePeriod::from_num_frames(
INTRA_PERIOD_FRAMES,
))
.num_threads(4),
) {
Ok(e) => {
eprintln!("[capture] encoder = openh264 software (5Mbps@60fps, 4T)");
EncoderKind::Soft(Box::new(e))
}
Err(e) => {
eprintln!("[capture] encoder init failed: {e}");
return;
}
}
} }
}; };
let mut last_frame: Option<Vec<u8>> = None; // 最近一帧的**裸载荷**与关键帧标志：静止重发时要用它们重建帧头，
// 不能直接重发整帧（那样 seq/ts_ms 会停在上次发送时刻）
let mut last_payload: Option<Vec<u8>> = None;
let mut last_kf = false;
let mut last_sent = Instant::now() - Duration::from_secs(10); let mut last_sent = Instant::now() - Duration::from_secs(10);
let min_frame = Duration::from_millis(33); // 30fps 上限 let min_frame = Duration::from_millis(16); // 60fps 上限
let mut scratch: Vec<u8> = Vec::new(); // 软编：行距不连续时的复用缓冲
// 帧序号：每发一帧 +1（含静止桌面的重复帧），供主控端做丢帧检测
let mut seq: u64 = 0;
// 编码耗时统计（每 120 帧输出一次均值，ms）
let mut stat_frames = 0u64;
let mut stat_convert_ms = 0u64;
let mut stat_encode_ms = 0u64;
let mut stat_skipped = 0u64; // 拥塞跳帧计数
// `$kf` 由调用点显式给出：NVENC 路径知道 `force_key`，软编路径同理。
// 不要在这里猜——首帧和 KeyFrameReq 都会产生 IDR，猜错会让解码端错过重同步点。
macro_rules! emit {
($payload:expr, $kf:expr) => {{
let header = VideoFrameHeader {
width: dup.width,
height: dup.height,
ts_ms: start.elapsed().as_millis() as u64,
seq,
keyframe: $kf,
};
let payload_vec: Vec<u8> = $payload;
let frame = encode_video_frame(&header, &payload_vec);
if tx.send(Some(frame.clone())).is_err() {
return; // 会话结束
}
last_payload = Some(payload_vec);
last_kf = $kf;
last_sent = Instant::now();
seq = seq.wrapping_add(1);
stat_frames += 1;
}};
}
loop { loop {
// 会话结束守卫：发送端已 drop 接收器时立即退出。
// 只靠 `tx.send` 失败退出是不够的——桌面完全静止时这里根本不发送，
// 线程会永远持有 duplication，导致下一个会话的 DuplicateOutput 失败
// （DXGI 每个输出仅允许一个活动实例）。
if tx.receiver_count() == 0 {
eprintln!("[capture] no receiver, stop");
return;
}
let iter_start = Instant::now(); let iter_start = Instant::now();
match dup.acquire(80) { match dup.acquire(80) {
Ok(true) => match dup.copy_frame() { Ok(true) => {
Ok(raw) => { // 拥塞反馈：发送侧排队深 → 跳过本帧（NVENC 少输入一帧，参考链连续；
// 主控端请求关键帧（解码异常恢复） // 解码端收到的帧序 = 编码端实际输出序，无花屏风险）
if keyreq.swap(false, Ordering::Relaxed) { //
encoder.force_intra_frame(); // ⚠️ 必须放在消费 keyreq **之前**：先前顺序是「先 swap 再判断拥塞」，
} // 一旦本帧因拥塞被跳过，已消费的 KeyFrameReq 就随 continue 一起丢失，
let header = VideoFrameHeader { // 解码端等不到 IDR（与下方静止分支同源的缺陷）。
width: dup.width, if congestion.swap(false, Ordering::Relaxed) {
height: dup.height, stat_skipped += 1;
ts_ms: start.elapsed().as_millis() as u64,
};
let yuv = YUVBuffer::from_bgra8_source(BgraSliceU8::new(
&raw,
(dup.width as usize, dup.height as usize),
));
match encoder.encode(&yuv) {
Ok(ebs) => {
let payload = ebs.to_vec();
let frame = encode_video_frame(&header, &payload);
if tx.send(Some(frame.clone())).is_err() {
return; // 会话结束
}
last_frame = Some(frame);
last_sent = Instant::now();
}
Err(e) => {
eprintln!("[capture] encode failed: {e}");
}
}
// 帧间隔节流到 ~30fps
let el = iter_start.elapsed(); let el = iter_start.elapsed();
if el < min_frame { if el < min_frame {
std::thread::sleep(min_frame - el); std::thread::sleep(min_frame - el);
} }
continue;
} }
Err(e) => { // 主控端请求关键帧（解码异常恢复）。走到这里一定会编码，
eprintln!("[capture] frame copy failed: {e}"); // 因此可以安全地消费标志。
break; // 重建 duplication let force_key = keyreq.swap(false, Ordering::Relaxed);
} match &mut encoder {
}, EncoderKind::Nvenc(enc) => {
Ok(false) => { let enc_start = Instant::now();
// 桌面静止：低频重发最后一帧（重复 P 帧，解码端画面不变） let r = enc.encode(force_key, start.elapsed().as_millis() as u64);
if last_sent.elapsed() > Duration::from_millis(500) { stat_encode_ms += enc_start.elapsed().as_millis() as u64;
if let Some(f) = last_frame.as_ref() { match r {
if tx.send(Some(f.clone())).is_err() { Ok(payload) => {
return; let kf = h264_contains_idr(&payload);
emit!(payload, kf)
}
Err(e) => eprintln!("[capture] nvenc encode failed: {e}"),
} }
last_sent = Instant::now();
} }
EncoderKind::Soft(enc) => {
if force_key {
enc.force_intra_frame();
}
// 映射窗口内直接做 BGRA→YUV（行距连续时零拷贝，否则复用 scratch）
let conv_start = Instant::now();
let dims = (dup.width as usize, dup.height as usize);
let conv = dup.with_mapped(|src, pitch| {
if pitch == dims.0 * 4 {
YUVBuffer::from_bgra8_source(BgraSliceU8::new(src, dims))
} else {
scratch.resize(dims.0 * dims.1 * 4, 0);
for (row, dst) in
scratch.chunks_exact_mut(dims.0 * 4).enumerate()
{
dst.copy_from_slice(
&src[row * pitch..row * pitch + dims.0 * 4],
);
}
YUVBuffer::from_bgra8_source(BgraSliceU8::new(&scratch, dims))
}
});
match conv {
Ok(yuv) => {
stat_convert_ms +=
conv_start.elapsed().as_millis() as u64;
let enc_start = Instant::now();
let r = enc.encode(&yuv);
stat_encode_ms += enc_start.elapsed().as_millis() as u64;
match r {
Ok(ebs) => {
let payload = ebs.to_vec();
let kf = h264_contains_idr(&payload);
emit!(payload, kf)
}
Err(e) => {
eprintln!("[capture] encode failed: {e}");
}
}
}
Err(e) => {
eprintln!("[capture] frame convert failed: {e}");
break;
}
}
}
}
if stat_frames > 0 && stat_frames % 120 == 0 {
eprintln!(
"[capture] avg convert {}ms, encode {}ms/frame, skipped {} (last 120)",
stat_convert_ms / 120,
stat_encode_ms / 120,
stat_skipped
);
stat_convert_ms = 0;
stat_encode_ms = 0;
stat_skipped = 0;
}
// 帧间隔节流到 ~60fps
let el = iter_start.elapsed();
if el < min_frame {
std::thread::sleep(min_frame - el);
}
}
Ok(false) => {
// 桌面静止：低频重发（丢包后恢复 + 会话保活）。
//
// ⚠️ 这里曾有三个缺陷，全部围绕「KeyFrameReq 被静默丢弃」：
// 1. `swap` 写在时间闸**外面**：请求若落在距上次发送 500ms 之内，
// 标志已被消费但分支不进入，该次请求直接丢失（手册 §6.1 第 3 条）。
// 2. 软编分支只重发旧码流、不看 force_key：永远吐不出 IDR（手册 §6.1 第 2 条）。
// 3. NVENC 分支虽然把 force_key 传给了编码器，但受第 1 条限制。
//
// 现在的语义：**关键帧请求绕过 500ms 节流**——它是错误恢复路径，
// 延迟一次 IDR 的代价是解码端持续黑屏，远高于多发一帧的带宽成本。
let force_key = keyreq.swap(false, Ordering::Relaxed);
let due = force_key || last_sent.elapsed() > Duration::from_millis(500);
if due {
match &mut encoder {
EncoderKind::Nvenc(enc) => {
// 重编当前 GPU 帧（内容未变，输出极小；force_key 时为 IDR）
let r = enc.encode(force_key, start.elapsed().as_millis() as u64);
match r {
Ok(payload) => {
let kf = h264_contains_idr(&payload);
emit!(payload, kf)
}
Err(e) => {
eprintln!("[capture] nvenc idle encode failed: {e}")
}
}
}
EncoderKind::Soft(enc) => {
// 软编：静止桌面没有新像素可编，只能重发最近一帧。
// 但 force_key 时必须真的重编出 IDR —— 重发旧 P 帧对
// 已经解不动的解码端毫无帮助。
//
// 裸像素来源：staging 纹理里仍是上一帧内容（DXGI 只在
// 有新帧时覆盖），重新 Map 一次即可，不必每帧额外拷 14MB。
if force_key {
enc.force_intra_frame();
let dims = (dup.width as usize, dup.height as usize);
let conv = dup.with_mapped(|src, pitch| {
if pitch == dims.0 * 4 {
YUVBuffer::from_bgra8_source(BgraSliceU8::new(
src, dims,
))
} else {
scratch.resize(dims.0 * dims.1 * 4, 0);
for (row, dst) in
scratch.chunks_exact_mut(dims.0 * 4).enumerate()
{
dst.copy_from_slice(
&src[row * pitch..row * pitch + dims.0 * 4],
);
}
YUVBuffer::from_bgra8_source(BgraSliceU8::new(
&scratch, dims,
))
}
});
match conv {
Ok(yuv) => match enc.encode(&yuv) {
Ok(ebs) => {
let payload = ebs.to_vec();
let kf = h264_contains_idr(&payload);
emit!(payload, kf)
}
Err(e) => eprintln!(
"[capture] idle keyframe encode failed: {e}"
),
},
Err(e) => {
eprintln!("[capture] idle keyframe convert failed: {e}")
}
}
} else if let Some(p) = last_payload.as_ref() {
// 非关键帧的保活重发：复用载荷但重建帧头（seq 必须递增）
let kf = last_kf;
emit!(p.clone(), kf)
}
}
}
last_sent = Instant::now();
} else { } else {
std::thread::sleep(Duration::from_millis(8)); std::thread::sleep(Duration::from_millis(8));
} }
agent/src/input.rs
+87 -6
View File
@@ -1,11 +1,12 @@
//! 输入注入：SendInput 实现（鼠标归一化绝对坐标 + 扫描码键盘）。 //! 输入注入：SendInput 实现（鼠标归一化绝对坐标 + 扫描码键盘 + v3 文本注入）。
use common::{InputEvent, MouseButton}; use common::{InputEvent, MouseButton};
use windows::Win32::UI::Input::KeyboardAndMouse::{ use windows::Win32::UI::Input::KeyboardAndMouse::{
SendInput, INPUT, INPUT_0, INPUT_KEYBOARD, INPUT_MOUSE, KEYBDINPUT, KEYEVENTF_EXTENDEDKEY, SendInput, INPUT, INPUT_0, INPUT_KEYBOARD, INPUT_MOUSE, KEYBDINPUT, KEYEVENTF_EXTENDEDKEY,
KEYEVENTF_KEYUP, KEYEVENTF_SCANCODE, MOUSEEVENTF_ABSOLUTE, MOUSEEVENTF_MIDDLEDOWN, KEYEVENTF_KEYUP, KEYEVENTF_SCANCODE, KEYEVENTF_UNICODE, MOUSEEVENTF_ABSOLUTE,
MOUSEEVENTF_MIDDLEUP, MOUSEEVENTF_MOVE, MOUSEEVENTF_RIGHTDOWN, MOUSEEVENTF_RIGHTUP, MOUSEEVENTF_MIDDLEDOWN, MOUSEEVENTF_MIDDLEUP, MOUSEEVENTF_MOVE, MOUSEEVENTF_RIGHTDOWN,
MOUSEEVENTF_VIRTUALDESK, MOUSEEVENTF_WHEEL, MOUSEINPUT, VIRTUAL_KEY, MOUSE_EVENT_FLAGS, MOUSEEVENTF_RIGHTUP, MOUSEEVENTF_VIRTUALDESK, MOUSEEVENTF_WHEEL, MOUSEINPUT, VIRTUAL_KEY,
MOUSE_EVENT_FLAGS,
}; };
use windows::Win32::UI::WindowsAndMessaging::{ use windows::Win32::UI::WindowsAndMessaging::{
GetSystemMetrics, SM_CXVIRTUALSCREEN, SM_CYVIRTUALSCREEN, SM_XVIRTUALSCREEN, GetSystemMetrics, SM_CXVIRTUALSCREEN, SM_CYVIRTUALSCREEN, SM_XVIRTUALSCREEN,
@@ -73,12 +74,79 @@ pub fn virtual_desktop() -> (i32, i32, i32, i32) {
} }
} }
/// v3 文本注入：把一段 UTF-16 code unit 作为键盘事件送入前台应用。
///
/// 语义来源（实读，不凭记忆）：
/// [KEYBDINPUT (winuser.h)](https://learn.microsoft.com/en-us/windows/win32/api/winuser/ns-winuser-keybdinput)
/// - `KEYEVENTF_UNICODE`（0x0004）：`wVk` **必须为 0**，`wScan` 即要送出的 Unicode 字符；
/// 系统合成 `VK_PACKET` 按键事件，经 `TranslateMessage` 转为带该字符的 `WM_CHAR`。
/// - 文档原文是 "This flag can only be combined with the KEYEVENTF_KEYUP flag"。
/// 本项目**仍先发不带 KEYUP 的 down**（与现有 `send_key` 的 down/up 成对形态一致），
/// 把那句读作「不能与 SCANCODE/EXTENDEDKEY 混用」是**我的推断，文档没有明说**。
/// 若真机验收发现 down 事件被系统丢弃（表现为一个字符进两次或不进），
/// 退路是只发 `KEYEVENTF_UNICODE | KEYEVENTF_KEYUP` 单事件——改一处 `mk` 即可。
///
/// 为什么一次性批量提交（而非逐字符一对 down/up）：
/// 代理对（U+1F600 = D83D DE00）的两个 code unit 必须在输入队列里**相邻**，
/// 远端才会把它们合成成一个字符。逐字符成对（down,up,down,up）会在两个
/// down 之间插入 up 事件；Windows 对 VK_PACKET 孤立代理项的缓冲规则文档
/// 没有描述，所以这里选择让两个 down 严格相邻的形态，由真机 emoji 验收（V7）定论。
///
/// 为什么是**一次** `SendInput` 调用（而不是 down 批 + up 批两次）：
/// [Keyboard Input Overview](https://learn.microsoft.com/en-us/windows/win32/inputdev/about-keyboard-input)
/// 的 Simulating Input 一节明确——单调用注入的一批事件"系统确保不会被其他输入事件
/// 打散"（the system ensures that no other input events intermingle with the
/// simulated events），两次调用之间则可能插入真实键盘/鼠标事件，代理对就有被打散风险。
/// 于是把 N 个 down + N 个 up 拼成一个数组一次提交，顺序仍是全部 down 先于全部 up。
///
/// 上限由调用方（`common::clamp_text_units`，4096）保证；这里对空输入直接返回，
/// 因为 `SendInput(0)` 会返回 0（失败）且没有任何注入语义。
fn send_text(units: &[u16]) {
if units.is_empty() {
return;
}
let mk = |scan: u16, up: bool| INPUT {
r#type: INPUT_KEYBOARD,
Anonymous: INPUT_0 {
ki: KEYBDINPUT {
wVk: VIRTUAL_KEY(0), // KEYEVENTF_UNICODE 要求 wVk=0（官方文档）
wScan: scan,
dwFlags: if up {
KEYEVENTF_UNICODE | KEYEVENTF_KEYUP
} else {
KEYEVENTF_UNICODE
},
time: 0,
dwExtraInfo: 0,
},
},
};
// 一次提交：全部 down，紧接全部 up。
let inputs: Vec<INPUT> = units
.iter()
.map(|&u| mk(u, false))
.chain(units.iter().map(|&u| mk(u, true)))
.collect();
let n = unsafe { SendInput(&inputs, std::mem::size_of::<INPUT>() as i32) };
// 可见失败后果：SendInput 返回「实际插入的事件数」，不等于请求数即失败
// （常见原因：桌面被 Secure UI 锁定、被 BlockInput 拦截）。
if n as usize != inputs.len() {
eprintln!("[input] SendInput 文本注入不完整: {n}/{}", inputs.len());
}
}
pub fn handle(ev: InputEvent) { pub fn handle(ev: InputEvent) {
match ev { match ev {
InputEvent::MouseMove { x, y } => { InputEvent::MouseMove { x, y } => {
// 主控端已按虚拟桌面归一化，这里直接透传 0..65535 // 归一化坐标系 = **被采集的那个输出**（capture 用 EnumOutputs(0)，
// 即主屏；AuthOk 的 width/height 也是它）。
// MOUSEEVENTF_ABSOLUTE 不带 VIRTUALDESK 时，0..=65535 恰好映射到
// 主显示器——与采集空间一致。
// 曾经带 VIRTUALDESK：多显示器机器上注入空间变成整个虚拟桌面
// （比主屏宽），手机画面与指针位置系统性错位（2026-10-30 实测）。
// 单显示器时两者重合，所以桌面端从未暴露此问题。
send_mouse( send_mouse(
MOUSEEVENTF_ABSOLUTE.0 | MOUSEEVENTF_VIRTUALDESK.0 | MOUSEEVENTF_MOVE.0, MOUSEEVENTF_ABSOLUTE.0 | MOUSEEVENTF_MOVE.0,
x as i32, x as i32,
y as i32, y as i32,
0, 0,
@@ -99,5 +167,18 @@ pub fn handle(ev: InputEvent) {
} }
InputEvent::KeyDown { scancode, extended } => send_key(scancode, extended, false), InputEvent::KeyDown { scancode, extended } => send_key(scancode, extended, false),
InputEvent::KeyUp { scancode, extended } => send_key(scancode, extended, true), InputEvent::KeyUp { scancode, extended } => send_key(scancode, extended, true),
InputEvent::Text { units } => {
// 上限截断必须可见（与 agent-core::dispatch_input 同口径；Windows 端
// 走自己的 handle()，不经过 agent-core，所以这里同源补齐——R-3 教训）。
let (units, truncated) = common::clamp_text_units(units);
if truncated {
eprintln!(
"[input] 文本超过上限，已截断到 {} 个 code unit",
common::MAX_TEXT_UNITS
);
}
// 只打印长度：注入日志不得含用户输入内容（安全红线）
send_text(&units);
}
} }
} }
agent/src/main.rs
+110 -20
View File
@@ -1,6 +1,8 @@
//! 被控端 Agent：DXGI 桌面复制采集 → LZ4 压缩 → QUIC 下行视频；上行接收输入事件注入。 //! 被控端 Agent：DXGI 桌面复制采集 → H.264 编码 → QUIC 下行视频；上行接收输入事件注入。
//! P0 优化：视频帧经 watch 单槽通道（latest-wins，网络慢时只发最新帧，不积压）； //! P0 优化：视频帧经 watch 单槽通道（latest-wins，网络慢时只发最新帧，不积压）；
//! 键鼠/控制走独立 QUIC 双向流，不与视频互相阻塞。 //! 键鼠/控制走独立 QUIC 双向流，不与视频互相阻塞。
//! 会话模型：**串行多会话**——accept 循环一次只服务一个主控，会话结束后回到 accept
//! 等下一个连接（进程不退出）。改成并发多会话需要先解决 DXGI 单实例独占，暂不做。
mod capture; mod capture;
mod input; mod input;
@@ -10,6 +12,7 @@ use common::{read_control, write_control, ControlMsg};
use rustls::pki_types::PrivatePkcs8KeyDer; use rustls::pki_types::PrivatePkcs8KeyDer;
use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc; use std::sync::Arc;
use std::time::Duration;
use tokio::io::AsyncWriteExt; use tokio::io::AsyncWriteExt;
use tokio::sync::{mpsc, watch}; use tokio::sync::{mpsc, watch};
@@ -43,12 +46,21 @@ async fn main() -> Result<()> {
// 传输参数调优： // 传输参数调优：
// - 收发窗口放大，避免高 RTT 下吞吐被 窗口/RTT 死锁（默认 512KB 流窗口） // - 收发窗口放大，避免高 RTT 下吞吐被 窗口/RTT 死锁（默认 512KB 流窗口）
// - 发送缓冲上限 2MB：QUIC 内部排队有界（约 1 帧多），配合上游 latest-wins 丢帧 // - 发送缓冲上限 64KB：QUIC 内部排队硬上限（实测 WiFi ~3Mbps 吞吐 → E2E ≤ ~170ms；
// 把端到端排队延迟压在 ~0.4s 内 // 突发吞吐 = 64KB/7ms ≈ 72Mbps，不限制带宽）。注意此窗口为连接级，流 2 输入/Ping
// 数据量极小不受影响。配合下方拥塞反馈跳帧，把排队压在数帧以内
let mut transport = quinn::TransportConfig::default(); let mut transport = quinn::TransportConfig::default();
transport.receive_window(quinn::VarInt::from_u32(16 * 1024 * 1024)); transport.receive_window(quinn::VarInt::from_u32(16 * 1024 * 1024));
transport.stream_receive_window(quinn::VarInt::from_u32(8 * 1024 * 1024)); transport.stream_receive_window(quinn::VarInt::from_u32(8 * 1024 * 1024));
transport.send_window(2 * 1024 * 1024); transport.send_window(64 * 1024);
// 空闲超时：主控端被强杀（非正常 close）时收不到它的 Ping，连接会一直占住
// 串行 accept，下一个连接要等几十秒才能建立。10s = 20 个 Ping 周期，
// 正常会话（Ping 每 500ms）永不触发，只用来回收死连接。
transport.max_idle_timeout(Some(
Duration::from_secs(10)
.try_into()
.expect("10s fits in VarInt"),
));
let mut server_config = quinn::ServerConfig::with_single_cert(vec![cert], key.into())?; let mut server_config = quinn::ServerConfig::with_single_cert(vec![cert], key.into())?;
server_config.transport_config(Arc::new(transport)); server_config.transport_config(Arc::new(transport));
@@ -58,26 +70,74 @@ async fn main() -> Result<()> {
)?; )?;
println!("[agent] listening 0.0.0.0:{port}, verify code = {code}"); println!("[agent] listening 0.0.0.0:{port}, verify code = {code}");
let conn = endpoint // 串行多会话：一次只服务一个主控，结束后回到 accept。
.accept() // 不做并发多会话——DXGI Desktop Duplication 每个输出仅允许一个活动实例，
.await // 第二个 DuplicateOutput 会让第一个收到 ACCESS_LOST。
.ok_or_else(|| anyhow::anyhow!("endpoint closed"))? // 会话失败（含认证失败）只记日志，不能让进程退出——否则每次连接都要重启被控端。
.await?; loop {
println!("[agent] peer connected: {}", conn.remote_address()); let Some(incoming) = endpoint.accept().await else {
println!("[agent] endpoint closed, exit");
return Ok(());
};
match incoming.await {
Ok(conn) => {
println!("[agent] peer connected: {}", conn.remote_address());
if let Err(e) = serve_session(conn, &code).await {
println!("[agent] session ended: {e:#}");
}
println!("[agent] waiting for next connection");
}
Err(e) => println!("[agent] handshake failed: {e}"),
}
}
}
/// 单次会话：认证 → 采集/发送/输入 → 会话结束。
/// 返回 Err 表示会话异常结束（含认证失败），调用方继续 accept。
async fn serve_session(conn: quinn::Connection, code: &str) -> Result<()> {
// 流 1：认证 + 视频下行 // 流 1：认证 + 视频下行
let (mut send, mut recv) = conn.accept_bi().await?; let (mut send, mut recv) = conn.accept_bi().await?;
// 认证 // 认证：先校验版本，再校验验证码。
// 顺序刻意如此——版本不一致时字段布局可能不同，验证码的解析结果不可信。
let first = read_control(&mut recv).await?; let first = read_control(&mut recv).await?;
let code_ok = matches!(&first, ControlMsg::Auth { code: c } if *c == code); let auth_err: Option<String> = match &first {
if !code_ok { ControlMsg::Auth {
write_control(&mut send, &ControlMsg::AuthFail("bad code".into())).await?; code: c,
proto_version,
} => {
if *proto_version != common::PROTO_VERSION {
Some(format!(
"protocol version mismatch: client={proto_version}, server={}",
common::PROTO_VERSION
))
} else if *c != code {
Some("bad code".into())
} else {
None
}
}
other => Some(format!("expected Auth, got {other:?}")),
};
if let Some(msg) = auth_err {
write_control(&mut send, &ControlMsg::AuthFail(msg.clone())).await?;
send.finish()?; send.finish()?;
anyhow::bail!("auth failed"); // 必须等对端读到 AuthFail 再关连接：直接返回会让进程带走 QUIC 缓冲区里
// 尚未发出的数据，客户端只会看到 "connection lost" 而不是 AuthFail。
// （agent-core 已有同款修复，此处同源补齐——手册 §6.1 第 1 条）
let _ = tokio::time::timeout(Duration::from_secs(5), conn.closed()).await;
anyhow::bail!(msg);
} }
let (w, h) = capture::desktop_size()?; let (w, h) = capture::desktop_size()?;
write_control(&mut send, &ControlMsg::AuthOk { width: w, height: h }).await?; write_control(
&mut send,
&ControlMsg::AuthOk {
width: w,
height: h,
proto_version: common::PROTO_VERSION,
},
)
.await?;
println!("[agent] auth ok, desktop {w}x{h}"); println!("[agent] auth ok, desktop {w}x{h}");
// 流 2：键鼠输入 + 控制上行（与视频分离，避免被视频积压阻塞） // 流 2：键鼠输入 + 控制上行（与视频分离，避免被视频积压阻塞）
@@ -87,14 +147,26 @@ async fn main() -> Result<()> {
let (tx, mut frames_rx) = watch::channel::<Option<Vec<u8>>>(None); let (tx, mut frames_rx) = watch::channel::<Option<Vec<u8>>>(None);
let keyreq = Arc::new(AtomicBool::new(false)); let keyreq = Arc::new(AtomicBool::new(false));
let keyreq_capture = keyreq.clone(); let keyreq_capture = keyreq.clone();
std::thread::spawn(move || capture::run(tx, keyreq_capture)); // 拥塞反馈标志：发送侧 write 超过单帧预算(20ms) → 置位，采集线程跳帧降注入率
let congestion = Arc::new(AtomicBool::new(false));
let congestion_capture = congestion.clone();
// 采集线程退出信号：会话结束必须等它真正退出才能 accept 下一个连接——
// DXGI Desktop Duplication 每个输出只允许一个活动实例，旧线程还持有 duplication
// 时新线程的 DuplicateOutput 会让旧实例收到 ACCESS_LOST（表现为下一个会话黑屏）。
let (done_tx, done_rx) = std::sync::mpsc::channel::<()>();
std::thread::spawn(move || {
capture::run(tx, keyreq_capture, congestion_capture);
let _ = done_tx.send(());
});
// 上行控制消息（Pong）经通道交给主循环写流 2，避免多任务争抢 // 上行控制消息（Pong）经通道交给主循环写流 2，避免多任务争抢
let (ctl_tx, mut ctl_rx) = mpsc::unbounded_channel::<ControlMsg>(); let (ctl_tx, mut ctl_rx) = mpsc::unbounded_channel::<ControlMsg>();
let t0 = std::time::Instant::now(); let t0 = std::time::Instant::now();
// 读循环（流 2）：输入事件 + Ping // 读循环（流 2）：输入事件 + Ping；Pong 附带视频下行方向的 QUIC 丢包统计
// （conn.stats().path 统计的是本端发送方向，agent 的发送方向即视频下行）
let keyreq_reader = keyreq.clone(); let keyreq_reader = keyreq.clone();
let conn_stats = conn.clone();
let reader = tokio::spawn(async move { let reader = tokio::spawn(async move {
loop { loop {
match read_control(&mut irecv).await { match read_control(&mut irecv).await {
@@ -104,7 +176,13 @@ async fn main() -> Result<()> {
} }
Ok(ControlMsg::Ping(t)) => { Ok(ControlMsg::Ping(t)) => {
let agent_ts = t0.elapsed().as_millis() as u64; let agent_ts = t0.elapsed().as_millis() as u64;
let _ = ctl_tx.send(ControlMsg::Pong { t, agent_ts }); let p = conn_stats.stats().path;
let _ = ctl_tx.send(ControlMsg::Pong {
t,
agent_ts,
lost: p.lost_packets,
sent: p.sent_packets,
});
} }
Ok(_) => {} Ok(_) => {}
Err(_) => break, Err(_) => break,
@@ -119,7 +197,14 @@ async fn main() -> Result<()> {
break; // 采集线程退出 break; // 采集线程退出
} }
if let Some(data) = frames_rx.borrow().clone() { if let Some(data) = frames_rx.borrow().clone() {
if send.write_all(&data).await.is_err() { let wt = std::time::Instant::now();
let ok = send.write_all(&data).await.is_ok();
// write 耗时 > 单帧预算(16.7ms 的 1.2 倍) → QUIC 窗口排队深，
// 通知采集线程跳帧，把注入率降到网络实际吞吐水平
if wt.elapsed() > std::time::Duration::from_millis(20) {
congestion.store(true, Ordering::Relaxed);
}
if !ok {
break; break;
} }
} }
@@ -137,6 +222,11 @@ async fn main() -> Result<()> {
} }
} }
reader.abort(); reader.abort();
// 关掉流并丢弃接收端，让采集线程下一次 send 失败而退出
let _ = send.finish();
drop(frames_rx);
// 等采集线程真正退出后再返回，调用方才会 accept 下一个连接（DXGI 独占，见上）
let _ = tokio::task::spawn_blocking(move || done_rx.recv_timeout(Duration::from_secs(2))).await;
println!("[agent] session closed"); println!("[agent] session closed");
Ok(()) Ok(())
} }
android-ffi/Cargo.toml
+20
View File
@@ -0,0 +1,20 @@
[package]
name = "android-ffi"
version = "0.1.0"
edition = "2021"
# 安卓主控的 JNI 薄层：把 viewer-core 的异步 Session 包成 Kotlin 可同步调用的句柄。
# 2026-10-29 裁决变更：原方案 A2 选 UniFFI，但其实训运行时不在 Maven Central /
# Mozilla maven（查证 404），需自编译发布，成本超过收益。改为裸 JNI——阶段 0
# （D:\working\android\phase0）已验证该链路真机可跑。接口刻意保持窄（连接/取帧/
# 发输入/销毁），将来若引入 UniFFI 也只是替换本层。
[lib]
name = "androidffi"
crate-type = ["cdylib"]
[dependencies]
viewer-core = { path = "../viewer-core" }
common = { path = "../common" }
anyhow.workspace = true
tokio.workspace = true
jni = "0.21"
android-ffi/src/lib.rs
+355
View File
@@ -0,0 +1,355 @@
//! android-ffi：viewer-core 的 JNI 薄层（阶段 1）。
//!
//! 线程模型（决定接口形态的原因）：
//! - Kotlin 解码线程调用 `nativeNextFrame` 同步取帧（内部 block_on，一次一帧）；
//! - Kotlin UI 线程调用 `nativeSend*` 发输入——tokio unbounded channel 入队不阻塞；
//! - 共用一个 current-thread runtime：`block_on(next_frame)` 期间 runtime 仍在驱动
//! QUIC 收发与 Ping；输入入队只是 channel send，不需要 runtime 参与。
//!
//! **锁纪律**：`nativeNextFrame` 在等帧期间**不得持有**会话锁——桌面静止时帧间隔
//! 可达 500ms，持锁会让 UI 线程的输入调用同步卡住。做法：短暂持锁克隆
//! `Arc<Session>` 与 `&'static Runtime` 引用后立即释放，再 block_on。
//! 代价：block_on 期间 destroy 会 use-after-free——阶段 1 由 Kotlin 侧
//! 「先停解码线程、后 destroy」的顺序纪律保证（登记在方案风险清单）。
//!
//! 句柄约定：`nativeConnect` 返回 Box 裸指针（永不为 0）作为 jlong；
//! `nativeDestroy` 释放。句柄释放后不得再使用，Kotlin 侧用置 0 保护。
//!
//! 帧返回约定：byte[]，布局 = [1 标志][8 seq LE][8 ts LE][4 w LE][4 h LE][payload]。
//! 定长头 25 字节；Kotlin 读 buf[0] 判断 keyframe（标志位复用为：0=P 帧, 1=IDR 帧，
//! 会话结束则整个返回 null）。
use common::InputEvent;
use jni::objects::{JCharArray, JClass, JObject, JString};
use jni::sys::{jbyteArray, jint, jlong, jstring};
use jni::JNIEnv;
use std::sync::{Arc, Mutex};
use viewer_core::{ConnectConfig, Session};
/// 帧头长度：keyframe 标志(1) + seq(8) + ts(8) + w(4) + h(4)
const FRAME_HEADER_LEN: usize = 25;
struct SessionBox {
session: Arc<Session>,
runtime: tokio::runtime::Runtime,
}
/// 连接被控端。成功返回句柄（>0），失败返回 0；错误消息经 `nativeLastError()` 取。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeConnect<'local>(
mut env: JNIEnv<'local>,
_class: JClass<'local>,
host: JString<'local>,
port: jint,
code: JString<'local>,
) -> jlong {
let host_s = match env.get_string(&host) {
Ok(s) => s.to_string_lossy().into_owned(),
Err(_) => {
set_last_error("host 参数不是合法字符串");
return 0;
}
};
let code_s = match env.get_string(&code) {
Ok(s) => s.to_string_lossy().into_owned(),
Err(_) => {
set_last_error("code 参数不是合法字符串");
return 0;
}
};
let built = std::panic::catch_unwind(|| {
let runtime = match tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
{
Ok(r) => r,
Err(e) => return Err(format!("runtime init: {e}")),
};
let cfg = ConnectConfig::direct(host_s, port as u16, code_s);
match runtime.block_on(Session::connect(cfg)) {
Ok(session) => Ok(SessionBox {
session: Arc::new(session),
runtime,
}),
Err(e) => Err(format!("{e:#}")),
}
});
match built {
Ok(Ok(boxed)) => {
set_last_error("");
Box::into_raw(Box::new(Mutex::new(boxed))) as jlong
}
Ok(Err(msg)) => {
set_last_error(&msg);
0
}
Err(_) => {
set_last_error("panic in nativeConnect");
0
}
}
}
fn set_last_error(msg: &str) {
let mut slot = LAST_ERROR.lock().unwrap();
if msg.is_empty() {
*slot = None;
} else {
*slot = Some(msg.to_string());
}
}
static LAST_ERROR: Mutex<Option<String>> = Mutex::new(None);
/// 取最近一次 nativeConnect 的错误消息（无错误返回 null）。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeLastError<'local>(
env: JNIEnv<'local>,
_class: JClass<'local>,
) -> jstring {
let msg = LAST_ERROR.lock().unwrap().take();
match msg {
Some(m) => env
.new_string(m)
.map(|s| s.into_raw())
.unwrap_or(std::ptr::null_mut()),
None => std::ptr::null_mut(),
}
}
/// 取下一帧（阻塞直到有帧或会话结束）。返回 null = 会话结束。
/// 仅允许解码线程调用；destroy 前必须先停止本线程（见文件头锁纪律）。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeNextFrame<'local>(
env: JNIEnv<'local>,
_class: JClass<'local>,
handle: jlong,
) -> jbyteArray {
// 短暂持锁：克隆 Arc 与拿 runtime 引用，随后立即释放——等帧不能挡输入。
let (session, runtime) = {
let Some(boxed) = handle_to_box(handle) else {
return std::ptr::null_mut();
};
let guard = lock_box(boxed);
(guard.session.clone(), &guard.runtime as *const tokio::runtime::Runtime)
};
// SAFETY: 见文件头「锁纪律」。Kotlin 保证解码线程存活期间不 destroy。
let rt = unsafe { &*runtime };
match rt.block_on(session.next_frame()) {
None => std::ptr::null_mut(),
Some(f) => {
let mut buf = vec![0u8; FRAME_HEADER_LEN + f.payload.len()];
buf[0] = f.keyframe as u8;
buf[1..9].copy_from_slice(&f.seq.to_le_bytes());
buf[9..17].copy_from_slice(&f.ts_ms.to_le_bytes());
buf[17..21].copy_from_slice(&f.width.to_le_bytes());
buf[21..25].copy_from_slice(&f.height.to_le_bytes());
buf[FRAME_HEADER_LEN..].copy_from_slice(&f.payload);
match env.byte_array_from_slice(&buf) {
Ok(a) => a.into_raw(),
Err(_) => std::ptr::null_mut(),
}
}
}
}
/// 会话统计打包：高 32 位 rtt_ms，低 32 位 frames_in 低 32 位。-1 = 句柄无效。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeStats(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) -> jlong {
let Some(boxed) = handle_to_box(handle) else {
return -1;
};
let guard = lock_box(boxed);
let st = guard.session.stats();
// 打包进 i64：高 32 位 rtt_ms、低 32 位 frames_in，Kotlin 侧移位还原
(((st.rtt_ms & 0xFFFF_FFFF) << 32) | (st.frames_in & 0xFFFF_FFFF)) as jlong
}
/// 被控端桌面尺寸，打包成 i64：高 32 位宽、低 32 位高。0 = 句柄无效。
/// Kotlin 侧必须在建解码器之前调用，用返回值定 SurfaceView 宽高比。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeDesktopSize(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) -> jlong {
let Some(boxed) = handle_to_box(handle) else {
return 0;
};
let guard = lock_box(boxed);
let (w, h) = guard.session.desktop_size();
(((w as u64 & 0xFFFF_FFFF) << 32) | (h as u64 & 0xFFFF_FFFF)) as jlong
}
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeSendMouseMove(
_env: JNIEnv,
_class: JClass,
handle: jlong,
x: jint,
y: jint,
) {
with_session(handle, |s| {
s.send_input(InputEvent::MouseMove {
x: x.clamp(0, 65535) as u16,
y: y.clamp(0, 65535) as u16,
})
});
}
/// button: 0=左 1=右 其他=中；down: 1=按下 0=抬起
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeSendMouseButton(
_env: JNIEnv,
_class: JClass,
handle: jlong,
button: jint,
down: jint,
) {
use common::MouseButton;
let b = match button {
0 => MouseButton::Left,
1 => MouseButton::Right,
_ => MouseButton::Middle,
};
with_session(handle, |s| {
s.send_input(if down != 0 {
InputEvent::MouseDown(b)
} else {
InputEvent::MouseUp(b)
})
});
}
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeSendWheel(
_env: JNIEnv,
_class: JClass,
handle: jlong,
dy: jint,
) {
with_session(handle, |s| s.send_input(InputEvent::Wheel { dy }));
}
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeSendKey(
_env: JNIEnv,
_class: JClass,
handle: jlong,
scancode: jint,
extended: jint,
down: jint,
) {
with_session(handle, |s| {
s.send_input(if down != 0 {
InputEvent::KeyDown {
scancode: scancode as u16,
extended: extended != 0,
}
} else {
InputEvent::KeyUp {
scancode: scancode as u16,
extended: extended != 0,
}
})
});
}
/// 协议 v3：文本上行。`units` 是 Kotlin 侧的 `CharArray`（UTF-16 code unit）。
///
/// 为什么不用 `JString`：jni 0.21 的 `get_string` 走 Modified-UTF-8 → Rust `String`，
/// 落单代理项（IME 提交边界常见）会在这次往返里丢失。`char[]` 经
/// `GetCharArrayRegion` 按 u16 直接拷贝，零转换（`jchar = u16`，见 jni-sys-0.3.1/lib.rs:15）。
///
/// 截断与空输入的判断在 `viewer_core::Session::send_text` 里做，与被控端同一口径。
/// 失败必须有可见后果：JNI 取数失败走 eprintln（Android 上落 stderr→logcat），
/// 不静默返回成功——用户对着远端打字却没字，最坏的就是「看起来发送了」。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeSendText<'local>(
// 不需要 `mut`：jni 0.21.1 的 `get_array_length` / `get_char_array_region`
// 都是 `&self`（jnienv.rs:1652 / :1887），与 `new_string` 之类的 `&mut self` 不同。
env: JNIEnv<'local>,
_class: JClass<'local>,
handle: jlong,
units: JObject<'local>,
) {
if units.as_raw().is_null() {
// Kotlin 侧不会传 null，但 JNI 入口的对端数据一律不信任（全局安全规则）
return;
}
let arr = JCharArray::from(units);
let len = match env.get_array_length(&arr) {
Ok(n) if n > 0 => n as usize,
Ok(_) => return,
Err(e) => {
eprintln!("[android-ffi] get_array_length failed: {e}");
return;
}
};
let mut buf = vec![0u16; len];
if let Err(e) = env.get_char_array_region(&arr, 0, &mut buf) {
eprintln!("[android-ffi] get_char_array_region failed: {e}");
return;
}
with_session(handle, |s| s.send_text(&buf));
}
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeRequestKeyframe(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
with_session(handle, |s| s.request_keyframe());
}
/// 释放会话。Kotlin 侧调用后必须把句柄置 0，且先停解码线程。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeDestroy(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
if handle == 0 {
return;
}
let real = handle as *mut Mutex<SessionBox>;
unsafe {
let _ = Box::from_raw(real);
}
}
fn handle_to_box(handle: jlong) -> Option<&'static Mutex<SessionBox>> {
if handle == 0 {
return None;
}
let real = handle as *mut Mutex<SessionBox>;
// 悬垂指针无法在 Rust 侧检测：阶段 1 由 Kotlin「destroy 后置 0 + 先停线程」
// 的顺序纪律保证。违反即 UB，登记在方案风险清单（阶段 2 可加代际表消除）。
// NonNull::new 是安全函数；解引用才是 unsafe 点
std::ptr::NonNull::new(real).map(|p| unsafe { p.as_ref() })
}
/// 锁中毒容忍：SessionBox 内部无跨线程共享可变状态（Session 各方法内部自带
/// 并发原语），中毒后继续用是安全的。
fn lock_box(b: &Mutex<SessionBox>) -> std::sync::MutexGuard<'_, SessionBox> {
match b.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
}
}
fn with_session(handle: jlong, f: impl FnOnce(&Session)) {
let Some(boxed) = handle_to_box(handle) else {
return;
};
let guard = lock_box(boxed);
f(&guard.session);
}
apps/android-viewer/.gitignore
+15
View File
@@ -0,0 +1,15 @@
# Gradle / Android 构建产物
.gradle/
build/
app/build/
local.properties
# Rust 交叉编译产物（由 build-android.ps1 生成到 app/libs）
app/libs/
# 验证截图
evidence/
# IDE
.idea/
*.iml
apps/android-viewer/app/build.gradle.kts
+56
View File
@@ -0,0 +1,56 @@
plugins {
id("com.android.application")
id("org.jetbrains.kotlin.android")
}
android {
namespace = "com.verify.remote"
compileSdk = 34
defaultConfig {
applicationId = "com.verify.remote"
minSdk = 26
targetSdk = 34
versionCode = 1
versionName = "0.1-phase1"
// 阶段 1 只测真机 arm64；x86_64 留给模拟器回归
ndk {
abiFilters += listOf("arm64-v8a")
}
}
buildTypes {
release {
isMinifyEnabled = false
}
}
compileOptions {
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
}
kotlinOptions {
jvmTarget = "11"
}
// .so 由外部脚本（build-android.ps1）预先拷到 app/libs/arm64-v8a/，
// 不让 Gradle 调 cargo——构建职责分离：Rust 侧失败与 Gradle 侧失败可立刻区分。
sourceSets {
getByName("main") {
jniLibs.srcDirs("libs")
}
}
packaging {
jniLibs {
// AGP 默认已按 useLegacyPackaging 处理；保持默认压缩+extractNativeLibs=false
// 由 AGP 保证 STORED+对齐，手动链的坑（lessons 记录）在 Gradle 下不存在
useLegacyPackaging = false
}
}
}
dependencies {
// 阶段 1 刻意零第三方 UI 依赖：纯 View 代码布局，减少变量。
// AndroidX 在阶段 2 做正式 UI 时再引入。
}
apps/android-viewer/app/src/main/AndroidManifest.xml
+22
View File
@@ -0,0 +1,22 @@
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
<!-- 阶段 1：局域网连 agent.exe 需要网络 -->
<uses-permission android:name="android.permission.INTERNET" />
<application
android:label="Remote Viewer"
android:hasCode="true">
<activity
android:name=".MainActivity"
android:exported="true"
android:keepScreenOn="true"
android:windowSoftInputMode="adjustPan"
android:configChanges="orientation|screenSize|keyboardHidden">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
</manifest>
apps/android-viewer/app/src/main/kotlin/com/verify/remote/H264Decoder.kt
+159
View File
@@ -0,0 +1,159 @@
package com.verify.remote
import android.media.MediaCodec
import android.media.MediaFormat
import android.util.Log
import android.view.Surface
import java.nio.ByteBuffer
/**
* MediaCodec H.264 解码器封装（阶段 1：同步模式 + Surface 直出）。
*
* 两条实测教训直接写进实现（来源见个人层 lessons / baseline）：
* 1. Exynos 等设备带 requires-allocate-on-input/output-ports Quirk，
* 必须用 getInputBuffer(i)/getOutputBuffer(i)，不用已废弃的数组版 API。
* 2. 参考帧链断裂后 P 帧必然解不出：解码线程维护 awaitingIdr 状态，
* 只喂 IDR 之后的帧；错误后 flush + requestKeyframe 并等 IDR（R-3）。
*
* MediaFormat 不设 csd：码流每个 IDR 自带 SPS/PPS（dump_stream 实测
* "SPS PPS IDR" 结构），解码器从码流内联参数集自举。
*/
class H264Decoder(private val surface: Surface) {
@Volatile
var stopped = false
private set
private var codec: MediaCodec? = null
private val bufferInfo = MediaCodec.BufferInfo()
/** 等待 IDR 标志：true 时 P 帧一律不喂解码器 */
private var awaitingIdr = true
private var configuredW = 0
private var configuredH = 0
/** 解码一帧。返回 false = 该帧被丢弃（等 IDR 中或缓冲满），不视为错误。 */
fun queue(frame: RustBridge.Frame): Boolean {
// 等待 IDR：参考链未建立/已断裂时，P 帧喂进去只会产生错误
if (awaitingIdr && !frame.keyframe) return false
if (frame.keyframe) awaitingIdr = false
val c = ensureCodec(frame) ?: return false
return try {
val idx = c.dequeueInputBuffer(INPUT_TIMEOUT_US)
if (idx < 0) return false // 输入满：阶段 1 直接丢，由消费方决定
val inBuf: ByteBuffer = c.getInputBuffer(idx) ?: return false
inBuf.clear()
inBuf.put(frame.payload)
c.queueInputBuffer(idx, 0, frame.payload.size, frame.tsMs * 1000, 0)
drainOnce(c)
true
} catch (e: MediaCodec.CodecException) {
Log.e(TAG, "codec error isRecoverable=${e.isRecoverable}", e)
invalidate()
false
} catch (e: IllegalStateException) {
Log.e(TAG, "codec illegal state", e)
invalidate()
false
}
}
/** 解码错误/跳帧后调用：flush 复位解码器（三级恢复第一级），
* 置等待标志；调用方负责 requestKeyframe。 */
fun invalidate() {
awaitingIdr = true
try {
codec?.flush()
} catch (e: Exception) {
// flush 失败说明解码器不可恢复：销毁，等下一个 IDR 重建
Log.w(TAG, "flush failed, releasing codec", e)
releaseCodec()
}
}
/** 确保解码器就绪。只在 keyframe 到来时才配置/重建（首配与重配同路径）。 */
private fun ensureCodec(frame: RustBridge.Frame): MediaCodec? {
val c = codec
if (c != null && configuredW == frame.width && configuredH == frame.height) {
return c
}
if (c != null) {
Log.i(TAG, "size changed ${configuredW}x$configuredH -> ${frame.width}x${frame.height}, reconfigure")
releaseCodec()
}
if (codec == null && frame.keyframe) {
val name = pickDecoderName()
val nc = if (name.isEmpty()) MediaCodec.createDecoderByType(MIME)
else MediaCodec.createByCodecName(name)
val fmt = MediaFormat.createVideoFormat(MIME, frame.width, frame.height)
nc.configure(fmt, surface, null, 0)
nc.start()
codec = nc
configuredW = frame.width
configuredH = frame.height
Log.i(TAG, "decoder configured ${frame.width}x${frame.height} name=${nc.name}")
}
return codec
}
/** 优先硬解；查不到就返回空串让系统自选（避免选中软件解码器）。 */
private fun pickDecoderName(): String {
return try {
android.media.MediaCodecList(android.media.MediaCodecList.REGULAR_CODECS)
.codecInfos
.firstOrNull { info ->
!info.isEncoder &&
info.supportedTypes.any { t -> t.equals(MIME, true) } &&
!info.name.startsWith("OMX.google.") &&
!info.name.startsWith("c2.android.")
}?.name ?: ""
} catch (e: Exception) {
""
}
}
private fun drainOnce(c: MediaCodec) {
while (!stopped) {
val idx = c.dequeueOutputBuffer(bufferInfo, 0)
when {
idx == MediaCodec.INFO_TRY_AGAIN_LATER -> return
idx == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED ->
Log.i(TAG, "output format: ${c.outputFormat}")
idx >= 0 -> {
// Surface 直出：渲染由解码器完成，不取图像数据
c.releaseOutputBuffer(idx, true)
}
else -> return
}
}
}
fun release() {
stopped = true
releaseCodec()
}
private fun releaseCodec() {
try {
codec?.stop()
} catch (e: Exception) {
Log.w(TAG, "stop ignored", e)
}
try {
codec?.release()
} catch (e: Exception) {
Log.w(TAG, "release ignored", e)
}
codec = null
configuredW = 0
configuredH = 0
}
companion object {
private const val TAG = "H264Decoder"
private const val MIME = "video/avc"
private const val INPUT_TIMEOUT_US = 4_000L
}
}
apps/android-viewer/app/src/main/kotlin/com/verify/remote/MainActivity.kt
+541
View File
@@ -0,0 +1,541 @@
package com.verify.remote
import android.app.Activity
import android.content.Context
import android.os.Bundle
import android.text.InputType
import android.util.Log
import android.view.Gravity
import android.view.KeyEvent
import android.view.SurfaceHolder
import android.view.View
import android.view.inputmethod.InputMethodManager
import android.widget.Button
import android.widget.EditText
import android.widget.FrameLayout
import android.widget.LinearLayout
import android.widget.LinearLayout.LayoutParams
import android.widget.TextView
import kotlin.concurrent.thread
/**
* 安卓主控端（阶段 2b）：正式连接界面 + 远控画面 + 触摸操作 + 软键盘文本输入。
*
* 两态：
* - 连接态：host / port / code 表单 + 状态文本（阶段 1/2a 只能靠 `am start --es host`
* 传参，本界面对它补位；intent extras 仍保留且优先级更高，构建脚本继续可用）。
* - 远控态：letterbox 画面 + 底部状态/操作条（帧数、RTT、诊断、键盘、断开）。
*
* 线程模型（与 android-ffi 的锁纪律对应）：
* - UI 线程：触摸/按键/IME 提交 → nativeSend*（入队不阻塞）；连接/断开
* - 解码线程：nativeNextFrame 阻塞取帧 → H264Decoder.queue
* - 断开顺序：**先置 stopDecoding 并 join 解码线程，再 destroy**
* （Rust 侧明确：解码线程存活期间 destroy 是 UB）
*
* 持久化：只存 host/port（连接成功后落盘）；**验证码不落盘**（2026-10-30 用户裁决）。
*/
class MainActivity : Activity(), RemoteSurfaceView.Listener {
private lateinit var surfaceView: RemoteSurfaceView
private lateinit var statusView: TextView
private lateinit var connectPanel: LinearLayout
private lateinit var remotePanel: LinearLayout
private lateinit var hostInput: EditText
private lateinit var portInput: EditText
private lateinit var codeInput: EditText
private lateinit var connectButton: Button
private lateinit var connectStatus: TextView
@Volatile
private var handle = 0L
@Volatile
private var stopDecoding = false
private var decodeThread: Thread? = null
/** 连接中标志：防重复点击起出第二个会话（句柄未回填前 handle 仍是 0） */
@Volatile
private var connecting = false
@Volatile
private var remoteW = 0
@Volatile
private var remoteH = 0
/** 最近一次输入事件（触摸/IME）诊断：logcat 被限流，状态栏是唯一可靠通道 */
@Volatile
private var lastTouchDbg = "-"
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
Log.i(TAG, "onCreate start")
surfaceView = RemoteSurfaceView(this).apply {
listener = this@MainActivity
onDebug = { msg -> lastTouchDbg = msg; refreshStatus() }
holder.addCallback(object : SurfaceHolder.Callback {
override fun surfaceCreated(holder: SurfaceHolder) {}
override fun surfaceChanged(h: SurfaceHolder, f: Int, w: Int, ht: Int) {}
override fun surfaceDestroyed(holder: SurfaceHolder) {
// Surface 没了解码器输出会失效：断开，回到连接界面重连
disconnect()
}
})
}
val root = LinearLayout(this).apply {
orientation = LinearLayout.VERTICAL
setBackgroundColor(0xFF000000.toInt())
}
connectPanel = buildConnectPanel()
remotePanel = buildRemotePanel()
val lp = LinearLayout.LayoutParams(LinearLayout.LayoutParams.MATCH_PARENT, 0, 1f)
root.addView(connectPanel, lp)
root.addView(remotePanel, LinearLayout.LayoutParams(LinearLayout.LayoutParams.MATCH_PARENT, 0, 1f))
setContentView(root)
// 键盘可见性只能靠窗口布局变化被动感知（没有公开 API 能直接查询）。
// 每次根视图布局完成都重算一次，覆盖「用户点键盘按钮」「IME 自己收起」两条路径。
root.viewTreeObserver.addOnGlobalLayoutListener { updateKeyboardVisibility() }
showConnectState("未连接")
// intent 入口优先级高于表单默认值：build-android.ps1 与调试脚本依赖它
val hostExtra = intent?.getStringExtra("host")
if (!hostExtra.isNullOrEmpty()) {
hostInput.setText(hostExtra)
portInput.setText(intent.getIntExtra("port", 47800).toString())
intent.getStringExtra("code")?.let { codeInput.setText(it) }
startConnect()
}
}
// ---- 连接态 UI ----
private fun buildConnectPanel(): LinearLayout {
val prefs = getSharedPreferences(PREFS, Context.MODE_PRIVATE)
// 只回填 host/port（上次**成功连接**过的值）；验证码每次手输
hostInput = editText(
"被控端 IP",
InputType.TYPE_CLASS_TEXT,
prefs.getString("host", "").orEmpty(),
)
portInput = editText(
"端口",
InputType.TYPE_CLASS_NUMBER,
prefs.getInt("port", 47800).toString(),
)
// 验证码刻意不从 prefs 读（不落盘）
codeInput = editText("验证码", InputType.TYPE_CLASS_TEXT, "")
connectButton = Button(this).apply {
text = "连接"
setOnClickListener { startConnect() }
}
connectStatus = TextView(this).apply {
textSize = 13f
setTextColor(0xFFBBBBBB.toInt())
setPadding(0, dp(8), 0, 0)
}
val title = TextView(this).apply {
text = "连接被控端"
textSize = 18f
setTextColor(0xFFFFFFFF.toInt())
setPadding(0, dp(16), 0, dp(8))
}
return LinearLayout(this).apply {
orientation = LinearLayout.VERTICAL
setPadding(dp(20), dp(24), dp(20), dp(24))
addView(title)
addView(hostInput)
addView(portInput)
addView(codeInput)
addView(connectButton)
addView(connectStatus)
}
}
private fun editText(hint: String, type: Int, initial: String) = EditText(this).apply {
inputType = type
this.hint = hint
setText(initial)
textSize = 16f
setTextColor(0xFFFFFFFF.toInt())
setHintTextColor(0xFF777777.toInt())
// 表单文字不能被画面抢占焦点
isSingleLine = true
}
// ---- 远控态 UI ----
private fun buildRemotePanel(): LinearLayout {
statusView = TextView(this).apply {
textSize = 12f
setPadding(dp(12), dp(6), dp(12), dp(6))
setBackgroundColor(0xCC000000.toInt())
setTextColor(0xFFFFFFFF.toInt())
maxLines = 3
}
// 画面居中、黑边填充。
// 用 FrameLayout + MATCH_PARENT（不是 WRAP_CONTENT）：FrameLayout 会把
// 子视图的 WRAP_CONTENT 当 MATCH_PARENT 处理，onMeasure 里拿到的
// MeasureSpec 就成了 EXACTLY 全尺寸，letterbox 比例收缩被直接忽略
// （2026-10-30 实测：视图被撑成 1080x2001，触摸按 2001 归一化 → 坐标全错）。
// MATCH_PARENT + gravity=center 时父容器传 AT_MOST，onMeasure 的
// setMeasuredDimension 才真正决定尺寸，再被 gravity 居中摆放。
val stage = FrameLayout(this).apply {
setBackgroundColor(0xFF000000.toInt())
addView(
surfaceView,
FrameLayout.LayoutParams(
FrameLayout.LayoutParams.MATCH_PARENT,
FrameLayout.LayoutParams.MATCH_PARENT,
Gravity.CENTER,
),
)
}
val keyboardButton = Button(this).apply {
text = "键盘"
setOnClickListener { showKeyboard() }
}
val disconnectButton = Button(this).apply {
text = "断开"
setOnClickListener { disconnect() }
}
val bar = LinearLayout(this).apply {
orientation = LinearLayout.HORIZONTAL
addView(keyboardButton, LayoutParams(0, LayoutParams.WRAP_CONTENT, 1f))
addView(disconnectButton, LayoutParams(0, LayoutParams.WRAP_CONTENT, 1f))
}
return LinearLayout(this).apply {
orientation = LinearLayout.VERTICAL
visibility = View.GONE
addView(stage, LayoutParams(LayoutParams.MATCH_PARENT, 0, 1f))
// 固定高度而非 WRAP_CONTENT：与 weight=1 的兄弟共存时，
// WRAP_CONTENT 可能被压成 0（实测 bounds=[0,0][0,0]，状态栏不可见）
addView(statusView, LayoutParams(LayoutParams.MATCH_PARENT, dp(64)))
addView(bar, LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.WRAP_CONTENT))
}
}
private fun showConnectState(msg: String) {
connecting = false
remotePanelVisible(false)
connectStatus.text = msg
}
private fun showRemoteState() {
remotePanelVisible(true)
connectPanel.visibility = View.GONE
}
private fun remotePanelVisible(show: Boolean) {
connectPanel.visibility = if (show) View.GONE else View.VISIBLE
remotePanel.visibility = if (show) View.VISIBLE else View.GONE
}
/**
* 显式唤起软键盘。不做自动弹：自动弹会挡画面，且 vivo ROM 行为不一致。
* SurfaceView 已在触摸模式下可聚焦（见 RemoteSurfaceView.init），
* 否则系统不会建立 InputConnection。
*/
private fun showKeyboard() {
if (handle == 0L) return
surfaceView.requestFocus()
val imm = getSystemService(Context.INPUT_METHOD_SERVICE) as InputMethodManager
imm.showSoftInput(surfaceView, InputMethodManager.SHOW_IMPLICIT)
lastTouchDbg = "showSoftInput 已请求（焦点=${surfaceView.hasFocus()}）"
refreshStatus()
}
// ---- RemoteSurfaceView.Listener ----
override fun sendMouseMove(x: Int, y: Int) {
val h = handle
if (h != 0L) RustBridge.nativeSendMouseMove(h, x, y)
}
override fun sendMouseButton(button: Int, down: Boolean) {
val h = handle
if (h != 0L) RustBridge.nativeSendMouseButton(h, button, if (down) 1 else 0)
}
override fun sendWheel(dy: Int) {
val h = handle
if (h != 0L) RustBridge.nativeSendWheel(h, dy)
}
/** 协议 v3：IME 提交的整段文本（UTF-16 code unit，含落单代理项原样透传） */
override fun sendText(units: CharArray) {
val h = handle
if (h != 0L) RustBridge.nativeSendText(h, units)
}
override fun sendKey(key: WinKey, down: Boolean) {
val h = handle
if (h != 0L) {
RustBridge.nativeSendKey(h, key.scancode, if (key.extended) 1 else 0, if (down) 1 else 0)
}
}
override fun isReadyForInput(): Boolean = handle != 0L
/**
* 软键盘是否可见（自测，不依赖 InputMethodManager 的间接信号）。
*
* 为什么必须自测：真机实测（vivo S6 + 讯飞，2026-10-30）键盘隐藏后
* `imm.isAcceptingText` **仍返回 true**——只要 InputConnection 还绑着它就一直 true。
* 用它判断"先关键盘还是先断会话"，会让第一次 BACK 空转、第二次也进不来
* （我们已 return true），会话再也断不掉。这是真机暴露的缺陷，不是推测。
*
* 做法：比较根窗口可见区域高度与窗口高度。键盘占位时可见区域显著变矮。
* 阈值用 15% 而不是固定像素，避免不同屏幕密度下误判。
*/
private var keyboardVisible = false
private val visibleFrame = android.graphics.Rect()
private fun updateKeyboardVisibility() {
val decor = window.decorView
decor.getWindowVisibleDisplayFrame(visibleFrame)
val screenH = decor.height.takeIf { it > 0 } ?: return
val visibleH = visibleFrame.height()
// 底部被 IME 吃掉的高度超过屏幕 15% → 认为键盘可见
val hiddenByIme = screenH - visibleH
val nowVisible = hiddenByIme > screenH * 0.15
if (nowVisible != keyboardVisible) {
keyboardVisible = nowVisible
lastTouchDbg = if (nowVisible) "键盘可见" else "键盘隐藏"
refreshStatus()
}
}
override fun onConfigurationChanged(newConfig: android.content.res.Configuration) {
super.onConfigurationChanged(newConfig)
updateKeyboardVisibility()
}
/**
* 按键转发（物理键盘 / adb 注入 / 无 IME 时的软键盘）：
* - 返回键：**键盘可见时**先收键盘，否则断开回表单（阶段 2a 语义延续）；
* - **可打印**的 `unicodeChar`（0x20 起、排除 0x7F）→ v3 **Text** 通道：
* 字符注入与远端键盘布局无关。注意 `getUnicodeChar()` 给的是单个
* UTF-16 code unit 而非码点，所以这条路只覆盖 BMP；增补平面字符由
* IME `commitText` 成对送达，不经这里；
* - 控制值（Enter/Tab/方向键…）与其余控制键 → Set-1 扫描码通道（down/up 配对）：
* 远端应用期待的是按键事件，不是 WM_CHAR；
* - 自动重复（`repeatCount > 0`，长按）只发一次 down：重复由远端键盘驱动，
* 本地重发会打爆消息队列。
*
* 与 IME 路径互斥：IME 显示时按键先被输入法消费（走 commitText），不会到这里；
* IME 隐藏时才走本方法。两条路同时生效会重复注入。
*
* 诊断：vivo logcat 被限流（lessons R-6），路由结果写状态栏，只报 keyCode
* 与去向，不报字符内容。
*/
override fun onKeyDown(keyCode: Int, event: KeyEvent?): Boolean {
if (keyCode == KeyEvent.KEYCODE_BACK) {
// 用自测的键盘可见性，不用 imm.isAcceptingText（见 updateKeyboardVisibility
// 注释：真机上键盘隐藏后它仍为 true，会让会话永远断不掉）
updateKeyboardVisibility()
if (keyboardVisible) {
val imm = getSystemService(Context.INPUT_METHOD_SERVICE) as InputMethodManager
imm.hideSoftInputFromWindow(surfaceView.windowToken, 0)
keyboardVisible = false
return true
}
if (handle != 0L) {
disconnect()
return true
}
return super.onKeyDown(keyCode, event)
}
val h = handle
if (h == 0L || event == null) return super.onKeyDown(keyCode, event)
// 只报 keyCode 与路由结果，不报字符内容（诊断通道会显示在屏幕上）
val cp = event.unicodeChar
if (cp >= 0x20 && cp != 0x7F) {
if (event.repeatCount == 0) RustBridge.nativeSendText(h, charArrayOf(cp.toChar()))
lastTouchDbg = "key code=$keyCode -> text 1u"
runOnUiThread { refreshStatus() }
return true
}
val key = WinScancode.of(keyCode) ?: run {
lastTouchDbg = "key code=$keyCode 未映射"
runOnUiThread { refreshStatus() }
return super.onKeyDown(keyCode, event)
}
if (event.repeatCount == 0) {
RustBridge.nativeSendKey(h, key.scancode, if (key.extended) 1 else 0, 1)
}
lastTouchDbg = "key code=$keyCode -> scan ${key.scancode} ext=${key.extended}"
runOnUiThread { refreshStatus() }
return true
}
override fun onKeyUp(keyCode: Int, event: KeyEvent?): Boolean {
val h = handle
if (keyCode == KeyEvent.KEYCODE_BACK || h == 0L || event == null) {
return super.onKeyUp(keyCode, event)
}
// 字符事件没有「抬起」语义（Text 通道是整段提交），只回扫控制键。
// 门槛与 onKeyDown 一致：不可打印的 unicodeChar（Enter/Tab）仍需发 key up，
// 否则 onKeyDown 发了 down、这里不发 up，远端会留下按住未释放的键。
val cp = event.unicodeChar
if (cp >= 0x20 && cp != 0x7F) return true
val key = WinScancode.of(keyCode) ?: return super.onKeyUp(keyCode, event)
RustBridge.nativeSendKey(h, key.scancode, if (key.extended) 1 else 0, 0)
return true
}
// ---- 连接 / 解码 ----
private fun startConnect() {
if (connecting || handle != 0L) return
val host = hostInput.text.toString().trim()
val code = codeInput.text.toString()
val port = portInput.text.toString().trim().toIntOrNull()
when {
host.isEmpty() -> {
connectStatus.text = "请填写被控端 IP"
return
}
port == null || port !in 1..65535 -> {
connectStatus.text = "端口必须是 1..65535"
return
}
code.isEmpty() -> {
connectStatus.text = "请填写验证码"
return
}
}
connecting = true
connectButton.isEnabled = false
showRemoteState()
statusView.text = "连接 $host:$port …"
thread(name = "connect") {
try {
val h = RustBridge.connect(host, port, code)
// 尺寸必须在解码前拿到：用它定 SurfaceView 宽高比。
// 若等首帧再改，视图会重新测量，把刚建好的解码器 surface 打掉。
val (w, ht) = RustBridge.desktopSize(h)
handle = h
remoteW = w
remoteH = ht
connecting = false
runOnUiThread {
connectButton.isEnabled = true
// 只有认证成功的会话才值得记住；验证码不落盘（用户裁决）
getSharedPreferences(PREFS, Context.MODE_PRIVATE)
.edit()
.putString("host", host)
.putInt("port", port)
.apply()
if (w > 0 && ht > 0) {
surfaceView.videoAspectRatio = w.toFloat() / ht
}
refreshStatus()
startDecodeLoop(h)
}
} catch (e: Exception) {
Log.e(TAG, "connect failed", e)
val msg = e.message ?: "连接失败"
runOnUiThread {
connectButton.isEnabled = true
handle = 0L
showConnectState("连接失败：$msg")
}
}
}
}
private fun startDecodeLoop(h: Long) {
stopDecoding = false
decodeThread = thread(name = "decode") {
val surface = surfaceView.holder.surface
var tries = 0
while (!surface.isValid && tries++ < 100) Thread.sleep(50)
if (!surface.isValid) {
Log.e(TAG, "surface never became valid")
return@thread
}
val dec = H264Decoder(surface)
var frames = 0L
var lastKeyReq = 0L
while (!stopDecoding) {
val f = try {
RustBridge.nextFrame(h)
} catch (e: Exception) {
Log.e(TAG, "nextFrame threw", e)
null
} ?: break
val fed = dec.queue(f)
if (!fed && !f.keyframe && System.currentTimeMillis() - lastKeyReq > 200) {
RustBridge.nativeRequestKeyframe(h) // 等 IDR 中：限频请求
lastKeyReq = System.currentTimeMillis()
}
frames++
if (frames % 120 == 0L) {
lastFrames = frames
lastRtt = RustBridge.rttMs(h)
runOnUiThread { refreshStatus() }
}
}
dec.release()
Log.i(TAG, "decode loop ended, frames=$frames")
runOnUiThread { refreshStatus() }
}
}
private fun disconnect() {
if (handle == 0L) return
stopDecoding = true
// 顺序纪律：等解码线程退出后才能 destroy（见 android-ffi 文件头）。
// 卡帧风险：agent 静止时 500ms 一帧、QUIC idle 超时后流必关，6s 足够覆盖。
decodeThread?.join(6000)
decodeThread = null
val h = handle
handle = 0L
remoteW = 0
remoteH = 0
lastFrames = 0
lastRtt = 0
RustBridge.destroy(h)
showConnectState("已断开")
}
override fun onDestroy() {
super.onDestroy()
disconnect()
}
/** 状态栏统一刷新：连接态 + 帧数/RTT + 最近输入诊断（logcat 被限流，这是唯一通道） */
@Volatile
private var lastFrames = 0L
@Volatile
private var lastRtt = 0L
private fun refreshStatus() {
if (handle == 0L) {
statusView.text = if (connecting) statusView.text else "未连接"
return
}
statusView.text = buildString {
append("已连接 v$PROTO_LABEL ${remoteW}x$remoteH 帧=$lastFrames RTT=${lastRtt}ms\n")
append("输入: $lastTouchDbg")
}
}
private fun dp(v: Int) = (v * resources.displayMetrics.density).toInt()
companion object {
private const val TAG = "MainActivity"
private const val PREFS = "connect"
/**
* 与协议版本保持一致：远控画面状态栏显示当前握手版本（方案 R5 要求）。
* 值随 `common::PROTO_VERSION` 人工同步——Android 侧不做线格式解析，
* 版本由 Rust 握手校验，这里只作显示与排障用。
*/
private const val PROTO_LABEL = "3"
}
}
apps/android-viewer/app/src/main/kotlin/com/verify/remote/RemoteInputConnection.kt
+121
View File
@@ -0,0 +1,121 @@
package com.verify.remote
import android.view.KeyEvent
import android.view.View
import android.view.inputmethod.BaseInputConnection
import android.view.inputmethod.EditorInfo
/**
* P1 IME 通道（loop-2b-plan §5，用户已裁决）：**本地没有编辑缓冲，远端才是权威**。
*
* 三条约定（与方案一致）：
* 1. `commitText` = 用户确认的整段文本 → 直接走协议 v3 `Text` 通道（布局无关、支持中文）；
* 2. `sendKeyEvent` = 硬键盘/控制键 → 既有扫描码通道（Enter/Backspace/方向键…）；
* 3. `deleteSurroundingText`：本地不知道远端内容，`before>0 && after==0`
* 是 IME 退格最常见形态 → 发对应次数的 Backspace；其余形态记录后放弃（U3）。
*
* composing（拼音未确认串）不显示到远端（U4）：`setComposingText` 吞掉但返回 true，
* 让 IME 维持本地组合状态，候选确认后仍会走 `commitText`。
*
* 日志纪律：debug 通道只报**长度与事件类型**，不报用户输入内容（安全红线）。
*/
class RemoteInputConnection(
target: View,
private val sendText: (CharArray) -> Unit,
private val sendKey: (WinKey, Boolean) -> Unit,
private val debug: (String) -> Unit,
) : BaseInputConnection(target, false) {
override fun commitText(text: CharSequence, newCursorPosition: Int): Boolean {
val units = text.toString().toCharArray()
if (units.isEmpty()) return true
sendText(units)
debug("commit ${units.size}u")
return true
}
override fun sendKeyEvent(event: KeyEvent): Boolean {
if (event.action == KeyEvent.ACTION_MULTIPLE) {
val chars: String? = event.characters
if (!chars.isNullOrEmpty()) {
sendText(chars.toCharArray())
debug("multiple ${chars.length}u")
}
return true
}
// 字符事件优先于扫描码：多数软键盘发的是 keyCode=KEYCODE_UNKNOWN +
// unicodeChar（adb `input text` 同款形态）。这一分支必须排在 WinScancode
// 查表**之前**——否则 UNKNOWN(0) 查不到表会被当「未映射」吞掉，字就丢了。
// 注意 `getUnicodeChar()` 返回的是单个 UTF-16 code unit（不是码点），
// 所以这条路只覆盖 BMP；增补平面字符由 `commitText` 成对送达。
// 只收**可打印**字符（空格起、排除 0x7F）：控制值（Enter/Tab）必须走扫描码，
// 远端应用期待的是按键事件而非 WM_CHAR。见 MainActivity.onKeyDown 同款门槛。
val cp = event.unicodeChar
if (event.action == KeyEvent.ACTION_DOWN && cp >= 0x20 && cp != 0x7F) {
sendText(charArrayOf(cp.toChar()))
debug("keychar 1u")
return true
}
val key = WinScancode.of(event.keyCode)
if (key == null) {
debug("IME 未映射键 code=${event.keyCode}")
return false
}
sendKey(key, event.action == KeyEvent.ACTION_DOWN)
return true
}
override fun deleteSurroundingText(beforeLength: Int, afterLength: Int): Boolean =
deleteBackspaces(beforeLength, afterLength)
override fun deleteSurroundingTextInCodePoints(beforeLength: Int, afterLength: Int): Boolean =
deleteBackspaces(beforeLength, afterLength)
private fun deleteBackspaces(before: Int, after: Int): Boolean {
if (before == 0 && after == 0) return true
if (after != 0) {
// U3：forward-delete 与「先删 after 再删 before」的形态本地无远端真值可依
debug("delete($before,$after)：after>0 形态未支持")
return false
}
val bk = WinScancode.of(KeyEvent.KEYCODE_DEL) ?: return false
repeat(before.coerceAtMost(MAX_BACKSPACE)) {
sendKey(bk, true)
sendKey(bk, false)
}
debug("backspace x$before")
return true
}
/** 吞掉 composing 但返回 true：IME 保持本地组合流程，候选确认仍会 commitText。 */
override fun setComposingText(text: CharSequence, newCursorPosition: Int): Boolean = true
override fun setComposingRegion(start: Int, end: Int): Boolean = true
override fun finishComposingText(): Boolean = true
// 本地无可信缓冲区（远端才是权威），一律回空串——比 null 温和，多数 IME 兼容。
override fun getTextBeforeCursor(n: Int, flags: Int): CharSequence = ""
override fun getTextAfterCursor(n: Int, flags: Int): CharSequence = ""
override fun getSelectedText(flags: Int): CharSequence = ""
override fun getCursorCapsMode(reqModes: Int): Int = 0
override fun performEditorAction(actionCode: Int): Boolean {
if (actionCode == EditorInfo.IME_ACTION_DONE) {
// 软键盘「完成/回车」在远控语义里就是 Enter
WinScancode.of(KeyEvent.KEYCODE_ENTER)?.let {
sendKey(it, true)
sendKey(it, false)
}
debug("IME_ACTION_DONE→Enter")
return true
}
debug("editorAction $actionCode 忽略")
return true
}
companion object {
/** 单次退格请求上限：防 IME 异常大值刷爆消息队列（与 agent 侧 WHEEL_MAX_CLICKS 同思路） */
private const val MAX_BACKSPACE = 64
}
}
apps/android-viewer/app/src/main/kotlin/com/verify/remote/RemoteSurfaceView.kt
+278
View File
@@ -0,0 +1,278 @@
package com.verify.remote
import android.content.Context
import android.text.InputType
import android.view.MotionEvent
import android.view.SurfaceView
import android.view.inputmethod.EditorInfo
import android.view.inputmethod.InputConnection
import kotlin.math.abs
import kotlin.math.hypot
/**
* 保持视频宽高比的 SurfaceView（letterbox），把触摸手势映射为远端鼠标事件，
* 并作为软键盘的宿主（协议 v3 文本通道 + 扫描码控制键）。
*
* 为什么视图尺寸必须等于画面尺寸：触摸坐标按视图宽高归一化到远端 0..=65535。
* 若视图被拉伸（阶段 1 用 MATCH_PARENT），画面与视图不再一一对应，手指位置与
* 远端指针位置就会错位；letterbox 后黑边不属于本视图，触摸天然只在画面内产生事件。
*
* 手势映射（桌面远控的通行方案，参考 Moonlight / Scrcpy 的交互形态，
* 实现为本项目独立编写）：
* - 单指轻点 = 左键单击
* - 单指按下并移动 = 左键拖拽（窗口拖动、文字选中）
* - 双指轻点 = 右键
* - 双指上下滑动 = 滚轮
*
* 刻意不做「手指即光标」的绝对定位：手指会完全挡住目标，桌面精细操作不可用。
*
* **IME 接线（阶段 2b）**：本视图是文本编辑器（`onCheckIsTextEditor`），
* 软键盘的提交经 `RemoteInputConnection` 上行。`isFocusableInTouchMode` 必须为
* true，否则触摸态拿不到焦点、系统不建立 InputConnection（阶段 2a 只有
* `isFocusable`，这就是当时软键盘弹不出来的原因之一）。
*/
class RemoteSurfaceView(context: Context) : SurfaceView(context) {
/** 由 Activity 实现，转发到 RustBridge */
interface Listener {
fun sendMouseMove(x: Int, y: Int)
fun sendMouseButton(button: Int, down: Boolean)
fun sendWheel(dy: Int)
/** 协议 v3：一段 UTF-16 code unit（IME 提交结果） */
fun sendText(units: CharArray)
/** 控制键（Enter/Backspace/方向键等）走扫描码通道 */
fun sendKey(key: WinKey, down: Boolean)
/** 连接态：只有已连接才声明自己是文本编辑器，避免未连接时误弹键盘 */
fun isReadyForInput(): Boolean
}
var listener: Listener? = null
/** 视频宽高比（w/h）。>0 时视图按此比例测量。 */
var videoAspectRatio = 0f
set(value) {
if (value > 0f && field != value) {
field = value
requestLayout()
}
}
private enum class Mode { IDLE, TAP, DRAG, TWO_FINGER }
private var mode = Mode.IDLE
private var downX = 0f
private var downY = 0f
private var lastX = 0f
private var lastY = 0f
private var downTime = 0L
private var twoFingerStartY = 0f
private var twoFingerDownTime = 0L
private var twoFingerMoved = false
private var wheelAccum = 0f
/** 判定「拖拽 vs 点击」的位移阈值：12dp */
private val dragSlopPx = resources.displayMetrics.density * 12f
init {
isFocusable = true
// IME 需要触摸态焦点；没有这行，点「键盘」按钮系统不会建立输入连接
isFocusableInTouchMode = true
}
// ---- IME 通道 ----
override fun onCheckIsTextEditor(): Boolean = listener?.isReadyForInput() == true
override fun onCreateInputConnection(outAttrs: EditorInfo): InputConnection? {
val l = listener ?: return null
if (!l.isReadyForInput()) return null
// TYPE_CLASS_TEXT：让输入法给出正常键盘（含中文候选）。
// 常量从 InputType 引用而非 EditorInfo：Kotlin 不保证经实现类名访问接口常量。
// 刻意不加 TYPE_TEXT_FLAG_NO_SUGGESTIONS——中文输入依赖候选词。
outAttrs.inputType = InputType.TYPE_CLASS_TEXT
// 抑制全屏/extract 模式抢走画面（API 29 上两个标志都还在，具体效果真机验证，见 U4/U5）
outAttrs.imeOptions =
EditorInfo.IME_FLAG_NO_EXTRACT_UI or EditorInfo.IME_FLAG_NO_FULLSCREEN
// 右下动作键文案。字段名是 actionLabel（javap android-34 核实，不是 imeActionLabel）
outAttrs.actionLabel = "发送"
return RemoteInputConnection(
target = this,
sendText = { l.sendText(it) },
sendKey = { k, down -> l.sendKey(k, down) },
debug = { msg -> onDebug?.invoke(msg) },
)
}
/**
* letterbox 测量：在父容器给出的可用空间内按视频宽高比取最大内接矩形。
*
* 关键：父容器给 WRAP_CONTENT 时 MeasureSpec 是 AT_MOST，getSize() 是
* **上限**而非实际可用尺寸。据此算出的尺寸必须自己 setMeasuredDimension，
* 否则父容器会把视图撑满（2026-10-30 实测：视图变成 1080x2001 而非
* 1080x607，触摸按 2001 归一化 → 坐标全错，点中心落到画面外）。
*/
override fun onMeasure(widthSpec: Int, heightSpec: Int) {
val ar = videoAspectRatio
if (ar <= 0f) {
super.onMeasure(widthSpec, heightSpec)
return
}
val availW = MeasureSpec.getSize(widthSpec)
val availH = MeasureSpec.getSize(heightSpec)
if (availW <= 0 || availH <= 0) {
super.onMeasure(widthSpec, heightSpec)
return
}
var w = availW
var h = (availW / ar).toInt()
if (h > availH) {
h = availH
w = (availH * ar).toInt()
}
setMeasuredDimension(w.coerceAtLeast(1), h.coerceAtLeast(1))
}
private fun normX(x: Float) = (x / width * 65535).toInt().coerceIn(0, 65535)
private fun normY(y: Float) = (y / height * 65535).toInt().coerceIn(0, 65535)
private fun moved(fromX: Float, fromY: Float, toX: Float, toY: Float) =
hypot((toX - fromX).toDouble(), (toY - fromY).toDouble()) > dragSlopPx
private fun midX(e: MotionEvent) =
if (e.pointerCount >= 2) (e.getX(0) + e.getX(1)) / 2f else e.x
private fun midY(e: MotionEvent) =
if (e.pointerCount >= 2) (e.getY(0) + e.getY(1)) / 2f else e.y
/** 诊断回调：把触摸情况显示到状态栏（本机 logcat 被系统限流，日志不可靠） */
var onDebug: ((String) -> Unit)? = null
override fun onTouchEvent(event: MotionEvent): Boolean {
val l = listener ?: run {
onDebug?.invoke("touch: listener=null")
return false
}
if (width == 0 || height == 0) return false
val act = when (event.actionMasked) {
MotionEvent.ACTION_DOWN -> "DOWN"
MotionEvent.ACTION_MOVE -> "MOVE"
MotionEvent.ACTION_UP -> "UP"
MotionEvent.ACTION_POINTER_DOWN -> "P2DOWN"
MotionEvent.ACTION_POINTER_UP -> "P2UP"
MotionEvent.ACTION_CANCEL -> "CANCEL"
else -> event.actionMasked.toString()
}
onDebug?.invoke(
"$act view=(${event.x.toInt()},${event.y.toInt()}) " +
"norm=(${normX(event.x)},${normY(event.y)}) ${width}x$height"
)
when (event.actionMasked) {
MotionEvent.ACTION_DOWN -> {
downX = event.x; downY = event.y
lastX = event.x; lastY = event.y
downTime = System.currentTimeMillis()
mode = Mode.TAP
// 先把远端指针移到位，点击/拖拽都以此为基准
l.sendMouseMove(normX(event.x), normY(event.y))
}
MotionEvent.ACTION_POINTER_DOWN -> {
// 第二指落下 = 双指手势开始。若已进入拖拽（远端鼠标按着），
// 必须先抬起，否则远端会留下一个被按住未释放的左键。
if (mode == Mode.DRAG) l.sendMouseButton(BTN_LEFT, false)
mode = Mode.TWO_FINGER
twoFingerMoved = false
wheelAccum = 0f
twoFingerStartY = midY(event)
lastX = midX(event); lastY = midY(event)
twoFingerDownTime = System.currentTimeMillis()
}
MotionEvent.ACTION_MOVE -> when (mode) {
Mode.TAP -> if (moved(downX, downY, event.x, event.y)) {
mode = Mode.DRAG
l.sendMouseButton(BTN_LEFT, true)
lastX = event.x; lastY = event.y
l.sendMouseMove(normX(event.x), normY(event.y))
}
Mode.DRAG -> {
lastX = event.x; lastY = event.y
l.sendMouseMove(normX(event.x), normY(event.y))
}
Mode.TWO_FINGER -> {
val cx = midX(event); val cy = midY(event)
if (abs(cy - twoFingerStartY) > dragSlopPx || moved(lastX, lastY, cx, cy)) {
twoFingerMoved = true
}
// 手指下移（cy 增大）= 内容向上翻 = Windows 滚轮向下 → dy 取负
wheelAccum += (cy - lastY)
val notches = (wheelAccum / WHEEL_PX_PER_NOTCH).toInt()
if (notches != 0) {
l.sendWheel(-notches * WHEEL_DELTA)
wheelAccum -= notches * WHEEL_PX_PER_NOTCH
}
lastX = cx; lastY = cy
}
Mode.IDLE -> {}
}
MotionEvent.ACTION_POINTER_UP -> {
// 双指之一抬起：先判定右键，再把剩余手指的移动视为已结束手势
if (mode == Mode.TWO_FINGER) {
maybeRightClick(l)
mode = Mode.IDLE
}
}
MotionEvent.ACTION_UP -> {
when (mode) {
Mode.TAP -> {
// 轻点 = 左键单击（位置已在 DOWN 时发送）
l.sendMouseButton(BTN_LEFT, true)
l.sendMouseButton(BTN_LEFT, false)
performClick()
}
Mode.DRAG -> l.sendMouseButton(BTN_LEFT, false)
Mode.TWO_FINGER -> maybeRightClick(l)
Mode.IDLE -> {}
}
mode = Mode.IDLE
}
MotionEvent.ACTION_CANCEL -> {
if (mode == Mode.DRAG) l.sendMouseButton(BTN_LEFT, false)
mode = Mode.IDLE
}
}
return true
}
/** 双指未移动且按住时间短 → 右键 */
private fun maybeRightClick(l: Listener) {
if (twoFingerMoved) return
if (System.currentTimeMillis() - twoFingerDownTime > TWO_FINGER_TAP_MAX_MS) return
l.sendMouseMove(normX(lastX), normY(lastY))
l.sendMouseButton(BTN_RIGHT, true)
l.sendMouseButton(BTN_RIGHT, false)
performClick()
}
@Suppress("ClickableViewAccessibility")
override fun performClick(): Boolean {
super.performClick()
return true
}
companion object {
const val BTN_LEFT = 0
const val BTN_RIGHT = 1
const val BTN_MIDDLE = 2
private const val WHEEL_DELTA = 120
private const val WHEEL_PX_PER_NOTCH = 60f
private const val TWO_FINGER_TAP_MAX_MS = 500L
}
}
apps/android-viewer/app/src/main/kotlin/com/verify/remote/RustBridge.kt
+109
View File
@@ -0,0 +1,109 @@
package com.verify.remote
/**
* viewer-core（Rust）的 JNI 声明。符号约定：Java_com_verify_remote_RustBridge_*，
* 与 android-ffi/src/lib.rs 一一对应。
*
* 句柄纪律（Rust 侧文件头有完整说明）：
* - nativeConnect 返回 >0 句柄；0 表示失败，错误消息取 [lastError]
* - destroy 后必须置 0；**destroy 前必须先停掉解码线程**（否则 use-after-free）
*/
object RustBridge {
private const val FRAME_HEADER_LEN = 25
init {
System.loadLibrary("androidffi")
}
private external fun nativeConnect(host: String, port: Int, code: String): Long
external fun nativeLastError(): String?
/** 返回 null = 会话结束；否则 [FRAME_HEADER_LEN] 字节头 + Annex-B 载荷 */
external fun nativeNextFrame(handle: Long): ByteArray?
external fun nativeStats(handle: Long): Long
external fun nativeDesktopSize(handle: Long): Long
external fun nativeSendMouseMove(handle: Long, x: Int, y: Int)
external fun nativeSendMouseButton(handle: Long, button: Int, down: Int)
external fun nativeSendWheel(handle: Long, dy: Int)
external fun nativeSendKey(handle: Long, scancode: Int, extended: Int, down: Int)
/**
* 协议 v3 文本上行。参数刻意用 `CharArray`（UTF-16 code unit）而不是 `String`：
* jni 侧 `get_string` 走 Modified-UTF-8 往返，IME 提交边界的落单代理项会在
* 往返中丢失；`char[]` 经 `GetCharArrayRegion` 按 u16 原样拷贝。
*/
external fun nativeSendText(handle: Long, units: CharArray)
external fun nativeRequestKeyframe(handle: Long)
external fun nativeDestroy(handle: Long)
/** 连接。失败抛 IllegalStateException（带 Rust 侧错误消息）。 */
fun connect(host: String, port: Int, code: String): Long {
val h = nativeConnect(host, port, code)
if (h == 0L) {
throw IllegalStateException(nativeLastError() ?: "connect failed")
}
return h
}
fun destroy(handle: Long) {
if (handle != 0L) nativeDestroy(handle)
}
/** 解出的帧（头字段已拆，payload 可直接喂 MediaCodec） */
data class Frame(
val keyframe: Boolean,
val seq: Long,
val tsMs: Long,
val width: Int,
val height: Int,
val payload: ByteArray,
)
/** 取帧 + 拆头。null = 会话结束。 */
fun nextFrame(handle: Long): Frame? {
val raw = nativeNextFrame(handle) ?: return null
if (raw.size < FRAME_HEADER_LEN) return null
return Frame(
keyframe = raw[0] != 0.toByte(),
seq = readLong(raw, 1),
tsMs = readLong(raw, 9),
width = readInt(raw, 17),
height = readInt(raw, 21),
payload = raw.copyOfRange(FRAME_HEADER_LEN, raw.size),
)
}
fun rttMs(handle: Long): Long {
val packed = nativeStats(handle)
return if (packed < 0) 0 else (packed ushr 32)
}
fun framesIn(handle: Long): Long {
val packed = nativeStats(handle)
return if (packed < 0) 0 else packed and 0xFFFF_FFFF
}
/**
* 被控端桌面尺寸 (w, h)。连接成功后、建解码器之前调用。
* 返回 (0,0) 表示句柄无效。
*/
fun desktopSize(handle: Long): Pair<Int, Int> {
val packed = nativeDesktopSize(handle)
if (packed == 0L) return 0 to 0
val w = ((packed ushr 32) and 0xFFFF_FFFF).toInt()
val h = (packed and 0xFFFF_FFFF).toInt()
return w to h
}
private fun readLong(b: ByteArray, off: Int): Long {
var v = 0L
for (i in 0 until 8) v = v or ((b[off + i].toLong() and 0xFF) shl (8 * i))
return v
}
private fun readInt(b: ByteArray, off: Int): Int {
return (b[off].toInt() and 0xFF) or
((b[off + 1].toInt() and 0xFF) shl 8) or
((b[off + 2].toInt() and 0xFF) shl 16) or
((b[off + 3].toInt() and 0xFF) shl 24)
}
}
apps/android-viewer/app/src/main/kotlin/com/verify/remote/WinScancode.kt
+126
View File
@@ -0,0 +1,126 @@
package com.verify.remote
import android.view.KeyEvent
/**
* Android KeyEvent.KEYCODE_* → Windows PS/2 Set-1 扫描码 + extended 标志。
*
* **扫描码值直接取自桌面 viewer 自己的映射（`viewer/src/keymap.rs`，项目内部事实）**，
* 不是凭记忆写的；Android 侧一律用命名常量 `KeyEvent.KEYCODE_*` 引用，
* 数值由编译器解析（已用 `javap -constants android.jar` 验证这些常量在 API 34 存在）。
*
* 分工（协议 v3 之后）：
* - 可打印字符优先走 **Text 通道**（`KeyEvent.unicodeChar` / IME `commitText`），
* 布局无关、中文可用；
* - 本表服务控制键（Enter/Tab/Backspace/方向键/功能键）与物理键盘，
* 走 `KeyDown/KeyUp` 扫描码通道。
*
* 不做 Shift/Ctrl 组合状态跟踪：软键盘的大写字母由 IME 直接 commit 字符，
* 不需要本地修饰键语义。物理键盘组合键（Ctrl+C 等）登记为后续事项。
*/
data class WinKey(val scancode: Int, val extended: Boolean)
object WinScancode {
fun of(keyCode: Int): WinKey? = when (keyCode) {
// 字母区（Set-1 make code，同 keymap.rs）
KeyEvent.KEYCODE_A -> WinKey(0x1E, false)
KeyEvent.KEYCODE_B -> WinKey(0x30, false)
KeyEvent.KEYCODE_C -> WinKey(0x2E, false)
KeyEvent.KEYCODE_D -> WinKey(0x20, false)
KeyEvent.KEYCODE_E -> WinKey(0x12, false)
KeyEvent.KEYCODE_F -> WinKey(0x21, false)
KeyEvent.KEYCODE_G -> WinKey(0x22, false)
KeyEvent.KEYCODE_H -> WinKey(0x23, false)
KeyEvent.KEYCODE_I -> WinKey(0x17, false)
KeyEvent.KEYCODE_J -> WinKey(0x24, false)
KeyEvent.KEYCODE_K -> WinKey(0x25, false)
KeyEvent.KEYCODE_L -> WinKey(0x26, false)
KeyEvent.KEYCODE_M -> WinKey(0x32, false)
KeyEvent.KEYCODE_N -> WinKey(0x31, false)
KeyEvent.KEYCODE_O -> WinKey(0x18, false)
KeyEvent.KEYCODE_P -> WinKey(0x19, false)
KeyEvent.KEYCODE_Q -> WinKey(0x10, false)
KeyEvent.KEYCODE_R -> WinKey(0x13, false)
KeyEvent.KEYCODE_S -> WinKey(0x1F, false)
KeyEvent.KEYCODE_T -> WinKey(0x14, false)
KeyEvent.KEYCODE_U -> WinKey(0x16, false)
KeyEvent.KEYCODE_V -> WinKey(0x2F, false)
KeyEvent.KEYCODE_W -> WinKey(0x11, false)
KeyEvent.KEYCODE_X -> WinKey(0x2D, false)
KeyEvent.KEYCODE_Y -> WinKey(0x15, false)
KeyEvent.KEYCODE_Z -> WinKey(0x2C, false)
// 数字主键区（1..9, 0；Android keycode 不连续，逐个写）
KeyEvent.KEYCODE_1 -> WinKey(0x02, false)
KeyEvent.KEYCODE_2 -> WinKey(0x03, false)
KeyEvent.KEYCODE_3 -> WinKey(0x04, false)
KeyEvent.KEYCODE_4 -> WinKey(0x05, false)
KeyEvent.KEYCODE_5 -> WinKey(0x06, false)
KeyEvent.KEYCODE_6 -> WinKey(0x07, false)
KeyEvent.KEYCODE_7 -> WinKey(0x08, false)
KeyEvent.KEYCODE_8 -> WinKey(0x09, false)
KeyEvent.KEYCODE_9 -> WinKey(0x0A, false)
KeyEvent.KEYCODE_0 -> WinKey(0x0B, false)
// 符号区（US 布局物理位置，同 keymap.rs）
KeyEvent.KEYCODE_SPACE -> WinKey(0x39, false)
KeyEvent.KEYCODE_ENTER -> WinKey(0x1C, false)
KeyEvent.KEYCODE_DEL -> WinKey(0x0E, false) // Backspace
KeyEvent.KEYCODE_FORWARD_DEL -> WinKey(0x53, true) // Delete（扩展）
KeyEvent.KEYCODE_TAB -> WinKey(0x0F, false)
KeyEvent.KEYCODE_ESCAPE -> WinKey(0x01, false)
KeyEvent.KEYCODE_CAPS_LOCK -> WinKey(0x3A, false)
KeyEvent.KEYCODE_GRAVE -> WinKey(0x29, false)
KeyEvent.KEYCODE_MINUS -> WinKey(0x0C, false)
KeyEvent.KEYCODE_EQUALS -> WinKey(0x0D, false)
KeyEvent.KEYCODE_LEFT_BRACKET -> WinKey(0x1A, false)
KeyEvent.KEYCODE_RIGHT_BRACKET -> WinKey(0x1B, false)
KeyEvent.KEYCODE_BACKSLASH -> WinKey(0x2B, false)
KeyEvent.KEYCODE_SEMICOLON -> WinKey(0x27, false)
KeyEvent.KEYCODE_APOSTROPHE -> WinKey(0x28, false)
KeyEvent.KEYCODE_COMMA -> WinKey(0x33, false)
KeyEvent.KEYCODE_PERIOD -> WinKey(0x34, false)
KeyEvent.KEYCODE_SLASH -> WinKey(0x35, false)
// 修饰键
KeyEvent.KEYCODE_SHIFT_LEFT -> WinKey(0x2A, false)
// 右 Shift **不是**扩展键：微软 Keyboard Input Overview 明确
// "The right-hand Shift key is not considered an extended key either;
// it has a separate scan code (0x36) instead."（本会话实读该页核实）
KeyEvent.KEYCODE_SHIFT_RIGHT -> WinKey(0x36, false)
KeyEvent.KEYCODE_CTRL_LEFT -> WinKey(0x1D, false)
KeyEvent.KEYCODE_CTRL_RIGHT -> WinKey(0x1D, true)
KeyEvent.KEYCODE_ALT_LEFT -> WinKey(0x38, false)
KeyEvent.KEYCODE_ALT_RIGHT -> WinKey(0x38, true)
KeyEvent.KEYCODE_META_LEFT -> WinKey(0x5B, true)
KeyEvent.KEYCODE_META_RIGHT -> WinKey(0x5C, true)
// 导航区（全部扩展键）
KeyEvent.KEYCODE_DPAD_UP -> WinKey(0x48, true)
KeyEvent.KEYCODE_DPAD_DOWN -> WinKey(0x50, true)
KeyEvent.KEYCODE_DPAD_LEFT -> WinKey(0x4B, true)
KeyEvent.KEYCODE_DPAD_RIGHT -> WinKey(0x4D, true)
KeyEvent.KEYCODE_MOVE_HOME -> WinKey(0x47, true)
KeyEvent.KEYCODE_MOVE_END -> WinKey(0x4F, true)
KeyEvent.KEYCODE_PAGE_UP -> WinKey(0x49, true)
KeyEvent.KEYCODE_PAGE_DOWN -> WinKey(0x51, true)
KeyEvent.KEYCODE_INSERT -> WinKey(0x52, true)
// 功能键（F11/F12 的 0x57/0x58 与 linux-agent keycode_of 注释同源）
KeyEvent.KEYCODE_F1 -> WinKey(0x3B, false)
KeyEvent.KEYCODE_F2 -> WinKey(0x3C, false)
KeyEvent.KEYCODE_F3 -> WinKey(0x3D, false)
KeyEvent.KEYCODE_F4 -> WinKey(0x3E, false)
KeyEvent.KEYCODE_F5 -> WinKey(0x3F, false)
KeyEvent.KEYCODE_F6 -> WinKey(0x40, false)
KeyEvent.KEYCODE_F7 -> WinKey(0x41, false)
KeyEvent.KEYCODE_F8 -> WinKey(0x42, false)
KeyEvent.KEYCODE_F9 -> WinKey(0x43, false)
KeyEvent.KEYCODE_F10 -> WinKey(0x44, false)
KeyEvent.KEYCODE_F11 -> WinKey(0x57, false)
KeyEvent.KEYCODE_F12 -> WinKey(0x58, false)
else -> null
}
}
apps/android-viewer/build-android.ps1
+85
View File
@@ -0,0 +1,85 @@
# Phase 1 build chain: Rust .so -> jniLibs -> Gradle debug APK -> install -> launch
#
# Keep this file ASCII-only (PS 5.1 reads BOM-less .ps1 as GBK; see lessons).
#
# Usage (from apps/android-viewer/):
# powershell -ExecutionPolicy Bypass -File build-android.ps1 [-Host_ 192.168.1.5] [-Port 47800] [-Code t] [-SkipRust]
# (param is Host_ because $Host is a PowerShell automatic variable)
#
# Prereqs (Windows dev box):
# - Rust + Android targets + cargo-ndk + NDK (see personal docs/baseline.md toolchain table)
# - JDK 17 + Gradle 8.14 (paths in $vars below, adjust to your machine)
# - local.properties with sdk.dir (gitignored; template: sdk.dir=<android sdk path>)
param(
[string]$Host_ = "192.168.1.5",
[int]$Port = 47800,
[string]$Code = "t",
[switch]$SkipRust
)
$ErrorActionPreference = "Stop"
# repo root = two levels up from this script (apps/android-viewer -> repo)
$proj = $PSScriptRoot
$repo = (Resolve-Path "$proj\..\..").Path
$env_ = "D:\working\environment"
$gradle = "D:\working\android\gradle\gradle-8.14.3\bin\gradle.bat"
$adb = "D:\working\android\sdk\platform-tools\adb.exe"
$env:RUSTUP_HOME = "$env_\rust"
$env:CARGO_HOME = "$env_\cargo"
$env:PATH = "$env_\mingw64\bin;$env_\cargo\bin;$env:PATH"
$env:ANDROID_HOME = "D:\working\android\sdk"
$env:ANDROID_NDK_HOME = "D:\working\android\sdk\ndk\30.0.16248370"
$env:GRADLE_USER_HOME = "D:\working\android\gradle-home"
$env:JAVA_HOME = "D:\working\java\jdk17"
if (-not (Test-Path "$proj\local.properties")) {
Set-Content "$proj\local.properties" 'sdk.dir=D\:\\working\\android\\sdk' -Encoding ascii
Write-Host "generated local.properties (sdk.dir)"
}
if (-not $SkipRust) {
Write-Host "=== 1/4 cargo ndk: android-ffi (arm64 release) ===" -ForegroundColor Cyan
Push-Location $repo
# cargo writes progress to stderr; PS 5.1 + ErrorActionPreference=Stop turns
# every stderr line into a terminating error. Relax for native calls and
# judge by exit code (same for gradle below).
$ErrorActionPreference = "Continue"
cargo ndk -t arm64-v8a build -p android-ffi --release 2>&1 |
Select-String "error|Finished|Building"
if ($LASTEXITCODE -ne 0) { throw "cargo ndk build failed (exit $LASTEXITCODE)" }
$ErrorActionPreference = "Stop"
Pop-Location
}
Write-Host "=== 2/4 copy .so into app/libs ===" -ForegroundColor Cyan
$so = "$repo\target\aarch64-linux-android\release\libandroidffi.so"
if (-not (Test-Path $so)) { throw "missing $so (run without -SkipRust)" }
New-Item -ItemType Directory -Force -Path "$proj\app\libs\arm64-v8a" | Out-Null
Copy-Item $so "$proj\app\libs\arm64-v8a\libandroidffi.so" -Force
Write-Host (" libandroidffi.so: {0:N0} KB" -f ((Get-Item $so).Length/1KB))
Write-Host "=== 3/4 gradle assembleDebug ===" -ForegroundColor Cyan
Push-Location $proj
$ErrorActionPreference = "Continue"
& $gradle :app:assembleDebug --console=plain 2>&1 | Select-Object -Last 25
$code = $LASTEXITCODE
$ErrorActionPreference = "Stop"
Pop-Location
if ($code -ne 0) { throw "gradle build failed (exit $code)" }
$apk = "$proj\app\build\outputs\apk\debug\app-debug.apk"
if (-not (Test-Path $apk)) { throw "apk not found: $apk" }
Write-Host (" APK: {0:N0} KB" -f ((Get-Item $apk).Length/1KB))
Write-Host "=== 4/4 install + launch ===" -ForegroundColor Cyan
$ErrorActionPreference = "Continue"
& $adb install -r $apk
if ($LASTEXITCODE -ne 0) { throw "install failed (phone must allow USB install)" }
& $adb logcat -c | Out-Null
& $adb shell am start -n com.verify.remote/.MainActivity `
--es host $Host_ --ei port $Port --es code $Code | Out-Null
$ErrorActionPreference = "Stop"
Write-Host "launched. watch: adb logcat -s MainActivity:H264Decoder RustBridge:* viewer-core:* AndroidRuntime:E"
Write-Host "after 10s capture screen: adb exec-out screencap -p > screen.png"
apps/android-viewer/build.gradle.kts
+4
View File
@@ -0,0 +1,4 @@
plugins {
id("com.android.application") version "8.9.2" apply false
id("org.jetbrains.kotlin.android") version "2.1.20" apply false
}
apps/android-viewer/gradle.properties
+4
View File
@@ -0,0 +1,4 @@
# 固定 JDK 17（不动全局 JAVA_HOME，遵守 2026-10-29 环境约束）
org.gradle.java.home=D\:\\working\\java\\jdk17
android.useAndroidX=false
org.gradle.jvmargs=-Xmx2g
apps/android-viewer/settings.gradle.kts
+17
View File
@@ -0,0 +1,17 @@
pluginManagement {
repositories {
google()
mavenCentral()
gradlePluginPortal()
}
}
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
repositories {
google()
mavenCentral()
}
}
rootProject.name = "remote-viewer"
include(":app")
common/src/lib.rs
+445 -10
View File
@@ -1,21 +1,45 @@
//! 通用协议与帧封装：控制消息（bincode + 长度前缀）与视频帧格式。 //! 通用协议与帧封装：控制消息（bincode + 长度前缀）与视频帧格式。
/// P2P 打洞信令协议（信令通道专用，不含中继；三端共享契约）
pub mod signaling;
use anyhow::Result; use anyhow::Result;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::io::{AsyncReadExt, AsyncWriteExt};
/// 媒体/控制协议版本。
///
/// **改动规则**：任何对 `ControlMsg` 或 `VideoFrameHeader` 字段的增删改都必须递增此值。
/// 理由：线格式是 bincode（按位置编码，无字段名），新增字段会让旧端解析失败或错位，
/// 且失败方式可能是静默的错误数据而非明确报错。递增后由 `Auth` 握手显式拒绝旧端，
/// 把"连上了但行为诡异"变成"连不上且提示升级"。
///
/// - v1：初版（无版本字段，隐式）
/// - v2：`Auth`/`AuthOk` 携带版本；帧头增加 `seq`/`keyframe`
/// - v3：新增 `InputEvent::Text`（文本注入通道，UTF-16 code unit）。
/// `InputEvent` 因携带 `Vec<u16>` 不再是 `Copy`；三端必须同步升级，
/// 旧端由 `Auth` 握手显式拒绝。
pub const PROTO_VERSION: u16 = 3;
/// 控制通道消息（双向） /// 控制通道消息（双向）
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
pub enum ControlMsg { pub enum ControlMsg {
/// 主控端认证（临时验证码，MVP 明文走 QUIC-TLS 加密通道） /// 主控端认证（临时验证码，MVP 明文走 QUIC-TLS 加密通道）。
Auth { code: String }, /// `proto_version` 必须等于 [`PROTO_VERSION`]，否则被控端回 `AuthFail` 并拒绝。
AuthOk { width: u32, height: u32 }, Auth { code: String, proto_version: u16 },
/// 认证通过。`proto_version` 回显被控端的版本，供主控端确认双方一致。
AuthOk {
width: u32,
height: u32,
proto_version: u16,
},
AuthFail(String), AuthFail(String),
KeyFrameReq, KeyFrameReq,
Ping(u64), Ping(u64),
/// t=主控端发送时刻（主控时钟），agent_ts=被控端收到时刻（被控时钟） /// t=主控端发送时刻（主控时钟），agent_ts=被控端收到时刻（被控时钟）
/// 两者配合可估计时钟偏移，进而用帧头 ts_ms 计算端到端延迟 /// 两者配合可估计时钟偏移，进而用帧头 ts_ms 计算端到端延迟
Pong { t: u64, agent_ts: u64 }, /// lost/sent = agent 发送方向（视频下行）的 QUIC 丢包/发包累计计数（conn.stats().path）
Pong { t: u64, agent_ts: u64, lost: u64, sent: u64 },
Input(InputEvent), Input(InputEvent),
} }
@@ -26,9 +50,12 @@ pub enum MouseButton {
Middle, Middle,
} }
/// 输入事件。鼠标坐标为虚拟桌面归一化 0..=65535； /// 输入事件。鼠标坐标为**被采集显示器**（主屏）归一化 0..=65535
/// （协议 v2 语义，v3 未改；`agent/src/input.rs` 的注入空间与此一致，见个人层 lessons R-4）；
/// 按键为 Set-1 扫描码，extended 标记扩展键（方向键/Home 等，需要 EXTENDEDKEY 标志）。 /// 按键为 Set-1 扫描码，extended 标记扩展键（方向键/Home 等，需要 EXTENDEDKEY 标志）。
#[derive(Debug, Clone, Copy, Serialize, Deserialize)] ///
/// v3 起本类型不再 `Copy`：`Text` 携带堆上 `Vec<u16>`。需要复制的地方用 `.clone()`。
#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum InputEvent { pub enum InputEvent {
MouseMove { x: u16, y: u16 }, MouseMove { x: u16, y: u16 },
MouseDown(MouseButton), MouseDown(MouseButton),
@@ -36,14 +63,60 @@ pub enum InputEvent {
Wheel { dy: i32 }, Wheel { dy: i32 },
KeyDown { scancode: u16, extended: bool }, KeyDown { scancode: u16, extended: bool },
KeyUp { scancode: u16, extended: bool }, KeyUp { scancode: u16, extended: bool },
/// v3 新增：**文本注入通道**。`units` 为 UTF-16 code unit 序列。
///
/// 为什么不是 `String`：Windows 注入侧（`KEYEVENTF_UNICODE`）与 Java 字符串的
/// 原生表示都是 UTF-16 code unit；用 `String` 要走 UTF-8 往返，遇到 IME 提交边界
/// 产生的落单代理项会**有损**。`Vec<u16>` 忠实透传，由接收端按 Win32 语义处理。
///
/// 为什么整段携带、而不是逐 code unit 一条消息：一次 IME 候选确认常是多个汉字；
/// 代理对（emoji 等增补平面字符）**必须在同一次注入批次里成对提交**才能被目标
/// 应用合成成一个字符，拆开注入必现乱码。
///
/// 上限 [`MAX_TEXT_UNITS`]：超长输入截断且两侧打警告日志（不静默丢弃）。
/// 另一道防线是 [`read_control`] 的 1 MiB 消息长度闸。
///
/// 日志纪律：中间层与注入端只允许打印 `units.len()`，**不得**打印用户输入内容。
Text { units: Vec<u16> },
} }
/// 视频帧头（载荷为 LZ4 压缩的 BGRA 像素，按行紧凑排列） /// [`InputEvent::Text`] 单次携带的 code unit 上限。
///
/// 与 `read_control` 的 1 MiB 消息闸是两道独立防线：前者防恶意对端把接收端分配打爆，
/// 后者防合法但异常长的 IME 提交把注入队列刷爆。4096 unit ≈ 8 KiB，
/// 远超正常 IME 一次提交（数十 unit），又足够小使单次 `SendInput` 批量仍可接受。
pub const MAX_TEXT_UNITS: usize = 4096;
/// 把文本输入截断到 [`MAX_TEXT_UNITS`]，返回 `(可注入的 units, 是否发生截断)`。
///
/// 调用方**必须**在截断为真时打印日志（可见后果）；本函数只负责机械截断，
/// 不自己打印——因为 `common` 同时被主控端与被控端复用，日志口径由调用方决定。
pub fn clamp_text_units(units: Vec<u16>) -> (Vec<u16>, bool) {
if units.len() > MAX_TEXT_UNITS {
let mut trimmed = units;
trimmed.truncate(MAX_TEXT_UNITS);
(trimmed, true)
} else {
(units, false)
}
}
/// 视频帧头。载荷为 H.264 Annex-B 码流（含 00 00 00 01 起始码）。
///
/// 字段语义：
/// - `width` / `height`：被控端桌面尺寸（逻辑尺寸，非编码对齐后的尺寸）
/// - `ts_ms`：被控端采集时刻（被控时钟），配合 Ping/Pong 估计的时钟偏移算端到端延迟
/// - `seq`：帧序号，从 0 单调递增，每发一帧 +1（含静止桌面的重复帧）。
/// 用于丢帧检测与弱网统计；注意它统计的是"发送序"，不是"显示序"
/// - `keyframe`：本帧是否 IDR。解码端（尤其 MediaCodec）需要它决定是否重置解码器，
/// 不必再自己扫 NAL 起始码判断
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
pub struct VideoFrameHeader { pub struct VideoFrameHeader {
pub width: u32, pub width: u32,
pub height: u32, pub height: u32,
pub ts_ms: u64, pub ts_ms: u64,
pub seq: u64,
pub keyframe: bool,
} }
async fn read_u32le<R: AsyncReadExt + Unpin>(r: &mut R) -> Result<u32> { async fn read_u32le<R: AsyncReadExt + Unpin>(r: &mut R) -> Result<u32> {
@@ -77,7 +150,9 @@ pub async fn read_control<R: AsyncReadExt + Unpin>(r: &mut R) -> Result<ControlM
Ok(bincode::deserialize(&buf)?) Ok(bincode::deserialize(&buf)?)
} }
/// 编码视频帧：[总长 u32][头长 u32][bincode 头][LZ4 载荷] /// 编码视频帧：[总长 u32][头长 u32][bincode 头][H.264 载荷]
///
/// `总长` 不含自身这 4 字节，等于 `4 + 头长 + 载荷长`。
pub fn encode_video_frame(header: &VideoFrameHeader, payload: &[u8]) -> Vec<u8> { pub fn encode_video_frame(header: &VideoFrameHeader, payload: &[u8]) -> Vec<u8> {
let hb = bincode::serialize(header).expect("serialize header"); let hb = bincode::serialize(header).expect("serialize header");
let total = 4 + hb.len() + payload.len(); let total = 4 + hb.len() + payload.len();
@@ -89,14 +164,374 @@ pub fn encode_video_frame(header: &VideoFrameHeader, payload: &[u8]) -> Vec<u8>
out out
} }
/// 单帧总长上限（256 MiB）。超过即视为协议错误或恶意输入。
const MAX_VIDEO_FRAME: usize = 256 << 20;
/// 判断一段 Annex-B H.264 码流里是否含 IDR 片（NAL 类型 5）。
///
/// 用途：填充 [`VideoFrameHeader::keyframe`]。
/// 为什么不问编码器要这个标志——NVENC 与 openh264 的 API 都能间接推出，
/// 但两者规则不同（首帧强制、`force_intra_frame`、GOP 自动 IDR 交织），
/// 而**实际输出**是唯一可信来源。直接看码流，两种编码器一套逻辑。
///
/// 只识别 3/4 字节起始码；`start-code emulation prevention` 由编码器保证，
/// 因此不会在 SPS/PPS 载荷里误判出起始码。
pub fn h264_contains_idr(payload: &[u8]) -> bool {
let mut i = 0usize;
while i + 3 < payload.len() {
let sc = if payload[i..].starts_with(&[0, 0, 0, 1]) {
4
} else if payload[i..].starts_with(&[0, 0, 1]) {
3
} else {
i += 1;
continue;
};
match payload.get(i + sc) {
// nal_unit_type 占首字节低 5 位
Some(&b) if b & 0x1F == 5 => return true,
Some(_) => i += sc + 1,
None => break,
}
}
false
}
/// 读一个视频帧。
///
/// 对端数据一律视为不可信输入：`total` 与 `hlen` 都必须做**上下界**校验，
/// 否则构造一个 `total < 4` 的帧就能让接收端在切片处 panic（进程级拒绝服务）。
pub async fn read_video_frame<R: AsyncReadExt + Unpin>( pub async fn read_video_frame<R: AsyncReadExt + Unpin>(
r: &mut R, r: &mut R,
) -> Result<(VideoFrameHeader, Vec<u8>)> { ) -> Result<(VideoFrameHeader, Vec<u8>)> {
let total = read_u32le(r).await? as usize; let total = read_u32le(r).await? as usize;
anyhow::ensure!(total < 256 << 20, "video frame too large: {total}"); // 下界：至少要容得下「头长字段(4) + 一个非空的 bincode 头」。
// 上界：防止对端声明超大长度导致巨量分配。
anyhow::ensure!(
(4..MAX_VIDEO_FRAME).contains(&total),
"video frame length out of range: {total}"
);
let mut buf = vec![0u8; total]; let mut buf = vec![0u8; total];
r.read_exact(&mut buf).await?; r.read_exact(&mut buf).await?;
let hlen = u32::from_le_bytes(buf[0..4].try_into().unwrap()) as usize;
let hlen = u32::from_le_bytes(buf[0..4].try_into().expect("已校验 total >= 4")) as usize;
// hlen 必须落在 [1, total-4] 内：0 无法反序列化，超出则切片越界
anyhow::ensure!(
hlen > 0 && hlen <= total - 4,
"video header length invalid: hlen={hlen}, total={total}"
);
let header: VideoFrameHeader = bincode::deserialize(&buf[4..4 + hlen])?; let header: VideoFrameHeader = bincode::deserialize(&buf[4..4 + hlen])?;
Ok((header, buf[4 + hlen..].to_vec())) Ok((header, buf[4 + hlen..].to_vec()))
} }
/// 把一个已编码的完整帧拆回 `(帧头, 载荷)`。
///
/// 用途：静止桌面重发时，被控端需要复用上一帧的**载荷**但重建**帧头**
/// （`seq` 必须递增、`ts_ms` 必须是本次发送时刻）。直接 `clone` 整个帧会让
/// 这两个字段停在上一次发送的时刻，破坏 `seq` 单调递增契约。
///
/// 只接受本模块 [`encode_video_frame`] 产出的字节；格式不符返回 `Err`。
pub fn split_video_frame(frame: &[u8]) -> Result<(VideoFrameHeader, &[u8])> {
anyhow::ensure!(frame.len() >= 8, "frame too short: {}", frame.len());
let hlen = u32::from_le_bytes(frame[4..8].try_into().expect("已校验 len >= 8")) as usize;
anyhow::ensure!(
hlen > 0 && hlen <= frame.len() - 8,
"header length invalid: hlen={hlen}, len={}",
frame.len()
);
let header: VideoFrameHeader = bincode::deserialize(&frame[8..8 + hlen])?;
Ok((header, &frame[8 + hlen..]))
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::io::duplex;
fn hdr(seq: u64, keyframe: bool) -> VideoFrameHeader {
VideoFrameHeader {
width: 1920,
height: 1080,
ts_ms: 12345,
seq,
keyframe,
}
}
// ---- 控制消息 ----
#[test]
fn control_roundtrip_preserves_version() {
let m = ControlMsg::Auth {
code: "123456".into(),
proto_version: PROTO_VERSION,
};
let bytes = encode_control(&m);
// 长度前缀 + 载荷
let len = u32::from_le_bytes(bytes[0..4].try_into().unwrap()) as usize;
assert_eq!(len, bytes.len() - 4);
let back: ControlMsg = bincode::deserialize(&bytes[4..]).unwrap();
match back {
ControlMsg::Auth { code, proto_version } => {
assert_eq!(code, "123456");
assert_eq!(proto_version, PROTO_VERSION);
}
other => panic!("期望 Auth，实际 {other:?}"),
}
}
#[tokio::test]
async fn control_write_read_roundtrip() {
let (mut a, mut b) = duplex(64 * 1024);
let msgs = [
ControlMsg::Auth {
code: "x".into(),
proto_version: PROTO_VERSION,
},
ControlMsg::AuthOk {
width: 800,
height: 600,
proto_version: PROTO_VERSION,
},
ControlMsg::KeyFrameReq,
ControlMsg::Ping(42),
ControlMsg::Pong {
t: 1,
agent_ts: 2,
lost: 3,
sent: 4,
},
];
for m in &msgs {
write_control(&mut a, m).await.unwrap();
let got = read_control(&mut b).await.unwrap();
assert_eq!(format!("{got:?}"), format!("{m:?}"));
}
}
// ---- v3 文本注入通道 ----
/// 版本守卫：v3 是本轮契约变更的登记点，改动必须显式经过这里。
#[test]
fn proto_version_is_three_after_text_event() {
assert_eq!(PROTO_VERSION, 3, "新增 InputEvent::Text 后必须递增到 3");
}
#[test]
fn text_event_roundtrips_utf16_units() {
// 「你好」+ 一个 emoji 的代理对（U+1F600 = D83D DE00）
let units: Vec<u16> = vec![0x4F60, 0x597D, 0xD83D, 0xDE00];
let m = ControlMsg::Input(InputEvent::Text {
units: units.clone(),
});
let bytes = encode_control(&m);
let back: ControlMsg = bincode::deserialize(&bytes[4..]).unwrap();
match back {
ControlMsg::Input(InputEvent::Text { units: got }) => assert_eq!(got, units),
other => panic!("期望 Input::Text，实际 {other:?}"),
}
}
/// 落单代理项必须原样往返：IME 提交边界可能给出孤立高/低代理项，
/// 用 `String` 承载会在这里有损（这正是选 `Vec<u16>` 的理由）。
#[test]
fn text_event_keeps_lone_surrogate() {
let units = vec![0xD83Du16];
let m = ControlMsg::Input(InputEvent::Text { units: units.clone() });
let bytes = encode_control(&m);
let back: ControlMsg = bincode::deserialize(&bytes[4..]).unwrap();
match back {
ControlMsg::Input(InputEvent::Text { units: got }) => assert_eq!(got, units),
other => panic!("期望 Input::Text，实际 {other:?}"),
}
}
#[test]
fn text_units_clamped_at_boundary() {
// 边界：恰好上限不截断；超一个才截断
let exact = vec![0x61u16; MAX_TEXT_UNITS];
let (kept, truncated) = clamp_text_units(exact);
assert_eq!(kept.len(), MAX_TEXT_UNITS);
assert!(!truncated);
let over = vec![0x61u16; MAX_TEXT_UNITS + 1];
let (kept, truncated) = clamp_text_units(over);
assert_eq!(kept.len(), MAX_TEXT_UNITS);
assert!(truncated, "超限必须返回截断标志，让调用方打印警告");
let (empty, truncated) = clamp_text_units(vec![]);
assert!(empty.is_empty());
assert!(!truncated);
}
/// 空 `Text` 也必须能往返（IME 可能提交空串；调用方负责决定忽略还是报错）。
#[test]
fn text_event_empty_roundtrips() {
let m = ControlMsg::Input(InputEvent::Text { units: vec![] });
let bytes = encode_control(&m);
let back: ControlMsg = bincode::deserialize(&bytes[4..]).unwrap();
match back {
ControlMsg::Input(InputEvent::Text { units }) => assert!(units.is_empty()),
other => panic!("期望 Input::Text，实际 {other:?}"),
}
}
/// 上限内的 `Text` 必须能通过 `read_control` 的长度闸；
/// 反向验证：伪造一个超限的声明长度必须在分配前被拒。
#[tokio::test]
async fn text_within_cap_passes_control_length_gate() {
let (mut a, mut b) = duplex(1 << 20);
let msg = ControlMsg::Input(InputEvent::Text {
units: vec![0x4F60; MAX_TEXT_UNITS],
});
write_control(&mut a, &msg).await.unwrap();
match read_control(&mut b).await.unwrap() {
ControlMsg::Input(InputEvent::Text { units }) => assert_eq!(units.len(), MAX_TEXT_UNITS),
other => panic!("期望 Input::Text，实际 {other:?}"),
}
}
// ---- 视频帧头 ----
#[test]
fn video_header_roundtrip_keeps_seq_and_keyframe() {
for (seq, kf) in [(0u64, true), (1, false), (u64::MAX, false)] {
let h = hdr(seq, kf);
let payload = b"\x00\x00\x00\x01\x65fake-nal";
let bytes = encode_video_frame(&h, payload);
let total = u32::from_le_bytes(bytes[0..4].try_into().unwrap()) as usize;
assert_eq!(total, bytes.len() - 4, "总长字段应不含自身");
let hlen = u32::from_le_bytes(bytes[4..8].try_into().unwrap()) as usize;
let back: VideoFrameHeader = bincode::deserialize(&bytes[8..8 + hlen]).unwrap();
assert_eq!(back.seq, seq);
assert_eq!(back.keyframe, kf);
assert_eq!(&bytes[8 + hlen..], payload);
}
}
#[tokio::test]
async fn video_frame_write_read_roundtrip() {
let (mut a, mut b) = duplex(256 * 1024);
let h = hdr(7, true);
let payload = vec![0xABu8; 4096];
let bytes = encode_video_frame(&h, &payload);
tokio::io::AsyncWriteExt::write_all(&mut a, &bytes).await.unwrap();
let (h2, p2) = read_video_frame(&mut b).await.unwrap();
assert_eq!(h2.seq, 7);
assert!(h2.keyframe);
assert_eq!((h2.width, h2.height), (1920, 1080));
assert_eq!(p2, payload);
}
// ---- 不可信输入（P3 的核心回归） ----
/// `total < 4` 必须报错而不是 panic：旧实现在这里 `buf[0..4]` 会越界 panic。
#[tokio::test]
async fn rejects_total_below_minimum() {
for total in [0u32, 1, 3] {
let (mut a, mut b) = duplex(64);
tokio::io::AsyncWriteExt::write_all(&mut a, &total.to_le_bytes())
.await
.unwrap();
let err = read_video_frame(&mut b).await.unwrap_err().to_string();
assert!(
err.contains("out of range"),
"total={total} 应被拒绝，实际错误: {err}"
);
}
}
/// `total == 4` 但没有头数据：hlen 读出来是 0，必须拒绝。
#[tokio::test]
async fn rejects_zero_header_length() {
let (mut a, mut b) = duplex(64);
let mut bytes = 4u32.to_le_bytes().to_vec();
bytes.extend_from_slice(&0u32.to_le_bytes());
tokio::io::AsyncWriteExt::write_all(&mut a, &bytes)
.await
.unwrap();
let err = read_video_frame(&mut b).await.unwrap_err().to_string();
assert!(err.contains("header length invalid"), "实际错误: {err}");
}
/// hlen 声明超过 total-4：旧实现会切片越界 panic。
#[tokio::test]
async fn rejects_header_length_beyond_frame() {
let (mut a, mut b) = duplex(64);
// total = 8 → body 必须正好 8 字节（hlen 字段 4 + 4 字节填充）
let mut bytes = 8u32.to_le_bytes().to_vec();
bytes.extend_from_slice(&999u32.to_le_bytes()); // hlen = 999 > total-4 = 4
bytes.extend_from_slice(&[0u8; 4]);
tokio::io::AsyncWriteExt::write_all(&mut a, &bytes)
.await
.unwrap();
let err = read_video_frame(&mut b).await.unwrap_err().to_string();
assert!(err.contains("header length invalid"), "实际错误: {err}");
}
/// 超过 256 MiB 上限必须在**分配前**拒绝，不能先 vec! 再检查。
#[tokio::test]
async fn rejects_oversized_frame_without_allocating() {
let (mut a, mut b) = duplex(64);
let total = (MAX_VIDEO_FRAME as u32) + 1;
tokio::io::AsyncWriteExt::write_all(&mut a, &total.to_le_bytes())
.await
.unwrap();
let err = read_video_frame(&mut b).await.unwrap_err().to_string();
assert!(err.contains("out of range"), "实际错误: {err}");
}
/// 头部字节损坏（非合法 bincode）应返回 Err 而非 panic。
#[tokio::test]
async fn rejects_corrupt_header_bytes() {
let (mut a, mut b) = duplex(64);
let garbage = [0xFFu8; 8];
let total = (4 + garbage.len()) as u32;
let mut bytes = total.to_le_bytes().to_vec();
bytes.extend_from_slice(&(garbage.len() as u32).to_le_bytes());
bytes.extend_from_slice(&garbage);
tokio::io::AsyncWriteExt::write_all(&mut a, &bytes)
.await
.unwrap();
assert!(read_video_frame(&mut b).await.is_err());
}
// ---- IDR 识别（P2 的 keyframe 标志来源） ----
#[test]
fn detects_idr_in_annex_b() {
// SPS(7) + PPS(8) + IDR 片(5)，4 字节起始码
let idr = [
0, 0, 0, 1, 0x67, 0xAA, // SPS
0, 0, 0, 1, 0x68, 0xBB, // PPS
0, 0, 0, 1, 0x65, 0xCC, // IDR
];
assert!(h264_contains_idr(&idr));
// 3 字节起始码也应识别
let idr3 = [0, 0, 1, 0x65, 0xCC];
assert!(h264_contains_idr(&idr3));
}
#[test]
fn rejects_non_idr_payloads() {
// 纯 P 片（NAL 类型 1）
let p_frame = [0, 0, 0, 1, 0x41, 0xDD];
assert!(!h264_contains_idr(&p_frame));
// 只有 SPS/PPS，没有片（关键帧边界可能落在下一个包，此时不应谎报 keyframe）
let params_only = [
0, 0, 0, 1, 0x67, 0xAA,
0, 0, 0, 1, 0x68, 0xBB,
];
assert!(!h264_contains_idr(&params_only));
assert!(!h264_contains_idr(&[]));
assert!(!h264_contains_idr(&[0, 0, 1])); // 起始码后无 NAL 字节
}
}
common/src/signaling.rs
+757
View File
@@ -0,0 +1,757 @@
//! P2P 打洞信令协议（`common/` 契约，agent / viewer / 信令服务三端共享）。
//!
//! ## 职责边界（重要，改动前必读）
//! - **信令服务器只做两件事**：① 把双方的外部视角（公网映射地址 + NAT 行为）拼起来回推给对端；
//! ② 下发同步打洞计划，让双方在同一绝对时刻开始发包。建连完成后信令与本次会话再无关系，
//! 不承载任何媒体流量。
//! - **本协议不含中继**（无 relay / TURN 消息）。打洞失败就是失败，因此必须把原因讲清楚
//! （见 [`PunchFailReason::user_hint`]），不允许让用户停在"连接中"。
//! - 会话建立后走 QUIC：**不要**再用 viewer 现有的 `SkipVerify`，改用
//! [`EndpointInfo::cert_fingerprint`] 做公钥钉扎。
//!
//! ## 线格式
//! 类型只依赖 `serde`，不绑定具体编码。建议信令走 WSS/443（最容易穿过企业防火墙），
//! 编码用 JSON 便于抓包排查；本模块单元测试用 bincode 做往返校验，两者不冲突。
//!
//! ## 一次会话的时序
//! ```text
//! agent 信令服务器 viewer
//! |-- Register(info) ------->|<------- Register(info) -----|
//! |<-- Registered(observed) -|------ Registered(observed) ->|
//! |<-- PeerReady(peer info) -|------ PeerReady(peer info) -->|
//! |<-- PunchPlan(start_in) --|------ PunchPlan(start_in) --->|
//! | （双方同时向对端候选列表发包，多轮） |
//! |<========= 打通：UDP 直连 → QUIC 握手 + 证书钉扎 =========>|
//! |--- PunchReport(Connected) ->|<---- PunchReport ----------|
//! ```
use serde::{Deserialize, Serialize};
use std::net::{IpAddr, SocketAddr};
// ---------------------------------------------------------------------------
// 常量
// ---------------------------------------------------------------------------
/// 信令协议版本。双方不一致时服务器应拒绝并提示升级，不要尝试兼容旧版。
pub const SIGNALING_VERSION: u16 = 1;
/// 打洞用的 UDP 端口，与 agent 当前监听端口一致。
/// 固定端口对两件事有直接好处：NAT 端口预测、以及用户手动做端口转发。
pub const PUNCH_UDP_PORT: u16 = 47800;
/// 信令默认端口。走 WSS/443 是因为企业防火墙通常只放行 80/443。
pub const SIGNALING_DEFAULT_PORT: u16 = 443;
/// 单次打洞的默认轮数。
pub const PUNCH_DEFAULT_ROUNDS: u8 = 3;
/// 打洞轮间隔（毫秒）。太短来不及让 NAT 建立映射，太长用户等不及。
pub const PUNCH_ROUND_INTERVAL_MS: u16 = 200;
/// 候选地址数量上限（含预测端口）。超出即截断，防止滥用与报文膨胀。
pub const MAX_CANDIDATES: usize = 32;
/// NAT 绑定保活周期的下限/上限（秒）。
pub const KEEPALIVE_MIN_S: u16 = 5;
pub const KEEPALIVE_MAX_S: u16 = 60;
/// 未测出 NAT 绑定超时时的保活默认值（秒）。家用 NAT 的 UDP 映射普遍 30s 左右超时。
pub const KEEPALIVE_DEFAULT_S: u16 = 15;
// ---------------------------------------------------------------------------
// 基础类型
// ---------------------------------------------------------------------------
/// 端点角色。打洞成功后 agent 作为 QUIC server，viewer 主动 connect。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum Role {
/// 被控端
Agent,
/// 主控端
Viewer,
}
impl Role {
pub fn peer(self) -> Role {
match self {
Role::Agent => Role::Viewer,
Role::Viewer => Role::Agent,
}
}
}
/// 候选地址的来源，决定打洞优先级。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum CandidateKind {
/// 本机网卡地址（同局域网直连用，最省事）
Host,
/// 经 UPnP / NAT-PMP / PCP 主动申请的端口映射（把对称 NAT 变成可直连）
Upnp,
/// 信令服务器观察到的公网映射（等价于一次 STUN binding，server-reflexive）
Reflexive,
/// 按 NAT 端口分配规律预测出的端口
Predicted,
}
/// 一个候选地址。IPv6 有公网地址时基本等同直连，因此固定排在 IPv4 之前。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub struct Candidate {
pub addr: SocketAddr,
pub kind: CandidateKind,
}
impl Candidate {
pub fn new(addr: SocketAddr, kind: CandidateKind) -> Self {
Self { addr, kind }
}
pub fn is_ipv6(&self) -> bool {
self.addr.is_ipv6()
}
/// 打分越高越先打。IPv6（+100）> UPnP 显式映射 > 本机地址 > 公网反射 > 端口预测。
pub fn score(&self) -> u16 {
let base = match self.kind {
CandidateKind::Upnp => 90,
CandidateKind::Host => 80,
CandidateKind::Reflexive => 60,
CandidateKind::Predicted => 40,
};
if self.is_ipv6() {
base + 100
} else {
base
}
}
}
/// 按优先级从高到低排序（`sort_by` 是稳定排序，同分保持调用方给出的顺序）。
pub fn sort_candidates(v: &mut [Candidate]) {
v.sort_by(|a, b| b.score().cmp(&a.score()));
}
/// 清洗对端送来的候选列表：丢弃端口 0、按地址去重、截断到 [`MAX_CANDIDATES`]。
///
/// 对端数据一律视为不可信输入；打洞阶段每个候选都会产生实际发包，必须设上限。
pub fn sanitize_candidates(v: Vec<Candidate>) -> Vec<Candidate> {
let mut out: Vec<Candidate> = Vec::with_capacity(v.len().min(MAX_CANDIDATES));
for c in v {
if c.addr.port() == 0 {
continue;
}
if out.iter().any(|e| e.addr == c.addr) {
continue;
}
if out.len() == MAX_CANDIDATES {
break;
}
out.push(c);
}
out
}
// ---------------------------------------------------------------------------
// NAT 画像与诊断
// ---------------------------------------------------------------------------
/// RFC 4787 §4.1 映射行为：同一个内网 socket 的公网端口是否随目标变化。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum MappingBehavior {
/// 端点无关（full cone / restricted cone）：打洞最理想
EndpointIndependent,
/// 地址相关
AddressDependent,
/// 地址+端口相关（即"对称 NAT"）
AddressAndPortDependent,
Unknown,
}
/// RFC 4787 §5.1 过滤行为：外部包能否进得来。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum FilteringBehavior {
EndpointIndependent,
AddressDependent,
AddressAndPortDependent,
Unknown,
}
/// 公网端口分配规律，决定端口预测是否可用。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum PortAllocation {
/// 保持内网端口
Preserved,
/// 递增分配（可预测，端口预测值得一试）
Sequential,
/// 随机（端口预测无效）
Random,
Unknown,
}
/// NAT 行为画像。信令握手阶段顺带测出，用于：① 决定打洞策略；② 失败时给用户解释。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub struct NatProfile {
pub mapping: MappingBehavior,
pub filtering: FilteringBehavior,
pub allocation: PortAllocation,
/// NAT 绑定超时（秒）。未测出为 `None`，此时用 [`KEEPALIVE_DEFAULT_S`]。
pub binding_timeout_s: Option<u16>,
/// 是否支持 hairpinning（同一 NAT 内的两端可互连）
pub hairpinning: bool,
}
impl Default for NatProfile {
fn default() -> Self {
Self {
mapping: MappingBehavior::Unknown,
filtering: FilteringBehavior::Unknown,
allocation: PortAllocation::Unknown,
binding_timeout_s: None,
hairpinning: false,
}
}
}
/// 打洞前景评估，直接用于 UI 提示与"是否值得继续重试"的决策。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum PunchOutlook {
/// 锥形 NAT 或公网 IPv6，基本必成
Likely,
/// 需要多轮重试 / 端口预测，可能成
Possible,
/// 对称 NAT 且端口不可预测，通常打不通
Unlikely,
/// 运营商级 NAT（CGNAT），无中继方案下无解 —— 不必浪费多轮打洞
Hopeless,
}
/// 运营商级 NAT 判定：RFC 6598 共享地址空间 `100.64.0.0/10`。
///
/// 命中即无中继不可达，应立刻给出"换网络"提示，而不是让用户等完所有打洞轮次。
pub fn is_cgnat(observed: IpAddr) -> bool {
match observed {
IpAddr::V4(v4) => {
let o = v4.octets();
o[0] == 100 && (64..=127).contains(&o[1])
}
IpAddr::V6(_) => false,
}
}
/// 综合 NAT 画像与观察到的公网地址，评估打洞前景。
///
/// `observed_public` 是信令服务器看到的本端公网映射（IPv6 时即本机全局地址）。
pub fn punch_outlook(nat: &NatProfile, observed_public: IpAddr) -> PunchOutlook {
// IPv6 没有 NAT 可言，只要双方都有公网 v6 就是真直连。
if observed_public.is_ipv6() {
return PunchOutlook::Likely;
}
if is_cgnat(observed_public) {
return PunchOutlook::Hopeless;
}
match nat.mapping {
MappingBehavior::EndpointIndependent => PunchOutlook::Likely,
MappingBehavior::AddressDependent => PunchOutlook::Possible,
MappingBehavior::AddressAndPortDependent => match nat.allocation {
// 对称 NAT 但端口可预测 → 端口预测 + 多轮重试仍有机会
PortAllocation::Sequential | PortAllocation::Preserved => PunchOutlook::Possible,
PortAllocation::Random | PortAllocation::Unknown => PunchOutlook::Unlikely,
},
// 测不出来就按"可能"处理，先试再说
MappingBehavior::Unknown => PunchOutlook::Possible,
}
}
/// 双方协商保活周期（秒）：取双方绑定超时的较小值再折半，并夹在
/// [`KEEPALIVE_MIN_S`]..=[`KEEPALIVE_MAX_S`] 之间。
///
/// 洞是靠出站包维持的，超时后会静默关闭 —— 表现为"连上了一会儿就断"。
pub fn keepalive_secs(mine: &NatProfile, theirs: &NatProfile) -> u16 {
let t = match (mine.binding_timeout_s, theirs.binding_timeout_s) {
(Some(a), Some(b)) => a.min(b),
(Some(a), None) => a,
(None, Some(b)) => b,
(None, None) => KEEPALIVE_DEFAULT_S,
};
(t / 2).clamp(KEEPALIVE_MIN_S, KEEPALIVE_MAX_S)
}
/// 服务器侧计算某个客户端的打洞启动延迟（相对 PunchPlan 送达时刻）。
///
/// 两侧 RTT 不同：若下发同一个延迟，RTT 大的一方会晚发包，洞就打不上。
/// 以双方最大 RTT 为基准、减去自身 RTT 的一半，即可在**不做时钟同步**的前提下
/// 让两侧的绝对开打时刻对齐（误差约等于单程抖动）。
///
/// ```text
/// start_in_ms = max_rtt_ms / 2 + margin_ms - own_rtt_ms / 2
/// ```
pub fn punch_start_delay_ms(own_rtt_ms: u16, max_rtt_ms: u16, margin_ms: u16) -> u16 {
let base = max_rtt_ms as u32 / 2 + margin_ms as u32;
let own_half = own_rtt_ms as u32 / 2;
base.saturating_sub(own_half).min(u16::MAX as u32) as u16
}
// ---------------------------------------------------------------------------
// 证书指纹（替代 viewer 现有的 SkipVerify）
// ---------------------------------------------------------------------------
/// 自签证书公钥指纹（SHA-256 的 32 字节裸值）。
///
/// agent 生成自签证书后把指纹经信令发布，viewer 用它对 QUIC 连接做**公钥钉扎**，
/// 从而去掉 `viewer/src/main.rs` 里的 `SkipVerify`（公网环境不可接受）。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub struct CertFingerprint(pub [u8; 32]);
impl CertFingerprint {
/// 小写十六进制（64 字符）
pub fn to_hex(&self) -> String {
self.0.iter().map(|b| format!("{b:02x}")).collect()
}
/// 解析 64 字符十六进制；长度不符或含非十六进制字符返回 `None`。
pub fn from_hex(s: &str) -> Option<Self> {
let s = s.trim();
if s.len() != 64 {
return None;
}
let mut out = [0u8; 32];
for (i, chunk) in s.as_bytes().chunks(2).enumerate() {
let hi = (chunk[0] as char).to_digit(16)?;
let lo = (chunk[1] as char).to_digit(16)?;
out[i] = (hi * 16 + lo) as u8;
}
Some(Self(out))
}
}
impl std::fmt::Display for CertFingerprint {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.to_hex())
}
}
// ---------------------------------------------------------------------------
// 端点信息与消息
// ---------------------------------------------------------------------------
/// 连通性相关的能力标志。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)]
pub struct Capabilities {
/// 已成功获得 UPnP / NAT-PMP / PCP 端口映射
pub upnp_mapped: bool,
/// 本机具备可用 IPv6 出口
pub ipv6_available: bool,
}
/// 一个端点的完整连通性描述，随 Register / PeerReady 交换。
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct EndpointInfo {
pub role: Role,
/// 本端打洞用的 UDP 监听端口（通常等于 [`PUNCH_UDP_PORT`]）
pub port: u16,
/// 候选地址，发送前请用 [`sort_candidates`] 排好序
pub candidates: Vec<Candidate>,
pub nat: NatProfile,
/// QUIC 证书公钥指纹，供对端钉扎
pub cert_fingerprint: CertFingerprint,
pub caps: Capabilities,
}
/// 打洞失败原因。**无中继方案的用户体验下限就在这里** —— 每种原因都要能翻译成
/// 用户能照做的一句话，见 [`PunchFailReason::user_hint`]。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum PunchFailReason {
/// 所有候选都无回包（最常见：双方都是地址+端口相关型 NAT）
NoCandidateReachable,
/// 双方均为对称 NAT 且端口不可预测
SymmetricNat,
/// 运营商级 NAT（100.64/10）
Cgnat,
/// 本机 UDP 出站被策略封锁（企业防火墙常见）
UdpBlocked,
/// 对端提前退出
PeerAborted,
/// 超时未收到任何回包
Timeout,
}
impl PunchFailReason {
/// 面向用户的中文处置建议。不要吞掉这个信息 —— 用户看不到原因就只会认为"这软件连不上"。
pub fn user_hint(self) -> &'static str {
match self {
PunchFailReason::NoCandidateReachable => {
"双方网络都无法直接互通。请在被控端路由器开启 UPnP 后重试，或双方改用 IPv6 网络。"
}
PunchFailReason::SymmetricNat => {
"当前网络是对称型 NAT，无法直连。请在被控端路由器开启 UPnP，或更换网络后重试。"
}
PunchFailReason::Cgnat => {
"当前处于运营商级 NAT（大内网），无法直连。请改用 IPv6 网络，或换一条能拿到公网地址的线路。"
}
PunchFailReason::UdpBlocked => {
"当前网络的防火墙拦截了 UDP 出站。请更换网络（如切换手机热点）后重试。"
}
PunchFailReason::PeerAborted => "对端已取消本次连接。",
PunchFailReason::Timeout => "连接超时，可能是网络波动。请重试一次。",
}
}
}
/// 打洞结果。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum PunchOutcome {
/// 打通：`peer` 为真正生效的对端地址，`rtt_ms` 为实测往返
Connected { peer: SocketAddr, rtt_ms: u16 },
Failed { reason: PunchFailReason },
}
impl PunchOutcome {
pub fn is_connected(&self) -> bool {
matches!(self, PunchOutcome::Connected { .. })
}
/// 已连通返回 `None`，失败返回给用户看的建议。
pub fn user_hint(&self) -> Option<&'static str> {
match self {
PunchOutcome::Connected { .. } => None,
PunchOutcome::Failed { reason } => Some(reason.user_hint()),
}
}
}
/// 信令层错误码。
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum SignalError {
/// [`SIGNALING_VERSION`] 不匹配（客户端需升级）
VersionMismatch,
/// 验证码错误或已过期
BadAuth,
SessionNotFound,
PeerOffline,
/// 候选地址数量超限
TooManyCandidates,
RateLimited,
Internal,
}
/// 客户端 → 信令服务器。
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub enum ClientMsg {
/// 注册并等待对端。`auth` 必须是一次性验证码（不是 MVP 里的固定 123456）。
Register {
version: u16,
/// 会话/设备标识；服务器据此配对两端
session: String,
auth: String,
info: EndpointInfo,
},
/// 候选变化补报：UPnP 映射稍后才成功、或网卡地址变化时用。
UpdateCandidates {
session: String,
candidates: Vec<Candidate>,
nat: NatProfile,
},
/// 打洞结果回报。服务器可据此记录有效路径、统计成功率。
PunchReport {
session: String,
plan_id: u32,
outcome: PunchOutcome,
},
/// 保活：维持信令长连接，同时刷新本端 NAT 绑定。
Keepalive { session: String },
Bye {
session: String,
reason: String,
},
}
/// 信令服务器 → 客户端。
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub enum ServerMsg {
/// 注册成功。`observed` 是服务器看到的本端公网映射，等价于一次 STUN binding；
/// `rtt_ms` 供打洞计划做时间对齐（见 [`punch_start_delay_ms`]）。
Registered {
session: String,
observed: SocketAddr,
/// 对端是否已在场。`false` 时应等待 [`ServerMsg::PeerReady`]。
peer_present: bool,
rtt_ms: u16,
},
/// 对端已就绪，附带其候选地址、NAT 画像与证书指纹。
PeerReady {
peer: EndpointInfo,
peer_observed: SocketAddr,
/// 打洞前景，可直接显示给用户（"很可能直连成功" / "当前网络可能无法直连"）
outlook: PunchOutlook,
},
/// 同步打洞计划。
///
/// `start_in_ms` 是**相对本消息**的延迟，服务器对两侧下发不同的值以使双方在同一
/// 绝对时刻开打 —— 因此本协议不要求两端时钟同步。
PunchPlan {
plan_id: u32,
start_in_ms: u16,
rounds: u8,
round_interval_ms: u16,
/// 对端候选（已排序、已清洗）；本端需对其中每个地址发包
peer_candidates: Vec<Candidate>,
},
/// 打洞结果确认（用于 UI 显示"直连成功，RTT xx ms"）。
PunchAck { plan_id: u32, outcome: PunchOutcome },
Error { code: SignalError, message: String },
}
// ---------------------------------------------------------------------------
// 测试
// ---------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
fn fp(byte: u8) -> CertFingerprint {
CertFingerprint([byte; 32])
}
fn info(role: Role) -> EndpointInfo {
let mut candidates = vec![
Candidate::new("1.2.3.4:47800".parse().unwrap(), CandidateKind::Reflexive),
Candidate::new("[2001:db8::1]:47800".parse().unwrap(), CandidateKind::Host),
Candidate::new("192.168.1.5:47800".parse().unwrap(), CandidateKind::Upnp),
];
sort_candidates(&mut candidates);
EndpointInfo {
role,
port: PUNCH_UDP_PORT,
candidates,
nat: NatProfile {
mapping: MappingBehavior::AddressDependent,
filtering: FilteringBehavior::AddressAndPortDependent,
allocation: PortAllocation::Sequential,
binding_timeout_s: Some(30),
hairpinning: false,
},
cert_fingerprint: fp(0xab),
caps: Capabilities {
upnp_mapped: true,
ipv6_available: true,
},
}
}
fn client_msgs() -> Vec<ClientMsg> {
vec![
ClientMsg::Register {
version: SIGNALING_VERSION,
session: "dev-1".into(),
auth: "0f3a91".into(),
info: info(Role::Agent),
},
ClientMsg::UpdateCandidates {
session: "dev-1".into(),
candidates: vec![Candidate::new(
"5.6.7.8:47800".parse().unwrap(),
CandidateKind::Predicted,
)],
nat: NatProfile::default(),
},
ClientMsg::PunchReport {
session: "dev-1".into(),
plan_id: 7,
outcome: PunchOutcome::Connected {
peer: "9.9.9.9:47800".parse().unwrap(),
rtt_ms: 42,
},
},
ClientMsg::PunchReport {
session: "dev-1".into(),
plan_id: 8,
outcome: PunchOutcome::Failed {
reason: PunchFailReason::Cgnat,
},
},
ClientMsg::Keepalive {
session: "dev-1".into(),
},
ClientMsg::Bye {
session: "dev-1".into(),
reason: "user quit".into(),
},
]
}
fn server_msgs() -> Vec<ServerMsg> {
vec![
ServerMsg::Registered {
session: "dev-1".into(),
observed: "203.0.113.9:51234".parse().unwrap(),
peer_present: true,
rtt_ms: 38,
},
ServerMsg::PeerReady {
peer: info(Role::Viewer),
peer_observed: "203.0.113.10:51235".parse().unwrap(),
outlook: PunchOutlook::Likely,
},
ServerMsg::PunchPlan {
plan_id: 9,
start_in_ms: 130,
rounds: PUNCH_DEFAULT_ROUNDS,
round_interval_ms: PUNCH_ROUND_INTERVAL_MS,
peer_candidates: info(Role::Viewer).candidates,
},
ServerMsg::PunchAck {
plan_id: 9,
outcome: PunchOutcome::Failed {
reason: PunchFailReason::SymmetricNat,
},
},
ServerMsg::Error {
code: SignalError::VersionMismatch,
message: "server expects v1".into(),
},
]
}
#[test]
fn bincode_roundtrip_client_msgs() {
for m in client_msgs() {
let bytes = bincode::serialize(&m).expect("serialize");
let back: ClientMsg = bincode::deserialize(&bytes).expect("deserialize");
assert_eq!(m, back);
}
}
#[test]
fn bincode_roundtrip_server_msgs() {
for m in server_msgs() {
let bytes = bincode::serialize(&m).expect("serialize");
let back: ServerMsg = bincode::deserialize(&bytes).expect("deserialize");
assert_eq!(m, back);
}
}
#[test]
fn candidates_sorted_ipv6_first_then_upnp() {
let mut v = vec![
Candidate::new("1.2.3.4:47800".parse().unwrap(), CandidateKind::Reflexive),
Candidate::new("[2001:db8::1]:47800".parse().unwrap(), CandidateKind::Host),
Candidate::new("192.168.1.5:47800".parse().unwrap(), CandidateKind::Upnp),
];
sort_candidates(&mut v);
assert!(v[0].is_ipv6(), "IPv6 候选必须排第一");
assert_eq!(v[1].kind, CandidateKind::Upnp);
assert_eq!(v[2].kind, CandidateKind::Reflexive);
}
#[test]
fn sanitize_dedups_and_bounds() {
let dup = Candidate::new("1.2.3.4:47800".parse().unwrap(), CandidateKind::Host);
let zero_port = Candidate::new("1.2.3.4:0".parse().unwrap(), CandidateKind::Host);
let mut v = vec![dup, dup, zero_port];
v.extend((0..MAX_CANDIDATES + 5).map(|i| {
Candidate::new(
format!("10.0.{}.{}:47800", i / 250, i % 250).parse().unwrap(),
CandidateKind::Predicted,
)
}));
let out = sanitize_candidates(v);
assert_eq!(out.len(), MAX_CANDIDATES);
assert!(out.iter().all(|c| c.addr.port() != 0));
let mut addrs: Vec<_> = out.iter().map(|c| c.addr).collect();
addrs.sort();
addrs.dedup();
assert_eq!(addrs.len(), out.len());
}
#[test]
fn outlook_covers_ipv6_cgnat_and_symmetric() {
let v6: IpAddr = "2001:db8::1".parse().unwrap();
let public: IpAddr = "203.0.113.9".parse().unwrap();
let cgnat: IpAddr = "100.64.1.1".parse().unwrap();
let mut nat = NatProfile::default();
assert_eq!(punch_outlook(&nat, v6), PunchOutlook::Likely);
assert_eq!(punch_outlook(&nat, cgnat), PunchOutlook::Hopeless);
assert!(is_cgnat("100.127.255.254".parse().unwrap()));
assert!(!is_cgnat("100.128.0.1".parse().unwrap()));
nat.mapping = MappingBehavior::EndpointIndependent;
assert_eq!(punch_outlook(&nat, public), PunchOutlook::Likely);
nat.mapping = MappingBehavior::AddressAndPortDependent;
nat.allocation = PortAllocation::Sequential;
assert_eq!(punch_outlook(&nat, public), PunchOutlook::Possible);
nat.allocation = PortAllocation::Random;
assert_eq!(punch_outlook(&nat, public), PunchOutlook::Unlikely);
}
#[test]
fn keepalive_halves_binding_timeout_and_clamps() {
let a = NatProfile {
binding_timeout_s: Some(30),
..Default::default()
};
let b = NatProfile {
binding_timeout_s: Some(20),
..Default::default()
};
assert_eq!(keepalive_secs(&a, &b), 10);
assert_eq!(keepalive_secs(&NatProfile::default(), &NatProfile::default()), 7);
let short = NatProfile {
binding_timeout_s: Some(4),
..Default::default()
};
assert_eq!(keepalive_secs(&short, &short), KEEPALIVE_MIN_S);
let long = NatProfile {
binding_timeout_s: Some(600),
..Default::default()
};
assert_eq!(keepalive_secs(&long, &long), KEEPALIVE_MAX_S);
}
#[test]
fn punch_start_delay_aligns_absolute_time() {
// RTT 大的一方应该更早出发，使两侧绝对开打时刻一致。
let slow = punch_start_delay_ms(200, 200, 100);
let fast = punch_start_delay_ms(20, 200, 100);
assert_eq!(slow, 100);
assert_eq!(fast, 190);
assert!(fast > slow);
}
#[test]
fn cert_fingerprint_hex_roundtrip() {
let f = fp(0x0f);
let hex = f.to_hex();
assert_eq!(hex.len(), 64);
assert_eq!(CertFingerprint::from_hex(&hex), Some(f));
assert_eq!(CertFingerprint::from_hex(&hex.to_uppercase()), Some(f));
assert_eq!(CertFingerprint::from_hex("00"), None);
assert_eq!(CertFingerprint::from_hex(&"z".repeat(64)), None);
}
#[test]
fn fail_reasons_all_have_user_hints() {
let all = [
PunchFailReason::NoCandidateReachable,
PunchFailReason::SymmetricNat,
PunchFailReason::Cgnat,
PunchFailReason::UdpBlocked,
PunchFailReason::PeerAborted,
PunchFailReason::Timeout,
];
for r in all {
assert!(!r.user_hint().is_empty());
let o = PunchOutcome::Failed { reason: r };
assert!(!o.is_connected());
assert!(o.user_hint().is_some());
}
let ok = PunchOutcome::Connected {
peer: "1.2.3.4:47800".parse().unwrap(),
rtt_ms: 30,
};
assert!(ok.is_connected());
assert!(ok.user_hint().is_none());
}
}
linux-agent/Cargo.toml
+16
View File
@@ -0,0 +1,16 @@
[package]
name = "linux-agent"
version = "0.1.0"
edition = "2021"
[dependencies]
agent-core = { path = "../agent-core" }
common = { path = "../common" }
anyhow.workspace = true
tokio.workspace = true
x11rb = { version = "0.13", features = ["xtest"] }
[dev-dependencies]
# 冒烟测试里要扮演主控端，自己发起 QUIC/TLS 连接
quinn.workspace = true
rustls.workspace = true
linux-agent/src/capture.rs
+147
View File
@@ -0,0 +1,147 @@
//! X11 屏幕采集。
//!
//! MVP 走核心协议 `GetImage`：零扩展依赖、零 unsafe，先把「采集 → 编码 → QUIC」整条链路打通。
//! 变更检测用整帧字节比较，桌面静止时不上行视频（省掉编码开销）。
//!
//! 二期优化锚点（已在 `ScreenCapturer` 语义内预留，无需改 agent-core）：
//! 1. **MIT-SHM**：共享内存替代 `GetImage` 的 socket 传输，省掉 X server 一次全画布拷贝；
//! 2. **XDamage**：只取变化矩形 + 只编码脏区，静止时零拷贝。
use agent_core::ScreenCapturer;
use anyhow::{anyhow, Result};
use std::time::{Duration, Instant};
use x11rb::connection::Connection;
use x11rb::protocol::xproto::{ConnectionExt as _, ImageFormat, VisualClass, Window};
use x11rb::rust_connection::RustConnection;
/// 帧节流上限，与 agent-core 默认编码档位（30fps）对齐
const FRAME_INTERVAL: Duration = Duration::from_millis(33);
pub struct X11Capturer {
conn: RustConnection,
root: Window,
width: u16,
height: u16,
/// 最近一帧 BGRA（既是变更检测基线，也是 copy_frame 的数据源）
last: Vec<u8>,
last_capture: Option<Instant>,
}
impl X11Capturer {
pub fn new() -> Result<Self> {
let (conn, screen_num) = x11rb::connect(None).map_err(|e| {
anyhow!(
"连接 X11 失败（DISPLAY={:?}）: {e}",
std::env::var("DISPLAY").ok()
)
})?;
// 先把需要的屏幕参数拷出来：随后 conn 要被移进结构体，不能再持有对它的借用
let (root, width, height, depth, visual_class, bpp) = {
let setup = conn.setup();
let screen = &setup.roots[screen_num];
// bits_per_pixel 不在 visual 上，而在 Setup 的 pixmap_formats 里（按 depth 索引）
let fmt = setup
.pixmap_formats
.iter()
.find(|f| f.depth == screen.root_depth)
.ok_or_else(|| anyhow!("找不到 depth={} 对应的像素格式", screen.root_depth))?;
let visual_class = screen
.allowed_depths
.iter()
.flat_map(|d| d.visuals.iter())
.find(|v| v.visual_id == screen.root_visual)
.map(|v| v.class)
.ok_or_else(|| anyhow!("找不到 root visual"))?;
(
screen.root,
screen.width_in_pixels,
screen.height_in_pixels,
screen.root_depth,
visual_class,
fmt.bits_per_pixel,
)
};
anyhow::ensure!(width > 0 && height > 0, "屏幕尺寸异常 {width}x{height}");
anyhow::ensure!(
bpp == 32,
"当前仅支持 32bpp 真彩采集（depth={depth}，实际 {bpp}bpp）"
);
// BGRA 字节序假设只在 TrueColor 下成立
anyhow::ensure!(
visual_class == VisualClass::TRUE_COLOR,
"当前仅支持 TrueColor 视觉（实际 {visual_class:?}）"
);
println!("[capture] X11 screen {width}x{height}, depth={depth}, {bpp}bpp");
Ok(Self {
conn,
root,
width,
height,
last: Vec::new(),
last_capture: None,
})
}
/// 抓一帧整屏到 `last`，返回画面是否发生变化
fn grab(&mut self) -> Result<bool> {
let reply = self
.conn
.get_image(
ImageFormat::Z_PIXMAP,
self.root,
0,
0,
self.width,
self.height,
!0,
)
.map_err(|e| anyhow!("get_image 请求失败: {e}"))?
.reply()
.map_err(|e| anyhow!("get_image 应答失败: {e}"))?;
let expect = self.width as usize * self.height as usize * 4;
anyhow::ensure!(
reply.data.len() == expect,
"帧数据长度异常：期望 {expect} 字节，实际 {}",
reply.data.len()
);
let changed = reply.data != self.last;
if changed {
self.last = reply.data;
}
Ok(changed)
}
}
impl ScreenCapturer for X11Capturer {
fn size(&self) -> (u32, u32) {
(self.width as u32, self.height as u32)
}
/// X11 核心协议没有「阻塞至新帧」语义，故这里按帧槽节流后主动抓帧：
/// 睡到下一个 1/30s 槽位 → 抓帧 → 与上一帧逐字节比较。
/// 因此 `timeout_ms` 在 Linux 实现中被忽略（agent-core 的 min_frame 节流也随之自然满足）。
fn acquire(&mut self, _timeout_ms: u32) -> Result<bool> {
if let Some(t) = self.last_capture {
let elapsed = t.elapsed();
if elapsed < FRAME_INTERVAL {
std::thread::sleep(FRAME_INTERVAL - elapsed);
}
}
self.last_capture = Some(Instant::now());
self.grab()
}
fn copy_frame(&mut self) -> Result<Vec<u8>> {
anyhow::ensure!(!self.last.is_empty(), "尚无帧数据可拷贝");
Ok(self.last.clone())
}
}
linux-agent/src/input.rs
+251
View File
@@ -0,0 +1,251 @@
//! X11 输入注入：XTest 扩展（Xorg / XWayland / Xvfb 均支持）。
//!
//! **键码映射是本模块的关键**：协议里键盘沿用 Windows 口味的 PS/2 Set-1 扫描码，
//! 而 X11 要的是 keycode，两套编号并非处处等价：
//! - 主键区：`keycode = scancode + 8`（evdev 偏移），已逐键核对；
//! - E0 扩展键：两条链路编号**完全不同**（如方向键 Set-1 是 0x48，evdev 侧 X11 keycode 是 111），必须查表。
//!
//! 鼠标沿用虚拟桌面归一化坐标 0..=65535：X11 根窗口天然横跨全部显示器，
//! 与 Windows 的 `MOUSEEVENTF_VIRTUALDESK` 语义一致，故映射到根窗口像素即可。
use agent_core::InputInjector;
use anyhow::{anyhow, Result};
use common::MouseButton;
use std::sync::Mutex;
use x11rb::connection::Connection;
use x11rb::protocol::xproto::{ConnectionExt as _, Window};
use x11rb::protocol::xtest::ConnectionExt as _;
use x11rb::rust_connection::RustConnection;
// X11 核心事件码（协议固定值）
const KEY_PRESS: u8 = 2;
const KEY_RELEASE: u8 = 3;
const BUTTON_PRESS: u8 = 4;
const BUTTON_RELEASE: u8 = 5;
const MOTION_NOTIFY: u8 = 6;
// X11 按钮编号：1=左 2=中 3=右 4=滚轮上 5=滚轮下
const BTN_LEFT: u8 = 1;
const BTN_MIDDLE: u8 = 2;
const BTN_RIGHT: u8 = 3;
const BTN_WHEEL_UP: u8 = 4;
const BTN_WHEEL_DOWN: u8 = 5;
/// Windows 滚轮一格 = 120（协议沿用该语义）
const WHEEL_DELTA: i32 = 120;
/// 单次滚轮最多合成的格数，防止异常值刷爆事件队列
const WHEEL_MAX_CLICKS: i32 = 20;
pub struct X11Injector {
/// `Mutex` 只为满足 `InputInjector: Sync`（X11 连接本身非 Sync），无竞争
conn: Mutex<RustConnection>,
root: Window,
width: i32,
height: i32,
}
impl X11Injector {
pub fn new() -> Result<Self> {
let (conn, screen_num) = x11rb::connect(None).map_err(|e| {
anyhow!(
"连接 X11 失败（DISPLAY={:?}）: {e}",
std::env::var("DISPLAY").ok()
)
})?;
// XTest 缺失就没有注入能力，启动即失败好过连上后才报错
let ext = conn
.query_extension(b"XTEST")
.map_err(|e| anyhow!("查询 XTEST 扩展失败: {e}"))?
.reply()
.map_err(|e| anyhow!("查询 XTEST 扩展应答失败: {e}"))?;
anyhow::ensure!(ext.present, "X server 未提供 XTEST 扩展，无法注入输入");
// 先拷出参数，避免借用 conn 的同时又把它移进结构体
let (root, width, height) = {
let screen = &conn.setup().roots[screen_num];
(
screen.root,
screen.width_in_pixels as i32,
screen.height_in_pixels as i32,
)
};
println!("[input] XTEST ready, root {width}x{height}");
Ok(Self {
conn: Mutex::new(conn),
root,
width,
height,
})
}
fn fake(&self, type_: u8, detail: u8, x: i32, y: i32) {
let conn = match self.conn.lock() {
Ok(c) => c,
Err(poisoned) => poisoned.into_inner(),
};
if let Err(e) = conn.xtest_fake_input(
type_,
detail,
0,
self.root,
x as i16,
y as i16,
0, // deviceid：0 = 核心设备
) {
eprintln!("[input] xtest_fake_input 失败: {e}");
return;
}
let _ = conn.flush();
}
fn click(&self, button: u8) {
self.fake(BUTTON_PRESS, button, 0, 0);
self.fake(BUTTON_RELEASE, button, 0, 0);
}
}
impl InputInjector for X11Injector {
fn mouse_move(&self, x: u16, y: u16) {
// detail=0 → 绝对坐标
let px = scale(x, self.width);
let py = scale(y, self.height);
self.fake(MOTION_NOTIFY, 0, px, py);
}
fn mouse_button(&self, b: MouseButton, down: bool) {
let button = match b {
MouseButton::Left => BTN_LEFT,
MouseButton::Middle => BTN_MIDDLE,
MouseButton::Right => BTN_RIGHT,
};
self.fake(
if down { BUTTON_PRESS } else { BUTTON_RELEASE },
button,
0,
0,
);
}
fn wheel(&self, dy: i32) {
// 协议沿用 Windows 语义：dy > 0 向上滚，一格 = 120
if dy == 0 {
return;
}
let clicks = (dy.abs() / WHEEL_DELTA).clamp(1, WHEEL_MAX_CLICKS);
let button = if dy > 0 { BTN_WHEEL_UP } else { BTN_WHEEL_DOWN };
for _ in 0..clicks {
self.click(button);
}
}
fn key(&self, scancode: u16, extended: bool, down: bool) {
let Some(keycode) = keycode_of(scancode, extended) else {
eprintln!("[input] 未映射的扫描码 scancode=0x{scancode:02X} extended={extended}");
return;
};
self.fake(
if down { KEY_PRESS } else { KEY_RELEASE },
keycode,
0,
0,
);
}
}
/// 虚拟桌面归一化坐标（0..=65535）→ 根窗口像素坐标
fn scale(v: u16, extent: i32) -> i32 {
if extent <= 1 {
return 0;
}
((v as i64 * (extent as i64 - 1)) / 65535) as i32
}
/// PS/2 Set-1 扫描码 → X11 keycode。
///
/// 非扩展键恒为 `scancode + 8`（当前 viewer 的 keymap 只发该区间内的键）；
/// E0 扩展键查表，编号与主键区规则无关。
fn keycode_of(scancode: u16, extended: bool) -> Option<u8> {
if extended {
// 已按 Xorg evdev 标准 keycode 分配核对
let kc = match scancode {
0x1C => 104, // 小键盘 Enter
0x1D => 105, // 右 Ctrl
0x35 => 106, // 小键盘 /
0x37 => 107, // PrintScreen
0x38 => 108, // 右 Alt
0x47 => 110, // Home
0x48 => 111, // ↑
0x49 => 112, // PageUp
0x4B => 113, // ←
0x4D => 114, // →
0x4F => 115, // End
0x50 => 116, // ↓
0x51 => 117, // PageDown
0x52 => 118, // Insert
0x53 => 119, // Delete
0x5B => 133, // 左 Super(Win)
0x5C => 134, // 右 Super(Win)
0x5D => 135, // Menu
_ => return None,
};
Some(kc)
} else {
// 0x01..=0x53 主键区，以及 0x57/0x58 的 F11/F12；其余为 Set-1 未定义位
match scancode {
0x01..=0x53 | 0x57 | 0x58 => Some((scancode + 8) as u8),
_ => None,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn main_block_uses_evdev_offset() {
assert_eq!(keycode_of(0x1E, false), Some(38)); // A
assert_eq!(keycode_of(0x02, false), Some(10)); // 1
assert_eq!(keycode_of(0x39, false), Some(65)); // Space
assert_eq!(keycode_of(0x1C, false), Some(36)); // Enter
assert_eq!(keycode_of(0x1D, false), Some(37)); // 左 Ctrl
assert_eq!(keycode_of(0x2A, false), Some(50)); // 左 Shift
assert_eq!(keycode_of(0x58, false), Some(96)); // F12
}
#[test]
fn extended_keys_are_not_offset() {
// 若误用 +8 规则，方向键会落到小键盘区（0x48+8=80=KP_8），这正是要防的坑
assert_eq!(keycode_of(0x48, true), Some(111)); // ↑
assert_ne!(keycode_of(0x48, true), Some(0x48 + 8));
assert_eq!(keycode_of(0x50, true), Some(116)); // ↓
assert_eq!(keycode_of(0x4B, true), Some(113)); // ←
assert_eq!(keycode_of(0x4D, true), Some(114)); // →
assert_eq!(keycode_of(0x47, true), Some(110)); // Home
assert_eq!(keycode_of(0x4F, true), Some(115)); // End
assert_eq!(keycode_of(0x49, true), Some(112)); // PageUp
assert_eq!(keycode_of(0x51, true), Some(117)); // PageDown
assert_eq!(keycode_of(0x53, true), Some(119)); // Delete
assert_eq!(keycode_of(0x5B, true), Some(133)); // 左 Super
}
#[test]
fn undefined_scancodes_are_rejected() {
assert_eq!(keycode_of(0x54, false), None);
assert_eq!(keycode_of(0x7F, false), None);
assert_eq!(keycode_of(0x99, true), None);
}
#[test]
fn coordinate_scaling_covers_full_range() {
assert_eq!(scale(0, 1920), 0);
// 归一化上界必须映射到 extent-1，否则指针永远到不了屏幕最右/最下
assert_eq!(scale(65535, 1920), 1919);
// 中点：32768 × 1919 ÷ 65535 = 959.49…，整数除法向下取整 → 959
assert_eq!(scale(32768, 1920), 959);
assert_eq!(scale(12345, 0), 0); // 退化尺寸不 panic
}
}
linux-agent/src/main.rs
+31
View File
@@ -0,0 +1,31 @@
//! Linux 被控端：X11 采集 + XTest 注入；会话层（QUIC/认证/编码/丢帧）复用 `agent-core`。
//!
//! 与 Windows 端的差异仅在这两个模块里，线格式与主控端完全一致，viewer 不需要任何改动。
//! 依赖 `DISPLAY` 环境变量（Xorg / XWayland 均可）。
mod capture;
mod input;
use agent_core::{parse_args, serve, EncoderSettings, InputInjector, ScreenCapturer, SessionConfig};
use anyhow::Result;
use std::sync::Arc;
#[tokio::main]
async fn main() -> Result<()> {
let (port, code) = parse_args()?;
let display = std::env::var("DISPLAY").unwrap_or_else(|_| "(未设置)".to_string());
println!("[linux-agent] DISPLAY={display}");
// 注入端连接常驻（启动即校验 XTEST 扩展是否可用，缺了就直接报错退出）
let injector: Arc<dyn InputInjector> = Arc::new(input::X11Injector::new()?);
serve(
SessionConfig { port, code },
EncoderSettings::default(),
// 采集端可被重建（X11 连接断开等），故用工厂闭包
|| Ok(Box::new(capture::X11Capturer::new()?) as Box<dyn ScreenCapturer>),
injector,
)
.await
}
linux-agent/tests/smoke.rs
+447
View File
@@ -0,0 +1,447 @@
//! Linux 被控端端到端冒烟测试。
//!
//! 需要 X server（Xvfb 即可），通过 `DISPLAY` 指定；推荐用 `scripts/linux-smoke.sh` 一键跑。
//!
//! 覆盖链路：X11 采集 → openh264 编码 → QUIC/TLS 下行 → 主控端解帧；
//! 外加验证码认证、Ping/Pong、关键帧请求，以及**输入注入的实证断言**
//! （注入鼠标移动后，从另一条 X11 连接查询指针真实落点，而不是只看进程没崩）。
use anyhow::{bail, Context, Result};
use common::{read_control, read_video_frame, write_control, ControlMsg, InputEvent};
use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier};
use rustls::pki_types::{CertificateDer, ServerName, UnixTime};
use rustls::{DigitallySignedStruct, SignatureScheme};
use std::io::{BufRead, BufReader};
use std::net::{TcpListener, ToSocketAddrs};
use std::process::{Child, Command, Stdio};
use std::sync::{mpsc, Arc};
use std::time::{Duration, Instant};
use x11rb::connection::Connection;
use x11rb::protocol::xproto::{ConnectionExt as _, Window};
use x11rb::rust_connection::RustConnection;
/// 与 viewer 一致：MVP 阶段跳过自签名证书校验
#[derive(Debug)]
struct SkipVerify;
impl ServerCertVerifier for SkipVerify {
fn verify_server_cert(
&self,
_end_entity: &CertificateDer<'_>,
_intermediates: &[CertificateDer<'_>],
_server_name: &ServerName<'_>,
_ocsp_response: &[u8],
_now: UnixTime,
) -> Result<ServerCertVerified, rustls::Error> {
Ok(ServerCertVerified::assertion())
}
fn verify_tls12_signature(
&self,
_message: &[u8],
_certificate: &CertificateDer<'_>,
_dss: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
Ok(HandshakeSignatureValid::assertion())
}
fn verify_tls13_signature(
&self,
_message: &[u8],
_certificate: &CertificateDer<'_>,
_dss: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
Ok(HandshakeSignatureValid::assertion())
}
fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
vec![
SignatureScheme::RSA_PKCS1_SHA256,
SignatureScheme::ECDSA_NISTP256_SHA256,
SignatureScheme::ED25519,
SignatureScheme::RSA_PSS_SHA256,
]
}
}
/// 被测进程的 RAII 句柄：测试无论成功失败都会收尸
struct AgentProc {
child: Child,
}
impl Drop for AgentProc {
fn drop(&mut self) {
let _ = self.child.kill();
let _ = self.child.wait();
}
}
fn expected_screen() -> (u32, u32) {
let s = std::env::var("SMOKE_SCREEN").unwrap_or_else(|_| "1280x720".to_string());
let (w, h) = s
.split_once('x')
.unwrap_or_else(|| panic!("SMOKE_SCREEN 需形如 1280x720，实际 {s:?}"));
(w.parse().expect("宽"), h.parse().expect("高"))
}
fn free_port() -> Result<u16> {
let l = TcpListener::bind("127.0.0.1:0")?;
Ok(l.local_addr()?.port())
}
fn spawn_agent(port: u16, code: &str) -> Result<AgentProc> {
let mut child = Command::new(env!("CARGO_BIN_EXE_linux-agent"))
.args(["--port", &port.to_string(), "--code", code])
.stdout(Stdio::piped())
.stderr(Stdio::inherit())
.spawn()
.context("启动 linux-agent 失败")?;
let stdout = child.stdout.take().expect("stdout 已 piped");
let (tx, rx) = mpsc::channel();
std::thread::spawn(move || {
for line in BufReader::new(stdout).lines().map_while(|l| l.ok()) {
println!("[agent] {line}");
let _ = tx.send(line);
}
});
// 等 QUIC 端口就绪
let deadline = Instant::now() + Duration::from_secs(30);
loop {
let left = deadline.saturating_duration_since(Instant::now());
if left.is_zero() {
bail!("等待 agent 监听超时（DISPLAY 是否已设置？）");
}
match rx.recv_timeout(left) {
Ok(line) if line.contains("listening") => return Ok(AgentProc { child }),
Ok(_) => continue,
Err(_) => bail!("等待 agent 监听超时，进程可能已退出"),
}
}
}
async fn connect(port: u16) -> Result<(quinn::Endpoint, quinn::Connection)> {
let tls = rustls::ClientConfig::builder()
.dangerous()
.with_custom_certificate_verifier(Arc::new(SkipVerify))
.with_no_client_auth();
let qc: quinn::crypto::rustls::QuicClientConfig = Arc::new(tls).try_into()?;
let mut transport = quinn::TransportConfig::default();
transport.receive_window(quinn::VarInt::from_u32(16 * 1024 * 1024));
transport.stream_receive_window(quinn::VarInt::from_u32(8 * 1024 * 1024));
let mut client_config = quinn::ClientConfig::new(Arc::new(qc));
client_config.transport_config(Arc::new(transport));
let mut endpoint = quinn::Endpoint::client("0.0.0.0:0".parse()?)?;
endpoint.set_default_client_config(client_config);
let addr = ("127.0.0.1", port)
.to_socket_addrs()?
.next()
.ok_or_else(|| anyhow::anyhow!("地址解析失败"))?;
let conn = endpoint.connect(addr, "localhost")?.await?;
// endpoint 必须与 conn 同生命周期，故一并返回
Ok((endpoint, conn))
}
fn has_start_code(payload: &[u8]) -> bool {
let head = &payload[..payload.len().min(64)];
head.windows(4).any(|w| w == [0, 0, 0, 1]) || head.windows(3).any(|w| w == [0, 0, 1])
}
/// H.264 NAL 类型：5 = IDR 片（关键帧）
const NAL_IDR: u8 = 5;
/// 按 Annex-B 起始码扫描，判断载荷里是否含指定类型的 NAL
fn contains_nal(payload: &[u8], nal_type: u8) -> bool {
let mut i = 0usize;
while i + 3 <= payload.len() {
let sc_len = if payload[i..].starts_with(&[0, 0, 0, 1]) {
4
} else if payload[i..].starts_with(&[0, 0, 1]) {
3
} else {
i += 1;
continue;
};
if i + sc_len >= payload.len() {
return false;
}
if payload[i + sc_len] & 0x1F == nal_type {
return true;
}
i += sc_len + 1;
}
false
}
/// 在根窗口左上角画一个**小**矩形，制造一次细微画面变化。
///
/// 刻意不用「整屏刷白」那类大变化：大变化会触发编码器的场景切换自动 IDR，
/// 于是「重发旧帧」和「真重编关键帧」拿到同样的字节，用例就失去区分度了。
fn perturb_screen(conn: &RustConnection, root: Window) -> Result<()> {
use x11rb::protocol::xproto::{CreateGCAux, Rectangle};
let gc = conn.generate_id()?;
conn.create_gc(
gc,
root,
&CreateGCAux::new().foreground(0x00FF_FFFF).graphics_exposures(0),
)?;
conn.poly_fill_rectangle(
root,
gc,
&[Rectangle {
x: 32,
y: 32,
width: 64,
height: 64,
}],
)?;
conn.free_gc(gc)?;
conn.flush()?;
println!("[smoke] 已扰动屏幕（根窗口画 64x64 白块）");
Ok(())
}
/// 读帧直到拿到一个非关键帧（跳过静止桌面的低频重发），返回该帧载荷
async fn read_until_p_frame(recv: &mut quinn::RecvStream, timeout: Duration) -> Result<Vec<u8>> {
let deadline = Instant::now() + timeout;
while Instant::now() < deadline {
let (_h, p) = tokio::time::timeout(Duration::from_secs(5), read_video_frame(recv))
.await
.context("等待扰动帧超时")??;
if !contains_nal(&p, NAL_IDR) {
return Ok(p);
}
}
bail!("扰动后未拿到非关键帧，用例前置条件不成立（扰动量是否太大？）")
}
/// 请求关键帧后，容错地等待一个 IDR：期间允许出现静止桌面的 P 帧重发。
/// 返回是否在超时内拿到 IDR。
async fn wait_for_idr(recv: &mut quinn::RecvStream, timeout: Duration) -> Result<bool> {
let deadline = Instant::now() + timeout;
while Instant::now() < deadline {
match tokio::time::timeout(Duration::from_secs(2), read_video_frame(recv)).await {
Ok(Ok((_h, p))) => {
if contains_nal(&p, NAL_IDR) {
return Ok(true);
}
}
_ => break,
}
}
Ok(false)
}
/// 归一化坐标 → 根窗口像素（与 linux-agent/src/input.rs 的 scale 完全一致）
fn scale(v: u16, extent: u32) -> i16 {
if extent <= 1 {
return 0;
}
((v as i64 * (extent as i64 - 1)) / 65535) as i16
}
fn open_display() -> Result<(RustConnection, Window)> {
let (conn, screen_num) = x11rb::connect(None)
.map_err(|e| anyhow::anyhow!("连接 X11 失败（DISPLAY 是否已设置？）: {e}"))?;
let root = conn.setup().roots[screen_num].root;
Ok((conn, root))
}
fn pointer_position(conn: &RustConnection, root: Window) -> Result<(i16, i16)> {
let r = conn.query_pointer(root)?.reply()?;
Ok((r.root_x, r.root_y))
}
fn wait_pointer(
conn: &RustConnection,
root: Window,
want: (i16, i16),
timeout: Duration,
) -> Result<(i16, i16)> {
let deadline = Instant::now() + timeout;
let mut last = pointer_position(conn, root)?;
while Instant::now() < deadline {
last = pointer_position(conn, root)?;
if (last.0 - want.0).abs() <= 2 && (last.1 - want.1).abs() <= 2 {
return Ok(last);
}
std::thread::sleep(Duration::from_millis(50));
}
Ok(last)
}
#[tokio::test]
async fn end_to_end_capture_encode_transport_input() -> Result<()> {
rustls::crypto::ring::default_provider()
.install_default()
.ok();
let (sw, sh) = expected_screen();
let port = free_port()?;
let _agent = spawn_agent(port, "smoke-code")?;
let (_endpoint, conn) = connect(port).await?;
// 流 1：认证，并核对被控端上报的桌面尺寸就是 Xvfb 的尺寸
let (mut send, mut recv) = conn.open_bi().await?;
write_control(
&mut send,
&ControlMsg::Auth {
code: "smoke-code".into(),
proto_version: common::PROTO_VERSION,
},
)
.await?;
match read_control(&mut recv).await? {
ControlMsg::AuthOk {
width,
height,
proto_version,
} => {
assert_eq!(proto_version, common::PROTO_VERSION, "被控端协议版本不符");
assert_eq!(
(width, height),
(sw, sh),
"agent 上报桌面尺寸与实际屏幕不符"
);
}
other => bail!("认证应成功，却收到 {other:?}"),
}
// 流 2：输入 + 控制
let (mut isend, mut irecv) = conn.open_bi().await?;
// Ping/Pong（延迟测量通道 + 下行丢包统计）
write_control(&mut isend, &ControlMsg::Ping(4242)).await?;
match read_control(&mut irecv).await? {
ControlMsg::Pong {
t,
agent_ts,
lost,
sent,
} => {
assert_eq!(t, 4242, "Pong 回显的时间戳不一致");
assert!(sent > 0, "下行发包计数应大于 0（视频流已在发）");
println!("[smoke] Pong 收到，agent_ts={agent_ts}，下行 sent={sent} lost={lost}");
}
other => bail!("期望 Pong，收到 {other:?}"),
}
// 首帧：证明采集 + 编码 + 线格式整条链路通了
let (h1, p1) = tokio::time::timeout(Duration::from_secs(15), read_video_frame(&mut recv))
.await
.context("等待首个视频帧超时")??;
assert_eq!((h1.width, h1.height), (sw, sh), "视频帧头尺寸不符");
assert!(!p1.is_empty(), "视频载荷为空");
println!(
"[smoke] 首帧 {}x{} ts={} 编码后 {} 字节，头部 {:02X?}",
h1.width,
h1.height,
h1.ts_ms,
p1.len(),
&p1[..p1.len().min(12)]
);
assert!(
has_start_code(&p1),
"载荷不像 H.264（前 16 字节 {:02X?}）",
&p1[..p1.len().min(16)]
);
// 关键帧恢复：先在**静止**桌面上请求关键帧。
// 为了有区分度，先扰动一下屏幕，让缓存里落入一个非关键帧（P 帧），否则
// 「重发旧帧」和「真重编关键帧」两种行为拿到的字节是一样的，测不出来。
let (probe, root) = open_display()?;
perturb_screen(&probe, root)?;
let pp = read_until_p_frame(&mut recv, Duration::from_secs(10)).await?;
println!(
"[smoke] 扰动帧 {} 字节，头部 {:02X?}（应为 P 帧，无 IDR）",
pp.len(),
&pp[..pp.len().min(12)]
);
// 前置条件自检：没有这一步，用例可能因为「扰动本身产生了 IDR」而虚假通过
assert!(
!contains_nal(&pp, NAL_IDR),
"扰动帧仍是 IDR，本用例无法区分重发旧帧与真重编关键帧"
);
tokio::time::sleep(Duration::from_millis(800)).await; // 等画面重新静止
write_control(&mut isend, &ControlMsg::KeyFrameReq).await?;
assert!(
wait_for_idr(&mut recv, Duration::from_secs(3)).await?,
"静止桌面请求关键帧后 3s 内未收到 IDR：解码端一旦出错将永远无法恢复"
);
println!("[smoke] 静止桌面请求关键帧 → 收到 IDR");
// 输入注入实证：注入到屏幕中心，再从独立 X11 连接查询指针真实落点
let before = pointer_position(&probe, root)?;
write_control(
&mut isend,
&ControlMsg::Input(InputEvent::MouseMove {
x: 32768,
y: 32768,
}),
)
.await?;
let want = (scale(32768, sw), scale(32768, sh));
let got = wait_pointer(&probe, root, want, Duration::from_secs(5))?;
println!("[smoke] 指针 {before:?} → {got:?}（期望 ≈{want:?}）");
assert!(
(got.0 - want.0).abs() <= 2 && (got.1 - want.1).abs() <= 2,
"XTest 注入未生效：期望指针 ≈{want:?}，实际 {got:?}"
);
// 键盘：链路不得因此崩溃（无窗口可断言落点）
write_control(
&mut isend,
&ControlMsg::Input(InputEvent::KeyDown {
scancode: 0x1E,
extended: false,
}),
)
.await?;
write_control(
&mut isend,
&ControlMsg::Input(InputEvent::KeyUp {
scancode: 0x1E,
extended: false,
}),
)
.await?;
tokio::time::sleep(Duration::from_millis(300)).await;
assert!(
conn.close_reason().is_none(),
"会话被意外关闭：{:?}",
conn.close_reason()
);
println!("[smoke] 全部断言通过");
Ok(())
}
#[tokio::test]
async fn rejects_wrong_verification_code() -> Result<()> {
rustls::crypto::ring::default_provider()
.install_default()
.ok();
let port = free_port()?;
let _agent = spawn_agent(port, "right-code")?;
let (_endpoint, conn) = connect(port).await?;
let (mut send, mut recv) = conn.open_bi().await?;
write_control(
&mut send,
&ControlMsg::Auth {
code: "wrong-code".into(),
proto_version: common::PROTO_VERSION,
},
)
.await?;
match read_control(&mut recv).await? {
ControlMsg::AuthFail(_) => Ok(()),
other => bail!("错误验证码竟被接受：{other:?}"),
}
}
scripts/linux-smoke.sh
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env bash
# Linux 被控端冒烟测试一键脚本：自建 Xvfb 虚拟显示 → 跑端到端测试 → 收摊。
#
# 用法（容器内或任意有 Xvfb 的 Linux）：
# bash scripts/linux-smoke.sh
# 可用环境变量覆盖：
# SMOKE_SCREEN=1920x1080 虚拟屏分辨率（默认 1280x720）
# SMOKE_DISPLAY=:99 X display 编号（默认 :99）
set -euo pipefail
SCREEN="${SMOKE_SCREEN:-1280x720}"
DISPLAY_NUM="${SMOKE_DISPLAY:-:99}"
export DISPLAY="$DISPLAY_NUM"
export SMOKE_SCREEN="$SCREEN"
XVFB_PID=""
cleanup() {
if [ -n "$XVFB_PID" ]; then
kill "$XVFB_PID" 2>/dev/null || true
wait "$XVFB_PID" 2>/dev/null || true
fi
}
trap cleanup EXIT
echo "[smoke] 启动 Xvfb $DISPLAY_NUM (${SCREEN}x24)"
Xvfb "$DISPLAY_NUM" -screen 0 "${SCREEN}x24" -nolisten tcp >/tmp/xvfb.log 2>&1 &
XVFB_PID=$!
ready=0
for _ in $(seq 1 50); do
if xdpyinfo -display "$DISPLAY_NUM" >/dev/null 2>&1; then
ready=1
break
fi
sleep 0.2
done
if [ "$ready" -ne 1 ]; then
echo "[smoke] Xvfb 启动失败，日志："
cat /tmp/xvfb.log 2>/dev/null || true
exit 1
fi
echo "[smoke] X server 就绪：$(xdpyinfo -display "$DISPLAY_NUM" | grep -E 'dimensions' | head -1)"
# 先跑纯逻辑单测（键码映射 / 坐标换算），不需要 X server
cargo test -p linux-agent --bins
# 再跑端到端用例：两个用例共用同一块虚拟屏与指针，必须串行
cargo test -p linux-agent --test smoke -- --nocapture --test-threads=1
echo "[smoke] 通过"
tools/bench.md
+37
View File
@@ -0,0 +1,37 @@
# 双机测试基线脚本（B1）
每个场景测试 60 秒，记录 HUD 读数填入《B侧设计文档.md》末尾的基线数据表。
## 准备
- 两端都用最新 release：`agent.exe` + `viewer.exe`（同目录带 3 个 mingw DLL）
- 被控机：`agent.exe --port 47800 --code 123456`
- 主控机：`viewer.exe --addr <被控机IP> --port 47800 --code 123456`
- 测试前确认：被控机亮屏有桌面；同一网络（记下是有线还是 WiFi 频段）
## 场景
| # | 场景 | 被控机操作 | 观察点 |
|---|---|---|---|
| 1 | 静止桌面 | 打开一个空白文档，不动 60s | FPS 应低（静止检测），E2E/JIT 应绿 |
| 2 | 视频播放 | 全屏播放任意 1080p 视频 | 码率峰值、FPS、丢包（LOSS 行） |
| 3 | 拖动窗口 | 持续快速拖动一个大窗口 60s | 最恶劣工况：E2E/P95/JIT 是否变红、DECFAIL 是否增长 |
## 记录方法
每个场景进行中截一张 HUD 图（Win+Shift+S 框选右上角），或肉眼逐行抄录：
```
FPS= RTT= E2E= JIT= P95= LOSS= DECFAIL= RATE=
```
## 附加记录
- 网络环境：有线 / WiFi 5GHz / WiFi 2.4GHz，信号强度
- 被控机 CPU 占用（任务管理器，agent.exe 一行）
- 编码器类型：当前 openh264 软编（A 侧换 NVENC 后需重测一轮对比）
## 注意
- LOSS 显示 `--` 表示第一个 Pong 还没回来或 Δsent=0（静止场景发送少，属正常）
- DECFAIL 在连接建立初期可能有 1-2 次（等第一个关键帧），后续不应增长；持续增长 = 关键帧自愈有问题，报给 A
tools/capture_evidence.ps1
+143
View File
@@ -0,0 +1,143 @@
# 2b evidence capture: byte-safe device screenshots of each acceptance state.
# ASCII-only source. Screenshots go through tools\shot.ps1 because PS 5.1 `>`
# writes UTF-16 and corrupts PNG bytes (see that file's header).
# Text/state assertions use uiautomator (lessons R-6: vivo logcat is throttled).
$ErrorActionPreference = "Stop"
$repo = (Resolve-Path "$PSScriptRoot\..").Path
$ev = "D:\working\projects\yuankong\evidence\2b"
$adb = "D:\working\android\sdk\platform-tools\adb.exe"
$phoneIp = "192.168.1.180"
function Adb-Shell {
param([Parameter(Mandatory)][string[]]$Cmd)
$old = $ErrorActionPreference
$ErrorActionPreference = "Continue"
try {
$out = & $adb shell @Cmd 2>$null
return @{ out = ($out -join "`n"); code = $LASTEXITCODE }
}
finally { $ErrorActionPreference = $old }
}
function Get-Ui {
[void](Adb-Shell @("uiautomator", "dump", "/sdcard/ui.xml"))
$old = $ErrorActionPreference
$ErrorActionPreference = "Continue"
try { & $adb pull /sdcard/ui.xml "$env:TEMP\cap-ui.xml" 2>$null | Out-Null }
finally { $ErrorActionPreference = $old }
if (-not (Test-Path "$env:TEMP\cap-ui.xml")) { return "" }
return [System.IO.File]::ReadAllText("$env:TEMP\cap-ui.xml", [System.Text.Encoding]::UTF8)
}
function Wait-Live([int]$Seconds) {
$deadline = (Get-Date).AddSeconds($Seconds)
while ((Get-Date) -lt $deadline) {
Start-Sleep -Milliseconds 900
$x = Get-Ui
if ($x -match "RTT=") { return $x }
}
return ""
}
function Shot([string]$name) {
$out = Join-Path $ev $name
& powershell -NoProfile -ExecutionPolicy Bypass -File "$repo\tools\shot.ps1" -Out $out
}
New-Item -ItemType Directory -Force -Path $ev | Out-Null
# drop the earlier broken captures (PS 5.1 '>' wrote them as UTF-16)
Get-ChildItem $ev -Filter "*.png" | ForEach-Object {
$b = [System.IO.File]::ReadAllBytes($_.FullName)
if ($b.Length -lt 4 -or ($b[0] -ne 0x89)) { Remove-Item $_.FullName -Force }
}
Write-Host "=== ensure fresh v3 agent ==="
$agentExe = "$repo\target\release\agent.exe"
$ag = Get-Process agent -ErrorAction SilentlyContinue
if ($ag -and (Get-Item $agentExe).LastWriteTime -gt $ag.StartTime) {
Stop-Process -Name agent -Force -ErrorAction SilentlyContinue
Start-Sleep -Milliseconds 800; $ag = $null
}
if (-not $ag) {
Start-Process $agentExe -ArgumentList "--port","47800","--code","t" -WindowStyle Hidden
Start-Sleep -Seconds 1
}
Write-Host "=== 1. connect form (V4 entry state) ==="
[void](Adb-Shell @("am","force-stop","com.verify.remote"))
[void](Adb-Shell @("am","start","-n","com.verify.remote/.MainActivity"))
Start-Sleep -Seconds 3
$ui = Get-Ui
if ($ui -notmatch "android\.widget\.EditText") { throw "form fields missing" }
Shot "4b-form.png"
Write-Host "=== 2. failure path (V5): unreachable host via intent ==="
[void](Adb-Shell @("am","force-stop","com.verify.remote"))
[void](Adb-Shell @("am","start","-n","com.verify.remote/.MainActivity",
"--es","host","192.168.1.199","--ei","port","47800","--es","code","t"))
# Measured 2026-10-30: an unreachable host returns to the form with a concrete
# reason after ~32s (quinn's default connect timeout). Budget 50s, not 30s -
# the earlier 30s budget made this step look like a failure when it was only
# still inside the timeout. UX follow-up (logged in the result record): during
# those 30s the panel says only "connecting" with no cancel affordance.
$deadline = (Get-Date).AddSeconds(50)
$err = ""
while ((Get-Date) -lt $deadline) {
Start-Sleep -Milliseconds 1000
$ui = Get-Ui
if ($ui -match "android\.widget\.EditText") {
$shown = [regex]::Matches($ui, 'text="([^"]+)"') | ForEach-Object { $_.Groups[1].Value }
$err = $shown -join " | "
break
}
}
if ($err) { Shot "4b-v5-error.png" } else { throw "V5: never returned to form with an error" }
Write-Host " error state visible: yes"
Write-Host "=== 3. live session (V4 via intent) ==="
[void](Adb-Shell @("am","force-stop","com.verify.remote"))
[void](Adb-Shell @("am","start","-n","com.verify.remote/.MainActivity",
"--es","host",$phoneIp,"--ei","port","47800","--es","code","t"))
$ui = Wait-Live 25
if (-not $ui) { throw "session never became live" }
Shot "4b-connected-stream.png"
Write-Host " $([regex]::Match($ui,'text="([^"]*RTT=[^"]*)"').Groups[1].Value)"
Write-Host "=== 4. keyboard shown (real IME bound to our InputConnection) ==="
$btns = [regex]::Matches($ui, '<node[^>]*class="android\.widget\.Button"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"')
if ($btns.Count -lt 1) { throw "no keyboard button" }
$b0 = $btns[0]
$kx = [int](([int]$b0.Groups[1].Value + [int]$b0.Groups[3].Value) / 2)
$ky = [int](([int]$b0.Groups[2].Value + [int]$b0.Groups[4].Value) / 2)
[void](Adb-Shell @("input","tap","$kx","$ky"))
Start-Sleep -Seconds 3
$ime = (Adb-Shell @("dumpsys","input_method")).out
if ($ime -notmatch "mInputShown=true") { throw "IME not shown" }
if ($ime -notmatch "mHaveConnection=true") { throw "no InputConnection bound" }
Shot "4b-ime-shown.png"
Write-Host "=== 5. IME commit reaches the pipeline (diagnostic shows commit) ==="
[void](Adb-Shell @("input","text","hello"))
Start-Sleep -Milliseconds 1200
[void](Adb-Shell @("input","keyevent","62")) # SPACE: accept any pending candidate
Start-Sleep -Seconds 2
$ui = Get-Ui
$diag = [regex]::Match($ui, 'text="([^"]*RTT=[^"]*)"').Groups[1].Value
Write-Host " phone status: $diag"
Shot "4b-ime-committed.png"
Write-Host "=== 6. disconnect (back key) ==="
[void](Adb-Shell @("input","keyevent","4")) # dismiss keyboard
Start-Sleep -Milliseconds 800
[void](Adb-Shell @("input","keyevent","4")) # disconnect
Start-Sleep -Seconds 2
$ui = Get-Ui
if ($ui -match "RTT=") { throw "still connected after two BACK presses" }
Shot "4b-disconnected.png"
Write-Host "=== CAPTURE COMPLETE ==="
Get-ChildItem $ev -Filter "4b-*.png" | ForEach-Object {
$b = [System.IO.File]::ReadAllBytes($_.FullName)
$sig = ($b[0..3] | ForEach-Object { $_.ToString("X2") }) -join " "
"{0,-28} {1,9} {2}" -f $_.Name, $b.Length, $sig
}
tools/ime_chinese_proof.ps1
+205
View File
@@ -0,0 +1,205 @@
# 2b V6 on-device CHINESE: drive the IME through pinyin + candidate selection.
# ASCII-only source. Non-ASCII expectations are built from code points.
#
# Measured path differences (2026-10-30, vivo S6 + iFlytek IME):
# - `adb input keyevent <letter>` behaves like a HARDWARE keyboard: the IME
# does not compose, the event lands in Activity.onKeyDown and our printable
# unicodeChar branch sends it as text. So keyevents CANNOT exercise Chinese
# candidate selection.
# - `adb input text <latin>` is delivered to the IME, which composes it;
# SPACE (keyevent 62) then accepts the highlighted candidate, and the Hanzi
# arrive at RemoteInputConnection.commitText.
# This script therefore types pinyin with `input text` and commits with SPACE.
#
# Acceptance: the Windows target window receives the exact Han characters the
# IME committed - Chinese input works end to end over protocol v3.
# Not covered: swipe/handwriting input, browsing the candidate list.
param(
[string]$PhoneIp = "192.168.1.180",
[int]$Port = 47800,
[string]$Code = "t"
)
$ErrorActionPreference = "Stop"
$repo = (Resolve-Path "$PSScriptRoot\..").Path
$adb = "D:\working\android\sdk\platform-tools\adb.exe"
$tmp = "$env:TEMP\remote-cjk2"
New-Item -ItemType Directory -Force -Path $tmp | Out-Null
$pidFile = "$tmp\target.pid"
$stopFile = "$tmp\stop.flag"
$resultFile = "$tmp\received.txt"
Remove-Item $pidFile,$stopFile,$resultFile -ErrorAction SilentlyContinue
Add-Type -TypeDefinition @"
using System;
using System.Runtime.InteropServices;
using System.Text;
public static class Fg5 {
[DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow();
[DllImport("user32.dll")] public static extern uint GetWindowThreadProcessId(IntPtr h, out uint pid);
[DllImport("user32.dll", CharSet=CharSet.Auto)] public static extern int GetWindowText(IntPtr h, StringBuilder s, int n);
public static string Desc() {
IntPtr h = GetForegroundWindow(); uint pid; GetWindowThreadProcessId(h, out pid);
var sb = new StringBuilder(256); GetWindowText(h, sb, 256);
return h.ToInt64() + " [" + sb.ToString() + "] pid=" + pid;
}
}
"@
function Adb-Shell {
param([Parameter(Mandatory)][string[]]$Cmd)
$old = $ErrorActionPreference
$ErrorActionPreference = "Continue"
try {
$out = & $adb shell @Cmd 2>$null
return @{ out = ($out -join "`n"); code = $LASTEXITCODE }
}
finally { $ErrorActionPreference = $old }
}
function Wait-File {
param([string]$Path, [int]$TimeoutMs = 20000)
$t = 0
while (-not (Test-Path $Path) -and $t -lt ($TimeoutMs / 150)) { Start-Sleep -Milliseconds 150; $t++ }
if (-not (Test-Path $Path)) { throw "timeout waiting for $Path" }
}
function Assert-Foreground([IntPtr]$hwnd, [string]$where) {
$deadline = (Get-Date).AddMilliseconds(2500)
while ((Get-Date) -lt $deadline) {
if ([Fg5]::GetForegroundWindow() -eq $hwnd) { return }
Start-Sleep -Milliseconds 50
}
throw "$where : foreground is $([Fg5]::Desc()), want target $($hwnd.ToInt64()) - refusing to inject"
}
function Get-Ui {
[void](Adb-Shell @("uiautomator", "dump", "/sdcard/ui.xml"))
$old = $ErrorActionPreference
$ErrorActionPreference = "Continue"
try { & $adb pull /sdcard/ui.xml "$tmp\ui.xml" 2>$null | Out-Null }
finally { $ErrorActionPreference = $old }
if (-not (Test-Path "$tmp\ui.xml")) { return "" }
return [System.IO.File]::ReadAllText("$tmp\ui.xml", [System.Text.Encoding]::UTF8)
}
function Read-Received {
if (-not (Test-Path $resultFile)) { return "" }
$b = [System.IO.File]::ReadAllBytes($resultFile)
$s = [System.Text.Encoding]::UTF8.GetString($b)
if ($s.Length -gt 0 -and $s[0] -eq [char]0xFEFF) { $s = $s.Substring(1) }
return $s
}
function Show-Cps([string]$s) {
($s.ToCharArray() | ForEach-Object { "U+{0:X4}" -f [int]$_ }) -join " "
}
Write-Host "=== 0. retire ALL earlier targets, verify none holds foreground ==="
$flags = @(Get-ChildItem -Path $env:TEMP -Recurse -Filter "stop.flag" -ErrorAction SilentlyContinue |
Where-Object { $_.FullName -like "*remote*cjk*" -or $_.FullName -like "*remote*proof*" })
foreach ($f in $flags) { (Get-Item $f.FullName).LastWriteTime = Get-Date }
if ($flags.Count -gt 0) { Start-Sleep -Seconds 3 }
$hosts = @(Get-CimInstance Win32_Process -Filter "Name='powershell.exe'" |
Where-Object { $_.CommandLine -match 'text_target\.ps1' -and $_.ProcessId -ne $PID })
foreach ($hx in $hosts) { Stop-Process -Id $hx.ProcessId -Force -ErrorAction SilentlyContinue }
if ($hosts.Count -gt 0) { Start-Sleep -Seconds 1 }
Write-Host " retired $($flags.Count) flag(s), killed $($hosts.Count) host(s); fg now: $([Fg5]::Desc())"
Write-Host "=== 1. agent freshness ==="
$agentExe = "$repo\target\release\agent.exe"
$ag = Get-Process agent -ErrorAction SilentlyContinue
if ($ag -and (Get-Item $agentExe).LastWriteTime -gt $ag.StartTime) {
Stop-Process -Name agent -Force -ErrorAction SilentlyContinue
Start-Sleep -Milliseconds 800; $ag = $null
}
if (-not $ag) {
Start-Process $agentExe -ArgumentList "--port",$Port,"--code",$Code -WindowStyle Hidden
Start-Sleep -Seconds 1
}
Write-Host "=== 2. fresh target window (empty field) ==="
Start-Process powershell.exe -ArgumentList "-NoProfile","-ExecutionPolicy","Bypass",`
"-File","$repo\tools\text_target.ps1","-PidFile",$pidFile,`
"-ResultFile",$resultFile,"-StopFile",$stopFile,"-GrabForeground"
Wait-File $pidFile
$parts = @((Get-Content $pidFile -Raw) -split "`n")
$tid = [int]$parts[0].Trim()
$hwnd = [IntPtr][long]$parts[1].Trim()
Assert-Foreground $hwnd "after launch"
$base0 = Read-Received
Write-Host " target pid=$tid hwnd=$($hwnd.ToInt64()) baseline=[$(Show-Cps $base0)]"
Write-Host "=== 3. phone session ==="
[void](Adb-Shell @("am","force-stop","com.verify.remote"))
[void](Adb-Shell @("am","start","-n","com.verify.remote/.MainActivity",
"--es","host",$PhoneIp,"--ei","port","$Port","--es","code",$Code))
$deadline = (Get-Date).AddSeconds(25)
$ui = ""
while ((Get-Date) -lt $deadline) { Start-Sleep -Milliseconds 900; $ui = Get-Ui; if ($ui -match "RTT=") { break } }
if ($ui -notmatch "RTT=") { throw "phone did not connect" }
Write-Host "=== 4. show IME (tap keyboard button) ==="
$btns = [regex]::Matches($ui, '<node[^>]*class="android\.widget\.Button"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"')
if ($btns.Count -lt 1) { throw "keyboard button not found" }
$b0 = $btns[0]
$kx = [int](([int]$b0.Groups[1].Value + [int]$b0.Groups[3].Value) / 2)
$ky = [int](([int]$b0.Groups[2].Value + [int]$b0.Groups[4].Value) / 2)
[void](Adb-Shell @("input","tap","$kx","$ky"))
Start-Sleep -Seconds 3
$ime = (Adb-Shell @("dumpsys","input_method")).out
if ($ime -notmatch "mInputShown=true") { throw "IME not shown" }
if ($ime -notmatch "mHaveConnection=true") { throw "IME not bound to our InputConnection" }
Write-Host " IME: $([regex]::Match($ime,'mCurId=(\S+)').Groups[1].Value)"
Write-Host "=== 4b/5. pinyin -> IME candidate -> commitText (mode-tolerant) ==="
# Both IMEs on this device may remember an ENGLISH mode, in which case letters
# pass through without composing - which looks identical to "our pipeline lost
# the Hanzi" unless you check. The IME's key views are NOT visible to
# uiautomator (its dump contains only our app's nodes), so the ab/zh toggle is
# reached by a calibrated tap: iFlytek's key sits at display (73,1048) in a
# 1080x2400 capture => device (146,2096) at 2x. Verified 2026-10-30.
# The toggle is STATEFUL, so this is written as an idempotent attempt loop:
# try, and only if we see letters-without-Hanzi do we flip the mode and retry.
# A stray tap here changes a keyboard setting, nothing destructive.
$modeToggle = @(146, 2096)
$expect = -join @([char]0x4F60, [char]0x597D) # NI HAO
$attempt = 0
$new = ""
while ($attempt -lt 2) {
$attempt++
$base = Read-Received
Assert-Foreground $hwnd "attempt ${attempt}: before pinyin"
[void](Adb-Shell @("input","text","nihao"))
Start-Sleep -Milliseconds 1500
$mid = Read-Received
Write-Host " attempt ${attempt} after pinyin (composing): [$(Show-Cps $mid)]"
Assert-Foreground $hwnd "attempt ${attempt}: before SPACE"
[void](Adb-Shell @("input","keyevent","62"))
Start-Sleep -Seconds 2
$got = Read-Received
$new = if ($got.Length -ge $base.Length) { $got.Substring($base.Length) } else { "" }
Write-Host " attempt ${attempt} newly received: [$(Show-Cps $new)]"
if ($new -match [regex]::Escape($expect)) { break }
if ($attempt -eq 1) {
Write-Host " no Hanzi yet -> flipping IME language mode and retrying once"
[void](Adb-Shell @("input","tap","$($modeToggle[0])","$($modeToggle[1])"))
Start-Sleep -Seconds 2
}
}
$diag = [regex]::Match((Get-Ui), 'text="([^"]*RTT=[^"]*)"').Groups[1].Value
Write-Host " phone status: $diag"
Write-Host " expected to contain: $(Show-Cps $expect)"
if ($new -notmatch [regex]::Escape($expect)) {
Write-Host " GROUND TRUTH: [$(Show-Cps $new)]"
Write-Host " U+006E.. letters only => IME stayed in English mode (re-calibrate the toggle tap)"
throw "V6 CHINESE FAIL: IME did not deliver Hanzi through commitText"
}
Write-Host " V6 PASS: pinyin -> IME candidate -> commitText -> v3 Text -> Windows Hanzi"
Write-Host "=== 6. cleanup ==="
[void](Adb-Shell @("input","keyevent","4"))
New-Item -ItemType File -Force -Path $stopFile | Out-Null
Start-Sleep -Milliseconds 500
Write-Host "=== CHINESE IME PROOF COMPLETE ==="
tools/ime_commit_proof.ps1
+182
View File
@@ -0,0 +1,182 @@
# 2b V6/V7 on-device IME path: does a real IME's commitText reach the remote
# desktop? ASCII-only source (see personal lessons on PS 5.1 + BOM-less .ps1).
#
# Difference from phone_chain_proof.ps1: that one drives Activity.onKeyDown with
# adb keyevents (IME hidden). Here the IME is SHOWN, so keys are consumed by the
# input method and reach us only through RemoteInputConnection.commitText /
# sendKeyEvent. That is the path Chinese candidates take, and it is the risk R1
# item in the plan (vivo ROM + third-party IME behaviour was never observed).
#
# What this proves mechanically: IME is bound to our InputConnection and its
# committed text lands in a Windows window we own, via protocol v3.
# What it cannot prove: human candidate-list selection (needs a finger); see
# the result record - V6/V7 full judgement is listed as manual acceptance.
#
# Usage:
# powershell -ExecutionPolicy Bypass -File tools\ime_commit_proof.ps1
param(
[string]$PhoneIp = "192.168.1.180",
[int]$Port = 47800,
[string]$Code = "t",
[string]$Text = "hi"
)
$ErrorActionPreference = "Stop"
$repo = (Resolve-Path "$PSScriptRoot\..").Path
$adb = "D:\working\android\sdk\platform-tools\adb.exe"
$tmp = "$env:TEMP\remote-ime-proof"
New-Item -ItemType Directory -Force -Path $tmp | Out-Null
$pidFile = "$tmp\target.pid"
$stopFile = "$tmp\stop.flag"
$resultFile = "$tmp\received.txt"
Remove-Item $pidFile,$stopFile,$resultFile -ErrorAction SilentlyContinue
Add-Type -TypeDefinition @"
using System;
using System.Runtime.InteropServices;
public static class FgApi2 {
[DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow();
}
"@
function Adb {
param([Parameter(Mandatory)][string[]]$AdbArgs)
$old = $ErrorActionPreference
$ErrorActionPreference = "Continue"
try {
$out = & $adb @AdbArgs 2>$null
return @{ out = ($out -join "`n"); code = $LASTEXITCODE }
}
finally { $ErrorActionPreference = $old }
}
function Adb-Shell { param([Parameter(Mandatory)][string[]]$Cmd) return Adb -AdbArgs (@("shell") + $Cmd) }
function Wait-File {
param([string]$Path, [int]$TimeoutMs = 20000)
$t = 0
while (-not (Test-Path $Path) -and $t -lt ($TimeoutMs / 150)) { Start-Sleep -Milliseconds 150; $t++ }
if (-not (Test-Path $Path)) { throw "timeout waiting for $Path" }
}
function Assert-Foreground([IntPtr]$hwnd, [string]$where) {
$deadline = (Get-Date).AddMilliseconds(2000)
while ((Get-Date) -lt $deadline) {
if ([FgApi2]::GetForegroundWindow() -eq $hwnd) { return }
Start-Sleep -Milliseconds 30
}
throw "$where : foreground=$([FgApi2]::GetForegroundWindow()) want=$($hwnd.ToInt64()) - refusing to inject"
}
function Get-PhoneUiText {
[void](Adb-Shell @("uiautomator", "dump", "/sdcard/ui.xml"))
$r = Adb -AdbArgs @("pull", "/sdcard/ui.xml", "$tmp\ui.xml")
if ($r.code -ne 0 -or -not (Test-Path "$tmp\ui.xml")) { return "" }
return [System.IO.File]::ReadAllText("$tmp\ui.xml", [System.Text.Encoding]::UTF8)
}
function Read-Received {
if (-not (Test-Path $resultFile)) { return "" }
$b = [System.IO.File]::ReadAllBytes($resultFile)
$s = [System.Text.Encoding]::UTF8.GetString($b)
if ($s.Length -gt 0 -and $s[0] -eq [char]0xFEFF) { $s = $s.Substring(1) }
return $s
}
Write-Host "=== 0. retire earlier targets (touch only existing stop flags) ==="
$flags = @(Get-ChildItem -Path $env:TEMP -Recurse -Filter "stop.flag" -ErrorAction SilentlyContinue |
Where-Object { $_.FullName -like "*remote*proof*" })
if ($flags.Count -gt 0) {
Write-Host " retiring $($flags.Count)"
foreach ($f in $flags) { (Get-Item $f.FullName).LastWriteTime = Get-Date }
Start-Sleep -Seconds 3
}
Write-Host "=== 1. fresh target window ==="
Start-Process powershell.exe -ArgumentList "-NoProfile","-ExecutionPolicy","Bypass",`
"-File","$repo\tools\text_target.ps1","-PidFile",$pidFile,`
"-ResultFile",$resultFile,"-StopFile",$stopFile,"-GrabForeground"
Wait-File $pidFile
$parts = @((Get-Content $pidFile -Raw) -split "`n")
$tid = [int]$parts[0].Trim()
$hwnd = [IntPtr][long]$parts[1].Trim()
Assert-Foreground $hwnd "after launch"
Write-Host " target pid=$tid hwnd=$($hwnd.ToInt64())"
Write-Host "=== 2. agent freshness + phone session ==="
$agentExe = "$repo\target\release\agent.exe"
$ag = Get-Process agent -ErrorAction SilentlyContinue
if ($ag -and (Get-Item $agentExe).LastWriteTime -gt $ag.StartTime) {
Stop-Process -Name agent -Force -ErrorAction SilentlyContinue
Start-Sleep -Milliseconds 800; $ag = $null
}
if (-not $ag) {
Start-Process $agentExe -ArgumentList "--port",$Port,"--code",$Code -WindowStyle Hidden
Start-Sleep -Seconds 1
}
[void](Adb-Shell @("am", "force-stop", "com.verify.remote"))
[void](Adb-Shell @("am", "start", "-n", "com.verify.remote/.MainActivity",
"--es", "host", $PhoneIp, "--ei", "port", "$Port", "--es", "code", $Code))
$deadline = (Get-Date).AddSeconds(25)
$xml = ""
while ((Get-Date) -lt $deadline) { Start-Sleep -Milliseconds 900; $xml = Get-PhoneUiText; if ($xml -match "RTT=") { break } }
if ($xml -notmatch "RTT=") { throw "phone did not connect" }
Write-Host " $([regex]::Match($xml,'text="([^"]*RTT=[^"]*)"').Groups[1].Value)"
Write-Host "=== 3. show the soft keyboard (tap the app's keyboard button) ==="
# Button coordinates come from the live hierarchy. Our remote-state bar adds
# "keyboard" before "disconnect", so index 0 is the keyboard button - asserted
# rather than assumed by checking the count and printing what we tapped.
$btns = [regex]::Matches($xml, '<node[^>]*class="android\.widget\.Button"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"')
if ($btns.Count -lt 1) { throw "no buttons found in remote state" }
$b0 = $btns[0]
$kx = [int](([int]$b0.Groups[1].Value + [int]$b0.Groups[3].Value) / 2)
$ky = [int](([int]$b0.Groups[2].Value + [int]$b0.Groups[4].Value) / 2)
Write-Host " tapping first button ($kx,$ky)"
[void](Adb-Shell @("input", "tap", "$kx", "$ky"))
Start-Sleep -Seconds 3
$ime = (Adb-Shell @("dumpsys", "input_method")).out
$shown = ($ime -match "mInputShown=true")
$bound = ($ime -match "mHaveConnection=true")
$curId = [regex]::Match($ime, "mCurId=(\S+)").Groups[1].Value
Write-Host " IME: shown=$shown boundToUs=$bound cur=$curId"
if (-not $shown) { throw "soft keyboard did not appear" }
if (-not $bound) { throw "no InputConnection bound - our onCheckIsTextEditor/onCreateInputConnection path failed" }
Write-Host "=== 4. type through the IME, expect commitText -> v3 Text -> Windows ==="
$before = Read-Received
Assert-Foreground $hwnd "before IME text"
[void](Adb-Shell @("input", "text", $Text))
Start-Sleep -Milliseconds 1500
# Latin input in most Chinese IMEs commits without a candidate pick; if the IME
# instead left it composing, nudge with SPACE to force a commit.
$got = Read-Received
if ($got -notmatch [regex]::Escape($Text)) {
Write-Host " not committed yet (got [$got]); sending SPACE to accept candidate"
[void](Adb-Shell @("input", "keyevent", "62")) # KEYCODE_SPACE
Start-Sleep -Milliseconds 1500
$got = Read-Received
}
Write-Host " target received: [$got] (expected to contain [$Text])"
$diag = [regex]::Match((Get-PhoneUiText), 'text="([^"]*RTT=[^"]*)"').Groups[1].Value
Write-Host " phone status: $diag"
if ($got -notmatch [regex]::Escape($Text)) {
throw "IME COMMIT FAIL: nothing from the IME reached the Windows target"
}
Write-Host " PASS: real IME -> RemoteInputConnection -> v3 Text -> Windows window"
Write-Host "=== 5. back key must close keyboard first, not the session ==="
[void](Adb-Shell @("input", "keyevent", "4"))
Start-Sleep -Seconds 2
$ime2 = (Adb-Shell @("dumpsys", "input_method")).out
$xml2 = Get-PhoneUiText
$stillLive = ($xml2 -match "RTT=")
$kbHidden = ($ime2 -match "mInputShown=false")
Write-Host " after BACK: session live=$stillLive keyboard hidden=$kbHidden"
if (-not ($kbHidden -and $stillLive)) { throw "BACK did not behave as keyboard-first-then-session" }
Write-Host " PASS: first BACK dismissed the keyboard, session kept"
Write-Host "=== 6. cleanup ==="
New-Item -ItemType File -Force -Path $stopFile | Out-Null
Start-Sleep -Milliseconds 500
Write-Host "=== IME COMMIT PROOF COMPLETE ==="
tools/phone_chain_proof.ps1
+271
View File
@@ -0,0 +1,271 @@
# 2b full-chain proof (phone side): Android key -> v3 Text channel -> agent
# SendInput -> dedicated Windows target window, read back from that window.
# ASCII-only source (PS 5.1 + BOM-less = GBK corruption; see personal lessons).
# State checks use ASCII markers ("RTT=" / EditText presence) rather than
# matching the app's Chinese labels, for the same reason.
#
# Why ONE atomic script: SendInput lands on the FOREGROUND window, and Windows'
# foreground lock means a background process cannot simply reclaim it. This
# box's interactive GUI steals foreground between separate tool calls, so
# launch -> assert foreground -> drive phone -> read back must run without
# yielding. Every injection is preceded by a foreground assertion: if the target
# lost foreground we ABORT rather than inject into some other application.
#
# Prereqs: vivo device online via adb, app installed, v3 agent build present.
# Usage:
# powershell -ExecutionPolicy Bypass -File tools\phone_chain_proof.ps1
param(
[string]$PhoneIp = "192.168.1.180",
[int]$Port = 47800,
[string]$Code = "t"
)
$ErrorActionPreference = "Stop"
$repo = (Resolve-Path "$PSScriptRoot\..").Path
$adb = "D:\working\android\sdk\platform-tools\adb.exe"
$tmp = "$env:TEMP\remote-phone-proof"
New-Item -ItemType Directory -Force -Path $tmp | Out-Null
$pidFile = "$tmp\target.pid"
$stopFile = "$tmp\stop.flag"
$resultFile = "$tmp\received.txt"
Remove-Item $pidFile,$stopFile,$resultFile -ErrorAction SilentlyContinue
Add-Type -TypeDefinition @"
using System;
using System.Runtime.InteropServices;
public static class FgApi {
[DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow();
}
"@
function Adb {
param([Parameter(Mandatory)][string[]]$AdbArgs)
# adb writes progress ("1 file pulled...") to stderr; PS 5.1 with
# $ErrorActionPreference=Stop turns every stderr line into a terminating
# error. Same workaround build-android.ps1 documents: relax for native
# calls, judge by exit code and captured stdout.
$old = $ErrorActionPreference
$ErrorActionPreference = "Continue"
try {
$out = & $adb @AdbArgs 2>$null
return @{ out = ($out -join "`n"); code = $LASTEXITCODE }
}
finally { $ErrorActionPreference = $old }
}
function Adb-Shell {
param([Parameter(Mandatory)][string[]]$Cmd)
return Adb -AdbArgs (@("shell") + $Cmd)
}
function Wait-File {
param([string]$Path, [int]$TimeoutMs = 20000)
$t = 0
while (-not (Test-Path $Path) -and $t -lt ($TimeoutMs / 150)) { Start-Sleep -Milliseconds 150; $t++ }
if (-not (Test-Path $Path)) { throw "timeout waiting for $Path" }
}
function Assert-Foreground([IntPtr]$hwnd, [string]$where) {
# Bounded wait, not a single instant check: the target re-asserts itself
# every 60ms in proof mode, so a momentary steal by the interactive desktop
# is recoverable. Still refuse to inject if it never comes back - a stray
# injection lands in whatever window DOES hold foreground.
$deadline = (Get-Date).AddMilliseconds(1500)
while ((Get-Date) -lt $deadline) {
if ([FgApi]::GetForegroundWindow() -eq $hwnd) { return }
Start-Sleep -Milliseconds 30
}
throw "$where : foreground=$([FgApi]::GetForegroundWindow()) want=$hwnd after 1.5s - refusing to inject"
}
function Read-Received {
Wait-File $resultFile 8000
$b = [System.IO.File]::ReadAllBytes($resultFile)
$s = [System.Text.Encoding]::UTF8.GetString($b)
if ($s.Length -gt 0 -and $s[0] -eq [char]0xFEFF) { $s = $s.Substring(1) }
return $s
}
function Get-PhoneUiText {
[void](Adb-Shell @("uiautomator", "dump", "/sdcard/ui.xml"))
$r = Adb -AdbArgs @("pull", "/sdcard/ui.xml", "$tmp\ui.xml")
if ($r.code -ne 0 -or -not (Test-Path "$tmp\ui.xml")) { return "" }
return [System.IO.File]::ReadAllText("$tmp\ui.xml", [System.Text.Encoding]::UTF8)
}
function Wait-For-LiveSession([int]$Seconds, [ref]$outXml) {
$deadline = (Get-Date).AddSeconds($Seconds)
while ((Get-Date) -lt $deadline) {
Start-Sleep -Milliseconds 900
$x = Get-PhoneUiText
if ($x -match "RTT=") { $outXml.Value = $x; return $true }
}
$outXml.Value = (Get-PhoneUiText)
return $false
}
Write-Host "=== 0. stale-agent guard (same rule as the desktop proof) ==="
$agentExe = "$repo\target\release\agent.exe"
$ag = Get-Process agent -ErrorAction SilentlyContinue
if ($ag -and (Get-Item $agentExe).LastWriteTime -gt $ag.StartTime) {
Write-Host " running agent predates current build; restarting it"
Stop-Process -Name agent -Force -ErrorAction SilentlyContinue
Start-Sleep -Milliseconds 800
$ag = $null
}
if (-not $ag) {
Start-Process $agentExe -ArgumentList "--port",$Port,"--code",$Code -WindowStyle Hidden
Start-Sleep -Seconds 1
Write-Host " agent started (v3)"
}
Write-Host "=== 1. launch target window (own process, empty field) ==="
# Retire leftovers from earlier runs WITHOUT killing them. Killing a foreground
# window hands foreground to the next app (this box's interactive GUI), and the
# foreground lock then refuses a brand-new background process any claim - the
# new target could never take foreground back.
# Signal only stop files that ALREADY EXIST: creating a fresh one here would
# hit the stop path of the run we are about to start, and that target would
# close itself the moment it launched (seen 2026-10-30: pid reported then gone).
$existingFlags = @(Get-ChildItem -Path $env:TEMP -Recurse -Filter "stop.flag" -ErrorAction SilentlyContinue |
Where-Object { $_.FullName -like "*remote*proof*" })
if ($existingFlags.Count -gt 0) {
Write-Host " retiring $($existingFlags.Count) earlier target(s) via their own stop file"
foreach ($f in $existingFlags) { (Get-Item $f.FullName).LastWriteTime = Get-Date }
Start-Sleep -Seconds 3
}
Start-Process powershell.exe -ArgumentList "-NoProfile","-ExecutionPolicy","Bypass",`
"-File","$repo\tools\text_target.ps1","-PidFile",$pidFile,`
"-ResultFile",$resultFile,"-StopFile",$stopFile,"-GrabForeground"
Wait-File $pidFile
# pid file carries "pid<n><hwnd>": the target writes its own handle once the
# form is realized, which removes the Get-Process MainWindowHandle race.
$parts = @((Get-Content $pidFile -Raw) -split "`n")
$tid = [int]$parts[0].Trim()
$hwnd = [IntPtr][long]$parts[1].Trim()
if ($hwnd -eq 0) { throw "target reported a null window handle" }
Assert-Foreground $hwnd "after launch"
Write-Host " target pid=$tid hwnd=$($hwnd.ToInt64()) foreground=yes"
Write-Host "=== 2. start app, connect via intent extras ==="
[void](Adb-Shell @("am", "force-stop", "com.verify.remote"))
[void](Adb-Shell @("am", "start", "-n", "com.verify.remote/.MainActivity",
"--es", "host", $PhoneIp, "--ei", "port", "$Port", "--es", "code", $Code))
$xmlRef = ""
if (-not (Wait-For-LiveSession 25 ([ref]$xmlRef))) { throw "phone never reported a live session (no RTT marker)" }
$script:xml = $xmlRef
Write-Host " phone: $([regex]::Match($script:xml,'text="([^"]*RTT=[^"]*)"').Groups[1].Value)"
Assert-Foreground $hwnd "after connect"
Write-Host "=== 3. phone keys A B C -> Activity.onKeyDown -> v3 Text channel ==="
# 29/30/31 = KEYCODE_A/B/C. adb-injected keys arrive with a printable
# unicodeChar, so the app routes them to nativeSendText (IME hidden). That is
# the product path introduced in 2b, not a stand-in for it.
foreach ($k in 29,30,31) {
Assert-Foreground $hwnd "before keyevent $k"
[void](Adb-Shell @("input", "keyevent", "$k"))
Start-Sleep -Milliseconds 250
}
$deadline = (Get-Date).AddSeconds(10)
$got = ""
while ((Get-Date) -lt $deadline) {
$got = Read-Received
if ($got -eq "abc") { break }
Start-Sleep -Milliseconds 300
}
Write-Host " target received: [$got]"
if ($got -ne "abc") {
Write-Host " foreground now: $([FgApi]::GetForegroundWindow()) target: $hwnd"
throw "FULL-CHAIN FAIL: expected [abc] got [$got]"
}
Write-Host " PASS: Android key -> v3 Text -> agent KEYEVENTF_UNICODE -> Windows window"
Write-Host "=== 4. phone ENTER (66): control key uses the scancode channel ==="
# Enter carries no printable unicodeChar, so it must go down the WinScancode
# path. Observable acceptance: session survives and keeps streaming (a
# single-line TextBox swallows Enter, so glyph-level proof is not possible).
Assert-Foreground $hwnd "before ENTER"
[void](Adb-Shell @("input", "keyevent", "66"))
Start-Sleep -Milliseconds 900
$xmlRef = ""
if (-not (Wait-For-LiveSession 6 ([ref]$xmlRef))) { throw "session died after ENTER" }
Write-Host " session alive: $([regex]::Match($xmlRef,'text="([^"]*RTT=[^"]*)"').Groups[1].Value)"
Write-Host "=== 5. phone BACK (4): disconnect and return to form ==="
[void](Adb-Shell @("input", "keyevent", "4"))
Start-Sleep -Seconds 2
$xml2 = Get-PhoneUiText
if ($xml2 -match "RTT=") { throw "BACK did not disconnect (still live session)" }
if ($xml2 -notmatch "android\.widget\.EditText") { throw "BACK did not return to the form" }
Write-Host " PASS: disconnected, form visible again"
Write-Host "=== 6. connect again FROM THE FORM (V4 through real UI, not intent) ==="
# Form typing uses keyevent, not 'adb input text': with the Chinese IME active,
# text injection gets composed (ASCII '.' becomes U+3002). We disable the IME,
# restore it in finally, and never assume a Chinese IME is absent.
$origIme = (Adb-Shell @("settings", "get", "secure", "default_input_method")).out.Trim()
Write-Host " original IME: [$origIme]"
if ($origIme -and $origIme -ne "null") { [void](Adb-Shell @("ime", "disable", $origIme)) }
try {
$xml3 = Get-PhoneUiText
$fields = [regex]::Matches($xml3, '<node[^>]*class="android\.widget\.EditText"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"')
if ($fields.Count -lt 3) { throw "expected 3 form fields, found $($fields.Count)" }
function Center($m) {
$a = [int]$m.Groups[1].Value; $b = [int]$m.Groups[2].Value
$c = [int]$m.Groups[3].Value; $d = [int]$m.Groups[4].Value
return @(([int](($a + $c) / 2)), ([int](($b + $d) / 2)))
}
$ipC = Center $fields[0]; $codeC = Center $fields[2]
# KEYCODE_0=7..KEYCODE_9=16, PERIOD=56, DEL=67, T=48 (A=29 -> T=29+19;
# all from javap against android-34, not memory)
[void](Adb-Shell @("input", "tap", "$($ipC[0])", "$($ipC[1])"))
Start-Sleep -Milliseconds 400
for ($i = 0; $i -lt 24; $i++) { [void](Adb-Shell @("input", "keyevent", "67")) }
foreach ($k in @(8,16,9,56,8,13,15,56,8,56,8,15,7)) { # 192.168.1.180
[void](Adb-Shell @("input", "keyevent", "$k"))
}
[void](Adb-Shell @("input", "tap", "$($codeC[0])", "$($codeC[1])"))
Start-Sleep -Milliseconds 400
for ($i = 0; $i -lt 6; $i++) { [void](Adb-Shell @("input", "keyevent", "67")) }
[void](Adb-Shell @("input", "keyevent", "48"))
$xml3 = Get-PhoneUiText
# Attribute order in uiautomator dumps is text= ... class= ... bounds=;
# matching "class=...text=" silently finds nothing (seen 2026-10-30: printed
# an empty form line). Parse per-node and pull attributes independently.
$vals = @()
foreach ($n in [regex]::Matches($xml3, '<node[^>]*>')) {
$v = $n.Value
if ($v -match 'class="android\.widget\.EditText"') {
$vals += [regex]::Match($v, 'text="([^"]*)"').Groups[1].Value
}
}
# NOTE: an EMPTY EditText reports its hint as text (verified: text="被控端 IP"),
# so "field text == typed value" is the assertion, not "field text non-empty".
Write-Host " form now: $($vals -join ' | ')"
if ($vals.Count -lt 3) { throw "expected 3 fields after typing, got $($vals.Count)" }
if ($vals[0] -ne "192.168.1.180") { throw "V4 FAIL: IP field is [$($vals[0])], typing did not land" }
$btn = [regex]::Match($xml3, '<node[^>]*class="android\.widget\.Button"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"')
if (-not $btn.Success) { throw "connect button not found" }
$bc = Center $btn
Assert-Foreground $hwnd "before form connect"
[void](Adb-Shell @("input", "tap", "$($bc[0])", "$($bc[1])"))
$xmlRef = ""
if (-not (Wait-For-LiveSession 25 ([ref]$xmlRef))) {
$shown = [regex]::Matches($xmlRef, 'text="([^"]+)"') | ForEach-Object { $_.Groups[1].Value }
throw "V4 FAIL: form connect did not reach a live session. screen=[$($shown -join ' | ')]"
}
Write-Host " V4 PASS via form UI: $([regex]::Match($xmlRef,'text="([^"]*RTT=[^"]*)"').Groups[1].Value)"
}
finally {
if ($origIme -and $origIme -ne "null") {
[void](Adb-Shell @("ime", "enable", $origIme))
[void](Adb-Shell @("ime", "set", $origIme))
Write-Host " IME restored: $((Adb-Shell @("settings", "get", "secure", "default_input_method")).out.Trim())"
}
}
Write-Host "=== 7. stop target ==="
New-Item -ItemType File -Force -Path $stopFile | Out-Null
Start-Sleep -Milliseconds 600
Write-Host "=== PHONE CHAIN PROOF COMPLETE ==="
tools/punch_probe.py
+365
View File
@@ -0,0 +1,365 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""UDP 打洞探针 —— 两台不同网络的机器各跑一个，用来摸清你们线路的底牌。
回答四个问题：
1. 这条线路的 NAT 能不能打洞？
2. 要几轮才能打通？（轮数越少说明 NAT 越宽松）
3. 对方的源端口有没有被 NAT 改写？（判断是不是对称 NAT）
4. 有没有公网 IPv6？IPv6 直连通不通？
用法
----
A 机（假设公网 1.2.3.4）:
py punch_probe.py --peer 2.3.4.5:47800
B 机（假设公网 2.3.4.5）:
py punch_probe.py --peer 1.2.3.4:47800
IPv6（双方都有公网 v6 时）:
py punch_probe.py --peer [2408:8207:1234::1]:47800
先查自己的公网映射（走 STUN，不需要对方）:
py punch_probe.py --stun
被动监听（不发包，只看能不能收到对方打进来的包）:
py punch_probe.py
说明
----
- 两台机器必须**同时**启动，脚本靠「双方同时向对方发包」来穿透 NAT。
- 先各自查公网 IP（浏览器搜 "my ip" 或 --stun），再互相填。
- 只依赖 Python 标准库。
"""
import argparse
import socket
import struct
import sys
import threading
import time
MAGIC = b"PUNCHv1|"
# ---------------------------------------------------------------- 地址解析
def parse_addr(s):
"""解析 'ip:port' 或 '[v6]:port' -> (host, port, family)"""
s = s.strip()
if s.startswith("["):
end = s.index("]")
host = s[1:end]
port = int(s[end + 2:])
return host, port, socket.AF_INET6
host, _, p = s.rpartition(":")
return host, int(p), socket.AF_INET
def fmt_addr(sockaddr):
host, port = sockaddr[0], sockaddr[1]
if ":" in host:
return "[%s]:%d" % (host, port)
return "%s:%d" % (host, port)
# ---------------------------------------------------------------- STUN
def stun_mapped(server="stun.l.google.com", port=19302, timeout=3.0, sock=None):
"""最小 STUN Binding 请求，返回 (ip, port)；失败返回 None。
sock 不为 None 时复用它（用来测「同一个本地端口对另一个目标」的映射端口，
这是判断 NAT 是不是对称型的关键）。
"""
txid = b"\x5a" * 12
req = struct.pack(">HHI12s", 0x0001, 0, 0x2112A442, txid)
try:
infos = socket.getaddrinfo(server, port, 0, socket.SOCK_DGRAM)
except OSError:
return None
fam, sockaddr = infos[0][0], infos[0][4]
own = sock is None
s = socket.socket(fam, socket.SOCK_DGRAM) if own else sock
s.settimeout(timeout)
try:
s.sendto(req, sockaddr)
data, _ = s.recvfrom(2048)
except OSError:
return None
finally:
if own:
s.close()
if len(data) < 20:
return None
off = 20
while off + 4 <= len(data):
atype, alen = struct.unpack(">HH", data[off:off + 4])
val = data[off + 4:off + 4 + alen]
if atype == 0x0020 and len(val) >= 8: # XOR-MAPPED-ADDRESS
famv = val[1]
xport = struct.unpack(">H", val[2:4])[0] ^ 0x2112
if famv == 0x01 and len(val) >= 8:
ip = bytes(a ^ b for a, b in zip(val[4:8], b"\x21\x12\xA4\x42"))
return socket.inet_ntoa(ip), xport
if famv == 0x02 and len(val) >= 20:
mask = b"\x21\x12\xA4\x42" + txid
ip = bytes(a ^ b for a, b in zip(val[4:20], mask))
return socket.inet_ntop(socket.AF_INET6, ip), xport
off += 4 + alen + ((4 - alen % 4) % 4)
return None
# ---------------------------------------------------------------- 信令
def discover_peer(sock, server, room, timeout=60.0):
"""注册到房间，等服务器回推对方的公网地址。
注意：这里**复用同一个 socket**——服务器看到的源地址，就是 NAT 为
「本机 → 服务器」这条流分配的映射。对锥形 NAT 而言，它与「本机 → 对端」
的映射相同，所以拿它去打洞是有效的；对对称 NAT 则不同，打不通属预期。
"""
host, port = server[0], server[1]
deadline = time.time() + timeout
last_send = 0.0
while time.time() < deadline:
now = time.time()
if now - last_send > 1.0:
try:
sock.sendto(b"REGISTER|" + room.encode("utf-8"), (host, port))
except OSError as e:
print(" 注册发送失败：%s" % e)
return None
last_send = now
try:
data, _src = sock.recvfrom(2048)
except socket.timeout:
continue
except ConnectionResetError:
continue
except OSError as e:
if getattr(e, "winerror", None) == 10054:
continue
return None
txt = data.decode("utf-8", "replace").strip()
if txt.startswith("PEER|"):
h, p, _f = parse_addr(txt[5:])
return (h, p)
return None
# ---------------------------------------------------------------- 主逻辑
class Probe:
def __init__(self, bind_host, port, family):
self.sock = socket.socket(family, socket.SOCK_DGRAM)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
if family == socket.AF_INET6:
try:
self.sock.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1)
except OSError:
pass
self.sock.bind((bind_host, port))
self.sock.settimeout(0.2)
# Windows 特有坑：曾经发过包给对方，但对方端口没开 → 会收到 ICMP 端口不可达，
# 之后下一次 recv 直接抛 WSAECONNRESET(10054)。不打这个 ioctl，
# 接收循环会被"自己之前的失败发送"打断，之后再也收不到包。
# （Rust 实现同样要处理：tokio recv 返回 ConnectionReset 时不能退出循环）
if sys.platform == "win32":
try:
self.sock.ioctl(socket.SIO_UDP_CONNRESET, False)
except (AttributeError, OSError):
pass
self.port = self.sock.getsockname()[1]
self.stop = threading.Event()
self.first_src = None
self.packets = 0
self.junk = 0
def receiver(self, ignore=()):
"""接收线程。
只认「以 MAGIC 开头」且「来自对端而非信令服务器」的包。
信令服务器的 PEER/WAIT 回复绝不能算作打通——否则客户端反复注册引来的
回复会造成假阳性（已踩过：把服务器的 PEER 回复误判成打洞成功）。
"""
while not self.stop.is_set():
try:
data, src = self.sock.recvfrom(2048)
except socket.timeout:
continue
except ConnectionResetError:
# ICMP 端口不可达留下的错误，忽略并继续收（绝不能 break）
continue
except OSError as e:
if getattr(e, "winerror", None) == 10054:
continue
break
# ---- 过滤：非打洞包一律丢弃
if not data.startswith(MAGIC) or (ignore and src[:2] in ignore):
self.junk += 1
continue
if self.first_src is None:
self.first_src = src
print("\n ✅ 收到对方的包！")
print(" 对方源地址（NAT 之后）= %s" % fmt_addr(src))
if src[1] != self.port:
print(" 对方源端口 = %d（本机端口 %d）；若双方约定端口相同，"
"说明 NAT 改写了端口" % (src[1], self.port))
else:
print(" 对方源端口与约定端口一致 → NAT 保留了端口")
self.packets += 1
if self.packets <= 3:
print(" [%s] 收到 %d 字节" % (fmt_addr(src), len(data)))
def send_rounds(self, peer_sockaddr, rounds, interval):
for i in range(1, rounds + 1):
if self.stop.is_set():
return
payload = MAGIC + struct.pack(">d", time.time())
try:
self.sock.sendto(payload, peer_sockaddr)
except OSError as e:
print(" 发送失败：%s" % e)
return
if i == 1 or i % 5 == 0 or self.first_src is not None:
mark = "✅" if self.first_src is not None else " "
print(" %s 第 %2d/%d 轮已发出" % (mark, i, rounds))
if self.first_src is not None:
return
time.sleep(interval)
def main():
ap = argparse.ArgumentParser(add_help=True, description="UDP 打洞探针")
ap.add_argument("--peer", help="对方地址 ip:port 或 [v6]:port；不给则只被动监听")
ap.add_argument("--port", type=int, default=47800, help="本地端口（默认 47800）")
ap.add_argument("--rounds", type=int, default=25, help="发包轮数（默认 25）")
ap.add_argument("--interval", type=float, default=0.2, help="轮间隔秒（默认 0.2）")
ap.add_argument("--grace", type=float, default=2.5,
help="发完后继续监听多少秒等回包（默认 2.5）")
ap.add_argument("--server", help="信令服务器 ip:port（与 --room 配合，自动配对）")
ap.add_argument("--room", help="房间名；同一房间的两个客户端会被配对")
ap.add_argument("--wait", type=float, default=60.0,
help="等对方加入房间的秒数（默认 60）")
ap.add_argument("--stun", action="store_true", help="只查询自己的公网映射后退出")
ap.add_argument("--stun-port", type=int, default=0,
help="配合 --stun：用指定本地端口发 STUN（与打洞端口一致时才能对比映射）")
args = ap.parse_args()
# ---- 只查 STUN
if args.stun:
s = None
if args.stun_port:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("0.0.0.0", args.stun_port))
print("[STUN] 用本机端口 %d 查询（与打洞同端口）..." % args.stun_port)
else:
print("[STUN] 查询公网映射（临时端口）...")
r = stun_mapped(sock=s)
if s:
s.close()
if r:
print(" 对 Google STUN 的映射 = %s:%d" % r)
if args.stun_port:
print(" 本机端口 = %d" % args.stun_port)
print(" 下一步：对比信令服务器日志里记录的源端口")
print(" 两者相同 → 端点无关映射（锥形 NAT）→ 打洞大概率成功")
print(" 两者不同 → 对称 NAT → 打洞困难，需要端口预测或中继")
else:
print(" 查询失败（STUN 19302 可能被封；可稍后重试）")
return 0
# ---- 决定地址族与模式
peer = None
if args.peer:
host, pport, family = parse_addr(args.peer)
peer = (host, pport)
bind_host = "::" if family == socket.AF_INET6 else "0.0.0.0"
mode = "双向打洞（手工填地址）"
elif args.server and args.room:
_h, _p, family = parse_addr(args.server)
bind_host = "::" if family == socket.AF_INET6 else "0.0.0.0"
mode = "信令自动配对 + 打洞"
else:
family = socket.AF_INET
bind_host = "0.0.0.0"
mode = "被动监听"
probe = Probe(bind_host, args.port, family)
print("=" * 60)
print(" UDP 打洞探针 模式=%s" % mode)
print(" 本机监听 %s:%d" % (bind_host, probe.port))
if peer:
print(" 发往 %s （%s）" % (fmt_addr(peer + (0, 0)),
"IPv6" if family == socket.AF_INET6 else "IPv4"))
print("=" * 60)
# ---- 信令配对：不用自己查公网 IP，服务器看到的就是你的公网映射
if args.server and args.room and peer is None:
srv = parse_addr(args.server)[:2]
print("\n向信令服务器 %s 注册 room=%s ..." % (fmt_addr(srv), args.room))
found = discover_peer(probe.sock, srv, args.room, timeout=args.wait)
if found is None:
print(" ❌ 超时：没等到对方加入同一个房间")
probe.sock.close()
return 1
peer = found
print(" ✅ 配对成功，对方公网地址 = %s" % fmt_addr(found))
# 信令服务器的回复（PEER/WAIT）绝不能算作打通，接收线程必须把它排除掉
ignore = set()
if args.server:
_sh, _sp, _sf = parse_addr(args.server)
ignore.add((_sh, _sp))
t = threading.Thread(target=probe.receiver, args=(ignore,), daemon=True)
t.start()
try:
if peer:
print("\n开始同时发包（若对方也在发包，第 1~3 轮通常就通）...")
probe.send_rounds(peer, args.rounds, args.interval)
# 对方可能比我晚启动：发完后继续监听，否则会把"晚到的成功"误判成失败
if probe.first_src is None:
print("\n发包结束，继续监听 %.1f 秒等对方回包 ..." % args.grace)
waited = 0.0
while waited < args.grace and probe.first_src is None:
time.sleep(0.1)
waited += 0.1
else:
print("\n被动监听中，Ctrl+C 退出 ...")
while not probe.stop.is_set():
time.sleep(0.3)
except KeyboardInterrupt:
pass
finally:
probe.stop.set()
time.sleep(0.3)
probe.sock.close()
print("\n" + "-" * 60)
if probe.first_src is not None:
print("结论：✅ 打洞成功")
print(" 对方实际源地址 = %s" % fmt_addr(probe.first_src))
print(" 共收到 %d 个包" % probe.packets)
else:
print("结论：❌ 本轮没有收到对方的打洞包")
if probe.junk:
print(" （另外收到 %d 个非打洞包并已忽略：那是信令服务器的回复，不算打通）"
% probe.junk)
print(" 可能原因（按概率排序）：")
print(" 1. 一端或两端是对称型 NAT / 运营商级 NAT(CGNAT) → 需要端口预测或中继")
print(" 2. 一方根本没发包（对方没启动、或地址填错/端口填错）")
print(" 3. UDP 被防火墙拦截（企业网络常见）")
print(" 4. 双方没有真正同时发包（时间差超过 NAT 表项寿命）")
print(" 下一步：两端各跑一次 --stun，看看是不是落在 100.64~100.127 网段（CGNAT）")
print("-" * 60)
return 0 if probe.first_src is not None else 1
if __name__ == "__main__":
sys.exit(main())
tools/rendezvous.py
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""极简 rendezvous（信令）服务器 —— POC 用
作用只有一个：**告诉双方「对方看到的你」是什么地址**。
双方注册到同一个房间后，服务器把各自观察到的公网地址回推给对方，
之后双方直接打洞，服务器不再参与（也不承载任何媒体流量）。
用法（跑在有公网 IP 的机器上，比如 43.140.215.244）:
py rendezvous.py --port 47999
客户端：
py punch_probe.py --server <服务器IP>:47999 --room test01 --port 47800
"""
import argparse
import socket
import sys
import time
# 房间成员多久没动静就清掉
MEMBER_TTL = 30.0
def fmt(sockaddr):
host, port = sockaddr[0], sockaddr[1]
return "[%s]:%d" % (host, port) if ":" in host else "%s:%d" % (host, port)
def main():
ap = argparse.ArgumentParser(description="极简 rendezvous 服务器")
ap.add_argument("--bind", default="0.0.0.0")
ap.add_argument("--port", type=int, default=47999)
args = ap.parse_args()
# 重定向到文件时 stdout 会全缓冲，日志要等进程退出才出现；强制行缓冲
try:
sys.stdout.reconfigure(line_buffering=True)
except (AttributeError, OSError):
pass
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((args.bind, args.port))
print("[rendezvous] 监听 udp://%s:%d" % (args.bind, args.port))
print("[rendezvous] 房间成员 TTL = %.0fs，Ctrl+C 退出\n" % MEMBER_TTL)
rooms = {} # room -> [(addr, last_seen)]
try:
while True:
data, src = sock.recvfrom(2048)
txt = data.decode("utf-8", "replace").strip()
now = time.time()
if not txt.startswith("REGISTER|"):
continue
room = txt.split("|", 1)[1].strip()
if not room:
continue
# 清掉过期成员 + 去掉本机的旧记录（支持重复注册）
members = [(a, t) for (a, t) in rooms.get(room, [])
if now - t < MEMBER_TTL and a[1] != src[1]]
members.append((src, now))
rooms[room] = members
print("[rendezvous] %s 注册 room=%s (现有 %d 人) 源地址=%s"
% (fmt(src), room, len(members), fmt(src)))
if len(members) >= 2:
a1, _ = members[0]
a2, _ = members[1]
sock.sendto(b"PEER|" + fmt(a2).encode(), a1)
sock.sendto(b"PEER|" + fmt(a1).encode(), a2)
print("[rendezvous] -> 配对成功：%s <==> %s"
% (fmt(a1), fmt(a2)))
else:
sock.sendto(b"WAIT", src)
print("[rendezvous] -> 等待第二个人加入 ...")
except KeyboardInterrupt:
print("\n[rendezvous] 退出")
finally:
sock.close()
return 0
if __name__ == "__main__":
sys.exit(main())
tools/shot.ps1
+23
View File
@@ -0,0 +1,23 @@
# Byte-safe device screenshot helper.
# ASCII-only. PS 5.1 `>` redirection writes UTF-16, which corrupts PNG bytes
# (measured 2026-10-30: `adb exec-out screencap -p > x.png` produced a file
# starting with FF FE). Capture on-device and pull instead.
param(
[Parameter(Mandatory = $true)][string]$Out,
[string]$Device = "/sdcard/shot.png"
)
$ErrorActionPreference = "Stop"
$adb = "D:\working\android\sdk\platform-tools\adb.exe"
$old = $ErrorActionPreference
$ErrorActionPreference = "Continue"
try {
& $adb shell screencap -p $Device | Out-Null
if ($LASTEXITCODE -ne 0) { throw "screencap failed ($LASTEXITCODE)" }
& $adb pull $Device $Out | Out-Null
if ($LASTEXITCODE -ne 0) { throw "pull failed ($LASTEXITCODE)" }
}
finally { $ErrorActionPreference = $old }
$b = [System.IO.File]::ReadAllBytes($Out)
$sig = ($b[0..3] | ForEach-Object { $_.ToString("X2") }) -join " "
if ($sig -ne "89 50 4E 47") { throw "not a PNG (sig=$sig): $Out" }
Write-Host ("OK {0} ({1:N0} bytes)" -f (Split-Path $Out -Leaf), $b.Length)
tools/text_injection_proof.ps1
+171
View File
@@ -0,0 +1,171 @@
# 2b V6/V7 proof: v3 text injection into a dedicated WinForms target window,
# read back from that process itself.
# ASCII-only source (PS 5.1 decodes BOM-less .ps1 as GBK; see personal lessons).
# Non-ASCII payloads are built from explicit code points, never literals.
#
# Usage:
# powershell -ExecutionPolicy Bypass -File tools\text_injection_proof.ps1
#
# Chain: viewer-core text_check --units <utf16> -> v3 agent SendInput
# KEYEVENTF_UNICODE -> foreground WinForms TextBox -> self-reported text.
#
# Why NOT Notepad (2026-10-30): Windows 11 Notepad hosts all tabs in ONE
# process and this dev box had a user document open in it - injecting "into
# Notepad" would mean injecting into the user's real file. The WinForms target
# is a separate process we own, opened on an empty field.
#
# Foreground handling (measured, not assumed): a form with TopMost=true that
# is activated at startup KEEPS foreground for seconds; AppActivate /
# SetForegroundWindow called from here actually STEALS it away. So we assert
# the target is foreground before injecting, and refuse to inject otherwise -
# a stray injection would land in whatever window was foreground.
$ErrorActionPreference = "Stop"
# Repo root derived from this script's location (tools/ is one level under the
# repo root) so the proof runs on any clone, not just the author's D: drive.
$repo = (Resolve-Path "$PSScriptRoot\..").Path
$tmp = "$env:TEMP\remote-text-proof"
$base = "D:\working\environment"
$env:RUSTUP_HOME = "$base\rust"
$env:CARGO_HOME = "$base\cargo"
$env:PATH = "$base\mingw64\bin;$base\cargo\bin;" + $env:PATH
Add-Type -TypeDefinition @"
using System;
using System.Runtime.InteropServices;
public static class WinFg {
[DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow();
}
"@
New-Item -ItemType Directory -Force -Path $tmp | Out-Null
$pidFile = "$tmp\target.pid"
$stopFile = "$tmp\stop.flag"
$resultFile = "$tmp\received.txt"
Remove-Item $pidFile,$stopFile,$resultFile -ErrorAction SilentlyContinue
function Wait-File {
param([string]$Path, [int]$TimeoutMs = 15000)
$tries = 0
while (-not (Test-Path $Path) -and $tries -lt ($TimeoutMs / 150)) {
Start-Sleep -Milliseconds 150; $tries++
}
if (-not (Test-Path $Path)) { throw "file did not appear: $Path" }
}
function Read-Received {
Wait-File $resultFile
$bytes = [System.IO.File]::ReadAllBytes($resultFile)
$text = [System.Text.Encoding]::UTF8.GetString($bytes)
if ($text.Length -gt 0 -and $text[0] -eq [char]0xFEFF) { $text = $text.Substring(1) }
return $text
}
function CodePointsToString([int[]]$cps) {
-join ($cps | ForEach-Object { [char]$_ })
}
function Show-Text([string]$s) {
# ASCII-safe console preview: non-ASCII rendered as U+XXXX
($s.ToCharArray() | ForEach-Object {
if ([int]$_ -ge 32 -and [int]$_ -lt 127) { $_ } else { "U+{0:X4}" -f [int]$_ }
}) -join ""
}
function Assert-TargetForeground([IntPtr]$hwnd) {
$fg = [WinFg]::GetForegroundWindow()
if ($fg -ne $hwnd) {
throw "foreground is $fg, not target $hwnd - refusing to inject (would hit another app)"
}
}
function Send-Units([int[]]$cps, [IntPtr]$hwnd, [int]$delayMs = 1500) {
$units = $cps -join ","
$exe = "$repo\target\debug\examples\text_check.exe"
if (-not (Test-Path $exe)) { throw "missing $exe (cargo build -p viewer-core --example text_check)" }
Assert-TargetForeground $hwnd
& $exe --port 47800 --code t --units $units --delay $delayMs --hold 800
$code = $LASTEXITCODE
Assert-TargetForeground $hwnd # 注入窗口在 --delay 期间必须仍是靶窗
if ($code -ne 0) { throw "text_check exited $code" }
Start-Sleep -Milliseconds 500
}
Write-Host "=== 0. ensure CURRENT v3 agent is the one running ==="
$ag = Get-Process agent -ErrorAction SilentlyContinue
$agentExe = "$repo\target\release\agent.exe"
if ($ag -and (Get-Item $agentExe).LastWriteTime -gt $ag.StartTime) {
Write-Host " running agent predates current build; restarting"
Stop-Process -Name agent -Force -ErrorAction SilentlyContinue
Start-Sleep -Milliseconds 800
$ag = $null
}
if (-not $ag) {
Start-Process -FilePath $agentExe -ArgumentList "--port","47800","--code","t" -WindowStyle Hidden
Start-Sleep -Seconds 1
Write-Host " agent started (v3)"
}
Write-Host "=== 1. launch WinForms target (own process, empty field) ==="
# NO -WindowStyle Hidden: the hidden host makes the form invisible, and an
# invisible window can never be foreground (measured 2026-10-30).
Start-Process powershell.exe -ArgumentList "-NoProfile","-ExecutionPolicy","Bypass",`
"-File","$repo\tools\text_target.ps1","-PidFile",$pidFile,`
"-ResultFile",$resultFile,"-StopFile",$stopFile
Wait-File $pidFile
# target.pid holds "pid<n><hwnd>" (see text_target.ps1 Shown handler); take line 1.
$parts = @((Get-Content $pidFile -Raw) -split "`n")
$tid = [int]$parts[0].Trim()
$tp = Get-Process -Id $tid
Start-Sleep -Milliseconds 1200
$hwnd = [IntPtr][long]$parts[1].Trim()
if ($hwnd -eq 0) { throw "target reported a null window handle" }
Write-Host " target pid=$tid hwnd=$hwnd title=[$($tp.MainWindowTitle)]"
Assert-TargetForeground $hwnd
Write-Host " target is foreground: yes"
Write-Host "=== 2. V6: four CJK BMP characters (Ni Hao Shi Jie) ==="
$cjk = @(0x4F60, 0x597D, 0x4E16, 0x754C)
Send-Units $cjk $hwnd
$got6 = Read-Received
$exp6 = CodePointsToString $cjk
Write-Host (" received [{0}] expected [{1}]" -f (Show-Text $got6), (Show-Text $exp6))
if ($got6 -ne $exp6) { throw "V6 FAIL" }
Write-Host " V6 PASS - BMP CJK arrived intact through v3 Text + KEYEVENTF_UNICODE"
Write-Host "=== 3. V7: exclaim + emoji U+1F600 (UTF-16 surrogate pair) appended ==="
$mix = @(0x0021, 0xD83D, 0xDE00)
Send-Units $mix $hwnd
$got7 = Read-Received
$exp7 = $exp6 + (CodePointsToString $mix)
Write-Host (" received len={0} [{1}]" -f $got7.Length, (Show-Text $got7))
Write-Host (" expected len={0} [{1}]" -f $exp7.Length, (Show-Text $exp7))
if ($got7 -ne $exp7) { throw "V7 FAIL - mismatch" }
if ($got7.Contains([char]0xFFFD)) { throw "V7 FAIL - U+FFFD present: pair got split" }
Write-Host " V7 PASS - surrogate pair survived as one character, no replacement char"
Write-Host "=== 4. negative guard: missing --units must NOT inject anything ==="
$before = Read-Received
$exe = "$repo\target\debug\examples\text_check.exe"
# Omit --units entirely (passing an empty string would be eaten by PowerShell
# argument handling and parsed as the next flag). viewer-core's parse_units
# rejects the degenerate input before any session is even opened.
# No 2>&1 here: viewer-core logs go to stderr by design; merging them turns
# every normal log line into a PowerShell error record.
& $exe --port 47800 --code t --delay 100 --hold 100 | Out-Null
$emptyCode = $LASTEXITCODE
Write-Host " exit=$emptyCode (expected non-zero: tool rejects empty --units)"
if ($emptyCode -eq 0) { throw "FAIL - empty units should be rejected by the tool" }
Start-Sleep -Milliseconds 400
$after = Read-Received
if ($after -ne $before) { throw "FAIL - target text changed unexpectedly: [$after]" }
Write-Host " guard PASS - no injection happened, target text unchanged"
Write-Host "=== 5. stop target ==="
New-Item -ItemType File -Force -Path $stopFile | Out-Null
$tries = 0
while ((Get-Process -Id $tid -ErrorAction SilentlyContinue) -and $tries -lt 40) {
Start-Sleep -Milliseconds 150; $tries++
}
Write-Host "=== PROOF COMPLETE: final target text = [$(Show-Text (Read-Received))] ==="
tools/text_target.ps1
+110
View File
@@ -0,0 +1,110 @@
# 2b V6/V7 proof target: a dedicated WinForms window we own.
# ASCII-only source (PS 5.1 decodes BOM-less .ps1 as GBK; see personal lessons).
#
# Launched by text_injection_proof.ps1. Shows a topmost empty TextBox, writes
# its current text to ResultFile (UTF-8 with BOM) on every change, exits when
# StopFile appears. The proof injects through the remote-control pipeline and
# polls ResultFile - so "what arrived on the controlled desktop" is reported
# by the receiving application itself, not by a screenshot or OCR.
param(
[Parameter(Mandatory = $true)][string]$PidFile,
[Parameter(Mandatory = $true)][string]$ResultFile,
[Parameter(Mandatory = $true)][string]$StopFile,
# SendInput lands on whatever is FOREGROUND. When the injector is driven
# from another machine (phone -> agent -> this desktop), nothing in this
# process keeps our window on top, so an unrelated window could eat the
# injection. With this switch the form re-asserts foreground every tick and
# the caller still asserts foreground before injecting.
[switch]$GrabForeground
)
$ErrorActionPreference = "Stop"
Add-Type -AssemblyName System.Windows.Forms
Add-Type -AssemblyName System.Drawing
# Event handler scriptblocks resolve script-scoped vars; make that explicit
# instead of relying on the param locals staying on a visible scope chain.
$script:PidFile = $PidFile
$script:ResultFile = $ResultFile
$script:StopFile = $StopFile
$script:lastWritten = $null
$form = New-Object System.Windows.Forms.Form
$form.Text = "RemoteTextTarget"
$form.ClientSize = New-Object System.Drawing.Size(420, 140)
$form.TopMost = $true
$form.StartPosition = "Manual"
$form.Location = New-Object System.Drawing.Point(80, 80)
$tb = New-Object System.Windows.Forms.TextBox
$tb.Multiline = $true
$tb.Font = New-Object System.Drawing.Font("Segoe UI", 16.0)
$tb.Dock = "Fill"
$form.Controls.Add($tb)
$script:GrabForeground = $GrabForeground
if ($script:GrabForeground) {
Add-Type -TypeDefinition @"
using System;
using System.Runtime.InteropServices;
public static class OwnForeground {
[DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow();
[DllImport("user32.dll")] public static extern bool SetForegroundWindow(IntPtr h);
[DllImport("user32.dll")] public static extern uint GetWindowThreadProcessId(IntPtr h, IntPtr pid);
[DllImport("kernel32.dll")] public static extern uint GetCurrentThreadId();
[DllImport("user32.dll")] public static extern bool AttachThreadInput(uint a, uint b, bool f);
// Deliberately NO keybd_event(VK_MENU). The usual 'tap ALT to beat the
// foreground lock' trick corrupts exactly what we measure: after ALT, Windows
// treats the next character as a menu MNEMONIC, so the first injected
// character is eaten. Measured 2026-10-30: ALT re-asserting every 60ms turned
// three injected characters into only 'b' in the target.
// Attaching input queues is a lock workaround that synthesises no key input.
public static bool Claim(IntPtr h) {
uint fgThread = GetWindowThreadProcessId(GetForegroundWindow(), IntPtr.Zero);
uint myThread = GetCurrentThreadId();
AttachThreadInput(myThread, fgThread, true);
bool ok = SetForegroundWindow(h);
AttachThreadInput(myThread, fgThread, false);
return ok && GetForegroundWindow() == h;
}
public static IntPtr Now() { return GetForegroundWindow(); }
}
"@
}
$timer = New-Object System.Windows.Forms.Timer
$timer.Interval = 150
$timer.Add_Tick({
if (Test-Path $script:StopFile) {
$form.Close()
return
}
if ($script:GrabForeground) {
# Re-assert only when actually lost (unconditional claiming every tick
# would fight the user for the desktop outside a proof run).
if ([OwnForeground]::Now() -ne $form.Handle) {
[void]$form.Activate()
[void][OwnForeground]::Claim($form.Handle)
}
}
if ($tb.Text -ne $script:lastWritten) {
$script:lastWritten = $tb.Text
$enc = New-Object System.Text.UTF8Encoding($true)
[System.IO.File]::WriteAllText($script:ResultFile, $tb.Text, $enc)
}
if (-not $tb.Focused) { [void]$tb.Focus() }
})
$form.Add_Shown({
$form.Activate()
[void]$tb.Focus()
# Report BOTH pid and window handle: a proof caller resolving the handle via
# Get-Process races with window creation and can read 0. Handle is stable
# here because the form is realized by the time Shown fires.
[System.IO.File]::WriteAllText($script:PidFile, "$PID`n" + $form.Handle.ToInt64())
})
$timer.Start()
[void]$form.ShowDialog()
$timer.Stop()
vendor/nvenc/.gitignore
+2
View File
@@ -0,0 +1,2 @@
/target
/.vscode
vendor/nvenc/Cargo.toml
Vendored
+63
View File
@@ -0,0 +1,63 @@
# THIS FILE IS AUTOMATICALLY GENERATED BY CARGO
#
# When uploading crates to the registry Cargo will automatically
# "normalize" Cargo.toml files for maximal compatibility
# with all versions of Cargo and also rewrite `path` dependencies
# to registry (e.g., crates.io) dependencies.
#
# If you are reading this file be aware that the original Cargo.toml
# will likely look very different (and much more reasonable).
# See Cargo.toml.orig for the original contents.
[package]
edition = "2024"
rust-version = "1.85.1"
name = "nvenc"
version = "0.1.0"
authors = ["AlsoSylv"]
build = false
autolib = false
autobins = false
autoexamples = false
autotests = false
autobenches = false
description = "Wrapper for the nvidia NVENC API"
readme = "README.md"
keywords = [
"nvidia",
"graphics",
"media",
"video",
"ffi",
]
categories = [
"multimedia",
"external-ffi-bindings",
]
license = "MIT"
repository = "https://github.com/AlsoSylv/nvenc"
[lib]
name = "nvenc"
path = "src/lib.rs"
# 上游 README 的示例是已过时的 Linux GLX 用法（Session<NeedsConfig> 等类型已不存在），
# 直接跑 doctest 会失败。本项目只做 vendor 补丁、不维护文档示例，故关闭。
doctest = false
[[example]]
name = "simple_encode"
path = "examples/simple_encode.rs"
[dependencies.bitfields]
version = "1.0.2"
[dependencies.libloading]
version = "0.8.9"
[target."cfg(windows)".dependencies.windows]
version = "0.62"
features = [
"Win32_Graphics_Dxgi_Common",
"Win32_Graphics_Direct3D",
"Win32_Graphics_Direct3D11",
]
vendor/nvenc/Cargo.toml.orig
Generated Vendored
+24
View File
@@ -0,0 +1,24 @@
[package]
name = "nvenc"
version = "0.1.0"
edition = "2024"
authors = ["AlsoSylv"]
description = "Wrapper for the nvidia NVENC API"
rust-version = "1.85.1"
readme = "README.md"
repository = "https://github.com/AlsoSylv/nvenc"
license = "MIT"
keywords = ["nvidia", "graphics", "media", "video", "ffi"]
categories = ["multimedia", "external-ffi-bindings"]
[dependencies]
libloading = "0.8.9"
bitfields = "1.0.2"
[target.'cfg(windows)'.dependencies.windows]
version = "0.62"
features = [
"Win32_Graphics_Dxgi_Common",
"Win32_Graphics_Direct3D",
"Win32_Graphics_Direct3D11",
]
vendor/nvenc/LICENSE.txt
Vendored
+27
View File
@@ -0,0 +1,27 @@
Notice Regarding Standards. I do not provide a license or sublicense to
any Intellectual Property Rights relating to any standards, including but not
limited to any audio and/or video codec technologies such as MPEG-2, MPEG-4;
AVC/H.264; HEVC/H.265; AAC decode/FFMPEG; AAC encode/FFMPEG; VC-1; and MP3
(collectively, the "Media Technologies"). For clarity, you will pay any
royalties due for such third party technologies, which may include the Media
Technologies that are owed as a result of me providing the Software to you.
Copyright 2025 AlsoSylv
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
vendor/nvenc/README.md
Vendored
+34
View File
@@ -0,0 +1,34 @@
### If you like my work and want to support what I do, support me on Ko-Fi 💜!
[![Ko-Fi](https://img.shields.io/badge/Ko--fi-F16061?style=for-the-badge&logo=ko-fi&logoColor=white)](https://ko-fi.com/cyberite)
---
![Crates.io MSRV](https://img.shields.io/crates/msrv/nvenc?style=for-the-badge) ![Crates.io License](https://img.shields.io/crates/l/nvenc?style=for-the-badge) ![GitHub Repo stars](https://img.shields.io/github/stars/AlsoSylv/nvenc?style=for-the-badge)
The following example is for Linux GLX
```rust
use nvenc::{session::Session, encoder::Encoder};
fn main() {
// Setup GLX context
let session: Session<NeedsConfig> = Session::open_gl();
let (session, config): (Session<NeedsInit>, NVencPresetConfig)
= session.get_preset_config(
NV_ENC_CODEC_H264_GUID,
NV_ENC_PRESET_P3_GUID,
NVencTuningInfo::LowLatency
);
let init_params = nvenc::session::InitParams {
encode_guid: NV_ENC_H264_GUID,
preset_guid: NV_ENC_PRESET_P3_GUID,
resolution: [1920, 1080],
aspect_ratio: [16, 9],
frame_rate: [30, 1],
tuning_info: NVencTuningInfo::LowLatency,
buffer_format: NVencBufferFormat::ARGB,
encode_config: &mut config.preset_cfg,
enable_ptd: true,
};
let encoder: Encoder = session.init(init_params);
}
```
vendor/nvenc/examples/simple_encode.rs
Vendored
+203
View File
@@ -0,0 +1,203 @@
use std::fs::File;
use std::io::Write;
use std::time::Instant;
use nvenc::bitstream::BitStream;
use nvenc::session::InitParams;
use nvenc::{
session::Session,
sys::guids::{NV_ENC_CODEC_H264_GUID, NV_ENC_PRESET_P3_GUID},
};
#[cfg(windows)]
use windows::Win32::Graphics::Direct3D11::{ID3D11Device, ID3D11Texture2D};
#[cfg(windows)]
pub fn setup() -> ID3D11Device {
let dxgi_factory: windows::Win32::Graphics::Dxgi::IDXGIFactory =
unsafe { windows::Win32::Graphics::Dxgi::CreateDXGIFactory() }.unwrap();
let dxgi_adapter = unsafe { dxgi_factory.EnumAdapters(0) }.unwrap();
let desc = unsafe { dxgi_adapter.GetDesc() }.unwrap();
println!("{:?}", windows::core::HSTRING::from_wide(&desc.Description));
let mut device = None;
let mut device_context = None;
unsafe {
use windows::Win32::{
Foundation::HMODULE,
Graphics::{
Direct3D::{D3D_DRIVER_TYPE_UNKNOWN, D3D_FEATURE_LEVEL_11_0},
Direct3D11::{D3D11_CREATE_DEVICE_FLAG, D3D11_SDK_VERSION, D3D11CreateDevice},
},
};
D3D11CreateDevice(
&dxgi_adapter,
D3D_DRIVER_TYPE_UNKNOWN,
HMODULE(std::ptr::null_mut()),
D3D11_CREATE_DEVICE_FLAG(0),
Some(&[D3D_FEATURE_LEVEL_11_0]),
D3D11_SDK_VERSION,
Some(&raw mut device),
None,
Some(&raw mut device_context),
)
}
.unwrap();
let device = device.unwrap();
let _device_context = device_context.unwrap();
device
}
#[cfg(windows)]
fn image(device: &ID3D11Device) -> ID3D11Texture2D {
let data: [u8; 1920 * 4] = std::array::from_fn(|idx| match idx % 4 {
0 => 255,
1 => 0,
2 => 0,
3 => 255,
_ => unreachable!(),
});
let mut texture = None;
let desc = windows::Win32::Graphics::Direct3D11::D3D11_TEXTURE2D_DESC {
Width: 1920,
Height: 1080,
MipLevels: 1,
ArraySize: 1,
Format: windows::Win32::Graphics::Dxgi::Common::DXGI_FORMAT_R8G8B8A8_UNORM_SRGB,
SampleDesc: windows::Win32::Graphics::Dxgi::Common::DXGI_SAMPLE_DESC {
Count: 1,
Quality: 0,
},
Usage: windows::Win32::Graphics::Direct3D11::D3D11_USAGE_DEFAULT,
BindFlags: windows::Win32::Graphics::Direct3D11::D3D11_BIND_SHADER_RESOURCE.0 as u32,
CPUAccessFlags: windows::Win32::Graphics::Dxgi::Common::DXGI_CPU_ACCESS_NONE,
MiscFlags: 0,
};
let data = windows::Win32::Graphics::Direct3D11::D3D11_SUBRESOURCE_DATA {
pSysMem: data.as_ptr() as _,
SysMemPitch: 1920 * 4,
SysMemSlicePitch: 0,
};
unsafe {
device.CreateTexture2D(
&raw const desc,
Some(&raw const data),
Some(&raw mut texture),
)
}
.unwrap();
texture.unwrap()
}
fn main() {
let device = setup();
#[cfg(target_os = "linux")]
compile_error!("Linux is currently unsupported in this example");
#[cfg(windows)]
let session: Session<nvenc::session::NeedsConfig> = Session::open_dx(&device).unwrap();
assert!(
session
.get_encode_codecs()
.unwrap()
.contains(&NV_ENC_CODEC_H264_GUID)
);
assert!(
session
.get_encode_presets(NV_ENC_CODEC_H264_GUID)
.unwrap()
.contains(&NV_ENC_PRESET_P3_GUID)
);
let (session, mut config) = session
.get_encode_preset_config_ex(
NV_ENC_CODEC_H264_GUID,
NV_ENC_PRESET_P3_GUID,
nvenc::sys::enums::NVencTuningInfo::LowLatency,
)
.unwrap();
let texture = image(&device);
config.preset_cfg.rc_params.rate_control_mode = nvenc::sys::enums::NVencParamsRcMode::VBR;
config.preset_cfg.rc_params.average_bit_rate = 10_000_000;
config.preset_cfg.gop_len = 0xffffffff;
config.preset_cfg.frame_interval_p = 1;
println!("P-Frames: {}", config.preset_cfg.frame_interval_p);
println!("Bitrate: {}", config.preset_cfg.rc_params.average_bit_rate);
let init_params = InitParams {
encode_guid: NV_ENC_CODEC_H264_GUID,
preset_guid: NV_ENC_PRESET_P3_GUID,
aspect_ratio: [16, 9],
encode_config: &mut config.preset_cfg,
tuning_info: nvenc::sys::enums::NVencTuningInfo::LowLatency,
buffer_format: nvenc::sys::enums::NVencBufferFormat::ARGB,
frame_rate: [30, 1],
resolution: [1920, 1080],
enable_ptd: true,
max_encoder_resolution: [0, 0],
};
println!("FPS: 30");
println!("Len: 600 frames");
println!("Expected time: {}", 600 / 30);
println!("Codec: H264");
println!("Tuning Info {:?}:", init_params.tuning_info);
let encoder = session.init_encoder(init_params).unwrap();
let registered = encoder
.register_resource_dx11(&texture, nvenc::sys::enums::NVencBufferFormat::ARGB, 0)
.unwrap();
let (processed, to_use) = std::sync::mpsc::sync_channel::<BitStream>(2);
let (re_use, to_process) = std::sync::mpsc::sync_channel::<BitStream>(2);
processed
.send(encoder.create_bitstream_buffer().unwrap())
.unwrap();
processed
.send(encoder.create_bitstream_buffer().unwrap())
.unwrap();
let handle = std::thread::spawn({
move || {
let mut out = File::create("output.h264").unwrap();
while let Ok(output) = to_process.recv() {
let lock = output.try_lock(true).unwrap();
out.write_all(lock.as_slice()).unwrap();
drop(lock);
if let Err(_) = processed.send(output) {
break;
}
}
}
});
let instant = Instant::now();
let mut i = 0;
while i < 600 {
if let Ok(output) = to_use.recv() {
encoder
.encode_picture(
&registered,
&output,
i,
instant.duration_since(Instant::now()).as_millis() as u64,
nvenc::sys::enums::NVencBufferFormat::ARGB,
nvenc::sys::enums::NVencPicStruct::Frame,
nvenc::sys::enums::NVencPicType::P,
None,
)
.unwrap();
re_use.send(output).unwrap();
i += 1;
}
}
drop(to_use);
drop(re_use);
handle.join().unwrap();
}
vendor/nvenc/src/lib.rs
Vendored
+86
View File
@@ -0,0 +1,86 @@
#![doc = include_str!("../README.md")]
use std::{mem::MaybeUninit, sync::OnceLock};
use libloading::Library;
use crate::sys::{
function_table::NVencFunctionList,
result::{NVencError, NVencResult},
version::NVENC_API_LIST_VERSION,
};
mod safe;
pub use safe::*;
/// Contains the original C Types, functions, enums, and constants
pub mod sys;
#[cfg(all(target_arch = "x86_64", target_os = "windows"))]
pub const NVENC_DLL: &str = "nvEncodeAPI64.dll";
#[cfg(all(target_arch = "x86", target_os = "windows"))]
pub const NVENC_DLL: &str = "nvEncodeAPI.dll";
#[cfg(target_os = "linux")]
/// Platform specific DLL name, `libnvidia-encode.so.1` on Linux, `nvEncodeAPI64.dll` on Windows
pub const NVENC_DLL: &str = "libnvidia-encode.so.1";
#[cfg(all(target_arch = "x86", windows))]
#[macro_export]
macro_rules! stdcall {
(fn $args:tt $(-> $ret:tt)?) => { unsafe extern "stdcall" fn $args $(-> $ret)? };
}
#[doc(hidden)]
#[cfg(not(all(target_arch = "x86", windows)))]
#[macro_export]
macro_rules! stdcall {
(fn $args:tt $(-> $ret:ty)?) => { unsafe extern "C" fn $args $(-> $ret)? };
}
pub(crate) static LIBRARY: OnceLock<NVENCLibrary> = OnceLock::new();
/// Struct containing the library handle, init and get_max_version number functions
pub struct NVENCLibrary {
#[allow(unused)]
// Lib must stay alive for nvenc to be used
lib: Library,
init: stdcall!(fn(list: *mut MaybeUninit<NVencFunctionList>) -> NVencResult),
get_max_version: stdcall!(fn(version: *mut u32) -> NVencResult),
}
/// Attempt to init NVENC, returns an error if the library or loading functions cannot be found
pub fn nvenc_init() -> Result<&'static NVENCLibrary, libloading::Error> {
if let Some(lib) = LIBRARY.get() {
Ok(lib)
} else {
let lib = unsafe { Library::new(NVENC_DLL) }?;
let init: stdcall!(fn(function_list: *mut MaybeUninit<NVencFunctionList>) -> NVencResult) =
*unsafe { lib.get(b"NvEncodeAPICreateInstance") }?;
let get_max_version: stdcall!(fn(version: *mut u32) -> NVencResult) =
*unsafe { lib.get(b"NvEncodeAPIGetMaxSupportedVersion") }?;
Ok(LIBRARY.get_or_init(|| NVENCLibrary {
lib,
init,
get_max_version,
}))
}
}
impl NVENCLibrary {
/// Create a new instance of the NVencFunctionList,
pub fn create_instance(&self) -> Result<NVencFunctionList, NVencError> {
let mut list = MaybeUninit::zeroed();
unsafe { *(list.as_mut_ptr() as *mut u32) = NVENC_API_LIST_VERSION };
unsafe { (self.init)(&raw mut list) }.into_error().unwrap();
Ok(unsafe { list.assume_init() })
}
pub fn get_max_version(&self) -> Result<u32, NVencError> {
let mut version = 0;
unsafe { (self.get_max_version)(&raw mut version) }.into_error()?;
let minor = version & 0x0F;
let version = (version >> 4) | minor << 28;
Ok(version)
}
}
vendor/nvenc/src/safe/bitstream.rs
Vendored
+53
View File
@@ -0,0 +1,53 @@
use std::{ffi::c_void, sync::Arc};
use crate::{safe::encoder::EncoderInternal, sys::result::NVencError};
pub struct BitStream {
pub(crate) buffer: *mut c_void,
pub(crate) encoder: Arc<EncoderInternal>,
}
impl Drop for BitStream {
fn drop(&mut self) {
println!("Dropping bitstream buffer");
self.encoder.destroy_bitstream_buffer(self.buffer).unwrap();
}
}
unsafe impl Send for BitStream {}
impl BitStream {
/// Attempts to lock the bit stream, if `wait` is true it will wait
/// otherwise a `LockBusy` Error may be returned, in which case the
/// client should retry in a few milliseconds
pub fn try_lock(&self, wait: bool) -> Result<BitStreamLockGuard<'_>, NVencError> {
let lock = self.encoder.lock_bit_stream_buffer(self.buffer, wait)?;
Ok(BitStreamLockGuard {
buffer: self,
data_ptr: lock.bitstream_buffer,
data_len: lock.bitstream_size_in_bytes,
})
}
}
/// Holds a reference to the `BitStream` and holds the data and associated fields
pub struct BitStreamLockGuard<'a> {
buffer: &'a BitStream,
data_ptr: *mut c_void,
data_len: u32,
}
impl BitStreamLockGuard<'_> {
pub fn as_slice(&self) -> &[u8] {
unsafe { std::slice::from_raw_parts(self.data_ptr as _, self.data_len as _) }
}
}
impl<'a> Drop for BitStreamLockGuard<'a> {
fn drop(&mut self) {
self.buffer
.encoder
.unlock_bit_stream_buffer(self.buffer.buffer)
.unwrap();
}
}
vendor/nvenc/src/safe/encoder.rs
Vendored
+517
View File
@@ -0,0 +1,517 @@
use std::ffi::{CStr, c_void};
use std::marker::PhantomData;
use std::mem::ManuallyDrop;
use std::sync::Arc;
use std::{mem::MaybeUninit, ptr::NonNull};
#[cfg(windows)]
use windows::Win32::Graphics::Direct3D11::ID3D11Texture2D;
use crate::input_buffer::InputBuffer;
use crate::safe::bitstream::BitStream;
use crate::sys::enums::{
NVencBufferFormat, NVencBufferUsage, NVencInputResourceType, NVencMemoryHeap, NVencPicStruct,
NVencPicType,
};
use crate::sys::structs::{
Guid, NV_ENC_CREATE_INPUT_BUFFER_VER, NV_ENC_LOCK_INPUT_BUFFER_VER, NVencCodecPicParams,
NVencCreateInputBuffer, NVencLockInputBuffer, NVencPicParamsH264, NVencRegisterResource,
};
use crate::sys::{
enums::{NVencPicFlags, NVencTuningInfo},
function_table::NVencFunctionList,
result::NVencError,
structs::{
NV_ENC_CONFIG_VER, NV_ENC_CREATE_BITSTREAM_BUFFER_VER, NV_ENC_LOCK_BITSTREAM_VER,
NV_ENC_MAP_INPUT_RESOURCE_VER, NV_ENC_PIC_PARAMS_VER, NV_ENC_PRESET_CONFIG_VER,
NV_ENC_REGISTER_RESOURCE_VER, NVencCreateBitstreamBuffer, NVencInitializeParams,
NVencLockBitStream, NVencPicParams, NVencPresetConfig,
},
};
pub struct Encoder {
pub(crate) encoder: Arc<EncoderInternal>,
pub(crate) marker: PhantomData<*mut ()>,
}
pub trait EncoderInput {
#[doc(hidden)]
fn as_ptr(&self) -> *mut c_void;
fn pitch(&self) -> u32;
fn height(&self) -> u32;
fn width(&self) -> u32;
}
impl EncoderInput for RegisteredResource {
fn as_ptr(&self) -> *mut c_void {
self.mapped
}
fn pitch(&self) -> u32 {
self.pitch
}
fn height(&self) -> u32 {
self.height
}
fn width(&self) -> u32 {
self.width
}
}
impl EncoderInput for InputBuffer {
fn as_ptr(&self) -> *mut c_void {
self.buffer
}
fn pitch(&self) -> u32 {
self.width
}
fn height(&self) -> u32 {
self.height
}
fn width(&self) -> u32 {
self.width
}
}
impl Encoder {
#[cfg(windows)]
/// Registers a DX11 texture as an nvenc input resource
pub fn register_resource_dx11(
&self,
resource: &ID3D11Texture2D,
format: NVencBufferFormat,
pitch: u32,
) -> Result<RegisteredResource, NVencError> {
self.encoder.register_resource_dx(resource, format, pitch)
}
/// # Safety
/// `resource` must not be null
/// `ty` and `format` should match the original buffer
pub unsafe fn register_resource_raw(
&self,
resource: *mut c_void,
format: NVencBufferFormat,
usage: NVencBufferUsage,
ty: NVencInputResourceType,
resolution: [u32; 2],
pitch: u32,
) -> Result<*mut c_void, NVencError> {
unsafe {
self.encoder
.register_resource_raw(resource, format, usage, ty, resolution, pitch)
}
}
/// Takes an encoder inout, and outputs a bit stream that can be read back
pub fn encode_picture<I: EncoderInput>(
&self,
input: &I,
output: &BitStream,
frame_count: usize,
timestamp: u64,
format: NVencBufferFormat,
pic_struct: NVencPicStruct,
ty: NVencPicType,
codec_params: Option<CodecPicParams>,
) -> Result<(), NVencError> {
self.encoder.encode_picture(
input,
output,
frame_count,
timestamp,
format,
pic_struct,
ty,
codec_params,
)
}
/// Ends the encoder session
pub fn end_encode(&self) -> Result<(), NVencError> {
self.encoder.end_encode()
}
/// Creates a bitstream bfufer that can be used for output storage
pub fn create_bitstream_buffer(&self) -> Result<BitStream, NVencError> {
self.encoder.create_bitstream_buffer()
}
pub fn create_input_buffer(
&self,
width: u32,
height: u32,
memory_heap: NVencMemoryHeap,
format: NVencBufferFormat,
) -> Result<InputBuffer, NVencError> {
Ok(InputBuffer {
encoder: self.encoder.clone(),
buffer: self
.encoder
.create_input_buffer(width, height, memory_heap, format)?,
width,
height,
})
}
}
pub(crate) struct EncoderInternal {
pub(super) encoder: NonNull<c_void>,
pub(super) function_list: NVencFunctionList,
}
/// The Nvidia encoder is technically thread-safe
unsafe impl Send for EncoderInternal {}
/// The Nvidia encoder handles syncing via `lock_bit_stream` which is all that is allowed to be Send
unsafe impl Sync for EncoderInternal {}
impl Drop for EncoderInternal {
fn drop(&mut self) {
println!("Dropping encoder, this should happen last");
unsafe { (self.function_list.nvenc_destroy_encoder)(self.encoder.as_ptr()) };
}
}
impl EncoderInternal {
pub(crate) fn get_encode_preset_config_ex(
&self,
codec: Guid,
preset: Guid,
tuning_info: NVencTuningInfo,
) -> Result<NVencPresetConfig, NVencError> {
let mut config: std::mem::MaybeUninit<NVencPresetConfig> = std::mem::MaybeUninit::zeroed();
unsafe { &mut *config.as_mut_ptr() }.version = NV_ENC_PRESET_CONFIG_VER;
unsafe { &mut *config.as_mut_ptr() }.preset_cfg.version = NV_ENC_CONFIG_VER;
unsafe {
(self.function_list.nvenc_get_encode_preset_config_ex)(
self.encoder.as_ptr(),
codec,
preset,
tuning_info,
&raw mut config,
)
.into_error()?
};
Ok(unsafe { config.assume_init() })
}
pub(crate) fn init_encoder(
&self,
mut init_params: NVencInitializeParams,
) -> Result<(), NVencError> {
unsafe {
(self.function_list.nvenc_initialize_encoder)(
self.encoder.as_ptr(),
&raw mut init_params,
)
}
.into_error()?;
Ok(())
}
#[cfg(windows)]
pub fn register_resource_dx(
self: &Arc<EncoderInternal>,
resource: &ID3D11Texture2D,
format: NVencBufferFormat,
pitch: u32,
) -> Result<RegisteredResource, NVencError> {
use windows::Win32::Graphics::Direct3D11::D3D11_TEXTURE2D_DESC;
use windows::core::Interface;
let mut desc = D3D11_TEXTURE2D_DESC::default();
unsafe { resource.GetDesc(&raw mut desc) };
let registered = unsafe {
self.register_resource_raw(
resource.as_raw(),
NVencBufferFormat::Undefined,
NVencBufferUsage::Image,
NVencInputResourceType::DirectX,
[desc.Width, desc.Height],
pitch,
)?
};
let mapped = unsafe { self.map_input_resource_raw(registered, format)? };
Ok(RegisteredResource {
encoder: self.clone(),
pitch,
width: desc.Width,
height: desc.Height,
registered,
mapped,
})
}
pub unsafe fn map_input_resource_raw(
&self,
resource: *mut c_void,
format: NVencBufferFormat,
) -> Result<*mut c_void, NVencError> {
let mut map: crate::sys::structs::NVencMapInputResource = unsafe { std::mem::zeroed() };
map.version = NV_ENC_MAP_INPUT_RESOURCE_VER;
map.registered_resource = resource;
map.mapped_buffer_format = format;
unsafe {
(self.function_list.nvenc_map_input_resource)(self.encoder.as_ptr(), &raw mut map)
}
.into_error()?;
Ok(map.mapped_resource)
}
pub unsafe fn register_resource_raw(
&self,
resource: *mut c_void,
format: NVencBufferFormat,
usage: NVencBufferUsage,
ty: NVencInputResourceType,
resolution: [u32; 2],
pitch: u32,
) -> Result<*mut c_void, NVencError> {
let mut register_resource: NVencRegisterResource = unsafe { std::mem::zeroed() };
register_resource.buffer_format = format;
register_resource.buffer_usage = usage;
register_resource.resource_type = ty;
register_resource.version = NV_ENC_REGISTER_RESOURCE_VER;
register_resource.width = resolution[0];
register_resource.height = resolution[1];
register_resource.pitch = pitch;
register_resource.sub_resource_index = 0;
register_resource.resource_to_register = resource;
unsafe {
(self.function_list.nvenc_register_resource)(
self.encoder.as_ptr(),
&raw mut register_resource,
)
.into_error()
}?;
Ok(register_resource.registered_resource)
}
pub unsafe fn unmap_input_resource(
&self,
mapped_resource: *mut c_void,
) -> Result<(), NVencError> {
unsafe {
(self.function_list.nvenc_unmap_input_resource)(self.encoder.as_ptr(), mapped_resource)
}
.into_error()
}
pub unsafe fn unregister_resource(
&self,
registered_resource: *mut c_void,
) -> Result<(), NVencError> {
unsafe {
(self.function_list.nvenc_unregister_resource)(
self.encoder.as_ptr(),
registered_resource,
)
}
.into_error()
}
pub fn create_input_buffer(
&self,
width: u32,
height: u32,
memory_heap: NVencMemoryHeap,
format: NVencBufferFormat,
) -> Result<*mut c_void, NVencError> {
let mut input = NVencCreateInputBuffer {
version: NV_ENC_CREATE_INPUT_BUFFER_VER,
width: width,
height: height,
memory_heap,
buffer_fmt: format,
rsvd: 0,
input_buffer: std::ptr::null_mut(),
p_sys_mem_buffer: std::ptr::null_mut(),
rsvd1: unsafe { std::mem::zeroed() },
rsvd2: unsafe { std::mem::zeroed() },
};
unsafe {
(self.function_list.nvenc_create_input_buffer)(self.encoder.as_ptr(), &raw mut input)
.into_error()?
};
Ok(input.input_buffer)
}
pub fn destroy_input_buffer(&self, buffer: *mut c_void) -> Result<(), NVencError> {
unsafe { (self.function_list.nvenc_destroy_input_buffer)(self.encoder.as_ptr(), buffer) }
.into_error()
}
pub fn lock_input_buffer(&self, buffer: *mut c_void) -> Result<(*mut c_void, u32), NVencError> {
let mut lock = NVencLockInputBuffer {
version: NV_ENC_LOCK_INPUT_BUFFER_VER,
bitflags: 0,
input_buffer: buffer,
buffer_data_ptr: std::ptr::null_mut(),
pitch: 0,
rsvd1: unsafe { std::mem::zeroed() },
rsvd2: unsafe { std::mem::zeroed() },
};
unsafe {
(self.function_list.nvenc_lock_input_buffer)(self.encoder.as_ptr(), &raw mut lock)
.into_error()?;
}
Ok((lock.buffer_data_ptr, lock.pitch))
}
pub fn unlock_input_buffer(&self, locked_buffer: *mut c_void) -> Result<(), NVencError> {
unsafe {
(self.function_list.nvenc_unlock_input_buffer)(self.encoder.as_ptr(), locked_buffer)
.into_error()
}
}
pub fn encode_picture<I: EncoderInput>(
&self,
input: &I,
output: &BitStream,
frame_count: usize,
timestamp: u64,
format: NVencBufferFormat,
pic_struct: NVencPicStruct,
ty: NVencPicType,
codec_params: Option<CodecPicParams>,
) -> Result<(), NVencError> {
let mut params: MaybeUninit<NVencPicParams> = MaybeUninit::zeroed();
let sys_codec = match codec_params {
None => NVencCodecPicParams { rsvd: [0; 256] },
Some(CodecPicParams::H264(codec)) => NVencCodecPicParams {
h264_pic_params: ManuallyDrop::new(codec),
},
};
unsafe { &mut *params.as_mut_ptr() }.version = NV_ENC_PIC_PARAMS_VER;
unsafe { &mut *params.as_mut_ptr() }.input_width = input.width();
unsafe { &mut *params.as_mut_ptr() }.input_height = input.height();
unsafe { &mut *params.as_mut_ptr() }.input_pitch = input.pitch();
unsafe { &mut *params.as_mut_ptr() }.output_bitstream = output.buffer;
unsafe { &mut *params.as_mut_ptr() }.input_buffer = input.as_ptr();
unsafe { &mut *params.as_mut_ptr() }.buffer_format = format;
unsafe { &mut *params.as_mut_ptr() }.picture_struct = pic_struct;
// [本项目补丁] 只设 picture_type=IDR 不足以让 NVENC 真正输出 IDR 切片，
// 必须同时置 encode_pic_flags |= NV_ENC_PIC_FLAG_FORCEIDR。
//
// 上游 0.1.0 未设置该标志，导致运行期请求关键帧时编码器静默忽略、
// 继续输出 P 帧（实测：picture_type=IDR 但输出 NAL type=1）。
// 首帧之所以"看起来正常"，是因为 frame_idx==0 本来就强制 IDR，与本字段无关。
//
// 语义区分：IntraRefresh 走 ForceIntra，其余 IDR 类走 ForceIDR。
// 必须在 `picture_type = ty` 之前计算，否则 ty 已被移动。
let pic_flags = match &ty {
NVencPicType::IDR => NVencPicFlags::ForceIDR as u32,
NVencPicType::IntraRefresh => NVencPicFlags::ForceIntra as u32,
_ => 0,
};
unsafe { &mut *params.as_mut_ptr() }.picture_type = ty;
unsafe { &mut *params.as_mut_ptr() }.encode_pic_flags = pic_flags;
unsafe { &mut *params.as_mut_ptr() }.frame_idx = frame_count as u32;
unsafe { &mut *params.as_mut_ptr() }.input_time_stamp = timestamp;
unsafe { &mut *params.as_mut_ptr() }.codec_pic_params = sys_codec;
match unsafe {
(self.function_list.nvenc_encode_picture)(self.encoder.as_ptr(), &raw mut params)
.into_error()
} {
Err(err) => {
let ptr =
unsafe { (self.function_list.nvenc_get_last_error)(self.encoder.as_ptr()) };
println!("{:?}", unsafe { CStr::from_ptr(ptr) });
Err(err)
}
Ok(()) => Ok(()),
}
}
pub fn end_encode(&self) -> Result<(), NVencError> {
let mut params: MaybeUninit<NVencPicParams> = MaybeUninit::zeroed();
unsafe { &mut *params.as_mut_ptr() }.version = NV_ENC_PIC_PARAMS_VER;
unsafe { &mut *params.as_mut_ptr() }.encode_pic_flags = NVencPicFlags::Eos as u32;
unsafe {
(self.function_list.nvenc_encode_picture)(self.encoder.as_ptr(), &raw mut params)
.into_error()
}?;
Ok(())
}
pub(crate) fn lock_bit_stream_buffer(
&self,
buffer: *mut c_void,
wait: bool,
) -> Result<NVencLockBitStream, NVencError> {
let mut lock: MaybeUninit<NVencLockBitStream> = MaybeUninit::zeroed();
unsafe { &mut *lock.as_mut_ptr() }.version = NV_ENC_LOCK_BITSTREAM_VER;
unsafe { &mut *lock.as_mut_ptr() }.output_bit_stream = buffer;
unsafe { &mut *lock.as_mut_ptr() }
.bit_fields
.set_do_not_wait(!wait);
unsafe { (self.function_list.nvenc_lock_bit_stream)(self.encoder.as_ptr(), &raw mut lock) }
.into_error()?;
Ok(unsafe { lock.assume_init() })
}
pub(crate) fn unlock_bit_stream_buffer(&self, buffer: *mut c_void) -> Result<(), NVencError> {
unsafe { (self.function_list.nvenc_unlock_bit_stream)(self.encoder.as_ptr(), buffer) }
.into_error()
}
pub fn create_bitstream_buffer(self: &Arc<Self>) -> Result<BitStream, NVencError> {
let mut params: NVencCreateBitstreamBuffer = unsafe { std::mem::zeroed() };
params.version = NV_ENC_CREATE_BITSTREAM_BUFFER_VER;
unsafe {
(self.function_list.nvenc_create_bit_stream_buffer)(
self.encoder.as_ptr(),
&raw mut params,
)
}
.into_error()?;
Ok(BitStream {
buffer: params.bitstream_buffer,
encoder: self.clone(),
})
}
pub(crate) fn destroy_bitstream_buffer(&self, buffer: *mut c_void) -> Result<(), NVencError> {
unsafe {
(self.function_list.nvenc_destory_bit_stream_buffer)(self.encoder.as_ptr(), buffer)
}
.into_error()
}
}
pub enum CodecPicParams {
H264(NVencPicParamsH264),
}
pub struct RegisteredResource {
registered: *mut c_void,
mapped: *mut c_void,
pitch: u32,
width: u32,
height: u32,
encoder: Arc<EncoderInternal>,
}
impl Drop for RegisteredResource {
fn drop(&mut self) {
unsafe { self.encoder.unmap_input_resource(self.mapped).unwrap() };
unsafe { self.encoder.unregister_resource(self.registered).unwrap() };
}
}
vendor/nvenc/src/safe/input_buffer.rs
Vendored
+60
View File
@@ -0,0 +1,60 @@
use std::{ffi::c_void, sync::Arc};
use crate::{encoder::EncoderInternal, sys::result::NVencError};
pub struct InputBuffer {
pub(crate) encoder: Arc<EncoderInternal>,
pub(crate) buffer: *mut c_void,
pub(crate) width: u32,
pub(crate) height: u32,
}
impl InputBuffer {
pub fn lock(&self) -> Result<InputBufferLock<'_>, NVencError> {
let (lock_ptr, pitch) = self.encoder.lock_input_buffer(self.buffer)?;
Ok(InputBufferLock {
input_buffer: self,
lock_ptr,
pitch,
})
}
}
impl Drop for InputBuffer {
fn drop(&mut self) {
self.encoder.destroy_input_buffer(self.buffer).unwrap()
}
}
pub struct InputBufferLock<'a> {
input_buffer: &'a InputBuffer,
lock_ptr: *mut c_void,
pitch: u32,
}
impl InputBufferLock<'_> {
pub unsafe fn data_ptr(&self) -> *mut u8 {
self.lock_ptr as _
}
pub fn width(&self) -> u32 {
self.input_buffer.width
}
pub fn height(&self) -> u32 {
self.input_buffer.height
}
pub fn pitch(&self) -> u32 {
self.pitch
}
}
impl Drop for InputBufferLock<'_> {
fn drop(&mut self) {
self.input_buffer
.encoder
.unlock_input_buffer(self.lock_ptr)
.unwrap();
}
}
vendor/nvenc/src/safe/mod.rs
Vendored
+8
View File
@@ -0,0 +1,8 @@
/// Contains the type for the safe BitStream buffer wrapper
pub mod bitstream;
/// Wrapper for the NVEncoder after creating an active session
pub mod encoder;
/// Wrapper around an NVenc input buffer
pub mod input_buffer;
/// Type-state wrapper around session initialization
pub mod session;
vendor/nvenc/src/safe/session.rs
Vendored
+198
View File
@@ -0,0 +1,198 @@
use std::{marker::PhantomData, sync::Arc};
use crate::{
encoder::{Encoder, EncoderInternal},
sys::{
enums::{NVencBufferFormat, NVencDeviceType, NVencTuningInfo},
result::NVencError,
structs::{
Guid, NV_ENC_INITIALIZE_PARAMS_VER, NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER,
NVencConfig, NVencInitializeParams, NVencOpenEncodeSessionExParams, NVencPresetConfig,
},
version::NVENC_API_VERSION,
},
};
pub struct Session<T> {
encoder: Arc<EncoderInternal>,
p: PhantomData<T>,
}
pub struct NeedsConfig;
pub struct NeedsInit;
#[cfg(windows)]
use windows::core::Interface;
impl Session<NeedsConfig> {
#[cfg(windows)]
pub fn open_dx(device: &impl Interface) -> Result<Self, NVencError> {
let lib = crate::nvenc_init().unwrap();
let function_list = lib.create_instance()?;
let mut session_params: NVencOpenEncodeSessionExParams = unsafe { std::mem::zeroed() };
session_params.version = NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER;
session_params.device_type = NVencDeviceType::DirectX;
session_params.api_version = NVENC_API_VERSION;
session_params.device = device.as_raw();
let mut encoder = std::ptr::null_mut();
unsafe {
(function_list.nvenc_open_encode_session_ex)(&raw mut session_params, &raw mut encoder)
.into_error()
}?;
assert!(!encoder.is_null());
let encoder = EncoderInternal {
encoder: unsafe { std::ptr::NonNull::new_unchecked(encoder) },
function_list,
};
Ok(Self {
encoder: Arc::new(encoder),
p: PhantomData,
})
}
pub fn open_gl() -> Result<Self, NVencError> {
let lib = crate::nvenc_init().unwrap();
let function_list = lib.create_instance()?;
let mut session_params: NVencOpenEncodeSessionExParams = unsafe { std::mem::zeroed() };
session_params.version = NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER;
session_params.device_type = NVencDeviceType::OpenGL;
session_params.api_version = NVENC_API_VERSION;
session_params.device = std::ptr::null_mut();
let mut encoder = std::ptr::null_mut();
unsafe {
(function_list.nvenc_open_encode_session_ex)(&raw mut session_params, &raw mut encoder)
.into_error()
}?;
assert!(!encoder.is_null());
let encoder = EncoderInternal {
encoder: unsafe { std::ptr::NonNull::new_unchecked(encoder) },
function_list,
};
Ok(Self {
encoder: Arc::new(encoder),
p: PhantomData,
})
}
pub fn get_encode_codecs(&self) -> Result<Box<[Guid]>, NVencError> {
let mut count = 0;
unsafe {
(self.encoder.function_list.nvenc_get_encoder_guid_count)(
self.encoder.encoder.as_ptr(),
&raw mut count,
)
.into_error()
}?;
let mut array = vec![Guid::default(); count as usize];
unsafe {
(self.encoder.function_list.nvenc_get_encoder_guids)(
self.encoder.encoder.as_ptr(),
array.as_mut_ptr(),
count,
&raw mut count,
)
}
.into_error()?;
array.truncate(count as usize);
Ok(array.into())
}
pub fn get_encode_presets(&self, codec: Guid) -> Result<Box<[Guid]>, NVencError> {
let mut count = 0;
unsafe {
(self.encoder.function_list.nvenc_get_encode_preset_count)(
self.encoder.encoder.as_ptr(),
codec.clone(),
&raw mut count,
)
.into_error()
}?;
let mut array = vec![Guid::default(); count as usize];
unsafe {
(self.encoder.function_list.nvenc_get_encode_preset_guids)(
self.encoder.encoder.as_ptr(),
codec,
array.as_mut_ptr(),
count,
&raw mut count,
)
}
.into_error()?;
array.truncate(count as usize);
Ok(array.into())
}
pub fn get_encode_preset_config_ex(
self,
codec: Guid,
preset: Guid,
tuning_info: NVencTuningInfo,
) -> Result<(Session<NeedsInit>, NVencPresetConfig), NVencError> {
self.encoder
.get_encode_preset_config_ex(codec, preset, tuning_info)
.map(|config| {
(
Session {
encoder: self.encoder,
p: PhantomData,
},
config,
)
})
}
}
pub struct InitParams<'a> {
pub encode_guid: Guid,
pub preset_guid: Guid,
pub resolution: [u32; 2],
pub aspect_ratio: [u32; 2],
pub frame_rate: [u32; 2],
pub tuning_info: NVencTuningInfo,
pub buffer_format: NVencBufferFormat,
pub encode_config: &'a mut NVencConfig,
pub enable_ptd: bool,
pub max_encoder_resolution: [u32; 2],
// TODO: Support for async encoding and bit fields
}
impl Session<NeedsInit> {
pub fn init_encoder(self, init_params: InitParams) -> Result<Encoder, NVencError> {
// This sucks, lol
let raw_init_params = NVencInitializeParams {
encode_guid: init_params.encode_guid,
preset_guid: init_params.preset_guid,
encode_width: init_params.resolution[0],
encode_height: init_params.resolution[1],
dar_width: init_params.aspect_ratio[0],
dar_height: init_params.aspect_ratio[1],
frame_rate_num: init_params.frame_rate[0],
frame_rate_den: init_params.frame_rate[1],
tuning_info: init_params.tuning_info,
buffer_format: init_params.buffer_format,
encode_config: init_params.encode_config as _,
enable_ptd: init_params.enable_ptd as u32,
version: NV_ENC_INITIALIZE_PARAMS_VER,
enable_encode_async: 0,
bit_flags: crate::sys::structs::NVencInitializeParamsBitfields::new(),
priv_data_size: 0,
rsvd: 0,
priv_data: std::ptr::null_mut(),
max_encode_width: init_params.max_encoder_resolution[0],
max_encode_height: init_params.max_encoder_resolution[1],
max_me_hint_counts_per_block: [Default::default(), Default::default()],
num_state_buffers: 0,
output_stats_level: crate::sys::enums::NVencOutputStatsLevel::None,
rsvd1: unsafe { std::mem::zeroed() },
rsvd2: unsafe { std::mem::zeroed() },
};
self.encoder.init_encoder(raw_init_params)?;
Ok(Encoder {
encoder: self.encoder,
marker: PhantomData,
})
}
}
vendor/nvenc/src/sys/enums.rs
Vendored
+481
View File
@@ -0,0 +1,481 @@
#[repr(C)]
pub enum NVencParamsFrameFieldMode {
Frame = 0x01,
Field = 0x02,
MBAFF = 0x03,
}
#[repr(C)]
pub enum NVencParamsRcMode {
ConstQP = 0x00,
VBR = 0x01,
CBR = 0x02,
}
#[repr(C)]
pub enum NVencMultiPass {
Disabled = 0x00,
TwoPassQuarterResolution = 0x01,
TwoPassFullResolution = 0x02,
}
#[repr(C)]
pub enum NVencStateRestoreType {
Full = 0x01,
RateControl = 0x02,
Encode = 0x03,
}
#[repr(C)]
pub enum NVencOutputStatsLevel {
None = 0,
BlockLevel = 1,
RowLevel = 2,
}
#[repr(C)]
pub enum NVencEmphasisMapLevel {
Level0 = 0x0,
Level1 = 0x1,
Level2 = 0x2,
Level3 = 0x3,
Level4 = 0x4,
Level5 = 0x5,
}
#[repr(C)]
pub enum NVencQPMapMode {
Disabled = 0x0,
Emphasis = 0x1,
Delta = 0x2,
/// Currently unsupported as of NVENC 13.0
Map = 0x3,
}
#[repr(C)]
pub enum NVencPicStruct {
Frame = 0x01,
FieldTopBottom = 0x02,
FieldBottomTop = 0x03,
}
#[repr(C)]
pub enum NVencDisplayPicStruct {
Frame = 0x0,
FieldTopBottom = 0x1,
FieldBottomTop = 0x2,
Doubling = 0x3,
Tripling = 0x4,
}
#[repr(C)]
pub enum NVencPicType {
P = 0x00,
B = 0x01,
IDR = 0x03,
BI = 0x04,
Skipped = 0x05,
IntraRefresh = 0x06,
NonRefP = 0x07,
Switch = 0x08,
UNKNOWN = 0xFF,
}
#[repr(C)]
pub enum NVencMVPrecision {
Default = 0x00,
FullPel = 0x01,
HalfPel = 0x02,
QuarterPel = 0x03,
}
#[repr(C)]
pub enum NVencBufferFormat {
Undefined = 0x00000000,
NV12 = 0x00000001,
YV12 = 0x00000010,
IYUV = 0x00000100,
YUV444 = 0x00001000,
YUV420_10Bit = 0x00010000,
YUV444_10Bit = 0x00100000,
ARGB = 0x01000000,
ARGB10 = 0x02000000,
AYUV = 0x04000000,
ABGR = 0x10000000,
ABGR10 = 0x20000000,
U8 = 0x40000000,
NV16 = 0x40000001,
P210 = 0x40000002,
}
#[repr(C)]
pub enum NvencLevel {
AutoSelect = 0,
}
// TODO: Translate level macro
#[repr(C)]
pub enum H264Level {
H264_1 = 10,
H264_1b = 9,
H264_11 = 11,
H264_12 = 12,
H264_13 = 13,
H264_2 = 20,
H264_21 = 21,
H264_22 = 22,
H264_3 = 30,
H264_31 = 31,
H264_32 = 32,
H264_4 = 40,
H264_41 = 41,
H264_42 = 42,
H264_5 = 50,
H264_51 = 51,
H264_52 = 52,
H264_60 = 60,
H264_61 = 61,
H264_62 = 62,
}
#[repr(C)]
pub enum HEVCLevel {
HEVC1 = 30,
HEVC2 = 60,
HEVC21 = 63,
HEVC3 = 90,
HEVC31 = 93,
HEVC4 = 120,
HEVC41 = 123,
HEVC5 = 150,
HEVC51 = 153,
HEVC52 = 156,
HEVC6 = 180,
HEVC61 = 183,
HEVC62 = 186,
HEVCMain = 0,
HEVCHigh = 1,
}
#[repr(C)]
pub enum AV1Level {
// Also AV1_0
AV1_2 = 0,
// Also AV1_1
AV1_21 = 1,
AV1_22 = 2,
AV1_23 = 3,
AV1_3 = 4,
AV1_31 = 5,
AV1_32 = 6,
AV1_33 = 7,
AV1_4 = 8,
AV1_41 = 9,
AV1_42 = 10,
AV1_43 = 11,
AV1_5 = 12,
AV1_51 = 13,
AV1_52 = 14,
AV1_53 = 15,
AV1_6 = 16,
AV1_61 = 17,
AV1_62 = 18,
AV1_63 = 19,
AV1_7 = 20,
AV1_71 = 21,
AV1_72 = 22,
AV1_73 = 23,
AV1AutoSelect = 24,
}
#[repr(C)]
pub enum NVencPicFlags {
ForceIntra = 0x1,
ForceIDR = 0x2,
OutputSpspps = 0x4,
Eos = 0x8,
DisableEncStateAdvance = 0x10,
OuputReconFrame = 0x20,
}
#[repr(C)]
pub enum NVencMemoryHeap {
AutoSelect = 0,
VID = 1,
SystemCached = 2,
SystemUncached = 3,
}
#[repr(C)]
pub enum NVencBFrameRefMode {
Disabled = 0x0,
Each = 0x1,
Middle = 0x2,
}
#[repr(C)]
pub enum NVencH264EntropyCodingMode {
AutoSelect = 0x0,
Cabac = 0x1,
Cavlc = 0x2,
}
#[repr(C)]
pub enum NVencH264BDirectMode {
Autoselect = 0x0,
Disable = 0x1,
Temporal = 0x2,
Spatial = 0x3,
}
#[repr(C)]
pub enum NVencH264FMOMode {
AutoSelect = 0x0,
Enable = 0x1,
Disable = 0x2,
}
#[repr(C)]
pub enum NVencH264AdaptiveTransformMode {
AutoSelect = 0x0,
Disable = 0x1,
Enable = 0x2,
}
#[repr(C)]
pub enum NVencStereoPackingMode {
None = 0x0,
Checkerboard = 0x1,
ColInterleave = 0x2,
RowInterleave = 0x3,
SideBySide = 0x4,
TopBottm = 0x5,
FrameSeq = 0x6,
}
#[repr(C)]
pub enum NVencInputResourceType {
DirectX = 0x0,
CudaDeivcePtr = 0x1,
CudaArray = 0x2,
OpenGLTex = 0x3,
}
#[repr(C)]
pub enum NVencBufferUsage {
Image = 0x0,
MotionVector = 0x1,
BitStream = 0x2,
OutputRecon = 0x4,
}
#[repr(C)]
#[derive(Clone)]
pub enum NVencDeviceType {
DirectX = 0x0,
Cuda = 0x1,
OpenGL = 0x2,
}
#[repr(C)]
pub enum NVencNumRefFrames {
AutoSelect = 0x0,
Frames1 = 0x1,
Frames2 = 0x2,
Frames3 = 0x3,
Frames4 = 0x4,
Frames5 = 0x5,
Frames6 = 0x6,
Frames7 = 0x7,
}
#[repr(C)]
pub enum NVencTemporalFilterLevel {
Level0 = 0,
Level4 = 4,
}
#[repr(C)]
pub enum NVencCaps {
NumMaxBFrames,
SupportedRateControlModes,
FieldEncoding,
MonoChrome,
FMO,
QPELMV,
BDirectMode,
CABAC,
AdaptiveTransform,
StereoMVC,
MaxTemporalLayers,
HierarchicalBFrames,
LevelMax,
LevelMin,
SeperateColorPlane,
WidthMax,
HeightMax,
TemporalSVC,
DynResChnage,
DynBitrateChange,
DynForceConstQP,
DynRcModeChange,
SubframeReadBack,
ConstrainedEncoding,
IntraRefresh,
CustomVBVBufSize,
DynamicSliceMode,
Invalidation,
PreProcSupport,
AsyncEncodeSupport,
MbNumMax,
MbPerSecMax,
YUV444Encode,
LoselessEncode,
SupportSao,
MeOnlyMode,
LookAhead,
TemporalAQ,
MaxLtrFrames,
SupportWeightedPrediction,
DynamicQueryEncoderCapacity,
BFrameRefMode,
EmphasisLevelMap,
WidthMin,
HeightMin,
MultipleRefFrames,
AlphaLayerEncoding,
NumEncoderEngines,
SingleSliceIntraRefresh,
DisableEncStateAdvance,
OutputReconSurface,
OutputBlockStats,
SupportTemporalFilter,
LookAheadLevel,
UndirectionalB,
MVHEVCEncode,
SupportYUV422Encode,
ExposedCount,
}
#[repr(C)]
pub enum NVencHEVCCuSize {
AutoSelect = 0,
Size8x8 = 1,
Size16x16 = 2,
Size32x32 = 3,
Size64x64 = 4,
}
#[repr(C)]
pub enum NVencAV1PartSize {
AutoSelect = 0,
Size4x4 = 1,
Size8x8 = 2,
Size16x16 = 3,
Size32x32 = 4,
Size64x64 = 5,
}
#[repr(C)]
pub enum NVencVUIVideoFormat {
Component = 0,
Pal = 1,
NTSC = 2,
SeCam = 3,
Mac = 4,
Unspecified = 5,
}
#[repr(C)]
pub enum NVencVUIColorPrimaries {
Undefined = 0,
BT709 = 1,
Unspecified = 2,
Reserved = 3,
BT470M = 4,
BT470BG = 5,
SMPTE170M = 6,
SMPTE240M = 7,
FIlm = 8,
BT2020 = 9,
SMPTE428 = 10,
SMPTE431 = 11,
SMPTE432 = 12,
JEDECP22 = 22,
}
#[repr(C)]
pub enum NVencVUITransCharacteristics {
Undefined = 0,
BT709 = 1,
Unspecified = 2,
Reserved = 3,
BT470M = 4,
BT470BG = 5,
SMPTE170M = 6,
SMPTE240M = 7,
Linear = 8,
Log = 9,
LogSqrt = 10,
IEC61966_2_4 = 11,
SRGB = 13,
BT2020_10 = 14,
BT2020_12 = 15,
SMPTE2084 = 16,
SMPTE428 = 17,
AribStdB67 = 18,
}
#[repr(C)]
pub enum NVencVUIMatrixCoeffs {
RGB = 0,
BT709 = 1,
Unspecified = 2,
Reserved = 3,
FCC = 4,
BT470BG = 5,
SMPTE170M,
SMPTE240M,
YCGCO = 8,
BT2020NCL = 9,
BT2020CL = 10,
SMPTE2085 = 11,
}
#[repr(C)]
pub enum NVencLookAheadLevel {
Level0 = 0,
Level1 = 1,
Level2 = 2,
Level3 = 3,
AutoSelect = 15,
}
#[repr(C)]
pub enum NVencBitDepth {
Invalid = 0,
Depth8 = 8,
Depth10 = 10,
}
#[derive(Debug)]
#[repr(C)]
pub enum NVencTuningInfo {
Undefined = 0,
HighQuality = 1,
LowLatency = 2,
UltraLowLatency = 3,
Lossless = 4,
UltraHighQuality = 5,
Count,
}
#[repr(C)]
pub enum NVencSplitEncodeMode {
Audo = 0,
AutoForced = 1,
TwoForced = 2,
ThreeForced = 3,
FourForced = 4,
Disable = 15,
}
vendor/nvenc/src/sys/function_table.rs
Vendored
+214
View File
@@ -0,0 +1,214 @@
use std::{ffi::c_char, mem::MaybeUninit, os::raw::c_void};
use crate::{
stdcall,
sys::{
enums::{NVencBufferFormat, NVencTuningInfo},
result::NVencResult,
structs::{
Guid, NVencCapsParam, NVencCreateBitstreamBuffer, NVencCreateInputBuffer,
NVencInitializeParams, NVencLockBitStream, NVencLockInputBuffer, NVencMapInputResource,
NVencOpenEncodeSessionExParams, NVencPicParams, NVencPresetConfig,
NVencRegisterResource,
},
},
};
#[repr(C)]
#[derive(Clone)]
pub struct NVencFunctionList {
pub version: u32,
rsvd: u32,
#[deprecated]
nvenc_open_encode_session: NvencOpenEncodeSession,
pub nvenc_get_encoder_guid_count: NvencGetEncodeGuidCount,
nvenc_get_encoder_profile_guid_count: NvencGetEncodeProfileGuidCount,
nvenc_get_encoder_profile_guids: NvencGetEncodeProfileGuids,
pub nvenc_get_encoder_guids: NvencGetEncodeGuids,
nvenc_get_input_format_count: NvencGetInputFormatCount,
nvenc_get_input_formats: NvencGetInputFormats,
nvenc_get_encode_caps: NvencGetEncodeCaps,
pub nvenc_get_encode_preset_count: NvencGetEncodePresetCount,
pub nvenc_get_encode_preset_guids: NvencGetEncodePresetGuids,
nvenc_get_encode_preset_config: NvencGetEncodePresetConfig,
pub nvenc_initialize_encoder: NvencInitializeEncoder,
pub nvenc_create_input_buffer: NvencCreateInputBuffer,
pub nvenc_destroy_input_buffer: NvencDestroyInputBuffer,
pub nvenc_create_bit_stream_buffer: NvencCreateBitStreamBuffer,
pub nvenc_destory_bit_stream_buffer: NvencDestroyBitStreamBuffer,
pub nvenc_encode_picture: NvencEncodePicture,
pub nvenc_lock_bit_stream: NvencLockBitStream,
pub nvenc_unlock_bit_stream: NvencUnlockBitStream,
pub nvenc_lock_input_buffer: NvencLockInputBuffer,
pub nvenc_unlock_input_buffer: NvencUnlockInputBuffer,
nvenc_get_encode_stats: NvencGetEncodeStats,
nvenc_get_sequence_params: NvencGetSequenceParams,
nvenc_register_async_event: NvencRegisterAsyncEvent,
nvenc_unregister_async_event: NvencUnregisterAsyncEvent,
pub nvenc_map_input_resource: NvencMapInputResource,
pub nvenc_unmap_input_resource: NvencUnmapInputResource,
pub nvenc_destroy_encoder: NvencDestroyEncoder,
nvenc_invalidate_ref_frames: NvencInvalidateRefFrames,
pub nvenc_open_encode_session_ex: NvencOpenEncodeSessionEx,
pub nvenc_register_resource: NvencRegisterResource,
pub nvenc_unregister_resource: NvencUnRegisterResource,
nvenc_reconfigure_encoder: NvencReconfigureEncoder,
rsvd1: *mut c_void,
nvenc_create_mv_buffer: NvencCreateMVBuffer,
nvenc_destory_mv_buffer: NvencDestoryMVBuffer,
nvenc_run_motion_estimation_only: NvencRunMotionEstimationOnly,
pub nvenc_get_last_error: NvencGetLastError,
nvenc_set_io_cuda_streams: NvencSetIOCudaStreams,
pub nvenc_get_encode_preset_config_ex: NvencGetEncodePresetConfigEx,
nvenc_get_sequence_params_ex: NvencGetSequenceParamsEx,
nvenc_store_encoder_state: NvencStoreEncoderState,
nvenc_look_ahead_picture: NvencLookAheadPicture,
rsvd2: [*mut c_void; 275],
}
type NvencOpenEncodeSession =
stdcall!(fn(device: *mut c_void, device_type: i32, encoder: *mut *mut c_void) -> NVencResult);
type NvencGetEncodeGuidCount =
stdcall!(fn(encoder: *mut c_void, encoder_guid_count: *mut u32) -> NVencResult);
type NvencGetEncodeGuids = stdcall!(
fn(
encoder: *mut c_void,
guids: *mut Guid,
guid_array_size: u32,
guid_count: *mut u32,
) -> NVencResult
);
type NvencGetEncodeProfileGuidCount = stdcall!(
fn(
encoder: *mut c_void,
encode_guid: Guid,
encoder_profile_guid_count: *mut u32,
) -> NVencResult
);
type NvencGetEncodeProfileGuids = stdcall!(
fn(
encoder: *mut c_void,
profile_guids: *mut Guid,
guid_array_size: u32,
guid_count: *mut u32,
) -> NVencResult
);
type NvencGetInputFormatCount = stdcall!(
fn(encoder: *mut c_void, encoder_guid: Guid, input_fmt_count: *mut u32) -> NVencResult
);
type NvencGetInputFormats = stdcall!(
fn(
encoder: *mut c_void,
input_formats: *mut NVencBufferFormat,
input_format_array_size: u32,
input_format_count: *mut u32,
) -> NVencResult
);
type NvencGetEncodeCaps = stdcall!(
fn(
encoder: *mut c_void,
encoder_guid: Guid,
caps_param: *mut NVencCapsParam,
caps_val: *mut i32,
) -> NVencResult
);
type NvencGetEncodePresetCount = stdcall!(
fn(
encoder: *mut c_void,
encoder_guid: Guid,
encoder_preset_guid_count: *mut u32,
) -> NVencResult
);
type NvencGetEncodePresetGuids = stdcall!(
fn(
encoder: *mut c_void,
encode_guid: Guid,
preset_guids: *mut Guid,
guid_array_size: u32,
encoder_preset_guid_count: *mut u32,
) -> NVencResult
);
type NvencGetEncodePresetConfig = stdcall!(
fn(
encoder: *mut c_void,
encode_guid: Guid,
preset_guid: Guid,
preset_config: *mut NVencPresetConfig,
) -> NVencResult
);
type NvencGetEncodePresetConfigEx = stdcall!(
fn(
encoder: *mut c_void,
encode_guid: Guid,
preset_guid: Guid,
tuning_info: NVencTuningInfo,
preset_config: *mut MaybeUninit<NVencPresetConfig>,
) -> NVencResult
);
type NvencInitializeEncoder = stdcall!(
fn(encoder: *mut c_void, create_encode_params: *mut NVencInitializeParams) -> NVencResult
);
type NvencCreateInputBuffer = stdcall!(
fn(
encoder: *mut c_void,
create_input_buffer_params: *mut NVencCreateInputBuffer,
) -> NVencResult
);
type NvencDestroyInputBuffer =
stdcall!(fn(encoder: *mut c_void, input_buffer: *mut c_void) -> NVencResult);
type NvencCreateBitStreamBuffer = stdcall!(
fn(
encoder: *mut c_void,
create_bit_stream_buffer_params: *mut NVencCreateBitstreamBuffer,
) -> NVencResult
);
type NvencDestroyBitStreamBuffer =
stdcall!(fn(encoder: *mut c_void, bitstream_buffer: *mut c_void) -> NVencResult);
type NvencEncodePicture = stdcall!(
fn(encoder: *mut c_void, encode_pic_params: *mut MaybeUninit<NVencPicParams>) -> NVencResult
);
type NvencLockBitStream = stdcall!(
fn(
encoder: *mut c_void,
lock_bitstream_buffer_params: *mut MaybeUninit<NVencLockBitStream>,
) -> NVencResult
);
type NvencUnlockBitStream =
stdcall!(fn(encoder: *mut c_void, bitstream_buffer: *mut c_void) -> NVencResult);
type NvencLockInputBuffer = stdcall!(
fn(encoder: *mut c_void, lock_input_buffer_params: *mut NVencLockInputBuffer) -> NVencResult
);
type NvencUnlockInputBuffer =
stdcall!(fn(encoder: *mut c_void, input_buffer: *mut c_void) -> NVencResult);
type NvencGetEncodeStats = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencGetSequenceParams = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencRegisterAsyncEvent = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencUnregisterAsyncEvent = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencMapInputResource = stdcall!(
fn(encoder: *mut c_void, map_input_resource_params: *mut NVencMapInputResource) -> NVencResult
);
type NvencUnmapInputResource =
stdcall!(fn(encoder: *mut c_void, mapped_input_buffer: *mut c_void) -> NVencResult);
type NvencDestroyEncoder = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencInvalidateRefFrames = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencOpenEncodeSessionEx = stdcall!(
fn(
open_session_ex_params: *mut NVencOpenEncodeSessionExParams,
encoder: *mut *mut c_void,
) -> NVencResult
);
type NvencRegisterResource = stdcall!(
fn(encoder: *mut c_void, register_res_params: *mut NVencRegisterResource) -> NVencResult
);
type NvencUnRegisterResource =
stdcall!(fn(encoder: *mut c_void, registered_res: *mut c_void) -> NVencResult);
type NvencReconfigureEncoder = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencCreateMVBuffer = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencDestoryMVBuffer = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencRunMotionEstimationOnly = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencGetLastError = stdcall!(fn(encoder: *mut c_void) -> *const c_char);
type NvencSetIOCudaStreams = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencGetSequenceParamsEx = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencStoreEncoderState = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencLookAheadPicture = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
vendor/nvenc/src/sys/guids.rs
Vendored
+146
View File
@@ -0,0 +1,146 @@
use crate::sys::structs::Guid;
pub const NV_ENC_CODEC_H264_GUID: Guid = Guid::from_values(
0x6bc82762,
0x4e63,
0x4ca4,
[0xaa, 0x85, 0x1e, 0x50, 0xf3, 0x21, 0xf6, 0xbf],
);
pub const NV_ENC_CODEC_HEVC_GUID: Guid = Guid::from_values(
0x790cdc88,
0x4522,
0x4d7b,
[0x94, 0x25, 0xbd, 0xa9, 0x97, 0x5f, 0x76, 0x3],
);
pub const NV_ENC_CODEC_AV1_GUID: Guid = Guid::from_values(
0x0a352289,
0x0aa7,
0x4759,
[0x86, 0x2d, 0x5d, 0x15, 0xcd, 0x16, 0xd2, 0x54],
);
pub const NV_ENC_CODEC_PROFILE_AUTOSELECT_GUID: Guid = Guid::from_values(
0xbfd6f8e7,
0x233c,
0x4341,
[0x8b, 0x3e, 0x48, 0x18, 0x52, 0x38, 0x3, 0xf4],
);
pub const NV_ENC_H264_PROFILE_BASELINE_GUID: Guid = Guid::from_values(
0x727bcaa,
0x78c4,
0x4c83,
[0x8c, 0x2f, 0xef, 0x3d, 0xff, 0x26, 0x7c, 0x6a],
);
pub const NV_ENC_H264_PROFILE_MAIN_GUID: Guid = Guid::from_values(
0x60b5c1d4,
0x67fe,
0x4790,
[0x94, 0xd5, 0xc4, 0x72, 0x6d, 0x7b, 0x6e, 0x6d],
);
pub const NV_ENC_H264_PROFILE_HIGH_GUID: Guid = Guid::from_values(
0xe7cbc309,
0x4f7a,
0x4b89,
[0xaf, 0x2a, 0xd5, 0x37, 0xc9, 0x2b, 0xe3, 0x10],
);
pub const NV_ENC_H264_PROFILE_HIGH_10_GUID: Guid = Guid::from_values(
0x8f0c337e,
0x186c,
0x48e9,
[0xa6, 0x9d, 0x7a, 0x83, 0x34, 0x08, 0x97, 0x5],
);
pub const NV_ENC_H264_PROFILE_HIGH_422_GUID: Guid = Guid::from_values(
0xff3242e9,
0x613c,
0x4295,
[0xa1, 0xe8, 0x2a, 0x7f, 0xe9, 0x4d, 0x81, 0x33],
);
pub const NV_ENC_H264_PROFILE_HIGH_444_GUID: Guid = Guid::from_values(
0x7ac663cb,
0xa598,
0x4960,
[0xb8, 0x44, 0x33, 0x9b, 0x26, 0x1a, 0x7d, 0x52],
);
pub const NV_ENC_H264_PROFILE_STEREO_GUID: Guid = Guid::from_values(
0x40847bf5,
0x33f7,
0x4601,
[0x90, 0x84, 0xe8, 0xfe, 0x3c, 0x1d, 0xb8, 0xb7],
);
pub const NV_ENC_H264_PROFILE_PROGRESSIVE_HIGH_GUID: Guid = Guid::from_values(
0xb405afac,
0xf32b,
0x417b,
[0x89, 0xc4, 0x9a, 0xbe, 0xed, 0x3e, 0x59, 0x78],
);
pub const NV_ENC_H264_PROFILE_CONSTRAINED_HIGH_GUID: Guid = Guid::from_values(
0xaec1bd87,
0xe85b,
0x48f2,
[0x84, 0xc3, 0x98, 0xbc, 0xa6, 0x28, 0x50, 0x72],
);
pub const NV_ENC_HEVC_PROFILE_MAIN_GUID: Guid = Guid::from_values(
0xb514c39a,
0xb55b,
0x40fa,
[0x87, 0x8f, 0xf1, 0x25, 0x3b, 0x4d, 0xfd, 0xec],
);
pub const NV_ENC_HEVC_PROFILE_MAIN10_GUID: Guid = Guid::from_values(
0xfa4d2b6c,
0x3a5b,
0x411a,
[0x80, 0x18, 0x0a, 0x3f, 0x5e, 0x3c, 0x9b, 0xe5],
);
pub const NV_ENC_HEVC_PROFILE_FREXT_GUID: Guid = Guid::from_values(
0x51ec32b5,
0x1b4c,
0x453c,
[0x9c, 0xbd, 0xb6, 0x16, 0xbd, 0x62, 0x13, 0x41],
);
pub const NV_ENC_AV1_PROFILE_MAIN_GUID: Guid = Guid::from_values(
0x5f2a39f5,
0xf14e,
0x4f95,
[0x9a, 0x9e, 0xb7, 0x6d, 0x56, 0x8f, 0xcf, 0x97],
);
pub const NV_ENC_PRESET_P1_GUID: Guid = Guid::from_values(
0xfc0a8d3e,
0x45f8,
0x4cf8,
[0x80, 0xc7, 0x29, 0x88, 0x71, 0x59, 0xe, 0xbf],
);
pub const NV_ENC_PRESET_P2_GUID: Guid = Guid::from_values(
0xf581cfb8,
0x88d6,
0x4381,
[0x93, 0xf0, 0xdf, 0x13, 0xf9, 0xc2, 0x7d, 0xab],
);
pub const NV_ENC_PRESET_P3_GUID: Guid = Guid::from_values(
0x36850110,
0x3a07,
0x441f,
[0x94, 0xd5, 0x36, 0x70, 0x63, 0x1f, 0x91, 0xf6],
);
pub const NV_ENC_PRESET_P4_GUID: Guid = Guid::from_values(
0x90a7b826,
0xdf06,
0x4862,
[0xb9, 0xd2, 0xcd, 0x6d, 0x73, 0xa0, 0x86, 0x81],
);
pub const NV_ENC_PRESET_P5_GUID: Guid = Guid::from_values(
0x21c6e6b4,
0x297a,
0x4cba,
[0x99, 0x8f, 0xb6, 0xcb, 0xde, 0x72, 0xad, 0xe3],
);
pub const NV_ENC_PRESET_P6_GUID: Guid = Guid::from_values(
0x8e75c279,
0x6299,
0x4ab6,
[0x83, 0x2, 0xb, 0x21, 0x5a, 0x33, 0x5c, 0xf5],
);
pub const NV_ENC_PRESET_P7_GUID: Guid = Guid::from_values(
0x84848c12,
0x6f71,
0x4c13,
[0x93, 0x1b, 0x53, 0xe2, 0x83, 0xf5, 0x79, 0x74],
);
vendor/nvenc/src/sys/mod.rs
Vendored
+6
View File
@@ -0,0 +1,6 @@
pub mod enums;
pub mod function_table;
pub mod guids;
pub mod result;
pub mod structs;
pub mod version;
vendor/nvenc/src/sys/result.rs
Vendored
+79
View File
@@ -0,0 +1,79 @@
#[derive(Debug, PartialEq, Eq)]
#[repr(C)]
pub enum NVencResult {
Success,
NoDevice,
UnsupportedDevice,
InvalidEncoderDevice,
InvalidDevice,
DeviceNotExist,
InvalidPtr,
InvalidParam,
InvalidEvent,
InvalidCall,
OutOfMemory,
NotInitialized,
UnsupportedParam,
LockBusy,
NotEnoughBuffer,
InvalidVersion,
MapFailed,
NeedMoreInput,
EncoderBusy,
EventNotRegisterd,
Generic,
IncompatibleClientKey,
Unimplemented,
ResourceRegisterFailed,
ResourceNotRegistered,
ResourceNotMapped,
NeedMoreOutput,
}
impl NVencResult {
pub fn into_error(self) -> Result<(), NVencError> {
NVencError::from_result(self)
}
}
impl NVencError {
pub fn from_result(value: NVencResult) -> Result<(), Self> {
if value == NVencResult::Success {
Ok(())
} else {
// Safety: They are one case different, so they always align
unsafe { Err(std::mem::transmute::<NVencResult, NVencError>(value)) }
}
}
}
#[derive(Debug, PartialEq, Eq)]
#[repr(C)]
pub enum NVencError {
NoDevice = NVencResult::NoDevice as isize,
UnsupportedDevice,
InvalidEncoderDevice,
InvalidDevice,
DeviceNotExist,
InvalidPtr,
InvalidEvent,
InvalidParam,
InvalidCall,
OutOfMemory,
NotInitialized,
UnsupportedParam,
LockBusy,
NotEnoughBuffer,
InvalidVersion,
MapFailed,
NeedMoreInput,
EncoderBusy,
EventNotRegisterd,
Generic,
IncompatibleClientKey,
Unimplemented,
ResourceRegisterFailed,
ResourceNotRegistered,
ResourceNotMapped,
NeedMoreOutput,
}
vendor/nvenc/src/sys/structs.rs
Vendored
+944
View File
@@ -0,0 +1,944 @@
use std::{ffi::c_void, mem::ManuallyDrop};
use crate::sys::{
enums::{
NVencAV1PartSize, NVencBFrameRefMode, NVencBitDepth, NVencBufferFormat, NVencBufferUsage,
NVencCaps, NVencDeviceType, NVencDisplayPicStruct, NVencH264AdaptiveTransformMode,
NVencH264BDirectMode, NVencH264EntropyCodingMode, NVencH264FMOMode, NVencHEVCCuSize,
NVencInputResourceType, NVencLookAheadLevel, NVencMVPrecision, NVencMemoryHeap,
NVencMultiPass, NVencNumRefFrames, NVencOutputStatsLevel, NVencParamsFrameFieldMode,
NVencParamsRcMode, NVencPicStruct, NVencPicType, NVencQPMapMode, NVencStateRestoreType,
NVencStereoPackingMode, NVencTemporalFilterLevel, NVencTuningInfo, NVencVUIColorPrimaries,
NVencVUIMatrixCoeffs, NVencVUITransCharacteristics, NVencVUIVideoFormat,
},
version::struct_version,
};
#[derive(Default, Clone, PartialEq, Eq)]
#[repr(C)]
pub struct Guid {
pub data1: u32,
pub data2: u16,
pub data3: u16,
pub data4: [u8; 8],
}
impl Guid {
pub const fn from_values(data1: u32, data2: u16, data3: u16, data4: [u8; 8]) -> Self {
Self {
data1,
data2,
data3,
data4,
}
}
pub const fn from_u128(uuid: u128) -> Self {
Self {
data1: (uuid >> 96) as u32,
data2: ((uuid >> 80) & 0xffff) as u16,
data3: ((uuid >> 64) & 0xffff) as u16,
data4: (uuid as u64).to_be_bytes(),
}
}
}
#[repr(C)]
pub struct NVencCapsParam {
version: u32,
caps_to_query: NVencCaps,
rsvd: [u32; 62],
}
pub const NV_ENC_CAPS_PARAM_VERSION: u32 = struct_version(1);
#[repr(C)]
pub struct NVencRestoreEncoderStateParams {
version: u32,
buffer: u32,
state: NVencStateRestoreType,
rsvd: u32,
output_bit_stream: *mut c_void,
completion_event: *mut c_void,
rsvd1: [u32; 64],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_RESTORE_STATE_PARAMS_VER: u32 = struct_version(2);
#[repr(C)]
pub struct NVencOutputStatsBlock {
version: u32,
qp: u8,
rsvd: [u8; 3],
bit_count: u32,
satd_cost: u32,
rsvd1: [u32; 12],
}
pub const NV_ENC_OUTPUT_STATS_BLOCK_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencOutputStatsRow {
version: u32,
qp: u8,
rsvd: [u8; 3],
bit_count: u32,
satd_cost: u32,
rsvd1: [u32; 12],
}
pub const NV_ENC_OUTPUT_STATS_ROW_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencEncodeOutParams {
version: u32,
bitstream_size_in_bytes: u32,
rsvd: [u32; 62],
}
pub const NV_ENC_ENCODE_OUT_PARAMS_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencLookAheadPicParams {
version: u32,
rsvd: u32,
input_buffer: *mut c_void,
picture_type: NVencPicType,
rsvd1: [u32; 63],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_LOOKAHEAD_PIC_PARAMS_VER: u32 = struct_version(2);
#[repr(C)]
pub struct NVencCreateInputBuffer {
pub version: u32,
pub width: u32,
pub height: u32,
pub memory_heap: NVencMemoryHeap,
pub buffer_fmt: NVencBufferFormat,
pub rsvd: u32,
pub input_buffer: *mut c_void,
pub p_sys_mem_buffer: *mut c_void,
pub rsvd1: [u32; 58],
pub rsvd2: [*mut c_void; 63],
}
pub const NV_ENC_CREATE_INPUT_BUFFER_VER: u32 = struct_version(2);
#[repr(C)]
pub struct NVencCreateBitstreamBuffer {
pub version: u32,
#[deprecated]
size: u32,
#[deprecated]
memory_heap: NVencMemoryHeap,
rsvd: u32,
pub bitstream_buffer: *mut c_void,
bitstream_buffer_ptr: *mut c_void,
rsvd1: [u32; 58],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_CREATE_BITSTREAM_BUFFER_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencMVector {
mvx: i16,
mvy: i16,
}
#[repr(C)]
pub struct NVencH264MVData {
mv: [NVencMVector; 4],
mb_type: u8,
partition_type: u8,
rsvd: u16,
mb_cost: u32,
}
#[repr(C)]
pub struct NVencHEVCMVData {
mv: [NVencMVector; 4],
cu_type: u8,
cu_size: u8,
partition_mode: u8,
last_cu_int_ctb: u8,
}
#[repr(C)]
pub struct NVencCreateMVBuffer {
version: u32,
rsvd: u32,
mv_buffer: *mut c_void,
rsvd1: [u32; 254],
rsvd2: [*mut c_void; 63],
}
pub const NV_ENC_CREATE_MV_BUFFER_VER: u32 = struct_version(2);
#[repr(C)]
pub struct NVencQP {
qp_inter_p: u32,
qp_inter_b: u32,
qp_intra: u32,
}
pub const MAX_NUM_VIEWS_MINUS_1: usize = 7;
#[repr(C)]
pub struct NVencRcParams {
version: u32,
pub rate_control_mode: NVencParamsRcMode,
const_qp: NVencQP,
pub average_bit_rate: u32,
max_bit_rate: u32,
vbv_buffer_size: u32,
vbv_initial_delay: u32,
// TODO: bit fields
bit_fields: NVencRcParamsBitFlags,
min_qp: NVencQP,
max_qp: NVencQP,
initial_rcqp: NVencQP,
temporal_layer_idx_mask: u32,
temporal_layer_qp: [u8; 8],
target_quality: u8,
target_quality_lsb: u8,
pub look_ahead_depth: u16,
low_delay_key_frame_scale: u8,
y_dc_qp_index_offset: i8,
u_dc_qp_index_offset: i8,
v_dc_qp_index_offset: i8,
qp_map_mode: NVencQPMapMode,
multi_pass: NVencMultiPass,
alpha_layer_bitrate_ratio: u32,
cb_qp_index_offset: i8,
cr_qp_index_offset: i8,
rsvd2: u16,
lookahead_level: NVencLookAheadLevel,
view_bitrate_ratios: [u8; MAX_NUM_VIEWS_MINUS_1],
rsvd3: u8,
rsvd1: u32,
}
#[repr(transparent)]
pub struct NVencRcParamsBitFlags(u32);
impl NVencRcParamsBitFlags {}
pub const NV_ENC_RC_PARAMS_VER: u32 = struct_version(1);
pub const MAX_NUM_CLOCK_TS: usize = 3;
#[repr(C)]
pub struct NVencClockTimeStampSet {
pub bitflags: u32,
pub time_offset: u32,
}
impl NVencClockTimeStampSet {}
#[repr(C)]
pub struct NVencTimeCode {
pub display_pic_struct: NVencDisplayPicStruct,
pub clock_timestamp: [NVencClockTimeStampSet; MAX_NUM_CLOCK_TS],
pub skip_clock_timestamp_insertion: u32,
}
pub const MULTIVIEW_MAX_NUM_REF_DISPLAY: usize = 32;
#[repr(C)]
pub struct HEVC3DReferenceDisplayInfo {
// TODO
pub bitflags: u32,
pub prec_ref_display_width: i32,
pub prec_ref_viewing_dist: i32,
pub num_ref_displays_minus1: i32,
pub left_view_id: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub right_view_id: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub exponent_ref_display_width: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub mantissa_ref_display_width: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub exponent_ref_viewing_distance: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub mantissa_ref_viewing_distance: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub num_sample_shift_plus512: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub additional_shift_present_flag: [u8; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub rsvd2: [u32; 4],
}
#[repr(C)]
pub struct ChromaPoints {
x: u16,
y: u16,
}
#[repr(C)]
pub struct MasteringDisplayInfo {
pub g: ChromaPoints,
pub b: ChromaPoints,
pub r: ChromaPoints,
pub white_point: ChromaPoints,
pub max_luma: u32,
pub min_luma: u32,
}
#[repr(C)]
pub struct ContentLightLevel {
pub max_content_light_level: u16,
pub max_pic_average_light_level: u16,
}
#[repr(C)]
pub struct NVencConfigH264VUIParameters {
overscan_info_present_flag: u32,
overscan_info: u32,
video_signal_type_present_flag: u32,
video_format: NVencVUIVideoFormat,
video_full_range_flag: u32,
color_description_present_flag: u32,
color_primaries: NVencVUIColorPrimaries,
transfer_characteristics: NVencVUITransCharacteristics,
color_matrix: NVencVUIMatrixCoeffs,
chroma_sample_location_flag: u32,
chroma_sample_location_top: u32,
chroma_sample_location_bot: u32,
bitstream_restriction_flag: u32,
timing_info_present_flag: u32,
num_unit_in_ticks: u32,
time_scale: u32,
rsvd: [u32; 12],
}
pub type NVencConfigHEVCVUIParamaters = NVencConfigH264VUIParameters;
#[repr(C)]
#[derive(Default)]
pub struct NVencExternalMeHintCountsPerBlockType {
pub bitflags: u32,
pub rsvd1: [u32; 3],
}
#[repr(transparent)]
pub struct NVencExternalMeHint(i32);
#[repr(C)]
pub struct NVencExternalMeSbHint {
flags1: i16,
flags2: i16,
flags3: i16,
}
#[repr(C)]
pub struct NVencConfigH264 {
bitflags: u32,
level: u32,
idr_period: u32,
seperate_color_plane_flag: u32,
disable_deblocking_filter_idc: u32,
num_termpoal_layers: u32,
sps_id: u32,
pps_id: u32,
adaptive_transform_mode: NVencH264AdaptiveTransformMode,
fmo_mode: NVencH264FMOMode,
bdirect_mode: NVencH264BDirectMode,
entropy_coding_mode: NVencH264EntropyCodingMode,
stereo_mode: NVencStereoPackingMode,
intra_refresh_period: u32,
intra_refresh_count: u32,
max_num_ref_frames: u32,
slice_mode: u32,
slice_mode_data: u32,
h264_vui_parameters: NVencConfigH264VUIParameters,
ltr_num_frames: u32,
ltr_trust_mode: u32,
chroma_format_idc: u32,
max_termpoal_layers: u32,
use_b_frames_as_ref: NVencBFrameRefMode,
num_ref_l0: NVencNumRefFrames,
num_ref_l1: NVencNumRefFrames,
output_bit_depth: NVencBitDepth,
input_bit_depth: NVencBitDepth,
tf_level: NVencTemporalFilterLevel,
rsvd1: [u32; 264],
rsvd: [*mut c_void; 64],
}
#[repr(C)]
pub struct NVencConfigHEVC {
level: u32,
tier: u32,
min_cu_size: NVencHEVCCuSize,
max_cu_size: NVencHEVCCuSize,
bitflags: u32,
idr_period: u32,
intra_refresh_cnt: u32,
max_num_ref_frames_in_dpb: u32,
ltr_num_frames: u32,
vps_id: u32,
sps_id: u32,
pps_id: u32,
slice_mode: u32,
slice_mode_data: u32,
max_temporal_layers_minus_1: u32,
hevc_vui_paramters: NVencConfigHEVCVUIParamaters,
ltr_trust_mode: u32,
use_b_frames_as_ref: NVencBFrameRefMode,
tf_level: NVencTemporalFilterLevel,
disable_deblocking_filter_idc: u32,
output_bit_depth: NVencBitDepth,
input_bit_depth: NVencBitDepth,
num_temporal_layers: u32,
num_views: u32,
rsvd1: [u32; 208],
rsvd2: [*mut c_void; 64],
}
pub const NV_MAX_TILE_COLS_AV1: usize = 64;
pub const NV_MAX_TILE_ROWS_AV1: usize = 64;
#[repr(C)]
pub struct NVencFilmGrainParamsAV1 {
bitfields: u32,
point_y_value: [u8; 14],
point_y_scaling: [u8; 14],
point_cb_value: [u8; 10],
point_cb_scaling: [u8; 10],
point_cr_value: [u8; 10],
point_cr_scaling: [u8; 10],
ar_coeffs_y_plus128: [u8; 24],
ar_coeffs_cb_plus128: [u8; 25],
ar_coeffs_cr_plus128: [u8; 25],
rsvd2: [u8; 2],
cb_mult: u8,
cb_luma_mult: u8,
cb_offset: u16,
cr_mult: u8,
cr_luma_mult: u8,
cr_offset: u16,
}
#[repr(C)]
pub struct NVencConfigAV1 {
level: u32,
tier: u32,
min_part_size: NVencAV1PartSize,
max_part_size: NVencAV1PartSize,
bitflags: u32,
idr_period: u32,
intra_refresh_period: u32,
intra_refresh_count: u32,
max_num_ref_frames_in_dpb: u32,
num_tiles_columns: u32,
num_tile_rows: u32,
rsvd2: u32,
tile_widths: *mut u32,
tile_heights: *mut u32,
max_temporal_layers_minus_1: u32,
color_primaries: NVencVUIColorPrimaries,
transfer_characteristics: NVencVUITransCharacteristics,
matrix_coefficients: NVencVUIMatrixCoeffs,
color_range: u32,
chroma_sample_position: u32,
use_b_frames_as_ref: NVencBFrameRefMode,
film_grain_params: NVencFilmGrainParamsAV1,
num_fwd_refs: NVencNumRefFrames,
num_bwd_refs: NVencNumRefFrames,
input_bit_depth: NVencBitDepth,
ltr_num_frames: u32,
num_temporal_layers: u32,
tf_level: NVencTemporalFilterLevel,
rsvd1: [u32; 230],
rsvd3: [*mut c_void; 62],
}
#[repr(C)]
pub struct NVencConfigH264MeOnly {
bitflags: u32,
rsvd1: [u32; 255],
rsvd2: [*mut c_void; 64],
}
#[repr(C)]
pub struct NVencConfigHEVCMeOnly {
rsvd: [u32; 256],
rsvd1: [*mut c_void; 64],
}
#[repr(C)]
pub union NVencCodecConfig {
h264: ManuallyDrop<NVencConfigH264>,
hevc: ManuallyDrop<NVencConfigHEVC>,
av1: ManuallyDrop<NVencConfigAV1>,
h264_meonly: ManuallyDrop<NVencConfigH264MeOnly>,
hevc_meonly: ManuallyDrop<NVencConfigHEVCMeOnly>,
rsvd: [u32; 320],
}
#[repr(C)]
pub struct NVencConfig {
pub version: u32,
pub profile_guid: Guid,
pub gop_len: u32,
pub frame_interval_p: i32,
mono_chrome_encoding: u32,
frame_field_mode: NVencParamsFrameFieldMode,
mv_precision: NVencMVPrecision,
pub rc_params: NVencRcParams,
encode_codec_config: NVencCodecConfig,
rsvd: [u32; 278],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_CONFIG_VER: u32 = struct_version(9) | (1 << 31);
#[bitfields::bitfield(u32)]
pub struct NVencInitializeParamsBitfields {
#[bits(1)]
pub report_slice_offsets: bool,
#[bits(1)]
pub enable_sub_frame_write: bool,
#[bits(1)]
pub enable_external_me_hints: bool,
#[bits(1)]
pub enable_me_only_mode: bool,
#[bits(1)]
pub enable_weighted_prediction: bool,
#[bits(4)]
pub split_encode_mode: u8,
#[bits(1)]
pub enable_output_in_vid_mem: bool,
#[bits(1)]
pub enable_recon_frame_output: bool,
#[bits(1)]
pub enable_output_stats: bool,
#[bits(1)]
pub enable_uni_directional_b: bool,
#[bits(19)]
pub rsvd: u32,
}
#[repr(C)]
pub struct NVencInitializeParams {
pub version: u32,
pub encode_guid: Guid,
pub preset_guid: Guid,
pub encode_width: u32,
pub encode_height: u32,
pub dar_width: u32,
pub dar_height: u32,
pub frame_rate_num: u32,
pub frame_rate_den: u32,
pub enable_encode_async: u32,
pub enable_ptd: u32,
pub bit_flags: NVencInitializeParamsBitfields,
pub(crate) priv_data_size: u32,
pub(crate) rsvd: u32,
pub(crate) priv_data: *mut c_void,
pub encode_config: *mut NVencConfig,
pub max_encode_width: u32,
pub max_encode_height: u32,
pub max_me_hint_counts_per_block: [NVencExternalMeHintCountsPerBlockType; 2],
pub tuning_info: NVencTuningInfo,
pub buffer_format: NVencBufferFormat,
pub num_state_buffers: u32,
pub output_stats_level: NVencOutputStatsLevel,
pub(crate) rsvd1: [u32; 284],
pub(crate) rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_INITIALIZE_PARAMS_VER: u32 = struct_version(7) | (1 << 31);
#[repr(C)]
pub struct NVencReconfigureParams {
version: u32,
rsvd: u32,
re_init_encode_params: NVencInitializeParams,
bitflags: u32,
rsvd2: u32,
}
pub const NV_ENC_RECONFIGURE_PARAMS_VER: u32 = struct_version(2) | (1 << 31);
#[repr(C)]
pub struct NVencPresetConfig {
pub version: u32,
rsvd: u32,
pub preset_cfg: NVencConfig,
rsvd1: [u32; 256],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_PRESET_CONFIG_VER: u32 = struct_version(5) | (1 << 31);
#[repr(C)]
pub struct NVencPicParamsMVC {
version: u32,
view_id: u32,
temporal_id: u32,
priority_id: u32,
rsvd1: [u32; 12],
rsvd2: [*mut c_void; 8],
}
pub const NV_ENC_PIC_PARAMS_MVC_VER: u32 = struct_version(1);
#[repr(C)]
pub union NVencPicParamsH264EXT {
pub mvc_pic_params: ManuallyDrop<NVencPicParamsMVC>,
pub rsvd1: [u32; 32],
}
#[repr(C)]
pub struct NVencSeiPayload {
pub payload_size: u32,
pub payload_type: u32,
pub payload: *mut u8,
}
pub type NVencH264SeiPayload = NVencSeiPayload;
#[repr(C)]
pub struct NVencPicParamsH264 {
pub display_poc_syntax: u32,
pub rsvd3: u32,
pub ref_pic_flag: u32,
pub color_plane_id: u32,
pub force_intra_refresh_with_frame_count: u32,
pub bitflags: u32,
pub slice_type_data: *mut u8,
pub slice_type_array_count: u32,
pub sei_payload_array_count: u32,
pub sei_payload_array: *mut NVencSeiPayload,
pub slice_mode: u32,
pub slice_mode_data: u32,
pub ltr_mark_framce_idx: u32,
pub ltr_use_frame_bitmap: u32,
pub ltr_usage_mode: u32,
pub force_intra_slice_count: u32,
pub force_intra_slice_idx: *mut u32,
pub h264_ext_pic_params: NVencPicParamsH264EXT,
pub time_code: NVencTimeCode,
pub rsvd: [u32; 202],
pub rsvd2: [*mut c_void; 61],
}
#[repr(C)]
pub struct NVencPicParamsHEVC {
pub display_poc_syntax: u32,
pub ref_pic_flag: u32,
pub temporal_id: u32,
pub force_intra_refresh_with_frame_count: u32,
pub bitflags: u32,
pub rsvd1: u32,
pub slice_type_data: *mut u8,
pub slice_type_array_count: u32,
pub slice_mode: u32,
pub slice_mode_data: u32,
pub ltr_mark_frame_idx: u32,
pub ltr_usage_frame_bitmap: u32,
pub ltr_usage_mode: u32,
pub sei_payload_array_count: u32,
pub rsvd: u32,
pub sei_payload_array: *mut NVencSeiPayload,
pub time_code: NVencTimeCode,
pub num_temporal_layers: u32,
pub view_id: u32,
pub _3d_reference_display_info: *mut HEVC3DReferenceDisplayInfo,
pub max_cll: *mut ContentLightLevel,
pub mastering_display: *mut MasteringDisplayInfo,
pub rsvd2: [u32; 234],
pub rsvd3: [*mut c_void; 58],
}
pub type NVencAV1OBUPayload = NVencSeiPayload;
#[repr(C)]
pub struct NVencPicParamsAV1 {
display_poc_syntax: u32,
ref_pic_flag: u32,
temporal_id: u32,
force_intra_refresh_with_frame_count: u32,
bitflags: u32,
num_tile_columns: u32,
num_tile_rows: u32,
rsvd: u32,
tile_widths: *mut u32,
tile_heights: *mut u32,
obu_payload_array_count: u32,
rsvd1: u32,
obu_payload_array: *mut NVencAV1OBUPayload,
file_grain_params: *mut NVencFilmGrainParamsAV1,
ltr_mark_frame_idx: u32,
ltr_use_frame_bitmap: u32,
num_temporal_layers: u32,
rsvd4: u32,
max_cll: *mut ContentLightLevel,
mastering_display: *mut MasteringDisplayInfo,
rsvd2: [u32; 242],
rsvd3: [*mut c_void; 59],
}
#[repr(C)]
pub union NVencCodecPicParams {
pub h264_pic_params: ManuallyDrop<NVencPicParamsH264>,
pub hevc_pic_params: ManuallyDrop<NVencPicParamsHEVC>,
av1_pic_params: ManuallyDrop<NVencPicParamsAV1>,
pub rsvd: [u32; 256],
}
#[repr(C)]
pub struct NVencPicParams {
pub version: u32,
pub input_width: u32,
pub input_height: u32,
pub input_pitch: u32,
pub encode_pic_flags: u32,
pub frame_idx: u32,
pub input_time_stamp: u64,
input_duration: u64,
pub input_buffer: *mut c_void,
pub output_bitstream: *mut c_void,
// TODO: Windows only, needed for async
completion_event: *mut c_void,
pub buffer_format: NVencBufferFormat,
pub picture_struct: NVencPicStruct,
pub picture_type: NVencPicType,
pub codec_pic_params: NVencCodecPicParams,
me_hint_counts_per_block: [NVencExternalMeHintCountsPerBlockType; 2],
me_external_hints: *mut NVencExternalMeHint,
rsvd2: [u32; 7],
rsvd5: [*mut c_void; 2],
qp_delta_map: *mut i8,
qp_deltra_map_size: u32,
rsvd_bit_fields: u32,
me_hint_ref_pic_dist: [u16; 2],
rsvd4: u32,
alpha_buffer: *mut c_void,
me_external_sb_hints: *mut NVencExternalMeSbHint,
me_sb_hints_count: u32,
state_buffer_idx: u32,
output_recon_buffer: *mut c_void,
rsvd3: [u32; 284],
rsvd6: [*mut c_void; 57],
}
pub const NV_ENC_PIC_PARAMS_VER: u32 = struct_version(7) | (1 << 31);
#[repr(C)]
pub struct NVencMeOnlyParams {
version: u32,
input_width: u32,
input_height: u32,
rsvd: u32,
input_buffer: *mut c_void,
reference_frame: *mut c_void,
mv_buffer: *mut c_void,
rsvd2: u32,
buffer_formtat: NVencBufferFormat,
view_id: u32,
me_hint_counts_per_block: [NVencExternalMeHintCountsPerBlockType; 2],
me_external_hints: NVencExternalMeHint,
rsvd1: [u32; 241],
rsvd3: [*mut c_void; 59],
}
pub const NV_ENC_MEONLY_PARAMS_VER: u32 = struct_version(4);
#[bitfields::bitfield(u32)]
pub struct NVencLockBitStreamBitFields {
#[bits(1)]
pub do_not_wait: bool,
#[bits(1)]
pub ltr_frame: bool,
#[bits(1)]
pub get_rc_stats: bool,
#[bits(29)]
rsvd: u32,
}
#[repr(C)]
pub struct NVencLockBitStream {
pub version: u32,
pub bit_fields: NVencLockBitStreamBitFields,
pub output_bit_stream: *mut c_void,
slice_offsets: *mut u32,
frame_idx: u32,
hw_encode_status: u32,
num_slices: u32,
pub bitstream_size_in_bytes: u32,
output_time_stamp: u64,
pub output_duration: u64,
pub bitstream_buffer: *mut c_void,
picture_type: NVencPicType,
picture_struct: NVencPicStruct,
framge_avg_qp: u32,
frame_satd: u32,
ltr_frame_idx: u32,
ltr_frame_bitmap: u32,
temporal_id: u32,
inter_mb_count: u32,
average_mvx: i32,
average_mvy: i32,
alpha_layer_size_in_bytes: u32,
output_stats_ptr_size: u32,
rsvd: u32,
output_stats_ptr: *mut c_void,
frame_idx_display: u32,
rsvd1: [u32; 219],
rsvd2: [*mut c_void; 63],
rsvd_interanl: [u32; 8],
}
pub const NV_ENC_LOCK_BITSTREAM_VER: u32 = struct_version(2) | (1 << 31);
#[repr(C)]
pub struct NVencLockInputBuffer {
pub version: u32,
pub bitflags: u32,
pub input_buffer: *mut c_void,
pub buffer_data_ptr: *mut c_void,
pub pitch: u32,
pub rsvd1: [u32; 251],
pub rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_LOCK_INPUT_BUFFER_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencMapInputResource {
pub version: u32,
sub_resource_idx: u32,
input_resource: *mut c_void,
pub registered_resource: *mut c_void,
pub mapped_resource: *mut c_void,
pub mapped_buffer_format: NVencBufferFormat,
rsvd1: [u32; 251],
rsvd: [*mut c_void; 63],
}
pub const NV_ENC_MAP_INPUT_RESOURCE_VER: u32 = struct_version(4);
#[repr(C)]
pub struct NVencInputResourceOpenGLTex {
texture: u32,
target: u32,
}
#[repr(C)]
pub struct NVencFencePointD3D12 {
version: u32,
rsvd: u32,
p_fence: *mut c_void,
wait_value: u64,
signal_value: u64,
bit_flags: u32,
rsvd1: [u32; 7],
}
pub const NV_ENC_FENCE_POINT_D3D12_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencInputResourceD3D12 {
version: u32,
rsvd: u32,
input_buffer: *mut c_void,
input_fence_point: NVencFencePointD3D12,
rsvd1: [u32; 16],
rsvd2: [*mut c_void; 16],
}
pub const NV_ENC_INPUT_RESOURCE_D3D12_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencOutputResourceD3D12 {
version: u32,
rsvd: u32,
output_buffer: *mut c_void,
output_fence_point: NVencFencePointD3D12,
rsvd1: [u32; 16],
rsvd2: [*mut c_void; 16],
}
pub const NV_ENC_OUTPUT_RESOURCE_D3D12_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencRegisterResource {
pub version: u32,
pub resource_type: NVencInputResourceType,
pub width: u32,
pub height: u32,
pub pitch: u32,
pub sub_resource_index: u32,
pub resource_to_register: *mut c_void,
pub registered_resource: *mut c_void,
pub buffer_format: NVencBufferFormat,
pub buffer_usage: NVencBufferUsage,
pub input_fence_point: *mut NVencFencePointD3D12,
pub chroma_offset: [u32; 2],
pub chroma_offset_in: [u32; 2],
pub(crate) rsvd1: [u32; 244],
pub(crate) rsvd2: [*mut c_void; 61],
}
pub const NV_ENC_REGISTER_RESOURCE_VER: u32 = struct_version(5);
#[repr(C)]
pub struct NVencStat {
version: u32,
rsvd: u32,
output_bit_stream: *mut c_void,
bit_stream_size: u32,
pic_type: u32,
last_valid_byte_offset: u32,
slice_offsets: [u32; 16],
pic_idx: u32,
frame_avg_qp: u32,
bit_flags: u32,
ltr_frame_idx: u32,
intra_mb_count: u32,
inter_mb_count: u32,
average_mv_x: i32,
average_mv_y: i32,
rsvd1: [u32; 227],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_STAT_VER: u32 = struct_version(2);
#[repr(C)]
pub struct NVencSequenceParamPayload {
version: u32,
in_buffer_size: u32,
sps_id: u32,
pps_id: u32,
spspps_buffer: *mut c_void,
output_spspps_payload_size: *mut u32,
rsvd: [u32; 250],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_SEQUENCE_PARAM_PAYLOAD_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencEventParams {
version: u32,
rsvd: u32,
completion_event: *mut c_void,
rsvd1: [u32; 254],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_EVENT_PARAMS_VER: u32 = struct_version(2);
#[repr(C)]
pub struct NVencOpenEncodeSessionExParams {
pub version: u32,
pub device_type: NVencDeviceType,
pub device: *mut c_void,
rsvd: *mut c_void,
pub api_version: u32,
rsvd1: [u32; 253],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER: u32 = struct_version(1);
vendor/nvenc/src/sys/version.rs
Vendored
+12
View File
@@ -0,0 +1,12 @@
//! Contains the nvenc version numbers, encoded as they are in C, for the version this crate was built against
pub const NVENC_MAJOR_VERSION: u16 = 13;
pub const NVENC_MINOR_VERSION: u8 = 0;
pub const NVENC_API_VERSION: u32 = NVENC_MAJOR_VERSION as u32 | (NVENC_MINOR_VERSION as u32) << 24;
pub const NVENC_API_LIST_VERSION: u32 = struct_version(2);
pub const fn struct_version(nvenc_api_version: u8) -> u32 {
NVENC_API_VERSION | (nvenc_api_version as u32) << 16 | 0x7 << 28
}
viewer-core/Cargo.toml
+22
View File
@@ -0,0 +1,22 @@
[package]
name = "viewer-core"
version = "0.1.0"
edition = "2021"
# 阶段 1 先做纯 Rust 库并在桌面 headless 验证；
# UniFFI / cdylib 在下一步（Android 接入）再加，隔离未知项。
[lib]
name = "viewer_core"
crate-type = ["lib"]
[dependencies]
common = { path = "../common" }
anyhow.workspace = true
tokio.workspace = true
quinn.workspace = true
rustls.workspace = true
[dev-dependencies]
# headless_check 用软解交叉验证收到的码流合法（库本身刻意不依赖 openh264，
# 解码在 Android 侧由 MediaCodec 承担）
openh264 = { version = "0.9.9", features = ["source"] }
viewer-core/examples/headless_check.rs
+149
View File
@@ -0,0 +1,149 @@
//! viewer-core 的桌面 headless 验证工具（阶段 1 验收用，不依赖 Android）。
//!
//! 证明三件事，逐条打印：
//! 1. viewer-core 能完成 QUIC 连接 + 协议 v2 认证握手；
//! 2. 帧流按契约交付（seq 单调递增、keyframe 标志与码流实际 NAL 一致）；
//! 3. 收到的码流能被 openh264 解出画面（即：Android 侧 MediaCodec 将拿到同样
//! 合法的 Annex-B——解码器不同，但码流合法性在这里就证明了）。
//!
//! 用法：
//! 终端1: target\release\agent.exe --port 47900 --code t
//! 终端2: cargo run -p viewer-core --bin headless_check -- --port 47900 --code t --frames 60
use anyhow::Result;
use std::time::Duration;
use viewer_core::{ConnectConfig, Session};
fn arg(name: &str) -> Option<String> {
let a: Vec<String> = std::env::args().collect();
a.iter()
.position(|x| x == name)
.and_then(|i| a.get(i + 1))
.cloned()
}
/// 与 dump_stream 同款：扫 Annex-B 起始码拿 NAL 类型，交叉校验 keyframe 标志
fn has_idr(payload: &[u8]) -> bool {
let mut i = 0usize;
while i + 3 < payload.len() {
let sc = if payload[i..].starts_with(&[0, 0, 0, 1]) {
4
} else if payload[i..].starts_with(&[0, 0, 1]) {
3
} else {
i += 1;
continue;
};
match payload.get(i + sc) {
Some(&b) if b & 0x1F == 5 => return true,
Some(_) => i += sc + 1,
None => break,
}
}
false
}
#[tokio::main]
async fn main() -> Result<()> {
let port: u16 = arg("--port")
.map(|s| s.parse())
.transpose()?
.unwrap_or(47800);
let code = arg("--code").unwrap_or_else(|| "123456".into());
// --host：默认 127.0.0.1；填局域网 IP 可顺带验证防火墙入站路径
let host = arg("--host").unwrap_or_else(|| "127.0.0.1".into());
let frames_wanted: u64 = arg("--frames")
.map(|s| s.parse())
.transpose()?
.unwrap_or(60);
let mut session = Session::connect(ConnectConfig::direct(&host, port, code)).await?;
println!("[check] 连接成功 host={host} peer={}", session.peer_addr());
let mut decoder = openh264::decoder::Decoder::new()?;
let mut last_seq: Option<u64> = None;
let mut got = 0u64;
let mut seq_violations = 0u64;
let mut kf_mismatches = 0u64;
let mut decoded = 0u64;
let mut decode_failed_before_idr = 0u64;
let mut awaiting_idr = true; // 与 Android 侧相同的状态机
let mut rtt_shown = false;
let deadline = std::time::Instant::now() + Duration::from_secs(30);
while got < frames_wanted && std::time::Instant::now() < deadline {
let Some(f) = tokio::time::timeout(Duration::from_secs(10), session.next_frame())
.await
.ok()
.flatten()
else {
break;
};
got += 1;
// 契约 2a：seq 单调递增
if let Some(prev) = last_seq {
if f.seq <= prev {
seq_violations += 1;
println!("[check] ⚠️ seq 回退: {prev} -> {}", f.seq);
}
}
last_seq = Some(f.seq);
// 契约 2b：keyframe 标志与码流一致
let real_kf = has_idr(&f.payload);
if f.keyframe != real_kf {
kf_mismatches += 1;
println!(
"[check] ⚠️ 帧头 keyframe={} 但码流实际 IDR={}（seq={}）",
f.keyframe, real_kf, f.seq
);
}
// 契约 3：与 Android 相同的状态机下解码必须成功
if awaiting_idr && !f.keyframe {
continue; // 等 IDR，P 帧不喂（这是 R-3 教训的实现）
}
if f.keyframe {
awaiting_idr = false;
}
match decoder.decode(&f.payload) {
Ok(_) => decoded += 1,
Err(e) => {
decode_failed_before_idr += 1;
println!("[check] ❌ seq={} 解码失败: {e}", f.seq);
awaiting_idr = true;
session.request_keyframe();
}
}
// RTT 出现即记录一次
if !rtt_shown {
let s = session.stats();
if s.rtt_ms > 0 {
println!("[check] RTT={}ms frames_in={} bytes_in={}", s.rtt_ms, s.frames_in, s.bytes_in);
rtt_shown = true;
}
}
}
let s = session.stats();
println!("\n========== 阶段 1 headless 验收 ==========");
println!("收到帧: {got}（目标 {frames_wanted}）");
println!("seq 违例: {seq_violations}");
println!("keyframe 标志不符: {kf_mismatches}");
println!("解码成功: {decoded} / 解码失败: {decode_failed_before_idr}");
println!("统计: rtt={}ms frames_in={} bytes_in={}", s.rtt_ms, s.frames_in, s.bytes_in);
let ok = got >= frames_wanted.min(30)
&& seq_violations == 0
&& kf_mismatches == 0
&& decoded > 0
&& decode_failed_before_idr == 0;
println!(
"结论: {}",
if ok { "✅ viewer-core 收帧链路通过" } else { "❌ 存在未满足项" }
);
session.close().await;
Ok(())
}
viewer-core/examples/text_check.rs
+90
View File
@@ -0,0 +1,90 @@
//! 协议 v3 文本注入链路的桌面真边界验证工具（loop-2b，不依赖 Android）。
//!
//! 证明的事：`Session::send_text` 经线格式到被控端后，`agent` 的
//! `KEYEVENTF_UNICODE` 注入把**同一段 UTF-16 code unit** 送进了前台应用。
//! 判据由外部读取目标窗口的实际文本内容（调用方负责），本工具只负责发送。
//!
//! 用法：
//! 终端1: target\release\agent.exe --port 47800 --code t
//! 终端2: 把目标应用（记事本）切到前台，然后
//! cargo run -p viewer-core --example text_check -- \
//! --port 47800 --code t --units 20320,22909,55357,56832
//!
//! `--units` 是十进制的 UTF-16 code unit 列表（逗号分隔）：
//! 「你好😀」= 20320,22909,55357,56832（😀 是代理对）。
//! 刻意不收自然语言字符串参数：命令行到 Rust 的编码往返会引入与本轮无关的变量，
//! 由调用方（PowerShell）从 .NET 字符串取 char[] 更忠实。
//!
//! `--delay`：连接后等待毫秒再注入。给调用方把目标窗口切到前台的时间——
//! SendInput 注入的是**前台**应用，窗口没在前台等于测了个寂寞。
//! `--hold`：注入后保持会话的毫秒数，等消息在远端消化完再关连接。
use anyhow::{bail, Result};
use std::time::Duration;
use viewer_core::{ConnectConfig, Session};
fn arg(name: &str) -> Option<String> {
let a: Vec<String> = std::env::args().collect();
a.iter()
.position(|x| x == name)
.and_then(|i| a.get(i + 1))
.cloned()
}
fn parse_units(spec: &str) -> Result<Vec<u16>> {
if spec.trim().is_empty() {
bail!(
"--units 缺失或为空：本工具要验证的是「有文本送出并被注入」，\
空输入没有验证价值（空 Text 在产品路径上由 Session::send_text 拦掉）"
);
}
spec.split(',')
.map(|s| s.trim())
.filter(|s| !s.is_empty())
.map(|s| {
s.parse::<u32>()
.ok()
.and_then(|v| u16::try_from(v).ok())
.ok_or_else(|| anyhow::anyhow!("--units 必须是 0..=65535 的十进制数，got {s:?}"))
})
.collect()
}
#[tokio::main]
async fn main() -> Result<()> {
let port: u16 = arg("--port")
.map(|s| s.parse())
.transpose()?
.unwrap_or(47800);
let code = arg("--code").unwrap_or_else(|| "123456".into());
let host = arg("--host").unwrap_or_else(|| "127.0.0.1".into());
let delay = arg("--delay")
.map(|s| s.parse())
.transpose()?
.unwrap_or(1500u64);
let hold = arg("--hold")
.map(|s| s.parse())
.transpose()?
.unwrap_or(800u64);
let units = parse_units(&arg("--units").unwrap_or_default())?;
let session = Session::connect(ConnectConfig::direct(&host, port, &code)).await?;
println!(
"[text_check] connected peer={} units={}",
session.peer_addr(),
units.len()
);
tokio::time::sleep(Duration::from_millis(delay)).await;
session.send_text(&units);
println!("[text_check] sent (只报长度，不打印内容)");
tokio::time::sleep(Duration::from_millis(hold)).await;
let st = session.stats();
println!(
"[text_check] frames={} rtt={}ms — 会话活着，注入已发给前台应用",
st.frames_in, st.rtt_ms
);
session.close().await;
Ok(())
}
viewer-core/src/lib.rs
+454
View File
@@ -0,0 +1,454 @@
//! viewer-core：安卓主控的连接 / 协议 / 收帧层（纯 Rust，平台无关）。
//!
//! 设计边界（对齐《多平台架构设计》与 2026-10-29 方案裁决）：
//! - **只做「怎么连 + 收发什么」**：QUIC 会话、认证握手、视频帧流、输入上行、
//! Ping/Pong、KeyFrameReq。解码（MediaCodec）、渲染（SurfaceView）、触摸映射
//! 全在 Kotlin 侧，本 crate 不碰像素，也不引入 openh264。
//! - **连接层是唯一的可替换点**：`PeerTarget::Direct` 现在只支持 `IP:端口` 局域网
//! 直连；将来设备码 + 信令 + 打洞只换 `connect()` 内部。**不定义 Relay 变体**
//! （中继已被团队裁决排除，见《多平台架构设计》§6.1）。
//! - 帧经 mpsc 通道交付，**不丢帧**：消费方（Kotlin）按 `keyframe` 标志自行决定
//! 跳帧策略，且跳帧后必须 `request_keyframe()` 并等 IDR（协议 v2 提供该字段）。
//! 丢帧决策放在这里会让 Rust 层承担它看不到的解码器状态。
//!
//! 安全现状：`insecure_skip_verify` 默认 true（与桌面 viewer 的 SkipVerify 一致，
//! 仅限局域网测试）。产品化前必须换成信令下发的证书指纹钉扎（`signaling.rs`
//! 已有 `CertFingerprint`），该项登记在方案阶段 4。
use anyhow::{bail, Context, Result};
use common::{
read_control, read_video_frame, write_control, ControlMsg, InputEvent, VideoFrameHeader,
PROTO_VERSION,
};
use rustls::client::danger::{
HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier,
};
use rustls::pki_types::{CertificateDer, ServerName, UnixTime};
use rustls::{DigitallySignedStruct, SignatureScheme};
use std::net::ToSocketAddrs;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::sync::mpsc;
/// 连接目标。当前只有局域网直连；设备码 + 打洞在后续阶段替换 `connect()` 内部实现。
#[derive(Debug, Clone)]
pub enum PeerTarget {
/// 局域网 / 可直连地址
Direct { host: String, port: u16 },
}
/// 一次会话的参数
#[derive(Debug, Clone)]
pub struct ConnectConfig {
pub target: PeerTarget,
/// 被控端验证码
pub code: String,
/// 仅测试用：跳过服务端证书校验（与桌面 viewer 行为一致）。
/// 公网/产品形态必须为 false 并启用证书钉扎。
pub insecure_skip_verify: bool,
}
impl ConnectConfig {
/// 局域网测试的便捷构造
pub fn direct(host: impl Into<String>, port: u16, code: impl Into<String>) -> Self {
Self {
target: PeerTarget::Direct {
host: host.into(),
port,
},
code: code.into(),
insecure_skip_verify: true,
}
}
}
/// 交付给消费方的一帧视频。字段语义与 `common::VideoFrameHeader` 一致，
/// payload 为标准 Annex-B H.264（可直接喂 MediaCodec）。
#[derive(Debug, Clone)]
pub struct Frame {
pub width: u32,
pub height: u32,
pub ts_ms: u64,
pub seq: u64,
pub keyframe: bool,
pub payload: Vec<u8>,
}
impl From<(VideoFrameHeader, Vec<u8>)> for Frame {
fn from((h, payload): (VideoFrameHeader, Vec<u8>)) -> Self {
Self {
width: h.width,
height: h.height,
ts_ms: h.ts_ms,
seq: h.seq,
keyframe: h.keyframe,
payload,
}
}
}
/// 会话状态快照（供 UI 显示，不做 EMA——那属于 Kotlin 的 HUD 层）
#[derive(Debug, Clone, Copy, Default)]
pub struct SessionStats {
/// 最近一次 Ping 往返（ms），0 = 还没测出
pub rtt_ms: u64,
/// 收到的帧数
pub frames_in: u64,
/// 收到的字节数（视频载荷累计）
pub bytes_in: u64,
}
/// MVP：跳过服务端证书校验（自签证书，仅局域网）。
#[derive(Debug)]
struct SkipVerify;
impl ServerCertVerifier for SkipVerify {
fn verify_server_cert(
&self,
_end_entity: &CertificateDer<'_>,
_intermediates: &[CertificateDer<'_>],
_server_name: &ServerName<'_>,
_ocsp_response: &[u8],
_now: UnixTime,
) -> std::result::Result<ServerCertVerified, rustls::Error> {
Ok(ServerCertVerified::assertion())
}
fn verify_tls12_signature(
&self,
_message: &[u8],
_certificate: &CertificateDer<'_>,
_dss: &DigitallySignedStruct,
) -> std::result::Result<HandshakeSignatureValid, rustls::Error> {
Ok(HandshakeSignatureValid::assertion())
}
fn verify_tls13_signature(
&self,
_message: &[u8],
_certificate: &CertificateDer<'_>,
_dss: &DigitallySignedStruct,
) -> std::result::Result<HandshakeSignatureValid, rustls::Error> {
Ok(HandshakeSignatureValid::assertion())
}
fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
vec![
SignatureScheme::RSA_PKCS1_SHA256,
SignatureScheme::ECDSA_NISTP256_SHA256,
SignatureScheme::ED25519,
SignatureScheme::RSA_PSS_SHA256,
]
}
}
/// 输入/控制命令（写入流 2）
enum Command {
Input(InputEvent),
KeyFrameReq,
Ping(u64),
}
/// 已建立的会话。Drop 时后台任务随连接关闭自然退出；显式 `close()` 更干净。
pub struct Session {
tx: mpsc::UnboundedSender<Command>,
/// 帧接收端。通道关闭 = 会话结束（被控端断开或流错误）。
/// Mutex 包一层是为了让 `next_frame` 取 `&self`：JNI 层里解码线程独占 recv，
/// 输入线程并发调 `send_input`（走无锁的 unbounded tx），两者互不阻塞。
frames: Arc<tokio::sync::Mutex<mpsc::Receiver<Frame>>>,
stats: Arc<std::sync::Mutex<SessionStats>>,
/// 被控端桌面尺寸（AuthOk 携带）。主控端据此在解码前定好视图宽高比——
/// 若等首帧再改尺寸会触发 SurfaceView 重新测量，把刚创建的解码器 surface 打掉。
desktop: (u32, u32),
/// 保持 Endpoint 存活（drop 会销毁连接）
_endpoint: quinn::Endpoint,
conn: quinn::Connection,
}
impl Session {
/// 建立连接并完成认证握手 + 首帧关键帧请求。
pub async fn connect(cfg: ConnectConfig) -> Result<Self> {
rustls::crypto::ring::default_provider()
.install_default()
.ok();
let tls = if cfg.insecure_skip_verify {
rustls::ClientConfig::builder()
.dangerous()
.with_custom_certificate_verifier(Arc::new(SkipVerify))
.with_no_client_auth()
} else {
// 证书钉扎尚未接入（方案阶段 4）。这里明确报错而不是静默放行。
bail!("insecure_skip_verify=false 需要证书钉扎，尚未实现（见方案阶段 4）");
};
let qc: quinn::crypto::rustls::QuicClientConfig = Arc::new(tls)
.try_into()
.context("build QUIC client config")?;
// 接收窗口放大：视频下行大流量，quinn 默认 512KB 流窗口会卡吞吐
// （与桌面 viewer / agent 的实测调优一致）
let mut transport = quinn::TransportConfig::default();
transport.receive_window(quinn::VarInt::from_u32(16 * 1024 * 1024));
transport.stream_receive_window(quinn::VarInt::from_u32(8 * 1024 * 1024));
let mut client_config = quinn::ClientConfig::new(Arc::new(qc));
client_config.transport_config(Arc::new(transport));
let mut endpoint = quinn::Endpoint::client("0.0.0.0:0".parse()?)?;
endpoint.set_default_client_config(client_config);
let PeerTarget::Direct { host, port } = &cfg.target;
let addr = (host.as_str(), *port)
.to_socket_addrs()?
.next()
.with_context(|| format!("resolve {host}:{port}"))?;
let conn = endpoint
.connect(addr, "localhost")?
.await
.with_context(|| format!("QUIC connect {addr}"))?;
// ---- 流 1：认证 + 视频下行 ----
let (mut v_send, mut v_recv) = conn.open_bi().await?;
write_control(
&mut v_send,
&ControlMsg::Auth {
code: cfg.code.clone(),
proto_version: PROTO_VERSION,
},
)
.await
.context("send Auth")?;
let desktop: (u32, u32);
match read_control(&mut v_recv).await.context("read AuthOk")? {
ControlMsg::AuthOk {
width,
height,
proto_version,
} => {
if proto_version != PROTO_VERSION {
bail!(
"protocol mismatch: agent={proto_version}, viewer-core={PROTO_VERSION}"
);
}
desktop = (width, height);
log(&format!("auth ok, remote desktop {width}x{height}"));
}
ControlMsg::AuthFail(e) => bail!("auth failed: {e}"),
m => bail!("unexpected first message: {m:?}"),
}
// ---- 流 2：输入 + 控制 + Ping/Pong ----
let (mut ctl_send, mut ctl_recv) = conn.open_bi().await?;
// 入会即请求 IDR：从 GOP 中间接入时裸 P 帧解不出来（桌面端同款教训）
write_control(&mut ctl_send, &ControlMsg::KeyFrameReq).await?;
let stats = Arc::new(std::sync::Mutex::new(SessionStats::default()));
// 帧通道：容量 2 足够——消费方（Kotlin）每帧都要喂解码器，
// 积压超过 2 帧说明该跳帧的是消费方的决策，不是这里的。
let (frame_tx, frames) = mpsc::channel::<Frame>(2);
{
let stats = stats.clone();
tokio::spawn(async move {
loop {
match read_video_frame(&mut v_recv).await {
Ok((h, payload)) => {
{
let mut s = stats.lock().unwrap();
s.frames_in += 1;
s.bytes_in += payload.len() as u64;
}
let frame: Frame = (h, payload).into();
if frame_tx.send(frame).await.is_err() {
break; // 消费方已退出
}
}
Err(e) => {
log(&format!("video stream ended: {e}"));
break;
}
}
}
});
}
// 命令通道 → 流 2 写任务
let (cmd_tx, mut cmd_rx) = mpsc::unbounded_channel::<Command>();
tokio::spawn(async move {
loop {
let Some(cmd) = cmd_rx.recv().await else {
break;
};
let msg = match cmd {
Command::Input(ev) => ControlMsg::Input(ev),
Command::KeyFrameReq => ControlMsg::KeyFrameReq,
Command::Ping(t) => ControlMsg::Ping(t),
};
if write_control(&mut ctl_send, &msg).await.is_err() {
break;
}
}
});
// 流 2 读任务：Pong → RTT
{
let stats = stats.clone();
tokio::spawn(async move {
loop {
match read_control(&mut ctl_recv).await {
Ok(ControlMsg::Pong { t, .. }) => {
let rtt = monotonic_ms().saturating_sub(t);
stats.lock().unwrap().rtt_ms = rtt;
}
Ok(_) => {}
Err(e) => {
log(&format!("control stream ended: {e}"));
break;
}
}
}
});
}
// Ping 任务：500ms 一次（与桌面 viewer 一致）
{
let cmd_tx_ping = cmd_tx.clone();
tokio::spawn(async move {
loop {
tokio::time::sleep(Duration::from_millis(500)).await;
if cmd_tx_ping.send(Command::Ping(monotonic_ms())).is_err() {
break;
}
}
});
}
Ok(Self {
tx: cmd_tx,
frames: Arc::new(tokio::sync::Mutex::new(frames)),
stats,
desktop,
_endpoint: endpoint,
conn,
})
}
/// 被控端桌面尺寸 `(width, height)`，来自 AuthOk，解码前即可用。
pub fn desktop_size(&self) -> (u32, u32) {
self.desktop
}
/// 取下一帧。`None` = 会话结束。
/// `&self`：JNI 层里解码线程独占 recv，输入线程可并发 `send_input`。
pub async fn next_frame(&self) -> Option<Frame> {
self.frames.lock().await.recv().await
}
/// 发送一个输入事件（非阻塞入队，由写任务串行发出）。
pub fn send_input(&self, ev: InputEvent) {
let _ = self.tx.send(Command::Input(ev));
}
/// 协议 v3：发送一段文本（UTF-16 code unit）到被控端注入。
///
/// 为什么参数是 `&[u16]` 而不是 `&str`：被控端的注入通道（Windows
/// `KEYEVENTF_UNICODE`）与主控端来源（Java 字符串）都是 UTF-16 code unit，
/// 中间任何一次 UTF-8 往返都会把 IME 提交边界的落单代理项变成替换字符。
/// 见 `common::InputEvent::Text` 的取舍说明。
///
/// 超限处理：截断到 [`common::MAX_TEXT_UNITS`] 并打日志——静默丢弃会让用户
/// 以为长文本已送达。日志只报长度，不报内容。
pub fn send_text(&self, units: &[u16]) {
if units.is_empty() {
return; // 空提交没有注入语义，也不该占用一条消息
}
let (units, truncated) = common::clamp_text_units(units.to_vec());
if truncated {
log(&format!(
"文本超过上限，已截断到 {} 个 code unit",
common::MAX_TEXT_UNITS
));
}
self.send_input(InputEvent::Text { units });
}
/// 请求被控端强制关键帧（解码器错误/跳帧后的恢复路径）。
pub fn request_keyframe(&self) {
let _ = self.tx.send(Command::KeyFrameReq);
}
/// 当前会话统计快照。
pub fn stats(&self) -> SessionStats {
*self.stats.lock().unwrap()
}
/// 被控端地址（日志/UI 用）
pub fn peer_addr(&self) -> String {
self.conn.remote_address().to_string()
}
/// 主动结束会话。
pub async fn close(self) {
let _ = self.conn.close(0u32.into(), b"viewer closing");
}
}
/// 单调毫秒时钟（进程启动起算）。Ping/Pong 两端共用同一口径即可测 RTT。
fn monotonic_ms() -> u64 {
use std::sync::OnceLock;
static T0: OnceLock<Instant> = OnceLock::new();
T0.get_or_init(Instant::now).elapsed().as_millis() as u64
}
/// 轻量日志出口。阶段 2 接 UniFFI 时替换为 Kotlin 侧回调。
fn log(msg: &str) {
eprintln!("[viewer-core] {msg}");
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn frame_from_header_preserves_contract_fields() {
let h = VideoFrameHeader {
width: 1920,
height: 1080,
ts_ms: 42,
seq: 7,
keyframe: true,
};
let payload = vec![0u8; 16];
let f: Frame = (h.clone(), payload.clone()).into();
assert_eq!((f.width, f.height), (1920, 1080));
assert_eq!(f.seq, 7);
assert!(f.keyframe);
assert_eq!(f.payload, payload);
}
#[test]
fn direct_config_defaults_to_insecure_lan_mode() {
let c = ConnectConfig::direct("192.168.1.5", 47800, "abc");
assert!(c.insecure_skip_verify);
match c.target {
PeerTarget::Direct { host, port } => {
assert_eq!(host, "192.168.1.5");
assert_eq!(port, 47800);
}
}
}
#[tokio::test]
async fn strict_verify_mode_is_explicit_unimplemented_error() {
// 安全红线不能静默：显式要求校验时必须是明确错误，而不是悄悄放行。
// 不用 unwrap_err()：它要求 Ok 类型（Session）实现 Debug，而连接对象
// 的 Debug 输出没有验证价值。
let cfg = ConnectConfig {
target: PeerTarget::Direct {
host: "127.0.0.1".into(),
port: 1,
},
code: "x".into(),
insecure_skip_verify: false,
};
match Session::connect(cfg).await {
Err(e) => assert!(e.to_string().contains("证书钉扎"), "实际错误: {e}"),
Ok(_) => panic!("insecure_skip_verify=false 必须明确报错，不得静默放行"),
}
}
}
viewer/src/main.rs
+300 -41
View File
@@ -81,8 +81,15 @@ fn now_ms() -> u64 {
T0.get().unwrap().elapsed().as_millis() as u64 T0.get().unwrap().elapsed().as_millis() as u64
} }
/// 帧到达间隔环形缓冲（取 p95 用）
struct GapRing {
last: Option<Instant>,
ring: [u32; 64],
idx: usize,
filled: usize,
}
/// 会话统计（HUD 数据源，全部原子量，跨任务共享） /// 会话统计（HUD 数据源，全部原子量，跨任务共享）
#[derive(Default)]
struct Stats { struct Stats {
/// 最近 RTT ms（0=未知） /// 最近 RTT ms（0=未知）
rtt: AtomicU64, rtt: AtomicU64,
@@ -95,6 +102,83 @@ struct Stats {
/// 累计收帧数 / 字节数 /// 累计收帧数 / 字节数
frames: AtomicU64, frames: AtomicU64,
bytes: AtomicU64, bytes: AtomicU64,
/// 抖动：相邻帧 |ΔE2E| 的 EMA（ms×16 定点存储避免浮点原子）
jitter_x16: AtomicU64,
prev_e2e: AtomicI64,
/// agent 发送侧（视频下行）QUIC 丢包/发包累计（最近一次 Pong）
agent_lost: AtomicU64,
agent_sent: AtomicU64,
/// H.264 解码失败计数（关键帧自愈验证指标）
dec_fail: AtomicU64,
/// 当前是否处于「参考链断裂、等待 IDR」状态（HUD 显示 RECOVER）
awaiting_idr: AtomicBool,
/// 帧到达间隔环形缓冲
gaps: std::sync::Mutex<GapRing>,
}
impl Default for Stats {
fn default() -> Self {
Self {
rtt: AtomicU64::new(0),
offset: AtomicI64::new(0),
offset_valid: AtomicBool::new(false),
e2e: AtomicU64::new(0),
e2e_valid: AtomicBool::new(false),
frames: AtomicU64::new(0),
bytes: AtomicU64::new(0),
jitter_x16: AtomicU64::new(0),
prev_e2e: AtomicI64::new(-1),
agent_lost: AtomicU64::new(0),
agent_sent: AtomicU64::new(0),
dec_fail: AtomicU64::new(0),
awaiting_idr: AtomicBool::new(true),
gaps: std::sync::Mutex::new(GapRing {
last: None,
ring: [0; 64],
idx: 0,
filled: 0,
}),
}
}
}
impl Stats {
/// 记录一次帧到达：更新间隔环形缓冲与抖动 EMA
fn on_frame_arrival(&self, est: Option<i64>) {
let now = Instant::now();
if let Ok(mut g) = self.gaps.lock() {
if let Some(t) = g.last {
let gap = (now - t).as_millis() as u32;
let i = g.idx;
g.ring[i] = gap;
g.idx = (i + 1) % g.ring.len();
g.filled = (g.filled + 1).min(g.ring.len());
}
g.last = Some(now);
}
// 抖动 = |本帧E2E - 上帧E2E| 的 EMA（RFC3550 风格，÷16 衰减）
if let Some(e) = est {
let prev = self.prev_e2e.load(Ordering::Relaxed);
self.prev_e2e.store(e, Ordering::Relaxed);
if prev >= 0 {
let d = (e - prev).unsigned_abs() as u64;
let old = self.jitter_x16.load(Ordering::Relaxed);
let sm = if old == 0 { d * 16 } else { old + d * 16 / 16 - old / 16 };
self.jitter_x16.store(sm, Ordering::Relaxed);
}
}
}
/// 帧间隔 p95（ms），样本不足返回 None
fn gap_p95(&self) -> Option<u32> {
let g = self.gaps.lock().ok()?;
if g.filled < 8 {
return None;
}
let mut v: Vec<u32> = g.ring[..g.filled].to_vec();
v.sort_unstable();
v.get(v.len() * 95 / 100).copied()
}
} }
fn main() -> Result<()> { fn main() -> Result<()> {
@@ -137,9 +221,27 @@ async fn run(addr: String, port: u16, code: String) -> Result<()> {
let (mut send, mut recv) = conn.open_bi().await?; let (mut send, mut recv) = conn.open_bi().await?;
write_control(&mut send, &ControlMsg::Auth { code }).await?; write_control(
&mut send,
&ControlMsg::Auth {
code,
proto_version: common::PROTO_VERSION,
},
)
.await?;
match read_control(&mut recv).await? { match read_control(&mut recv).await? {
ControlMsg::AuthOk { width, height } => { ControlMsg::AuthOk {
width,
height,
proto_version,
} => {
// 版本已在 Auth 里声明，这里再核对一次：防止被控端是旧版却"接受"了认证
if proto_version != common::PROTO_VERSION {
bail!(
"protocol version mismatch: server={proto_version}, client={}",
common::PROTO_VERSION
);
}
println!("[viewer] auth ok, remote desktop {width}x{height}"); println!("[viewer] auth ok, remote desktop {width}x{height}");
// 流 2：键鼠输入 + Ping/Pong 独立通道，不与视频流竞争 // 流 2：键鼠输入 + Ping/Pong 独立通道，不与视频流竞争
let (isend, irecv) = conn.open_bi().await?; let (isend, irecv) = conn.open_bi().await?;
@@ -160,48 +262,134 @@ fn session(
let (w, h) = (width as usize, height as usize); let (w, h) = (width as usize, height as usize);
let stats = Arc::new(Stats::default()); let stats = Arc::new(Stats::default());
// 读任务（流 1）：H.264 解码 → RGB → mpsc；同时统计端到端延迟与流量 // 读任务（流 1）：H.264 解码 → RGB → 单槽 latest-wins；同时统计端到端延迟与流量
let (frame_tx, frame_rx) = std::sync::mpsc::channel::<Vec<u8>>(); // 帧格式 (src_w, src_h, rgb)；(0,0,_) = 解码失败占位，触发 KeyFrameReq
// 单槽而非无界队列：渲染跟不上 60fps 到达率时旧帧直接被覆盖，绝不积压
//（无界 mpsc 在 NVENC 60fps 下会无限积压 → 延迟线性增长 + 渲染 drain 时
// 长时间阻塞窗口消息泵 → 窗口假死、HUD 停刷新）
let latest: Arc<std::sync::Mutex<Option<(u32, u32, Vec<u8>)>>> =
Arc::new(std::sync::Mutex::new(None));
{ {
let stats = stats.clone(); let stats = stats.clone();
let latest = latest.clone();
let mut decoder = openh264::decoder::Decoder::new() let mut decoder = openh264::decoder::Decoder::new()
.map_err(|e| anyhow::anyhow!("decoder init failed: {e}"))?; .map_err(|e| anyhow::anyhow!("decoder init failed: {e}"))?;
let mut rgb_buf: Vec<u8> = Vec::new();
let mut last_kfreq = Instant::now();
let mut skip_stale = 0u64;
let mut last_arr: Option<Instant> = None;
// 「等待 IDR」状态：参考链一旦断裂（跳帧或解码失败），后续 P 帧**必然**
// 解不出来。此时必须停止喂解码器，只等 IDR 重新同步。
//
// 之前没有这个状态，后果是：跳掉一帧后，每个 P 帧都被送进 decoder →
// 每个都失败 → 每个都打日志 → 每个都触发一次 KeyFrameReq。
// 表现为「decode failed」刷屏，且解码器反复在错误状态里挣扎。
//
// 放在 Stats 里（而非局部变量）以便 HUD 显示 RECOVER 状态。
// 初始 true：入会时已发 KeyFrameReq，拿到第一个 IDR 前不该硬解。
tokio::spawn(async move { tokio::spawn(async move {
let mut size_logged = false;
loop { loop {
match common::read_video_frame(&mut recv).await { match common::read_video_frame(&mut recv).await {
Ok((hdr, payload)) => { Ok((hdr, payload)) => {
stats.bytes.fetch_add(payload.len() as u64, Ordering::Relaxed); stats.bytes.fetch_add(payload.len() as u64, Ordering::Relaxed);
stats.frames.fetch_add(1, Ordering::Relaxed); stats.frames.fetch_add(1, Ordering::Relaxed);
// 与上一帧的到达间隔：≤100ms 才算"突发到达"。
// 桌面静止时 agent 以 2Hz 空闲重发，帧龄天然 ≥500ms，
// 不能按陈旧帧丢弃——否则静止桌面所有帧被跳过 →
// 永久黑屏 + KeyFrameReq/IDR 死循环（v0.8 实测教训）
let burst = matches!(last_arr, Some(t) if t.elapsed() <= Duration::from_millis(100));
last_arr = Some(Instant::now());
// 端到端延迟 = 接收时刻 - 时钟偏移 - 采集时刻（EMA 平滑） // 端到端延迟 = 接收时刻 - 时钟偏移 - 采集时刻（EMA 平滑）
let mut est = None;
let mut stale = false;
if stats.offset_valid.load(Ordering::Relaxed) { if stats.offset_valid.load(Ordering::Relaxed) {
let est = let e =
now_ms() as i64 - stats.offset.load(Ordering::Relaxed) now_ms() as i64 - stats.offset.load(Ordering::Relaxed)
- hdr.ts_ms as i64; - hdr.ts_ms as i64;
if (0..5000).contains(&est) { if (0..5000).contains(&e) {
let old = stats.e2e.load(Ordering::Relaxed) as i64; // 积压判定必须同时满足：突发到达（背靠背）+ 帧龄>250ms。
let sm = if stats.e2e_valid.load(Ordering::Relaxed) { // QUIC 可靠有序，积压须逐帧读出但可不解码——跳过解码
(old * 4 + est) / 5 // 让队列以网络速度排空
} else { stale = burst && e > 250;
est // 空闲重发帧的帧龄不代表显示延迟，不计入 EMA
}; if burst || e < 250 {
stats.e2e.store(sm as u64, Ordering::Relaxed); let old = stats.e2e.load(Ordering::Relaxed) as i64;
stats.e2e_valid.store(true, Ordering::Relaxed); let sm = if stats.e2e_valid.load(Ordering::Relaxed) {
(old * 4 + e) / 5
} else {
e
};
stats.e2e.store(sm as u64, Ordering::Relaxed);
stats.e2e_valid.store(true, Ordering::Relaxed);
est = Some(sm);
}
} }
} }
stats.on_frame_arrival(est);
if stale {
skip_stale += 1;
if skip_stale % 60 == 1 {
eprintln!(
"[viewer] backlog: stale burst frame skipped (total {skip_stale})"
);
}
// 跳帧直接切断 H.264 参考链：之后所有 P 帧都解不出来。
// 立刻转入「等待 IDR」并请求关键帧，绝不能让后续 P 帧
// 继续喂给解码器（那是必然失败的徒劳尝试）。
stats.awaiting_idr.store(true, Ordering::Relaxed);
if last_kfreq.elapsed() >= Duration::from_millis(200) {
last_kfreq = Instant::now();
if let Ok(mut slot) = latest.lock() {
*slot = Some((0, 0, Vec::new()));
}
}
continue;
}
// 参考链未恢复时，P 帧一律跳过——此为正确性要求，不是优化。
if stats.awaiting_idr.load(Ordering::Relaxed) && !hdr.keyframe {
skip_stale += 1;
// 补发请求，防止首次 KeyFrameReq 丢失（时间闸限频）
if last_kfreq.elapsed() >= Duration::from_millis(200) {
last_kfreq = Instant::now();
if let Ok(mut slot) = latest.lock() {
*slot = Some((0, 0, Vec::new()));
}
}
continue;
}
// 拿到 IDR 才能重新开始解码
if hdr.keyframe {
stats.awaiting_idr.store(false, Ordering::Relaxed);
}
match decoder.decode(&payload) { match decoder.decode(&payload) {
Ok(Some(yuv)) => { Ok(Some(yuv)) => {
let (dw, dh) = yuv.dimensions(); let (dw, dh) = yuv.dimensions();
let mut rgb = vec![0u8; dw * dh * 3]; let (dw, dh) = (dw as u32, dh as u32);
yuv.write_rgb8(&mut rgb); if !size_logged {
if frame_tx.send(rgb).is_err() { size_logged = true;
break; eprintln!(
"[viewer] decoded frame {dw}x{dh} (window {}x{})",
hdr.width, hdr.height
);
}
rgb_buf.clear();
rgb_buf.resize(dw as usize * dh as usize * 3, 0);
yuv.write_rgb8(&mut rgb_buf);
// latest-wins：直接覆盖旧帧，无积压
if let Ok(mut slot) = latest.lock() {
*slot = Some((dw, dh, rgb_buf.clone()));
} }
} }
Ok(None) => {} Ok(None) => {}
Err(e) => { Err(e) => {
// 解码异常：请求被控端强制关键帧 // 解码异常：参考链已不可信，转入等待 IDR
stats.awaiting_idr.store(true, Ordering::Relaxed);
stats.dec_fail.fetch_add(1, Ordering::Relaxed);
eprintln!("[viewer] decode failed: {e}, request keyframe"); eprintln!("[viewer] decode failed: {e}, request keyframe");
let _ = frame_tx.send(Vec::new()); // 空帧占位唤醒渲染循环 if let Ok(mut slot) = latest.lock() {
*slot = Some((0, 0, Vec::new()));
}
} }
} }
} }
@@ -220,9 +408,11 @@ fn session(
tokio::spawn(async move { tokio::spawn(async move {
loop { loop {
match read_control(&mut irecv).await { match read_control(&mut irecv).await {
Ok(ControlMsg::Pong { t, agent_ts }) => { Ok(ControlMsg::Pong { t, agent_ts, lost, sent }) => {
let r = now_ms(); let r = now_ms();
stats.rtt.store(r.saturating_sub(t), Ordering::Relaxed); stats.rtt.store(r.saturating_sub(t), Ordering::Relaxed);
stats.agent_lost.store(lost, Ordering::Relaxed);
stats.agent_sent.store(sent, Ordering::Relaxed);
// agent_ts ≈ (t+r)/2 时刻的 agent 时钟读数 → 偏移 EMA // agent_ts ≈ (t+r)/2 时刻的 agent 时钟读数 → 偏移 EMA
let est = agent_ts as i64 - ((t + r) / 2) as i64; let est = agent_ts as i64 - ((t + r) / 2) as i64;
let old = stats.offset.load(Ordering::Relaxed); let old = stats.offset.load(Ordering::Relaxed);
@@ -235,7 +425,10 @@ fn session(
stats.offset_valid.store(true, Ordering::Relaxed); stats.offset_valid.store(true, Ordering::Relaxed);
} }
Ok(_) => {} Ok(_) => {}
Err(_) => break, Err(e) => {
eprintln!("[viewer] control stream closed: {e} (RTT/LOSS 将停止更新)");
break;
}
} }
} }
}); });
@@ -255,9 +448,11 @@ fn session(
let push = |tx: &tokio::sync::mpsc::UnboundedSender<Vec<u8>>, m: ControlMsg| { let push = |tx: &tokio::sync::mpsc::UnboundedSender<Vec<u8>>, m: ControlMsg| {
let _ = tx.send(common::encode_control(&m)); let _ = tx.send(common::encode_control(&m));
}; };
// 入会立即请求 IDR：避免从 GOP 中间接入收到裸 P 帧解码失败导致启动黑屏
push(&msg_tx, ControlMsg::KeyFrameReq);
let mut window = Window::new( let mut window = Window::new(
"mvp-remote viewer (Esc 退出)", "mvp-remote v0.8.1-CatchUp (Esc 退出)",
w, w,
h, h,
WindowOptions { WindowOptions {
@@ -278,6 +473,8 @@ fn session(
let mut hud_bytes = 0u64; let mut hud_bytes = 0u64;
let mut hud_fps = 0u64; let mut hud_fps = 0u64;
let mut hud_mbps = 0f64; let mut hud_mbps = 0f64;
let mut hud_lost = 0u64;
let mut hud_sent = 0u64;
loop { loop {
if !window.is_open() || window.is_key_down(Key::Escape) { if !window.is_open() || window.is_key_down(Key::Escape) {
@@ -290,18 +487,15 @@ fn session(
push(&msg_tx, ControlMsg::Ping(now_ms())); push(&msg_tx, ControlMsg::Ping(now_ms()));
} }
// 1) 渲染最新帧（只保留最新，丢弃积压）；空帧=解码异常，请求关键帧 // 1) 渲染最新帧（单槽 latest-wins：每迭代只取最新一帧，天然防积压）
loop { if let Ok(mut slot) = latest.lock() {
match frame_rx.try_recv() { if let Some((sw, sh, raw)) = slot.take() {
Ok(raw) => { if raw.is_empty() {
if raw.is_empty() { push(&msg_tx, ControlMsg::KeyFrameReq);
push(&msg_tx, ControlMsg::KeyFrameReq); } else if sw > 0 && sh > 0 {
} else if raw.len() == w * h * 3 { blit(&mut buffer, w, h, &raw, sw as usize, sh as usize);
blit(&mut buffer, &raw); dirty = true;
dirty = true;
}
} }
Err(_) => break,
} }
} }
@@ -367,7 +561,9 @@ fn session(
} }
} }
// 6) HUD：每秒刷新一次指标，画在右上角 // 6) HUD：指标每秒刷新一次；每秒至少强制呈现一次（视频冻结/黑屏时 HUD 依然可见）
let mut need_present = dirty;
let (hud_loss, hud_jit, hud_p95, hud_dec);
if hud_last.elapsed() >= Duration::from_secs(1) { if hud_last.elapsed() >= Duration::from_secs(1) {
let f = stats.frames.load(Ordering::Relaxed); let f = stats.frames.load(Ordering::Relaxed);
let b = stats.bytes.load(Ordering::Relaxed); let b = stats.bytes.load(Ordering::Relaxed);
@@ -375,12 +571,47 @@ fn session(
hud_mbps = (b.saturating_sub(hud_bytes)) as f64 * 8.0 / 1e6; hud_mbps = (b.saturating_sub(hud_bytes)) as f64 * 8.0 / 1e6;
hud_frames = f; hud_frames = f;
hud_bytes = b; hud_bytes = b;
// 丢包率 = Δlost / Δsent（agent 发送方向，随 Pong 同步）
let l = stats.agent_lost.load(Ordering::Relaxed);
let s = stats.agent_sent.load(Ordering::Relaxed);
hud_loss = if s > hud_sent {
((l - hud_lost) as f64 / (s - hud_sent) as f64 * 100.0).min(99.9)
} else {
-1.0
};
hud_lost = l;
hud_sent = s;
hud_jit = stats.jitter_x16.load(Ordering::Relaxed) / 16;
hud_p95 = stats.gap_p95();
hud_dec = stats.dec_fail.load(Ordering::Relaxed);
hud_last = Instant::now(); hud_last = Instant::now();
need_present = true; // HUD 面板每秒至少重绘一次
} else {
hud_loss = f64::NAN;
hud_jit = 0;
hud_p95 = None;
hud_dec = 0;
} }
if dirty { if need_present {
let rtt = stats.rtt.load(Ordering::Relaxed); let rtt = stats.rtt.load(Ordering::Relaxed);
let e2e = stats.e2e.load(Ordering::Relaxed); let e2e = stats.e2e.load(Ordering::Relaxed);
let e2e_ok = stats.e2e_valid.load(Ordering::Relaxed); let e2e_ok = stats.e2e_valid.load(Ordering::Relaxed);
let loss_str = if hud_loss.is_nan() {
"--".to_string()
} else if hud_loss < 0.0 {
"--".to_string()
} else {
format!("{hud_loss:.1}%")
};
let loss_c = if hud_loss.is_nan() || hud_loss < 0.0 {
C_GRAY
} else {
lat_color(hud_loss < 1.0, hud_loss < 5.0)
};
let jit_c = lat_color(hud_jit < 10, hud_jit < 30);
let p95_c = hud_p95
.map(|p| lat_color(p < 60, p < 150))
.unwrap_or(C_GRAY);
draw_hud( draw_hud(
&mut buffer, &mut buffer,
w, w,
@@ -392,6 +623,25 @@ fn session(
&if e2e_ok { format!("{e2e}ms") } else { "--".into() }, &if e2e_ok { format!("{e2e}ms") } else { "--".into() },
lat_color(e2e < 100, e2e < 200), lat_color(e2e < 100, e2e < 200),
), ),
("JIT", &format!("{hud_jit}ms"), jit_c),
("P95", &hud_p95.map(|p| format!("{p}ms")).unwrap_or("--".into()), p95_c),
("LOSS", &loss_str, loss_c),
("DECFAIL", &format!("{hud_dec}"), if hud_dec == 0 { C_GRAY } else { C_RED }),
(
// 参考链断裂、正在等 IDR 的显式指示：此时画面冻结是**预期行为**，
// 不是故障。没有这个指示，用户会以为软件卡死。
"STATE",
&if stats.awaiting_idr.load(Ordering::Relaxed) {
"RECOVER".to_string()
} else {
"OK".to_string()
},
if stats.awaiting_idr.load(Ordering::Relaxed) {
C_AMBER
} else {
C_GREEN
},
),
("RATE", &format!("{:.0}MBP", hud_mbps), C_GRAY), ("RATE", &format!("{:.0}MBP", hud_mbps), C_GRAY),
("RES", &format!("{w}x{h}"), C_GRAY), ("RES", &format!("{w}x{h}"), C_GRAY),
], ],
@@ -473,6 +723,7 @@ fn glyph(c: u8) -> [u8; 5] {
b':' => [0x00, 0x36, 0x36, 0x00, 0x00], b':' => [0x00, 0x36, 0x36, 0x00, 0x00],
b'.' => [0x00, 0x60, 0x60, 0x00, 0x00], b'.' => [0x00, 0x60, 0x60, 0x00, 0x00],
b'-' => [0x08, 0x08, 0x08, 0x08, 0x08], b'-' => [0x08, 0x08, 0x08, 0x08, 0x08],
b'%' => [0x13, 0x18, 0x04, 0x0A, 0x11],
b' ' => [0x00; 5], b' ' => [0x00; 5],
_ => [0x00; 5], _ => [0x00; 5],
} }
@@ -546,9 +797,17 @@ fn draw_hud(buffer: &mut [u32], w: usize, lines: &[(&str, &str, u32)]) {
} }
} }
/// RGB888 字节（openh264 解码输出）→ minifb 0RGB u32 缓冲 /// RGB888 字节（openh264 解码输出）→ minifb 0RGB u32 缓冲。
fn blit(buffer: &mut [u32], raw: &[u8]) { /// 支持源尺寸与窗口不同（解码器可能对齐到 16 倍数，如 1080→1088）：
for (px, chunk) in buffer.iter_mut().zip(raw.chunks_exact(3)) { /// 逐行拷贝重叠区域，源小于窗口时右侧/下方留黑。
*px = ((chunk[0] as u32) << 16) | ((chunk[1] as u32) << 8) | chunk[2] as u32; fn blit(buffer: &mut [u32], w: usize, h: usize, raw: &[u8], dw: usize, dh: usize) {
let cw = w.min(dw);
let ch = h.min(dh);
for row in 0..ch {
let src = &raw[row * dw * 3..row * dw * 3 + cw * 3];
let dst = &mut buffer[row * w..row * w + cw];
for (px, chunk) in dst.iter_mut().zip(src.chunks_exact(3)) {
*px = ((chunk[0] as u32) << 16) | ((chunk[1] as u32) << 8) | chunk[2] as u32;
}
} }
} }
借鉴映射与落地规范.md
+301
View File
@@ -0,0 +1,301 @@
# 借鉴映射与落地规范
> 配套《多平台架构设计.md》。本文回答三个问题：**每个模块怎么设计、看谁、抄什么、不抄什么。**
> 状态：架构部分已定稿；第 2 章的「逆向需求清单」等同事的逆向资料回填。
---
## 0. 使用方式
| 角色 | 看哪几章 |
|---|---|
| A（被控侧） | §3.5 platform、§3.6 agent-core、§4 表中 agent 相关行 |
| B（主控+协议） | §3.1–3.4、§3.7 viewer-core、§5 许可 |
| 移动端（待补人） | §3.4 media（解码后端）、§3.8 apps |
| 全员 | §2 逆向红线（踩了会出大事） |
---
## 1. 两条铁律
1. **架构由需求决定，不由参考项目决定。** 参考项目用来解决"这一块业界怎么做的"，不是用来决定"我们做成什么样"。
2. **能抄的是算法和协议思路，抄不到的是平台集成和 UI。** 后者是体力活，任何参考项目都救不了（它们自己也是硬写的）。
---
## 2. 逆向资料的用法与红线
### 2.1 红线（务必转告同事）
| 可以做 | 不可以做 |
|---|---|
| 记录**功能清单、交互流程、界面布局** | 把反编译产物、反汇编清单提交进仓库 |
| 记录**用户可见的错误提示、状态显示** | 逐字复制对方的 UI 文案、图标、素材 |
| 推断**产品模型**（账号/设备/授权怎么设计） | 抄对方私有协议的具体编码（抄了也不能互通） |
| 记录**网络行为的宏观特征**（是否中转、连接耗时） | 提取并移植对方的算法实现 |
**仓库里只提交"行为结论"，不提交"逆向产物"。** 建议逆向资料单独放一个**不纳入 git** 的目录（或内部网盘），文档里只写结论。这条不是洁癖——商用产品里出现竞品反编译代码是实打实的法律风险。
### 2.2 逆向需求清单（直接发给同事）
**A. 产品功能面（最重要，优先做）**
1. 主控端完整交互流程截图流：登录 → 设备列表 → 连接（输密码/免密）→ 会话中 → 断开
2. 每个平台的功能矩阵：Win/Linux/Android/iOS 各有哪些功能、哪些缺失
3. 会话中工具栏全清单：分辨率切换、多屏、文件传输、剪贴板、音频开关、画质/帧率档位、游戏模式
4. 设置项全清单（分"被控设置"和"主控设置"两类）
5. 错误提示文案全集 + 断线重连时的用户可见行为
**B. 身份与授权模型**
6. 设备 ID 的形式（长度/字符集）、在哪里查看、能否自定义
7. 授权方式有几种：临时密码 / 固定密码 / 免密白名单 / 账号直接授权？默认是哪种
8. 账号体系：是否必须登录、登录后设备列表怎么呈现、一台设备能否被多账号访问
9. 被控端的"允许谁连"控制（黑名单/白名单/单次确认弹窗）
**C. 被控端形态**
10. 是不是系统服务？开机自启吗？锁屏/注销/UAC 提权界面能不能连？
11. 安装包形态、是否需要管理员权限、卸载残留
**D. 网络行为（只看宏观，不要深入协议）**
12. 连接时用户是否能看到"直连 / 中转"标识
13. 典型连接耗时、转中继时的画质变化
14. 端口/协议宏观特征：UDP 还是 TCP 为主、监听端口是否固定
**E. 商业化（可选但有用）**
15. 免费额度/收费点/增值功能边界
### 2.3 逆向资料回填位置
| 资料 | 回填到 |
|---|---|
| A 组 | 本文 §6「产品范围基线」+ 里程碑取舍 |
| B/C 组 | §3.3 session（身份模型影响状态机） |
| D 组 | §3.2 conn（验证我们的打洞/中继策略） |
| E 组 | 不在本文，属于商业决策 |
---
## 3. 模块级设计规范
### 3.1 `proto` — 协议
**职责**：信令消息、控制消息、媒体容器格式。三端（agent/viewer/信令服务）共享契约。
**设计点**
- **双协议分离**：信令协议（`signaling.rs`，文本友好、低频、可 JSON）与媒体控制协议（`lib.rs`，二进制、高频）**不要混**。前者要便于抓包排查，后者要极致紧凑。
- **版本协商**：`SIGNALING_VERSION` 不一致直接拒绝，不做兼容。媒体协议用"能力协商"而非版本号（见 §3.3）。
- **消息演进**：现用 bincode。二期若换 protobuf，**必须一次性切换**，不要两套并存。
- **单帧长度上限**：现有 `ensure!(len < 1<<20)` 这类边界检查要保留，且信令侧对端候选列表必须经 `sanitize_candidates`。
**看谁 / 抄什么**
| 项目 | 抄什么 | 不抄 |
|---|---|---|
| RustDesk `libs/hbb_common` | 消息分类方式（信令/中继/控制分层） | 具体协议编码（AGPL + 抄了不互通） |
| Moonlight `moonlight-common-c` | **控制消息与媒体分离**的成熟做法 | 它的私有控制协议 |
**产出物**：`crates/proto/src/{lib.rs, signaling.rs}`
---
### 3.2 `conn` — 连接层（最值得借鉴的一块）
**职责**：信令客户端、打洞编排、路径选择、QUIC 会话、保活。
**设计点**
- **`PathProvider` 抽象**（已在《打洞与信令设计.md》定义）：现在只实现 Direct，类型上预留 Relay。
- **打洞编排**：候选收集（v6/Host/UPnP/Reflexive/Predicted）→ 排序 → 信令交换 → 同步打洞 → 落选清理。
- **同一 socket 双角色**：打洞用的 UDP socket 必须与后续 QUIC 是同一个，否则洞不在同一映射上。
- **保活**：按双方 `binding_timeout_s` 折半，5–60s。**这是"连一会儿就断"的根治点。**
- **不用 QUIC 握手当探测**：先用轻量 UDP 探测包确认通路（秒级失败），再 `connect`。
**看谁 / 抄什么**
| 项目 | 抄什么 | 许可 |
|---|---|---|
| **iroh** | **整套**：打洞编排、中继回退、地址发现。技术栈与你们重合（Rust + quinn） | Apache-2.0 ✅ **可直接依赖** |
| webrtc-rs / pion/ice | ICE 状态机、候选配对（candidate pairing）策略 | MIT/Apache ✅ |
| Tailscale / headscale | DERP 中继的**回退判定条件**与降级策略 | BSD-3 ✅ |
| RustDesk `rendezvous_mediator` 相关 | 信令 ↔ 打洞的**编排顺序**（读思路） | AGPL ⚠️ 只读 |
| libjuice / libnice | ICE 精简实现（作为对照，不必依赖） | MPL / LGPL |
> **建议**：`conn` 优先评估**直接依赖 iroh**。它解决的正是你们纠结的"打洞要不要中继"，而且给了"直连优先、失败回退"的完整实现。自研仅当 iroh 有硬伤（体积、可控性、特定 NAT 场景）时才考虑。
**产出物**：`crates/conn/`
---
### 3.3 `session` — 会话编排（"一体程序"的核心）
**职责**：会话状态机、角色仲裁、能力协商、生命周期管理、被控端准入控制。
**设计点**
- **状态机双线**：主控线 / 被控线各自独立，同一进程可并发（被控中也能主控别人）。
- **角色仲裁**：打洞阶段双方对称，**谁先打通谁定角色**（被控方做 QUIC server），不预设。
- **能力协商**：被控端上报硬件能力（可用编码器、最大分辨率、帧率上限、是否支持音频），主控端选择。**用能力协商代替版本号**——旧客户端缺字段就是"不支持"，天然向后兼容。
- **准入控制**：单会话/多会话、白名单、单次确认弹窗。**这一条直接来自 §2 逆向需求 B 组第 7/9 项**，等资料回填后定稿。
- **失败出口**：`PunchFailReason::user_hint()` 必须一路透传到 UI，不允许出现无解释的"连接中"。
**看谁 / 抄什么**
| 项目 | 抄什么 |
|---|---|
| RustDesk `src/server` / `src/client` | 会话生命周期与连接管理的分层方式（读思路） |
| Sunshine + Moonlight | host/client 两侧的**能力协商**与配置下发流程 |
| — | 状态机本身建议自己画，不要套用别人的（各家产品逻辑差异大） |
**产出物**：`crates/session/`
---
### 3.4 `media` — 编解码抽象
**职责**：编码/解码 trait、各平台后端、零拷贝通路、码率自适应闭环。
**设计点**
- **不对称策略**：**编码用 FFmpeg**（只在 Win/Linux，桌面端不心疼包体），**解码用平台原生**（四个主控平台都要，移动端必须硬解且要零拷贝）。
- **trait 收口**（见《多平台架构设计.md》§4 决策 4），上层不允许出现平台判断。
- **零拷贝**：解码器直接输出到渲染表面（D3D11 纹理 / `SurfaceView` / `CAMetalLayer`），中间不做 CPU 拷贝。
- **软硬兜底**：硬编失败回落 openh264；硬解失败回落软解（桌面可，移动端应直接提示而非硬撑）。
- **码率自适应闭环**：接收端统计（丢包/抖动/帧间隔 p95）→ 反馈编码端 → AIMD 调码率。协议已留 `RateReport`（见 B侧设计文档 B2）。
**看谁 / 抄什么**
| 项目 | 抄什么 | 许可 |
|---|---|---|
| **libwebrtc** `modules/congestion_controller/goog_cc` | **GCC 拥塞控制**的完整思路——这是码率自适应的教科书 | BSD-3 ✅ 可参考可移植 |
| **libwebrtc** `modules/video_coding/jitter_buffer` | 抖动缓冲与丢包重排 | BSD-3 ✅ |
| **Sunshine** `src/video` + `src/platform` | **抓屏→编码→打包**的完整管线组织，尤其是多编码器抽象与回落 | GPL-3.0 ⚠️ 只读 |
| **Moonlight** `moonlight-common-c` | 接收端解码→渲染→HUD 的管线，以及**丢包恢复策略** | GPL-3.0 ⚠️ 只读 |
| FFmpeg | NVENC/QSV/AMF/VAAPI 的调用姿势与参数 | LGPL/GPL ⚠️ 注意构建选项 |
| openh264 | 软编兜底（已在用） | BSD ✅ |
**产出物**：`crates/media/`
---
### 3.5 `platform` — L0 平台层（抄不到，只能自己写）
**职责**：抓屏、注入、渲染表面、音频 IO、权限、常驻。
**为什么抄不到**：这层是各平台 API 的体力活。参考项目能给的是"用哪个 API、踩过什么坑"，不是可复用代码。
**设计点**
| 平台 | 关键坑 |
|---|---|
| Win 抓屏 | DXGI DD 在**会话切换/RDP 断开**时失效，必须处理 `DXGI_ERROR_ACCESS_LOST` 并重建 |
| Win 注入 | 服务进程注入到用户会话需要 `CreateProcessAsUser` + 会话隔离处理；UAC 提权界面默认收不到输入（需 `uiAccess` 签名） |
| Win 常驻 | 服务 + 用户会话代理双进程，锁屏可连靠的是**服务侧保持会话** |
| Linux 抓屏 | X11 用 XShm/XComposite；**Wayland 必须走 PipeWire + xdg-desktop-portal**（有授权弹窗，无法静默） |
| Linux 注入 | X11 用 XTest；**Wayland 用 libei/uinput**（uinput 需 root 或 udev 规则） |
| Linux 常驻 | systemd 服务 + 会话切换处理（Wayland 的 portal 授权是最大障碍） |
| 渲染表面 | 四平台各一套；**必须支持"表面尺寸≠视频尺寸"的缩放与拉伸模式** |
**看谁**
| 项目 | 看什么 |
|---|---|
| Sunshine `src/platform/{windows,linux}` | 抓屏与注入的分平台组织方式、Wayland 处理 |
| RustDesk `libs/scrap` | 同样的抓屏抽象（多平台统一 trait）——**这是它最有价值的部分** |
| libei / xdg-desktop-portal 官方文档 | Wayland 的正规姿势（别用野路子） |
**产出物**：`crates/platform/`
---
### 3.6 `agent-core` / `viewer-core` — 两条链路
**职责**：把现有 `agent`/`viewer` 的 main 逻辑抽成库，供一体程序按角色调用。
**设计点**
- **`agent-core`**：capture → encode → 打包 → 发送；输入接收 → 注入。A 已在进行中（`agent-core/src/lib.rs` 398 行）。
- **`viewer-core`**：接收 → 解码 → 渲染表面；输入采集 → 发送。
- **生命周期**：两者都不允许自己决定退出，由 `session` 层驱动。
- **单槽 latest-wins**：现有做法保留（网络慢时只发最新帧），这是正确设计。
**看谁**：RustDesk（抽库方式）、Sunshine/Moonlight（管线组织）。
**产出物**：`crates/agent-core/`、`crates/viewer-core/`
---
### 3.7 `apps/*` — UI 与常驻（第二个抄不到的地方）
**职责**：UI 壳、常驻服务、安装包。
**设计点**
- **硬约束**：所有业务逻辑在 Rust，UI 只调一层稳定 API（决定了将来能否换 UI 框架）。
- **视频走原生子表面**，绝不在 WebView 里逐帧渲染。
- **一体 = 一个安装包 + 两个进程**：前台 UI + 后台被控服务，本地 IPC。
- **移动端只有前台进程**（纯主控）。
**看谁**
| 项目 | 抄什么 |
|---|---|
| Tauri v2 官方 mobile 示例 | WKWebView/Android WebView 与原生 surface 的叠加方式 |
| RustDesk `flutter/` | 同一套 UI 适配多端的组织方式（它用 Flutter，可对照取舍） |
| Moonlight `moonlight-android` / `moonlight-ios` | **移动端主控的完整交互**：虚拟鼠标、手势映射、软键盘、性能档位 |
**产出物**：`apps/desktop/`、`apps/android/`、`apps/ios/`、`apps/daemon-win/`、`apps/daemon-linux/`
---
## 4. 借鉴映射总表
| 模块 | 主参考 | 许可 | 用法 |
|---|---|---|---|
| 打洞 / 中继 / 地址发现 | **iroh** | Apache-2.0 | **直接依赖** |
| ICE / STUN / TURN | webrtc-rs、pion、coturn | MIT/BSD | 依赖或参考 |
| 中继回退策略 | Tailscale / headscale | BSD-3 | 读思路 |
| 拥塞控制 / 抖动缓冲 | **libwebrtc** | BSD-3 | 读思路，重写 |
| 抓屏→编码管线 | **Sunshine** | GPL-3.0 | 只读 |
| 接收端解码→渲染 | **Moonlight** | GPL-3.0 | 只读 |
| 整机架构 / 抽库方式 | **RustDesk** | **AGPL-3.0** | **只读，绝不抄代码** |
| 移动端主控交互 | Moonlight mobile | GPL-3.0 | 只读 |
| Rust↔移动端绑定 | UniFFI | MPL-2.0 | 依赖 |
---
## 5. 许可结论与依赖策略
| 许可 | 能否进产品 | 项目 |
|---|---|---|
| MIT / Apache-2.0 / BSD | ✅ 可依赖、可移植 | iroh、webrtc-rs、pion、coturn、libwebrtc、UniFFI(MPL 弱) |
| LGPL | ⚠️ 可动态链接，改库要开源 | FFmpeg、GStreamer、libnice |
| **GPL / AGPL** | ❌ **不可进代码** | **RustDesk(AGPL)、Sunshine/Moonlight(GPL)** |
**AGPL 特别提示**：网络服务也算分发。引入 RustDesk 代码 = 你们整个产品连服务端都要开源。
**依赖准入流程**（补充进《Git协作规范.md》的合并规则）：新增依赖必须在 PR 描述里写明**许可证 + 是否修改其源码**，由另一人确认。
---
## 6. 产品范围基线（待逆向资料回填）
> 拿到 §2.2 的 A 组资料后填这里，用于定"做哪些功能、砍哪些平台"。
| 功能 | UU 有 | 我们一期 | 备注 |
|---|---|---|---|
| 桌面画面 | | ✅ | 已有 |
| 键鼠控制 | | ✅ | 已有 |
| 音频 | | ? | |
| 文件传输 | | ? | |
| 剪贴板同步 | | ? | |
| 多屏 | | ? | |
| 移动端主控 | | ? | 人力问题见《多平台架构设计.md》§8 |
| 无人值守 | | ? | 安全含义，需固定密码+白名单 |
---
## 7. 待拍板
1. `conn` 是否**直接依赖 iroh**（而不是自研打洞）——建议先花 2 天做技术验证
2. 逆向资料回来后，"产品范围基线"的取舍（尤其移动端）
3. `media` 编码侧是否引入 FFmpeg（包体/构建成本 vs 省下的适配工作）
4. 依赖准入流程是否写进《Git协作规范.md》
多平台架构设计.md
+315
View File
@@ -0,0 +1,315 @@
# 多平台架构设计
> 需求：**主控端** Windows / Linux / Android / iOS；**被控端** Windows / Linux。
> 桌面端（Win/Linux）主控+被控合为一体；移动端仅主控。
> 前置文档：《打洞与信令设计.md》《B侧设计文档.md》《构建运行手册.md》
---
## 0. 一句话架构
**一个 Rust 核心 + 一层薄平台适配 + 可替换的 UI 壳。**
业务逻辑（连接/打洞/会话/协议）100% 在 Rust 里跨平台共享；只有**抓屏、编解码、注入、渲染表面**四件事分平台写；UI 层只管"壳"，**永远不碰视频流**。
---
## 1. 纵向分层
```
┌──────────────────────────────────────────────────────────┐
│ L4 UI 壳 Win/Linux/Android/iOS 各自实现 │
│ 设备列表 / 设置 / 连接管理 / 悬浮控件 │
│ ★ 绝不放视频渲染 │
├──────────────────────────────────────────────────────────┤
│ L3 会话层 session 纯 Rust，全平台共享 │
│ 会话状态机、角色仲裁、能力协商、keepalive 调度 │
├──────────────────────────────────────────────────────────┤
│ L2 连接层 conn 纯 Rust，全平台共享 │
│ 信令客户端、打洞编排、中继接口(预留)、QUIC(quinn)多流 │
├──────────────────────────────────────────────────────────┤
│ L1 媒体层 media Rust + 平台后端 │
│ 编码/解码/色彩转换/缩放/音频(Opus) trait 化 │
├──────────────────────────────────────────────────────────┤
│ L0 平台层 platform 唯一必须分平台的地方 │
│ 抓屏 / 注入 / 音频IO / 渲染表面 / 权限 / 常驻 │
└──────────────────────────────────────────────────────────┘
```
**分界线原则**：L1 以上任何代码不允许出现 `#[cfg(windows)]`。平台差异只能通过 L0 的 trait 暴露。
---
## 2. 仓库结构
```
remote/
├── crates/
│ ├── proto/ # 协议：信令 + 控制 + 媒体容器（现 common/）
│ ├── conn/ # 信令客户端 / 打洞编排 / QUIC 会话 / 中继接口
│ ├── session/ # 会话状态机 + 角色仲裁（"一体"的核心）
│ ├── agent-core/ # 被控链路：capture → encode → send [A 已在做]
│ ├── viewer-core/ # 主控链路：recv → decode → surface
│ ├── media/ # 编解码 trait + 各平台后端 + openh264 软兜底
│ └── platform/ # L0 实现，按 cfg 编译各平台
├── apps/
│ ├── desktop/ # Tauri v2 壳：Win/Linux 一体程序
│ ├── android/ # 主控（Tauri mobile 或原生 Kotlin）
│ ├── ios/ # 主控（Tauri mobile 或原生 Swift）
│ ├── daemon-win/ # Windows 服务：被控常驻
│ └── daemon-linux/ # systemd：被控常驻
├── bindings/ # UniFFI 生成的 Kotlin/Swift 绑定
└── tools/ # bench / 信令服务器
```
`agent-core` 与 `viewer-core` 的拆分是**一体化的前提**：现在的 `agent`/`viewer` 是两个 main，必须先变成库，才能被同一个进程按角色调用。A 正在做的 `agent-core` 抽取就是在铺这条路。
---
## 3. 平台能力矩阵
| 能力 | Windows | Linux | Android | iOS |
|---|---|---|---|---|
| 抓屏 | DXGI Desktop Duplication | X11 (XShm) / Wayland (PipeWire portal) | — | — |
| 硬编 | NVENC / QSV / AMF | VAAPI / NVENC | — | — |
| 硬解 | D3D11VA / NVDEC | VAAPI | MediaCodec | VideoToolbox |
| 渲染表面 | D3D11 子窗口 | X11/GL 子窗口 | SurfaceView | UIView + CAMetalLayer |
| 键鼠注入 | SendInput | XTest / uinput / libei | — | — |
| 键鼠采集 | Raw Input | evdev | 触摸→鼠标映射 | 触摸→鼠标映射 |
| 音频出 | WASAPI loopback | PulseAudio monitor | — | — |
| 音频入/放 | WASAPI | PulseAudio | AudioTrack | AVAudioEngine |
| 常驻服务 | Windows Service | systemd | — | — |
| **角色** | **主 + 被** | **主 + 被** | **仅主** | **仅主** |
移动端没有"被控"，所以 Android/iOS **不需要**抓屏、编码、注入、服务化——少了一半工作量，这是需求里最省事的一条。
---
## 4. 六个关键决策
### 决策 1：视频渲染绝不走 UI 框架
从 WebView / Compose / Flutter 里逐帧渲染 60fps 视频必然掉帧、增延迟。
**做法**：UI 框架只画"壳"，视频走**原生子表面**叠加在 UI 之上——
Win: 子 HWND + D3D11；Linux: 子窗口 + GL；Android: `SurfaceView`；iOS: `UIView` + `CAMetalLayer`。
MediaCodec / VideoToolbox 都能**直接解码到该 Surface，零拷贝**。这是 Parsec/ToDesk 这类产品的通行做法。
### 决策 2：UI 框架 —— 推荐 Tauri v2，但必须可替换
| 方案 | 覆盖 | 评价 |
|---|---|---|
| **Tauri v2** | Win/Linux/Android/iOS | ✅ 推荐：Rust 原生、一套 UI 四端、IPC 天然 |
| Flutter | 四端 | UI 强，但 Rust 桥（flutter_rust_bridge）+ 平台视图叠加更绕 |
| 各平台原生 ×4 | 四端 | 性能最好，**2-3 人团队不可能维护** |
选 Tauri v2 的理由：你们本来就写 Rust，视频走原生表面后 WebView 只做表单和列表，性能足够。
**代价**：Tauri 移动端还不够成熟（WKWebView 与原生 surface 叠加、生命周期）。所以——
> **硬性约束：所有业务逻辑必须在 Rust 里，UI 只通过一层稳定 API 调用。**
> 这样移动端若体验不达标，可以单点退化成原生 Kotlin/Swift + UniFFI，桌面仍用 Tauri，核心零改动。
### 决策 3：Rust ↔ 移动端绑定用 UniFFI
6 个平台角色组合下手写 JNI/FFI 会失控。UniFFI 从 Rust 生成 Kotlin/Swift 绑定，含 async、错误类型、对象生命周期，是当前标准解法。
### 决策 4：编码用 FFmpeg，解码用平台原生
不对称，但都有理由：
- **编码（只在 Win/Linux 被控端）**：用 FFmpeg 统一接管 NVENC/QSV/AMF/VAAPI，openh264 软编兜底。桌面端不心疼包体，省下大量适配。
- **解码（四个主控平台都要）**：直接用平台原生 API。
- 移动端**必须**硬解（软解烧电池、发热、跑不动 1080p60）
- 平台原生能零拷贝直出渲染表面，FFmpeg 反而多一次拷贝
- 绕开 FFmpeg 在 Android/iOS 上的交叉编译地狱和许可问题
`media` crate 用 trait 收口：
```rust
pub trait VideoDecoder {
fn configure(&mut self, cfg: &VideoConfig, target: SurfaceHandle) -> Result<()>;
fn decode(&mut self, packet: &[u8]) -> Result<Option<FrameHandle>>;
}
pub trait VideoEncoder {
fn configure(&mut self, cfg: &EncodeConfig) -> Result<()>;
fn encode(&mut self, frame: &FrameHandle, out: &mut Vec<u8>) -> Result<()>;
fn request_keyframe(&mut self);
}
```
### 决策 5："一体程序" = 一个安装包 + 两个进程
用户视角是一个 app，实现上职责分离：
```
┌────────────────────────┐ ┌──────────────────────────┐
│ 前台进程（UI） │ IPC │ 后台常驻（被控服务） │
│ 主控画面 / 设备列表 │<──────>│ 抓屏·编码·注入·保活 │
│ 退出即结束 │ 本地管道│ 开机自启 / 锁屏可连 │
└────────────────────────┘ └──────────────────────────┘
▲ ▲
│ 移动端只有这一个 │ 仅 Win/Linux
```
理由：**两种角色的生命周期天然冲突**——被控要"开机就在、没人操作也在"，主控要"跟窗口走、关了就走"。硬塞进一个进程会导致主控一退出就掉线。
IPC 用命名管道（Win）/ Unix socket（Linux），或复用 QUIC 走本地回环。
### 决策 6：中继接口必须预留 —— 哪怕现在不实现
`conn` 层定义：
```rust
pub trait PathProvider {
/// 返回一条可用数据通道：直连优先，失败时可回退
async fn establish(&self, peer: &PeerInfo) -> Result<Path>;
}
pub enum Path { Direct { addr: SocketAddr }, Relay { endpoint: EndpointId } }
```
现在只实现 `Direct`。**但类型上留出 `Relay`**，将来补中继不必动会话层。
> ⚠️ **这条与"不做中继"的决定直接相关，见 §6。**
---
## 5. 会话状态机（L3 核心）
一体程序两条线：
```
主控线: Idle → Signalling → Punching → Handshaking → Streaming → Closing
被控线: Idle → Registered → Incoming → Punching → Handshaking → Serving → Closing
↑ 同一进程两线可并发（被控中也能主控别人）
```
要点：
- 打洞阶段**双方对称**：谁都不预设客户端/服务端，各自向对端候选发包，谁先打通谁定角色（被控方做 QUIC server）
- 会话能力协商：分辨率、编码器、码率上限、是否带音频——由**被控端**根据硬件能力给候选，主控端选
- 单进程内用 tokio 一runtime 跑多会话；被控同时只服务一个主控（可配置）
---
## 6. ✅ 已裁决：不做中继（2026-10-28）
> **裁决结论**：公司不提供中继服务器，中继不做。以下保留原始分析以免后人重蹈讨论。
**手机在蜂窝网络下几乎 100% 处于运营商级 NAT（CGNAT），打洞打不通。**
这不是实现问题，是网络事实——`100.64.0.0/10` 那段地址根本没给你可映射的公网端口。
后果：**"主控端支持安卓/iOS" + "不做中继" = 手机只能在 WiFi 下用。**
地铁上、外面、用流量，一律连不上。
三条出路：
| 方案 | 代价 | 说明 |
|---|---|---|
| **A. 只靠 IPv6** | 部分用户不可用 | 国内三大运营商 v6 覆盖率高，手机 v6 可用性远好于 v4 公网 |
| B. 补一个只为移动端服务的中继 | 视频带宽成本 | **已排除：公司不提供额外服务器** |
| C. 接受"手机只能 WiFi" | 产品力受限 | 零成本，但要写进产品说明 |
### 6.1 裁决内容与边界（2026-10-28）
| 项 | 裁决 |
|---|---|
| 中继 / TURN / relay 服务器 | **不做**。公司不提供额外服务器，且无预算计划 |
| IPv6 | **一等公民**。候选地址排序 v6 优先，`signaling.rs` 已实现 |
| 中继接口预留 | **不预留**。不写 `PathProvider` trait、不写 `Path::Relay` 变体 |
| 连接层边界 | **仍要做**。理由见 §6.2，与中继无关 |
**为什么不留中继接口**：留接口的收益是「将来补中继不用动会话层」，但既然中继已被排除，
`Path::Relay` 这类变体永远不会被构造，只会产生 `dead_code` 警告，要么 `#[allow]` 要么日后删除。
**零收益的预留就是纯负担。**
### 6.2 连接层边界与中继接口是两件事
这两件事常被混为一谈，必须分开：
| | 内容 | 现在做不做 |
|---|---|---|
| **连接层边界** | 把"怎么连"（信令/打洞/直连）与"连上干什么"（收流/解码/渲染）分开 | ✅ **要做** |
| **中继接口** | `enum Path { Direct, Relay }` 这类类型预留 | ❌ **不做** |
连接层边界本身就有独立价值，与中继无关：
1. **连接方式确定会变**：局域网 `IP:端口` → 将来设备码 + 打洞，这是已定的路线；
2. **安卓主控要复用**：`viewer-core` 收流解码渲染不应知道对面是怎么连上的；
3. **可测性**：解码逻辑能脱离真连接单独测试。
做法是把它抽成一个函数边界（输入目标描述、返回 `quinn::Connection`），
**不是**定义 trait 或枚举。
### 6.3 对产品的影响（必须写进对外说明）
| 网络环境 | 能否连接 |
|---|---|
| 双方任一侧无公网 IPv6，且处于 CGNAT 后 | ❌ **无法连接，无解** |
| 双方都有公网 IPv6 | ✅ 可直连 |
| 局域网 / 双方都有可打洞的公网 IPv4 | ✅ 可直连 |
| 对称 NAT（双方） | ❌ 通常不通 |
**这不是 bug，是已知边界。** 用户在外网连不上时，UI 必须给出可执行提示
（`PunchFailReason::user_hint()` 已提供文案），不能停留在"连接中"。
---
## 7. 分期路线（不要一次做 6 个组合）
| 里程碑 | 内容 | 预估 | 依赖 |
|---|---|---|---|
| **M1** | Win↔Win 打通：信令 + 打洞 + 证书钉扎 | 2-3 周 | 协议已就绪 |
| **M2** | Win 一体化：agent-core 抽库、会话状态机、前台/后台双进程 | 2-3 周 | M1 |
| **M3** | Win 被控服务化（开机即在线、锁屏可连、UAC 处理） | 2 周 | M2 |
| **M4** | Linux 被控（X11 先行，Wayland/PipeWire 预留） | 2-3 周 | M2，A 在做 |
| **M5** | 桌面 UI（Tauri v2 壳 + 原生视频表面） | 3 周 | M2 |
| **M6** | **Android 主控**（MediaCodec + UniFFI + 触摸映射） | 3-4 周 | M5 |
| **M7** | **iOS 主控**（VideoToolbox + SwiftUI/UniFFI） | 3-4 周 | M6 |
| **M8** | Linux 主控（复用 M5 的桌面 UI） | 1 周 | M5 |
| 穿插 | 音频(Opus) / 文件传输 / 多屏 | 各 1-2 周 | — |
**先做 Android 再做 iOS**：不用苹果开发者账号、真机调试方便、热重载快、审核风险低。iOS 放最后。
---
## 8. ⚠️ 必须说清楚的人力问题
现有团队 **2-3 人**，目前连"Win↔Win 打通"都还没做完。而这份需求是：
- 2 个被控平台（抓屏 + 编码 + 注入 + 服务化）× 各自两套显示服务器（Linux X11/Wayland）
- 4 个主控平台（解码 + 渲染 + 输入 + UI）× 各自一套原生集成
- 外加信令、打洞、音频、文件、多屏、账号体系
**这是 6-10 人年的工作量，2-3 人做不完。** 不是唱衰，是排期事实——Android + iOS 两个平台的 UI 和平台集成，单独拿出来就够一个人全职做半年。
可行的收缩方式（选一个）：
1. **砍掉 iOS**（苹果审核 + 真机调试 + 开发者账号成本最高，收益最小）
2. **移动端只做 Android**，iOS 用 Web 端替代（但 Web 端无法硬解，体验差）
3. **桌面优先，做深做透**，移动端放到"产品验证之后"，先招人
4. 移动端主控**外包**，核心协议和桌面端自己掌握
我的建议是 **3 或 1**：先把 Win/Linux 两端做成一款真的好用的产品（含打洞、服务化、音频、文件），移动端等有用户了再投入。现在同时铺 6 个平台，最大概率是每个都半成品。
---
## 9. 给 A / B 的分工调整建议
| 人 | 主责 | 说明 |
|---|---|---|
| A | `platform/` 被控侧：Win 抓屏/编码/注入/服务化 + Linux 被控 | 与现状一致 |
| B | `proto/` + `conn/` + `session/` + `viewer-core/` + 桌面 UI | 与现状一致 |
| — | **移动端（Android/iOS）** | **无人** ← 这是缺口 |
现有分工在"桌面 + 协议"上是自洽的，**移动端是纯增量，必须补人或砍需求**。
---
## 10. 开放问题（需要拍板）
> §10 第 1 条已于 2026-10-28 裁决，见 §6。
1. ~~中继：确认走 §6 的 A 方案（v6 优先、留接口不实现）？~~
**已裁决：不做中继、不留接口、v6 优先、但保留连接层边界。详见 §6.1 / §6.2。**
2. 移动端：先 Android 还是直接砍 iOS？
3. UI 框架：确认 Tauri v2，还是先原生桌面 + 后面再说移动？
4. 账号体系：现在是"输验证码"，要支持设备列表就得有账号/设备 ID 体系——什么时候做？
5. 被控服务化后，主控端要不要支持"无人值守开机连接"（有安全含义，需要固定密码 + 白名单）？
打洞与信令设计.md
+124
View File
@@ -0,0 +1,124 @@
# 打洞与信令设计（无中继方案）
> 状态：协议契约已落在 `common/src/signaling.rs`，已合入 main（`703eb64`）。
> 决策：**不做中继（无 TURN / coturn）**，服务器只保留信令 + STUN 职能。
> **裁决确认（2026-10-28）**：公司不提供中继服务器，本决策为最终结论，
> 同时**不预留中继接口**（理由见《多平台架构设计.md》§6.1 / §6.2）。
> 关联：`构建运行手册.md` §7 第 4 项、`Git协作规范.md` 二·B 第 ③ 项。
## 1. 结论先行：信令不是中继
| | 信令服务器 | 中继服务器（已砍掉） |
|---|---|---|
| 承载媒体流量 | ❌ 不承载 | ✅ 全部承载 |
| 生命周期 | 只在建连那几秒 | 整个会话期间 |
| 成本 | 一台最低配 VPS 足够 | 视频流带宽，主要成本项 |
| 必要性 | **必须**（无它则无法交换地址） | 可选（可用率兜底） |
砍掉中继 ≠ 不要服务器。信令必须保留，因为它解决的是死循环：双方都在 NAT 后，
各自只知道自己内网地址，**"自己的公网映射"这个信息只有第三方发包过来时才能看见**。
## 2. 无中继的代价：可用率，不是功能
打洞成功率由 NAT 类型决定，与代码质量基本无关：
| NAT 类型 | 典型场景 | 无中继 |
|---|---|---|
| Full cone / Restricted cone | 家用路由器默认 | ✅ 很高 |
| Port-restricted cone | 家用宽带常见 | ✅ 需同时发包 |
| Symmetric NAT | 部分企业网关 | ⚠️ 仅端口可预测时可行 |
| CGNAT（`100.64.0.0/10`） | 移动/社区宽带 | ❌ 无解（IPv6 除外） |
预期 P2P 成功率 **80–90%**，其余连不上。
**产品边界（需写进对外说明）**：目标用户框定为「家用宽带 + IPv6 可用 + 可开 UPnP」；
企业网络 / CGNAT 标注为不支持。这样剩余失败就不是 bug，是已知边界。
## 3. 打洞流程（协议见 `common/src/signaling.rs`）
```
agent 信令服务器 viewer
|-- Register(info) ------->|<------- Register(info) -----|
|<-- Registered(observed) -|------ Registered(observed) ->| observed = 公网映射（等价一次 STUN binding）
|<-- PeerReady(peer info) -|------ PeerReady(peer info) -->| + 打洞前景 outlook
|<-- PunchPlan(start_in) --|------ PunchPlan(start_in) --->| 两侧 start_in 不同，绝对时刻对齐
| （双方同时向对端候选列表发包，多轮） |
|<========= 打通：UDP 直连 → QUIC 握手 + 证书钉扎 =========>|
|--- PunchReport(Connected) ->|<---- PunchReport ----------|
```
### 3.1 候选地址与优先级
`Candidate::score()`：IPv6（+100）> UPnP 显式映射(90) > 本机地址(80) > 公网反射(60) > 端口预测(40)。
**IPv6 是无中继策略的最强后盾** —— 双方都有公网 v6 就是真直连，成功率≈100%，
因此候选列表按 v6 → v4 排序，`EndpointInfo` 两类都带。
### 3.2 同时发包的时间对齐（不需要时钟同步）
两侧 RTT 不同，若下发同一个延迟，RTT 大的一方晚发包就打不上。服务器按
`punch_start_delay_ms(own_rtt, max_rtt, margin)` 给两侧算不同的 `start_in_ms`：
```
start_in_ms = max_rtt_ms / 2 + margin_ms - own_rtt_ms / 2
```
`start_in_ms` 是**相对消息送达**的延迟，所以两端时钟不需要同步，误差≈单程抖动。
### 3.3 保活
NAT 的 UDP 映射默认 30s 左右超时，洞是出站包维持的，超时后静默关闭
（表现为"连上了一会儿就断"）。`keepalive_secs()` 取双方 `binding_timeout_s`
较小值折半，夹在 5–60s。
## 4. 提高成功率的实现清单（按性价比）
1. **IPv6 优先**：候选排序已实现；agent 侧需枚举全局 v6 地址填 `CandidateKind::Host`。
2. **UPnP / NAT-PMP / PCP**：agent 启动时就尝试为 `PUNCH_UDP_PORT(47800)` 申请映射，
成功后以 `CandidateKind::Upnp` 补报（走 `ClientMsg::UpdateCandidates`）。
一次成功就把对称 NAT 变成可直连，收益最大。
3. **双方同时发包**：见 §3.2，这是打洞能成的关键。
4. **端口预测**：`PortAllocation::Sequential/Preserved` 时才值得做，`Predicted` 候选由
`sanitize_candidates` 统一清洗（去重、丢端口 0、上限 32 个）。
5. **多轮重试**：`rounds=3 / 200ms`，一次不通不代表不通。
**不要走 TCP 打洞**：QUIC 只跑 UDP，TCP simultaneous open 成功率低，性价比不值得。
## 5. 实现约束（两端各自注意）
- **别拿 QUIC 握手当探测**：`quinn::Endpoint::connect` 打不通会持续重试，失败判定被
idle timeout 拖很久。应先用同一 UDP socket 发轻量探测包确认通路（秒级），
确认后再 `connect`；端口扫描也放这一层，便宜且可并行。
- **UDP socket 复用**：打洞用的 socket 要和后续 QUIC 用同一个，否则洞不在同一个映射上。
`quinn::Endpoint` 支持同时 `accept()` 与 `connect()`，可以在一个 Endpoint 上做完整流程。
- **固定端口**：`47800` 对端口预测和用户手动端口转发都友好，不要随机端口。
## 6. 安全（本次必须同步改，否则公网不可用）
1. **证书钉扎**：`viewer/src/main.rs` 的 `SkipVerify` 必须替换为对
`EndpointInfo.cert_fingerprint` 的校验（`CertFingerprint` 已定义 hex 编解码）。
2. **认证**：`ClientMsg::Register.auth` 必须是一次性验证码（替代固定 `123456`/`888888`），
带过期与失败限速，否则公网等于没有认证。
3. **候选列表视为不可信输入**：一律经 `sanitize_candidates` 后再发包。
## 7. 失败必须有明确出口（无中继的用户体验下限）
没有中继兜底，就必须把"为什么连不上"讲清楚，不允许用户停在"连接中"。
`PunchFailReason::user_hint()` 已给出每种原因对应的可执行建议，例如：
- `Cgnat` → "当前处于运营商级 NAT（大内网），无法直连。请改用 IPv6 网络……"
- `SymmetricNat` → "请在被控端路由器开启 UPnP 后重试。"
- `UdpBlocked` → "当前网络的防火墙拦截了 UDP 出站，请切换手机热点后重试。"
`punch_outlook()` 还能在**开打之前**就给出前景（`Likely/Possible/Unlikely/Hopeless`），
`Hopeless`（CGNAT）时可直接跳过打洞轮次并立刻提示，不让用户白等。
## 8. 后续动作
| 顺序 | 事项 | 负责 |
|---|---|---|
| 1 | `common/src/signaling.rs` 过审合入 main | B 提 PR / A review |
| 2 | 信令服务最小实现（WSS/443，配对 + 回推 observed + 下发 PunchPlan） | B |
| 3 | agent：候选收集（v6/Host/UPnP/Reflexive）、同步打洞、保活 | A |
| 4 | viewer：候选收集、同步打洞、改为证书钉扎 | B |
| 5 | 双机联调：家庭宽带 / 手机热点 / CGNAT 三种网络各跑一轮 | A + B |
打洞验证记录.md
+135
View File
@@ -0,0 +1,135 @@
# 打洞连通性验证记录
> 目的：用最小成本确认「自研 P2P 打洞」在本项目目标网络下是否可行，再决定要不要投入实现。
> 工具：`tools/punch_probe.py`（探针）、`tools/rendezvous.py`（极简信令服务器）
> 结论：**本机线路为锥形 NAT + 端口保留，打洞可行性高。**
---
## 1. 测试环境
| 角色 | 地址 | 说明 |
|---|---|---|
| 信令服务器 | `47.98.245.66:47999/udp` | 阿里云 ECS，Ubuntu 24.04.2，python 3.12.3 |
| 客户端 | 出口 `116.3.201.115` | 联通线路，位于 NAT 之后 |
| 工具 | `tools/rendezvous.py` + `tools/punch_probe.py` | 纯 Python 标准库 |
部署位置：`/root/uu-punch-poc/`（与服务器上其他服务完全隔离）
---
## 2. 验证过程与结果
| # | 验证项 | 结果 |
|---|---|---|
| 1 | 信令服务器回环注册 | ✅ 能注册、能记录 |
| 2 | 服务器进程/监听 | ✅ `0.0.0.0:47999`，长期运行稳定 |
| 3 | **跨公网注册** | ✅ 服务器日志出现 `116.3.201.115:47800` |
| 4 | **双方配对与地址回推** | ✅ `配对成功：116.3.201.115:47800 <==> :47801` |
| 5 | 同一 NAT 内两实例互打 | ❌ 失败（见 §4.1，属 hairpinning 限制） |
| 6 | **NAT 映射行为判定** | ✅ **端点无关映射 + 端口保留**（见 §3） |
---
## 3. 关键结论：NAT 类型 = 锥形 + 端口保留
用**同一个本地端口 47800** 分别打两个完全不同的目标，观察公网映射：
| 目标 | 观察到的公网映射 |
|---|---|
| Google STUN（境外） | `116.3.201.115:47800` |
| 自建信令服务器 `47.98.245.66:47999` | `116.3.201.115:47800` |
**不同目标 → 同一个公网端口 → 端点无关映射（Endpoint-Independent Mapping，即锥形 NAT），且端口原样保留。**
按打洞成功率排序：
```
锥形 + 端口保留 ← 本机（最优）
锥形 + 端口改写 成功率高
对称 NAT 困难，需端口预测
CGNAT 基本无解
```
**推论**：信令服务器观察到的地址，对该客户端是**可复用的真实映射**，对端直接向它发包即可打洞，无需端口预测。这是最理想的情形。
复现命令：
```powershell
# 一定要带 --stun-port，用与打洞相同的端口，否则映射不可比
py tools\punch_probe.py --stun --stun-port 47800
```
---
## 4. 踩到的坑（Rust 实现必须处理）
### 4.1 hairpinning：同一公网 IP 下两实例互打会失败
两个客户端都在 `116.3.201.115` 后面，互相打对方的公网地址，需要路由器支持 **NAT 环回（hairpinning）**。本线路不支持。
**这不代表真实打洞会失败**——真实场景是两台机器位于**不同**公网 IP 之后。此测试只能验证"信令配对"链路，不能验证穿透能力。
### 4.2 Windows：ICMP 端口不可达会打断接收循环
对方尚未启动时发包 → 收到 ICMP port unreachable → 下一次 `recv` 抛 `WSAECONNRESET(10054)`。若把它当致命错误 `break`，接收循环就此死掉，之后对方真打进来也收不到。
处理方式（两层）：
```python
sock.ioctl(socket.SIO_UDP_CONNRESET, False) # 1) 关闭该行为
# 2) 收到该错误时 continue，绝不能 break
```
**Rust 对应**：`tokio` 的 `recv_from` 会返回 `ConnectionReset`，同样不能 `?` 或 `break`。
### 4.3 发完就退出会漏掉晚到的回包
对方可能比自己晚启动。发完最后一包立即关闭 socket，会把"晚到的成功"误判为失败。
→ 发完后保留 grace 期（默认 2.5s）继续监听。
### 4.4 部署层的两道防火墙
| 层 | 现象 | 处理 |
|---|---|---|
| 主机 ufw | `INPUT DROP`，无规则则静默丢弃 | `ufw allow 47999/udp` |
| 云安全组 | 包到不了网卡（tcpdump 零捕获） | 控制台加 UDP 入方向规则 |
**排查手法**：`tcpdump -n -i any udp port <port>` 零捕获 = 包在到达主机前就被丢，问题在云安全组而不在主机防火墙。
> 注意：首次配置时误把安全组规则设成了 **TCP**，UDP 包仍被丢弃。协议类型必须选 UDP。
### 4.5 假阳性：信令回复被误判为"打通"
客户端在拿到对端地址前会反复注册，服务器的 `PEER` / `WAIT` 回复可能在 `discover_peer` 返回之后才到达。若接收线程不做区分，会把**服务器发来的回复**当成"对方打进来了"，输出假的"✅ 打洞成功"。
实测踩到的形态：报告的"对方源地址"是**信令服务器的 IP:端口**（`47.98.245.66:47999`），包长 22 字节——而真正的打洞包固定是 **16 字节**（`PUNCHv1|` + 8 字节时间戳）。
**判据（Rust 实现必须照做）**：
1. 只认以协议魔数开头的包，长度必须是约定值
2. 显式排除信令服务器的 `IP:端口`
3. 结论里如果出现"非打洞包 N 个"，说明收到了信令回复，**不能**据此判定成功
---
## 5. 尚未验证
| 项 | 为什么重要 | 怎么验 |
|---|---|---|
| **两台不同网络的机器互打** | 唯一能证明真实穿透能力的测试 | 一台机器 + 一台连手机热点的笔记本 |
| 手机热点（CGNAT）场景 | 决定"无中继"方案对移动端是否成立 | 同上，热点那端即为 CGNAT |
| IPv6 直连 | 若两端都有公网 v6，可绕开 NAT | `--peer [v6地址]:端口` |
| 对端 NAT 是否同样友好 | 打洞需要两端配合 | 在对方网络重复 §3 的映射判定 |
**下一步优先级：手机热点组合 > IPv6 > 其他。**
---
## 6. 对实现的影响
1. **信令服务器的职责就此确认**：注册、配对、回推观察到的源地址。不承载媒体，建连后即退出。
2. **信令与打洞必须复用同一个 socket**：服务器看到的映射端口，必须就是打洞时使用的端口，否则映射不可复用。
- Rust 实现中：该 socket 需继续交给 `quinn::Endpoint` 使用（一次绑定，三处复用：信令 / 打洞 / QUIC）。
3. **失败判定要快**：先做 NAT 行为探测，若判定为对称 NAT 直接给出结论，不必耗完多轮。
4. **保活**：NAT 表项会超时，建连后需按 `binding_timeout_s` 折半做 keepalive（协议中已定义 `keepalive_secs()`）。
构建运行手册.md
+135 -12
View File
@@ -1,37 +1,53 @@
# MVP 远程控制 Demo · 构建运行手册 # MVP 远程控制 Demo · 构建运行手册
> 项目路径：`D:\wordkb\2026-10-22-15-52-47\mvp-remote\` > 项目路径：`D:\code\remote\`
> 形态：Rust workspace，三个 crate：`common`（协议）、`agent`（被控端）、`viewer`（主控端） > 形态：Rust workspace：`common`（协议）、`agent-core`（平台无关会话层，当前供 linux-agent 使用）、`agent`（Windows 被控端）、`linux-agent`（Linux 被控端）、`viewer`（主控端）
## 1. 功能范围（量级一 Demo） ## 1. 功能范围（量级一 Demo）
- ✅ DXGI Desktop Duplication 全屏采集（GPU 拷贝，脏区/硬编码留待二期） - ✅ DXGI Desktop Duplication 全屏采集 + NVENC 硬编（无 N 卡时自动回落 openh264 软编）
- ✅ LZ4 压缩 + QUIC 加密传输（自签名证书，传输层已加密） - ✅ QUIC 加密传输（自签名证书，传输层已加密）
- ✅ 验证码认证（默认 123456） - ✅ 验证码认证（默认 123456）
- ✅ 键鼠远程控制（SendInput 注入，虚拟桌面绝对坐标 + 扫描码） - ✅ 键鼠远程控制（SendInput 注入，虚拟桌面绝对坐标 + 扫描码）
- ✅ 桌面静止时低频重发（保持画面新鲜） - ✅ 桌面静止时低频重发（保持画面新鲜）
- ❌ 暂无：硬件编码、P2P 打洞/中继（当前仅局域网/可直连地址）、音频、文件传输、Linux 被控 - ✅ Linux 被控端（X11 采集 + XTest 注入，见第 8 节）
- ❌ 暂无：P2P 打洞/中继（当前仅局域网/可直连地址）、音频、文件传输
## 2. 环境要求 ## 2. 环境要求
- Windows 8+（被控端需要 DXGI Duplication；主控端 Win7 亦可） - Windows 8+（被控端需要 DXGI Duplication；主控端 Win7 亦可）
- 工具链已全部部署在 D 盘（无需再装）： - Rust（GNU host：`x86_64-pc-windows-gnu`）+ MinGW-w64，已验证可用组合：
- Rust (GNU)：`uurev/tools/rust/cargo/bin` - `rustup default stable-x86_64-pc-windows-gnu`，装到 `%USERPROFILE%\.cargo\bin`
- MinGW-w64 14.2：`uurev/tools/mingw64/bin` - MinGW-w64：`D:\mingw64\bin`（gcc 16.2.0）
- 构建时 PATH 需同时包含上面两个目录（缺 MinGW 会在链接 openh264 的 C++ 时失败）
- **cargo 网络**：国内直连 crates.io 会超时，且 **cargo 不读 Windows 系统代理**，必须显式告诉它。
已在本机 `%USERPROFILE%\.cargo\config.toml` 写入（Clash 之类混合端口，按实际改）：
```toml
[http]
proxy = "http://127.0.0.1:7897"
```
⚠️ **不要用国内镜像做源替换**：实测 USTC 镜像的索引 `cksum` 与其实际下发的 `.crate`
字节**不一致**（`nvenc 0.1.0`：索引写 `3227c3b0…`，包体实际 `ab6fde0e…`），
会报 `failed to verify the checksum of nvenc`；且源替换会改变 registry 身份，
使 `Cargo.lock` 里已有的 checksum 与全部缓存失配。**走官方源 + 代理最稳。**
> 注：旧版本手册写的 `D:\wordkb\...\uurev\tools\rust` 路径已不存在，按上面重新准备工具链即可。
## 3. 构建 ## 3. 构建
```bash ```bash
export RUSTUP_HOME="D:\\wordkb\\2026-10-22-15-52-47\\uurev\\tools\\rust\\rustup" # Git Bash 写法：PATH 需含 rustup 的 cargo 与 MinGW-w64
export CARGO_HOME="D:\\wordkb\\2026-10-22-15-52-47\\uurev\\tools\\rust\\cargo" export PATH="$USERPROFILE/.cargo/bin:/d/mingw64/bin:$PATH"
export PATH="/d/wordkb/2026-10-22-15-52-47/uurev/tools/rust/cargo/bin:/d/wordkb/2026-10-22-15-52-47/uurev/tools/mingw64/bin:$PATH" cd /d/code/remote # 换成你自己的工作目录 / worktree
cd /d/wordkb/2026-10-22-15-52-47/mvp-remote
cargo build --release cargo build --release
``` ```
产物： 产物：
- 被控端 `target/release/agent.exe` - 被控端 `target/release/agent.exe`
- 主控端 `target/release/viewer.exe` - 主控端 `target/release/viewer.exe`
- 编码基准（可选）`target/release/encbench.exe`
## 4. 运行（两台机器，或同机双开验证） ## 4. 运行（两台机器，或同机双开验证）
@@ -67,6 +83,30 @@ viewer.exe --addr <被控端IP> --port 47800 --code 888888
4. 高 DPI/多屏：当前只采集主输出（EnumOutputs(0)）。 4. 高 DPI/多屏：当前只采集主输出（EnumOutputs(0)）。
5. 主控端窗口固定为远端分辨率，颜色序 BGRA→0RGB 若异常可在 `blit()` 一键翻转。 5. 主控端窗口固定为远端分辨率，颜色序 BGRA→0RGB 若异常可在 `blit()` 一键翻转。
### 6.1 待修问题（已定位，尚未修，均在 Windows `agent/`）
1. **错误验证码时 `AuthFail` 送不达**：`agent/src/main.rs` 写完 `AuthFail` 后立刻
`send.finish()` 并返回，进程退出会把 QUIC 缓冲区里尚未发出的数据一起带走，
客户端只会看到 `connection lost` 而不是 `AuthFail`。
修法：`finish()` 后等对端读到并挂断（`conn.closed()` + 超时）再退出。
2. **软编路径下静止桌面仍不响应关键帧请求**（`04c02d9` 只修了 NVENC 那一半）：
`agent/src/capture.rs` 的 `Ok(false)`（画面未变）分支里，`force_key` 被
`keyreq.swap` 消费后，只有 `EncoderKind::Nvenc` 分支拿它去重编 IDR；
`EncoderKind::Soft` 分支仍然只是「重发最后一帧码流副本」，标志被直接丢弃。
后果：**无 N 卡 / 驱动异常（走软编兜底）的机器上，解码端一旦出错，静止画面下
永远拿不到 IDR，表现为黑屏。**
修法：软编路径同样保留最近一帧裸像素，`force_key` 时重编一个真关键帧。
3. **单次 `KeyFrameReq` 可能被静默吞掉**：同一个 `keyreq.swap(false, …)` 写在了
`if last_sent.elapsed() > 500ms` 的**外面**，因此请求若落在距上次发送 500ms 之内，
标志已被消费但分支不进入，该次请求就丢了，必须等下一次请求才可能生效
（主控端目前周期性重发，故表现为延迟而非永久黑屏）。
修法：把 `swap` 移进时间闸内，或先在闸外预判、进闸后再消费。
> 第 1、2 条在 `feature/agent-linux` 分支的 `agent-core` 里已经实现（**Linux 侧生效**），
> 但按"不与他线争抢 `agent/`"的约定，**本期没有改动 Windows 代码**。
## 7. 下一阶段任务队列（Phase 1 剩余） ## 7. 下一阶段任务队列（Phase 1 剩余）
1. H.264：接 openh264（软编兜底）→ NVENC/QSV（硬编优先） 1. H.264：接 openh264（软编兜底）→ NVENC/QSV（硬编优先）
@@ -74,3 +114,86 @@ viewer.exe --addr <被控端IP> --port 47800 --code 888888
3. QUIC 多流：视频/控制/键鼠分离流（键鼠零阻塞） 3. QUIC 多流：视频/控制/键鼠分离流（键鼠零阻塞）
4. P2P 打洞 + coturn 中继（自建信令服务器） 4. P2P 打洞 + coturn 中继（自建信令服务器）
5. 被控端服务化（Windows Service + 会话切换处理） 5. 被控端服务化（Windows Service + 会话切换处理）
## 8. Linux 被控端（linux-agent）
### 8.1 现状与边界（重要）
本期**只在新增文件里做，未改动 Windows `agent/`**，原因见下：
| crate / 文件 | 职责 | 当前使用者 |
|---|---|---|
| `agent-core/src/lib.rs` | QUIC/TLS 握手、传输调优、验证码认证、会话/select 循环、latest-wins 单槽、输入分发，以及软件编码视频管线（openh264） | **仅 linux-agent** |
| `linux-agent/src/capture.rs` | X11 `GetImage` 采集 + 整帧变更检测 | Linux |
| `linux-agent/src/input.rs` | XTest 注入 + 扫描码→X11 keycode 映射 | Linux |
| `agent/`（Windows） | 保留原有自持会话循环 + NVENC/软编双路编码，**本期未动** | Windows |
**为什么 Windows 端没有一并接过来**：`agent/` 的 NVENC 路径是「GPU 内 BGRA→H.264 零转换零拷贝」，
编码器必须直接吃 DXGI 的 GPU 纹理；而 `agent-core` 目前的接缝是 `ScreenCapturer`（返回 CPU 上的
BGRA 字节），硬接会把零拷贝优势废掉。正确的接缝应是「平台视频管线产出**已编码帧**」
（`VideoSource` 语义），待 `agent/` 由 NVENC 那条线稳定后再重切，届时应把 `openh264` 从
`agent-core` 移到各平台 crate。
在此之前 Windows 与 Linux 各持一份会话循环，属于**有意的过渡态**：`common/` 协议改动需要两处同步
（例如本期跟着补的 `Pong{lost,sent}`）。
### 8.2 构建
Windows 侧照旧（`default-members` 已排除 linux-agent，产物仍是 agent.exe / viewer.exe）：
```bash
export PATH="$USERPROFILE/.cargo/bin:/d/mingw64/bin:$PATH" # Git Bash 写法
cargo build --release
```
Linux 侧显式指定包（跨平台交叉编译不推荐，用容器最省事）：
```bash
cargo build --release -p linux-agent
```
### 8.3 运行
```bash
# 被控 Linux 机器（Xorg 或 XWayland 均可，依赖 DISPLAY 环境变量）
./linux-agent --port 47800 --code 123456
# 主控端（Windows）命令行不变
viewer.exe --addr <被控机IP> --port 47800 --code 123456
```
### 8.4 冒烟测试（不需要图形桌面）
```bash
bash scripts/linux-smoke.sh
```
脚本自建 Xvfb 虚拟屏后跑**单测 + 端到端**两层：
- 单测（不需要 X server）：键码映射、坐标换算等纯逻辑；
- 端到端：X11 采集 → openh264 编码 → QUIC 下行解帧、验证码认证（含**错误验证码被拒**）、
Ping/Pong（含下行丢包统计字段）、静止桌面下的关键帧恢复，以及**输入注入实证**——
注入鼠标移动后，从另一条 X11 连接读回指针真实坐标。
> 单测那层别省：坐标换算的取整边界就是靠它才发现的（集成测试有 ±2 像素容差，
> 期望值写错也照样绿）。
> 静止关键帧那条用例做过**变异测试**：把修复退回会失败、恢复后通过，确认断言不是摆设。
> 注意扰动量必须小（画 64x64 小方块）——整屏刷白会触发编码器场景切换自动 IDR，
> 于是"重发旧帧"和"真重编关键帧"拿到同样的字节，用例就失去区分度。
### 8.5 键码映射（两端最关键的差异）
协议里键盘沿用 Windows 口味的 PS/2 Set-1 扫描码，而 X11 要的是 keycode，两者**并非处处等价**：
- 主键区：满足 `keycode = scancode + 8`；
- **E0 扩展键编号体系完全不同**（`↑` 是 0x48，但 X11 keycode 是 111 而不是 0x50）——必须查表。
映射表在 `linux-agent/src/input.rs::keycode_of`，带单元测试防回归。
### 8.6 Linux 侧已知限制
1. 仅支持 32bpp TrueColor 视觉（绝大多数桌面满足），其他格式启动即明确报错而非静默出错；
2. 采集走核心协议 `GetImage`，每帧经 X socket 全量传输，静止桌面也会持续轮询抓帧；
**MIT-SHM 共享内存 + XDamage 脏区**是下一步优化（锚点已写在 `linux-agent/src/capture.rs` 顶部注释）；
3. 仅覆盖 X11 / XWayland；纯 Wayland（无 XWayland）需另走 PipeWire 门户；
4. 键位表按 Xorg evdev 标准 keycode 编写，自定义键盘布局可能需要调整。