wangyukun/remote
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: wangyukun/remote:feature/common-punch-signaling
wangyukun/remote:main wangyukun/remote:feature/android-viewer-core wangyukun/remote:feature/proto-versioning wangyukun/remote:feature/common-punch-signaling wangyukun/remote:feature/agent-linux wangyukun/remote:backup/agent-linux-0f3f1cf
..
compare: wangyukun/remote:5079d33b0a020e321fad82b5ac1e6a4fbbc1424f
wangyukun/remote:feature/android-viewer-core wangyukun/remote:feature/proto-versioning wangyukun/remote:feature/common-punch-signaling wangyukun/remote:feature/agent-linux wangyukun/remote:main wangyukun/remote:backup/agent-linux-0f3f1cf
20 Commits
feature/common-punch-signaling .. 5079d33b0a
Author SHA1 Message Date
郑博文 5079d33b0a fix(agent): 显式 max_idle_timeout=10s——强杀主控端后 10 秒内可重连 
问题（实测复现）
串行 accept 下，主控端被强杀（force-stop，非正常 close）时 QUIC 连接
不会立即消亡：agent 还在向死连接发帧，新连接只能排队，要等 quinn 默认
idle 超时（约 30s）才能建立。表现为「杀掉 App 重开，一直连不上」。

修法
TransportConfig 显式设 max_idle_timeout=10s。正常会话每 500ms 一个 Ping
永不触发该超时，只回收死连接。10s = 20 个 Ping 周期，抗抖动余量足够。

验证（vivo S6 实测）
- 会话1 auth ok -> force-stop App -> 立即重连
- 10.2s 后 peer connected + auth ok（修复前 ~30s）
 2026-10-30 10:38:04 +08:00
郑博文 fe558a1c1e feat(android): 状态栏触摸诊断通道（vivo logcat 被限流的对策） 
本机（vivo S6 + 办公 Windows）logcat 只能抓到 4 行，App 日志完全不可靠，
排查触摸/坐标问题时没有任何可用通道。

- RemoteSurfaceView 加 onDebug 回调：ACTION_DOWN/MOVE/UP 的视图坐标与
  发出的 norm 归一化坐标实时上报
- MainActivity 统一 refreshStatus()：连接态 + 帧数/RTT + 最近触摸合并成
  一处刷新（原来 8 处各自 statusView.text 互相覆盖，且引用过已删的 showHint）
- 删掉临时步骤标记与 lastMeasureInfo（诊断价值已被 onDebug 覆盖）

实测：状态栏正确显示 UP view=(643,222) norm=(39017,23968) 1080x607，
据此确认了 letterbox 视图尺寸与归一化换算全对，把排查焦点正确引向
agent 注入侧（VIRTUALDESK，已另行修复）。
 2026-10-30 10:35:21 +08:00
郑博文 6a6f989595 fix(agent): 绝对坐标注入去掉 VIRTUALDESK——注入空间与采集空间对齐 
根因（实测确认）
capture 用 EnumOutputs(0) 只采**主屏**，AuthOk 报告的 width/height 也是主屏；
但 input 注入带 MOUSEEVENTF_VIRTUALDESK，归一化 0..=65535 被映射到**整个虚拟
桌面**。双显示器机器上虚拟桌面（主屏 2560x1440 + 副屏 1920x1080 在左）比主屏
宽得多，手机画面与指针位置系统性错位——点主屏任务栏，指针落到别处。
单显示器时两个空间重合，所以桌面 viewer 联调从未暴露。

修法
MOUSEEVENTF_ABSOLUTE 不带 VIRTUALDESK 时，0..=65535 恰好映射到主显示器，
与采集空间一致。协议注释「鼠标坐标为虚拟桌面归一化」的语义由「整个虚拟桌面」
修正为「被采集的输出」——桌面 viewer 是绝对定位透传，行为不变。

验证（vivo S6 -> 本机双屏，修复后）
- 手机拖微信标题栏：窗口跟随手指移动（drag-before/after 截图对比）
- App 状态栏 norm 坐标与视图坐标换算一致（view=(643,222) norm=(39017,23968)）
 2026-10-30 10:30:53 +08:00
郑博文 5047618c01 feat(android): 触摸映射 + letterbox 显示（阶段 2a 代码部分） 
问题：阶段 1 的 SurfaceView 用 MATCH_PARENT，2560x1440 的视频被拉进
1080x2400 竖屏，画面变形。变形状态下手指位置与远端指针不再一一对应，
触摸映射无从谈起。所以两者必须一起做。

新增 RemoteSurfaceView
- onMeasure 按视频宽高比 letterbox：视图尺寸 = 画面尺寸，黑边不属于视图，
  触摸天然只在画面内产生事件，坐标归一化到 0..=65535 才准确
- 手势：单指轻点=左键、单指拖动=左键拖拽、双指轻点=右键、双指滑动=滚轮
- 刻意不做「手指即光标」的绝对跟随：手指会完全挡住目标，精细操作不可用
- 边界处理：进入拖拽后落下第二指，先抬起左键再切双指手势，
  否则远端会留下一个被按住未释放的按键

协议侧（不改线格式，只加只读查询）
- viewer-core::Session 记录 AuthOk 的桌面尺寸，新增 desktop_size()
- android-ffi 新增 nativeDesktopSize(handle) -> i64（高32=宽，低32=高）
- RustBridge.desktopSize() 拆包
- 尺寸必须在建解码器之前拿到：等首帧再改视图尺寸会触发重新测量，
  把刚创建的解码器 surface 打掉

MainActivity
- 接入 RemoteSurfaceView.Listener；连接成功后先取尺寸设宽高比，再起解码线程
- 返回键 = 断开（不退出 App，便于重连）；去掉阶段 1 的双击断开
  （双击现在语义上是「两次左键点击」，会和远端交互冲突）

验证：cargo build --workspace 通过；cargo test 全绿（common 20 + viewer-core 3）。
真机手势验收待手机重新连接后补。
 2026-10-30 09:51:38 +08:00
郑博文 f119300a57 feat(agent): 串行多会话——会话结束不再退出进程 
问题（阶段 2 的硬阻塞）
accept() 只接一次，会话结束进程退出。阶段 1 一次验收看不出来，
但阶段 2 要反复测试输入，每测一次都要手动重启被控端。
（即公共阻碍清单里的 P5，此前判断为「只影响调试效率」，
 现在它阻塞迭代，提前做掉。）

改动
1. main：会话体抽成 serve_session()，外层 accept 循环。
   会话失败（含认证失败）只记日志不退出进程。
   明确不做并发多会话：DXGI Desktop Duplication 每个输出仅允许一个
   活动实例，第二个 DuplicateOutput 会让第一个收到 ACCESS_LOST。
2. 认证失败路径补 conn.closed() 等待（agent-core 已有同款修复，
   此处同源补齐 = 手册 §6.1 第 1 条）：否则客户端只看到 connection
   lost 而不是 AuthFail。
3. capture 循环加 tx.receiver_count()==0 守卫。
   只靠「send 失败退出」不够——桌面完全静止时那个分支根本不发送，
   线程会永远持有 duplication，下一个会话的 DuplicateOutput 直接失败
   （表现为新会话黑屏）。agent-core 同源修复。
4. 会话收尾：finish 流 + drop 接收端 + 等采集线程退出（2s 超时）后才
   返回，调用方才 accept 下一个连接。

验证（连续两个会话，中间不重启 agent）
  会话1：收到帧 40/40，结论 ✅
  agent 存活：✅
  会话2：收到帧 40/40，结论 ✅
  日志：session closed -> waiting for next connection -> peer connected
        -> auth ok；stderr 有 [capture] no receiver, stop 证明线程真退出
 2026-10-30 09:31:44 +08:00
郑博文 58bb66f4c2 feat(android): 阶段 1 安卓主控 App 工程（已真机验收出画面） 
Kotlin 层三个文件（刻意零第三方依赖，纯 View 布局——阶段 1 只验证链路）：
- RustBridge.kt：android-ffi 的 JNI 声明 + 帧头拆解（25 字节定长头 + Annex-B 载荷）
- H264Decoder.kt：MediaCodec 同步模式 + Surface 直出；内置两条实测教训——
  只用 getInputBuffer(i)（Exynos Quirk）、awaitingIdr 状态机（R-3 跳帧断参考链）
- MainActivity.kt：连接/解码循环/触摸绝对坐标映射；断开顺序遵守
  「先停解码线程 join 后 destroy」的 FFI 纪律

构建链（build-android.ps1，一条命令：cargo ndk -> .so -> gradle assembleDebug -> 安装启动）：
- 路径自适应（ 定位仓库根），local.properties 自动生成
- gradle.properties 固定 JDK 17，不动全局 JAVA_HOME

真机验收（vivo S6 / 192.168.1.163 -> PC / 192.168.1.180:47800）：
- agent 日志：peer connected + auth ok, desktop 2560x1440
- 手机截图：Windows 桌面（WPS 文件列表）已渲染在 SurfaceView 上
  证据：evidence/screen-connected.png（本地，不入库）

已知环境限制（记录在个人层 baseline）：本机 Windows 防火墙的 agent.exe
程序级放行规则对 UDP 入站不生效（临时关防火墙即通），QQPC 驱动级防护
静默拦 TCP 入站（管理员无法停止/杀死）。阶段 2 前需落定长期方案：
端口级防火墙规则或换开发机。
 2026-10-29 19:59:43 +08:00
郑博文 64d497e577 test(viewer-core): headless_check 增加 --host，可指定被控端地址 
默认仍 127.0.0.1。加 --host 后能用局域网 IP 跑同一验收，
便于排查「桌面通、手机不通」这类只影响入站路径的问题
（注：本机回环连接不经过防火墙，只有跨主机连接才验证防火墙路径）。
 2026-10-29 19:18:11 +08:00
郑博文 44767ed370 feat(android-ffi): viewer-core 的 JNI 薄层，arm64 交叉编译通过（QUIC 栈上手机的第一道关） 
方案变更说明（A2 落地修正）：原裁决选 UniFFI，但实查 Maven Central 与
maven.mozilla.org 均无 uniffi Kotlin 运行时构件（404），自编译发布成本超过收益。
改为裸 JNI：阶段 0（真机）已验证该链路，接口刻意保持窄（连接/取帧/发输入/销毁），
将来若引入 UniFFI 只替换本层。

线程与锁纪律（决定接口形态）：
- nativeNextFrame 等帧期间不持会话锁（clone Arc<Session> 后即释放），
  桌面静止 500ms 帧间隔不会卡住 UI 线程的输入调用
- destroy 前必须先停解码线程，Kotlin 侧顺序纪律保证（已登记风险清单）
- 句柄 = Box 裸指针（永不为 0），destroy 后置 0

帧返回约定：byte[] = [keyframe 标志 1][seq 8][ts 8][w 4][h 4][Annex-B payload]，
定长 25 字节头，Kotlin 切片零 JNI 往返。

viewer-core 适配：Session.frames 改 Arc<tokio Mutex>，next_frame 取 &self，
使解码线程与输入线程可并发借用同一 Session。

验证：
- 桌面 cargo build -p android-ffi 零警告
- cargo ndk -t arm64-v8a --release：quinn/ring/rustls 交叉编译全部通过
  （消除阶段 1 最大未知：QUIC 栈能否编到 Android）
- libandroidffi.so 4.2MB，10 个 Java_com_verify_remote_RustBridge_* 符号导出正确
 2026-10-29 18:27:34 +08:00
郑博文 34df7b070b feat(viewer-core): 新增安卓主控的连接/协议/收帧层（阶段 1 桌面验证通过） 
定位（2026-10-29 方案裁决 A1-A4）：
纯 Rust、平台无关，只做「怎么连 + 收发什么」——QUIC 会话、协议 v2 认证、
视频帧流、输入上行、Ping/Pong、KeyFrameReq。解码/渲染/触摸全在 Kotlin 侧，
本 crate 不碰像素、不依赖 openh264（dev-dependency 仅验证工具用）。

设计要点
- PeerTarget::Direct 是唯一的连接形态；不做中继、不定义 Relay 变体
  （《多平台架构设计》§6.1 裁决）。将来设备码+打洞只换 connect() 内部。
- 帧经容量 2 的 mpsc 交付，**不丢帧**：跳帧决策属于看得到解码器状态的消费方
  （Kotlin），Rust 层丢帧会重演 R-3 教训（跳帧断参考链）。
- insecure_skip_verify=false 时明确报错而非静默放行——证书钉扎登记在方案
  阶段 4，安全红线不允许默默降级。
- 双流结构与桌面 viewer 一致（流1 认证+视频，流2 输入+控制），入会即请求 IDR。

验证（headless，桌面→桌面 127.0.0.1）
- 单测 3 passed（帧契约、默认配置、strict-verify 显式报错）
- examples/headless_check：300 帧，seq 违例 0、keyframe 标志不符 0、
  解码成功 300/300、RTT 路径工作（RTT=1ms 出现）
- 全 workspace 回归：common 20 passed，构建全绿
 2026-10-29 18:14:27 +08:00
郑博文 1837658c85 test(agent): dump_stream 增加 --save，可导出裸 H.264 码流供外部解码器验证 
目的
安卓主控方案的唯一高风险未知项是「agent 发出的码流能否被 MediaCodec 解码」。
要验证它，需要一份脱离 QUIC/自定义帧头的**裸 Annex-B 码流文件**，供手机端
或任何标准解码器直接读取。

改动
- 新增 --save <路径>：把各帧载荷按到达顺序拼接成 Annex-B 文件
- 存的是**载荷**而非完整帧：外部解码器只认标准 Annex-B，不认识本项目的
  长度前缀与 bincode 帧头，故不能存 encode_video_frame 的输出
- 保存时打印字节数与 IDR 数量，便于快速判断样本是否可用

不做的事
不引入独立的码流解析库，IDR 统计复用文件内已有的起始码扫描逻辑，
避免为一次性诊断工具增加依赖。

验证
cargo build --workspace 退出码 0；cargo test --workspace --exclude linux-agent 全绿。
实际使用：导出 90 帧、437823 字节、含 2 个 IDR，该文件已用于安卓 MediaCodec
兼容性验证（vivo S6 / Exynos 硬解，87/89 帧解出）。
 2026-10-29 16:13:58 +08:00
郑博文 6e83bda8b8 perf(agent): 关键帧周期从 4s/10s 统一到 1s，缩短丢帧后的兜底恢复窗口 
背景
协议 v2 与 vendor nvenc 补丁之后，KeyFrameReq 已成为**真正生效**的恢复主路径
（此前 NVENC 缺 FORCEIDR 标志，请求被静默忽略）。因此周期 IDR 的角色变了：
从「唯一恢复手段」降级为「兜底手段」，只用于防「请求连续丢失」这种极端情况。
既然是兜底，就没必要配得那么长。

问题：三处周期不一致，且都过长
- agent/src/capture.rs NVENC：gop_len = 240 帧 @60fps = 4 秒
- agent/src/capture.rs 软编：    intra_period = 300 帧 @60fps = 5 秒
- agent-core/src/lib.rs Linux：  intra_period = 300 帧 @30fps = **10 秒**

同一产品在三种编码后端上的恢复等待时间差 2.5 倍，且 Linux 端最差。

改动
- 新增 agent/src/capture.rs::INTRA_PERIOD_FRAMES = 60（Windows 两个编码器共用）
- agent-core 的 EncoderSettings::default() 改为按 max_fps 推导（1 秒对应帧数），
  避免以后调帧率时周期又漂移

代价
IDR 是整帧内编码，实测一次约 95-120KB，而 P 帧约 3-10KB。1 秒一次 IDR 约
增加 0.8Mbps 开销，在 5Mbps 码率下可接受。

验证（dump_stream 观察 150 帧）
  [  0] seq=0   IDR   ← 首帧
  [ 60] seq=60  IDR   ← 间隔精确 60 帧
  [120] seq=120 IDR
@60fps 即每 1 秒一个自动 IDR，与 gop_len=60 配置一致。
 2026-10-28 16:35:44 +08:00
郑博文 8410a8de2a docs: 记录「不做中继」的最终裁决（2026-10-28） 
背景：公司决定不提供中继服务器，中继方案从「暂缓」变为「排除」。

《多平台架构设计.md》
- §6 从「⚠️ 必须现在拍板」改为「✅ 已裁决」，保留原始分析但标注结论
- 新增 §6.1 裁决内容：不做中继、不预留中继接口、IPv6 一等公民、
  连接层边界仍要做
- 新增 §6.2 澄清一件常被混淆的事：**连接层边界 ≠ 中继接口**
  - 连接层边界（把「怎么连」与「连上干什么」分开）：要做，与中继无关，
    因为连接方式确定会变（IP→设备码）、安卓端要复用、解码逻辑要可单测
  - 中继接口（Path::Relay 这类类型预留）：不做。中继已排除，这些变体
    永远不会被构造，只会产生 dead_code 警告——零收益的预留就是纯负担
- 新增 §6.3 对产品的影响：明确哪些网络环境连不上，并说明这不是 bug
  而是已知边界，UI 必须给出可执行提示
- §10 第 1 条标记为已裁决

《打洞与信令设计.md》
- 状态更新为「已合入 main（703eb64）」
- 补充裁决确认，指向 §6.1/§6.2

补充事实：此前代码中并不存在 PathProvider / Path::Relay，唯一提到 relay 的
signaling.rs:7 是在声明「本协议不含中继」。因此本次是「确认不留」而非
「拆除预留」。
 2026-10-28 16:35:32 +08:00
郑博文 192513dd2a fix(viewer): 跳帧后进入「等待 IDR」状态，不再把必然失败的 P 帧喂给解码器 
问题
H.264 的 P 帧依赖前一帧的参考图像。主控端的积压跳帧逻辑（读出但不解码）
会切断参考链，此后所有 P 帧都解不出来，且不会自行恢复——必须收到 IDR。
旧实现跳帧后只是 continue，后续 P 帧仍被送进 decoder，导致：

- 每帧解码失败一次 → stderr 刷屏「decode failed」
- 每次失败都触发一次 KeyFrameReq → 请求风暴
- 解码器反复在错误状态里挣扎

实测（修复前，10 秒会话）：decode failed 刷屏数十条。

修复
引入 awaiting_idr 状态，语义是「参考链已断，等 IDR 重新同步」：

- 跳帧时置位，并请求关键帧
- 置位期间跳过所有 P 帧（这是正确性要求，不是优化）
- 收到 hdr.keyframe=true 的帧才清位并恢复解码
- 解码出错时也置位
- 初始值为 true：入会时已发 KeyFrameReq，拿到第一个 IDR 前不该硬解

该实现依赖协议 v2 新增的 hdr.keyframe 字段（从实际码流 NAL 类型判定），
因此必须在协议改动之后合入。

HUD 新增 STATE 行显示 OK / RECOVER。RECOVER 期间画面冻结是**预期行为**，
没有这个指示用户会误以为软件卡死。

验证（同机 agent+viewer，25 秒）
- decode failed：0 次（修复前刷屏）
- stderr 总行数：3（仅首帧日志 + 2 条积压提示）
- 窗口稳定无假死

已知局限（下一步）
本修复解决的是「跳帧后不再做无用功」，但跳帧本身仍会造成画面冻结直到
IDR 到达。NVENC 的 gop_len=240、@60fps 即 4 秒才有一个自动 IDR，恢复窗口
偏长。缩短 GOP 或降低跳帧触发频率是后续优化项。
 2026-10-28 16:10:40 +08:00
郑博文 e778ff73cc test(agent): 新增 dump_stream 码流诊断工具 
用途：不依赖 viewer 的 Minifb 窗口，直接抓 agent 下发的码流并逐帧打印
NAL 结构与解码结果，用于定位解码恢复类问题。

能力
- 打印每帧的 seq / keyframe / 字节数 / NAL 类型（SPS PPS IDR P）
- --skip-at N：模拟主控端「积压跳帧」——读出但不解码，用于验证跳帧是否
  切断参考链（实测确认：跳掉一帧后所有 P 帧持续解码失败，直到收到 IDR）
- 交叉校验帧头 keyframe 标志与码流实际 NAL 类型是否一致
- 末尾补发 KeyFrameReq 并观察 IDR 是否真的回来

用法
  cargo run --release -p agent --bin dump_stream -- --port 47800 --code 123456
  cargo run --release -p agent --bin dump_stream -- --port 47800 --code 123456 --skip-at 1

注意：KeyFrameReq 必须写在流 2（isend）——agent 的读循环只监听流 2，
写到流 1 会被静默忽略（本工具第一版踩过，误判「修复无效」）。

这个工具是发现 nvenc FORCEIDR 缺陷的直接手段，保留以备回归排查。
 2026-10-28 16:05:41 +08:00
郑博文 e814cb64e7 fix(agent): vendor nvenc 并补 FORCEIDR 标志——修复运行期关键帧请求被静默忽略 
根因（实测确认，非推测）
上游 nvenc 0.1.0 的 encode_picture 只设置 picture_type=NV_ENC_PIC_TYPE_IDR，
却从未设置 encode_pic_flags=NV_ENC_PIC_FLAG_FORCEIDR。NVENC 会据此忽略请求、
继续输出 P 帧。首帧之所以「看起来正常」，是因为 frame_idx==0 本来就强制 IDR，
与该字段无关——这也是该缺陷长期未被发现的原因。

证据（同一 frame_count=4，修复前后）
- 修复前：输出 NAL 首字节 0x61 → 0x61 & 0x1F = 1 = 普通 P 片
- 修复后：输出 NAL 首字节 0x65 → 0x65 & 0x1F = 5 = IDR 片

影响
这解释了构建运行手册 §6.1 记录的「静止桌面黑屏」：NVENC 路径的重编 IDR 从未
真正生效，KeyFrameReq 拿不到 IDR 就无法从解码错误中恢复。

改动
- 新增 vendor/nvenc（上游为 MIT 许可，可自由修改分发，符合商用要求）
- vendor/nvenc/src/safe/encoder.rs：picture_type=IDR 时置 ForceIDR；
  IntraRefresh 时置 ForceIntra。改动处有「本项目补丁」注释标记
- agent/Cargo.toml：nvenc 改为 path 依赖指向 vendor 版本

建议后续向上游提 PR（github.com/AlsoSylv/nvenc），以降低长期维护成本。

验证：跳帧后发 KeyFrameReq，第 3 帧即收到 IDR 并解码成功
 2026-10-28 16:05:41 +08:00
郑博文 9e50bc6237 feat(common): 协议 v2——版本号 + 帧头 seq/keyframe + 长度下界校验 
为接入安卓主控做准备，先把三端共享契约改到能承载多客户端。改动集中在
common/src/lib.rs，三端适配在同一提交内完成以保证可编译。

协议版本（PROTO_VERSION=2）
- ControlMsg::Auth/AuthOk 增加 proto_version 字段；认证顺序改为「先版本后验证码」：
  版本不一致时字段布局可能不同，验证码的解析结果不可信
- 任一端版本不符即 AuthFail 并明确报错，把「连上了但行为诡异」变成「连不上且提示升级」

视频帧头（VideoFrameHeader）
- 新增 seq: u64（每次实际发送递增，含静止桌面重发）与 keyframe: bool
- keyframe 由新增的 common::h264_contains_idr() 从实际 Annex-B 码流判定，
  不依赖编码器 API 推断——NVENC 与 openh264 规则不同，实际输出才是唯一可信来源
- 该字段是 MediaCodec 判断解码器重置时机的必要条件，避免客户端自己写 NAL 扫描

不可信输入防护
- read_video_frame 补齐长度下界：total 必须 >= 4，hlen 必须落在 [1, total-4]
- 旧实现只校验上界，构造 total<4 的帧会让接收端在 buf[0..4] 处 panic（进程级DoS）
- 测试从 0 增至 11 个，覆盖序列化往返与 6 类越界/损坏输入

顺带修复（与上述契约直接相关）
- 新增 common::split_video_frame()：静止桌面重发时复用载荷但重建帧头，
  否则 seq/ts_ms 会停在上次发送时刻，破坏 seq 单调递增契约
- agent-core 与 agent 两端的静止重发路径均改用该函数

验证：cargo test -p common → 20 passed（9 signaling + 11 新增）
 2026-10-28 16:05:41 +08:00
郑博文 703eb64fc0 merge: 合入 feature/common-punch-signaling（P2P 信令协议 + 打洞探针 + 4 份架构设计文档） 2026-10-28 15:14:56 +08:00
郑博文 4533968057 merge: 合入 feature/agent-linux（测试期望值修正 + 冒烟脚本纳入单测 + cargo 网络配置文档） 2026-10-28 15:14:51 +08:00
yu bae871c78c fix(linux-agent): 修正坐标换算单测期望值 + 冒烟脚本纳入单测 
- scale(32768,1920) 期望值应为 959（32768×1919÷65535=959.49 向下取整），
  原写成 960 —— 是**测试期望值错**，实现本身正确（0→0、65535→extents-1 均通过）；
  集成测试因带 ±2 像素容差未能暴露，是跑单测才抓到的
- scripts/linux-smoke.sh 原先只跑 --test smoke，漏掉 --bins 单测；现两层都跑
- 手册 §8.4 说明补充单测这一层及其必要性

验证：bash scripts/linux-smoke.sh → 单测 4 passed + 端到端 2 passed
 2026-10-23 15:04:56 +08:00
yu 49f85db9f9 docs: 修正 cargo 网络配置建议（代理而非镜像源）+ 更新构建命令路径 
- 实测 USTC 镜像索引 cksum 与下发的 .crate 字节不一致（nvenc 0.1.0），
  源替换会触发 checksum 校验失败并改变 registry 身份；改为 [http] proxy 走官方源
- 构建命令里 D:\\wordkb\\... 的失效路径替换为可用写法
- 已实测：Windows cargo build --release 通过（agent.exe/viewer.exe/encbench.exe）
  且 cargo test -p linux-agent 在容器内 2 passed
 2026-10-23 14:10:20 +08:00
51 changed files with 6309 additions and 116 deletions
Expand all files Collapse all files

No files matched your search

Cargo.lock
Generated
+155 -17
View File
@@ -31,6 +31,17 @@ dependencies = [
"tokio",
]
[[package]]
name = "android-ffi"
version = "0.1.0"
dependencies = [
"anyhow",
"common",
"jni 0.21.1",
"tokio",
"viewer-core",
]
[[package]]
name = "anyhow"
version = "1.0.104"
@@ -76,7 +87,7 @@ dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"thiserror",
"thiserror 2.0.21",
]
[[package]]
@@ -121,6 +132,12 @@ dependencies = [
"shlex",
]
[[package]]
name = "cesu8"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6d43a04d8753f35258c91f8ec639f792891f748a1edbd759cf1dcea3382ad83c"
[[package]]
name = "cfg-if"
version = "1.0.5"
@@ -387,6 +404,22 @@ dependencies = [
"web-sys",
]
[[package]]
name = "jni"
version = "0.21.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a87aa2bb7d2af34197c04845522473242e1aa17c12f4935d5856491a7fb8c97"
dependencies = [
"cesu8",
"cfg-if",
"combine",
"jni-sys 0.3.1",
"log",
"thiserror 1.0.69",
"walkdir",
"windows-sys 0.45.0",
]
[[package]]
name = "jni"
version = "0.22.4"
@@ -396,10 +429,10 @@ dependencies = [
"cfg-if",
"combine",
"jni-macros",
"jni-sys",
"jni-sys 0.4.1",
"log",
"simd_cesu8",
"thiserror",
"thiserror 2.0.21",
"walkdir",
"windows-link",
]
@@ -417,6 +450,15 @@ dependencies = [
"syn 2.0.119",
]
[[package]]
name = "jni-sys"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41a652e1f9b6e0275df1f15b32661cf0d4b78d4d87ddec5e0c3c20f097433258"
dependencies = [
"jni-sys 0.4.1",
]
[[package]]
name = "jni-sys"
version = "0.4.1"
@@ -628,8 +670,6 @@ checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441"
[[package]]
name = "nvenc"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3227c3b09b1e042dab7d8d82cea77c0404e7d0c2bbb92e7e87cb319749cd20dc"
dependencies = [
"bitfields",
"libloading",
@@ -766,7 +806,7 @@ dependencies = [
"rustc-hash",
"rustls",
"socket2",
"thiserror",
"thiserror 2.0.21",
"tokio",
"tracing",
"web-time",
@@ -790,7 +830,7 @@ dependencies = [
"rustls-pki-types",
"rustls-platform-verifier",
"slab",
"thiserror",
"thiserror 2.0.21",
"tinyvec",
"tracing",
"web-time",
@@ -975,7 +1015,7 @@ checksum = "1167586491e2b18b8bfbb293e8180ec17c201c4f076d7cb3070ca964e7598f98"
dependencies = [
"core-foundation",
"core-foundation-sys",
"jni",
"jni 0.22.4",
"log",
"once_cell",
"rustls",
@@ -1233,13 +1273,33 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "thiserror"
version = "1.0.69"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52"
dependencies = [
"thiserror-impl 1.0.69",
]
[[package]]
name = "thiserror"
version = "2.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09e52cb86a36cede5cb101bf8908837b3e4c6e5e59fe7fd85c23fb56200d189e"
dependencies = [
"thiserror-impl",
"thiserror-impl 2.0.21",
]
[[package]]
name = "thiserror-impl"
version = "1.0.69"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
@@ -1364,6 +1424,18 @@ dependencies = [
"tokio",
]
[[package]]
name = "viewer-core"
version = "0.1.0"
dependencies = [
"anyhow",
"common",
"openh264",
"quinn",
"rustls",
"tokio",
]
[[package]]
name = "walkdir"
version = "2.5.0"
@@ -1680,13 +1752,22 @@ dependencies = [
"windows-link",
]
[[package]]
name = "windows-sys"
version = "0.45.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "75283be5efb2831d37ea142365f009c02ec203cd29a3ebecbc093d52315b66d0"
dependencies = [
"windows-targets 0.42.2",
]
[[package]]
name = "windows-sys"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
dependencies = [
"windows-targets",
"windows-targets 0.52.6",
]
[[package]]
@@ -1698,20 +1779,35 @@ dependencies = [
"windows-link",
]
[[package]]
name = "windows-targets"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e5180c00cd44c9b1c88adb3693291f1cd93605ded80c250a75d472756b4d071"
dependencies = [
"windows_aarch64_gnullvm 0.42.2",
"windows_aarch64_msvc 0.42.2",
"windows_i686_gnu 0.42.2",
"windows_i686_msvc 0.42.2",
"windows_x86_64_gnu 0.42.2",
"windows_x86_64_gnullvm 0.42.2",
"windows_x86_64_msvc 0.42.2",
]
[[package]]
name = "windows-targets"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
dependencies = [
"windows_aarch64_gnullvm",
"windows_aarch64_msvc",
"windows_i686_gnu",
"windows_aarch64_gnullvm 0.52.6",
"windows_aarch64_msvc 0.52.6",
"windows_i686_gnu 0.52.6",
"windows_i686_gnullvm",
"windows_i686_msvc",
"windows_x86_64_gnu",
"windows_x86_64_gnullvm",
"windows_x86_64_msvc",
"windows_i686_msvc 0.52.6",
"windows_x86_64_gnu 0.52.6",
"windows_x86_64_gnullvm 0.52.6",
"windows_x86_64_msvc 0.52.6",
]
[[package]]
@@ -1723,18 +1819,36 @@ dependencies = [
"windows-link",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "597a5118570b68bc08d8d59125332c54f1ba9d9adeedeef5b99b02ba2b0698f8"
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
[[package]]
name = "windows_aarch64_msvc"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e08e8864a60f06ef0d0ff4ba04124db8b0fb3be5776a5cd47641e942e58c4d43"
[[package]]
name = "windows_aarch64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
[[package]]
name = "windows_i686_gnu"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c61d927d8da41da96a81f029489353e68739737d3beca43145c8afec9a31a84f"
[[package]]
name = "windows_i686_gnu"
version = "0.52.6"
@@ -1747,24 +1861,48 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
[[package]]
name = "windows_i686_msvc"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "44d840b6ec649f480a41c8d80f9c65108b92d89345dd94027bfe06ac444d1060"
[[package]]
name = "windows_i686_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
[[package]]
name = "windows_x86_64_gnu"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8de912b8b8feb55c064867cf047dda097f92d51efad5b491dfb98f6bbb70cb36"
[[package]]
name = "windows_x86_64_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "26d41b46a36d453748aedef1486d5c7a85db22e56aff34643984ea85514e94a3"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
[[package]]
name = "windows_x86_64_msvc"
version = "0.42.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
[[package]]
name = "windows_x86_64_msvc"
version = "0.52.6"
Cargo.toml
+3 -1
View File
@@ -2,7 +2,9 @@
resolver = "2"
# agent-core：平台无关会话层 + 软件编码视频管线（当前仅 linux-agent 使用）
# linux-agent：Linux 被控端（X11 + XTest）
members = ["common", "agent-core", "agent", "linux-agent", "viewer"]
# viewer-core：安卓主控的连接/协议/收帧层（桌面 headless 可验证；Android 经 JNI 复用）
# android-ffi：viewer-core 的 JNI 薄层（cdylib，交叉编译成 libandroidffi.so）
members = ["common", "agent-core", "agent", "linux-agent", "viewer", "viewer-core", "android-ffi"]
# 默认成员保持 Windows 三件套，`cargo build --release` 行为与改动前完全一致；
# Linux 被控端显式构建：`cargo build --release -p linux-agent`
default-members = ["common", "agent", "viewer"]
agent-core/src/lib.rs
+103 -25
View File
@@ -11,8 +11,8 @@
use anyhow::Result;
use common::{
encode_video_frame, read_control, write_control, ControlMsg, InputEvent, MouseButton,
VideoFrameHeader,
encode_video_frame, h264_contains_idr, read_control, write_control, ControlMsg, InputEvent,
MouseButton, VideoFrameHeader,
};
use openh264::encoder::{
BitRate, Encoder, EncoderConfig, FrameRate, IntraFramePeriod, RateControlMode, UsageType,
@@ -79,11 +79,14 @@ pub struct EncoderSettings {
impl Default for EncoderSettings {
fn default() -> Self {
// 对标 UU：屏幕内容实时档；关键帧周期 300 帧 ≈ 10s
// 关键帧周期：以**时间**为准而非固定帧数，避免帧率变化后恢复窗口漂移。
// 1 秒 @30fps = 30 帧。主恢复路径是 KeyFrameReq（协议 v2 后已确认生效），
// 自动 IDR 只是兜底；此前 300 帧 @30fps 意味着兜底窗口长达 10 秒。
let max_fps = 30.0f32;
Self {
bitrate_bps: 5_000_000,
max_fps: 30.0,
intra_period: 300,
max_fps,
intra_period: (max_fps as u32).max(1),
}
}
}
@@ -125,23 +128,30 @@ enum LoopEnd {
}
/// 编码一帧并推入下行通道。返回 false 表示通道已关闭（会话结束）。
///
/// `seq` 由调用方持有并自增：帧序号必须覆盖**每一次实际发送**（含静止桌面重发），
/// 不能由编码器计数（编码器只知道编码了多少帧，不知道发了多少帧）。
fn encode_send(
encoder: &mut Encoder,
raw: &[u8],
(w, h): (u32, u32),
ts_ms: u64,
seq: u64,
tx: &watch::Sender<Option<Vec<u8>>>,
last_frame: &mut Option<Vec<u8>>,
) -> bool {
let header = VideoFrameHeader {
width: w,
height: h,
ts_ms,
};
let yuv = YUVBuffer::from_bgra8_source(BgraSliceU8::new(raw, (w as usize, h as usize)));
match encoder.encode(&yuv) {
Ok(ebs) => {
let frame = encode_video_frame(&header, &ebs.to_vec());
let payload = ebs.to_vec();
let header = VideoFrameHeader {
width: w,
height: h,
ts_ms,
seq,
keyframe: h264_contains_idr(&payload),
};
let frame = encode_video_frame(&header, &payload);
if tx.send(Some(frame.clone())).is_err() {
return false;
}
@@ -178,8 +188,17 @@ fn capture_session(
// 保留最近一帧裸像素：静止桌面收到关键帧请求时，仍能重编一个真关键帧
let mut last_raw: Option<Vec<u8>> = None;
let mut last_sent = Instant::now() - Duration::from_secs(10);
// 帧序号：覆盖每一次实际发送（含静止桌面重发），供主控端做丢帧检测
let mut seq: u64 = 0;
loop {
// 会话结束守卫：发送端已 drop 接收器时立即退出。
// 与 agent/src/capture.rs 同源——静止桌面分支不发送，只靠 send 失败退出
// 会让线程在画面不动时永不退出（X11 无 DXGI 式独占，危害较小但同样泄漏线程）。
if tx.receiver_count() == 0 {
eprintln!("[capture] no receiver, stop");
return LoopEnd::ChannelClosed;
}
let iter_start = Instant::now();
match cap.acquire(80) {
Ok(true) => match cap.copy_frame() {
@@ -189,9 +208,10 @@ fn capture_session(
encoder.force_intra_frame();
}
let ts = start.elapsed().as_millis() as u64;
if !encode_send(&mut encoder, &raw, (w, h), ts, tx, &mut last_frame) {
if !encode_send(&mut encoder, &raw, (w, h), ts, seq, tx, &mut last_frame) {
return LoopEnd::ChannelClosed; // 会话结束
}
seq = seq.wrapping_add(1);
last_raw = Some(raw);
last_sent = Instant::now();
// 帧间隔节流到 max_fps
@@ -206,24 +226,56 @@ fn capture_session(
}
},
Ok(false) => {
// 静止桌面：先看关键帧请求，再决定是否走 500ms 保活节流。
//
// ⚠️ 这里必须保证「消费了标志就一定发送」：先前写法是
// `if force && last_raw.is_some() { ... }`，当 force 为真但
// last_raw 还是 None（刚建会话、尚未抓到任何一帧）时，标志被
// swap 消费掉却没有发送，解码端的这次请求就永久丢失。
let force = keyreq.swap(false, Ordering::Relaxed);
if force && last_raw.is_some() {
let can_key = last_raw.is_some();
if force && !can_key {
// 还没有裸帧可重编：把请求还回去，让下一轮（拿到首帧后）继续生效
keyreq.store(true, Ordering::Relaxed);
eprintln!("[capture] KeyFrameReq 暂存：尚无裸帧可重编");
std::thread::sleep(Duration::from_millis(8));
} else if force {
// 静止桌面下的关键帧恢复：画面没变不代表可以不给关键帧——
// 解码端一旦出错，若只会收到旧 P 帧就永远恢复不了
// 解码端一旦出错，若只会收到旧 P 帧就永远恢复不了。
// 关键帧请求绕过 500ms 节流：它是错误恢复路径，延迟一次 IDR
// 的代价是持续黑屏，远高于多发一帧的带宽成本。
encoder.force_intra_frame();
let raw = last_raw.as_ref().expect("上面已判断 is_some");
let raw = last_raw.as_ref().expect("can_key 已保证");
let ts = start.elapsed().as_millis() as u64;
if !encode_send(&mut encoder, raw, (w, h), ts, tx, &mut last_frame) {
if !encode_send(&mut encoder, raw, (w, h), ts, seq, tx, &mut last_frame) {
return LoopEnd::ChannelClosed;
}
seq = seq.wrapping_add(1);
last_sent = Instant::now();
} else if last_sent.elapsed() > Duration::from_millis(500) && last_frame.is_some() {
// 桌面静止：低频重发最后一帧（重复 P 帧，解码端画面不变）
// 桌面静止：低频重发最后一帧（重复 P 帧，解码端画面不变）。
// 帧头必须重建：直接 clone 旧帧会让 seq/ts_ms 停在上次发送时刻，
// 破坏 seq 单调递增契约。
let f = last_frame.as_ref().expect("上面已判断 is_some");
if tx.send(Some(f.clone())).is_err() {
return LoopEnd::ChannelClosed;
match common::split_video_frame(f) {
Ok((mut hdr, payload)) => {
hdr.seq = seq;
hdr.ts_ms = start.elapsed().as_millis() as u64;
let frame = encode_video_frame(&hdr, payload);
if tx.send(Some(frame)).is_err() {
return LoopEnd::ChannelClosed;
}
seq = seq.wrapping_add(1);
last_sent = Instant::now();
}
Err(e) => {
// 自己产出的帧解析失败 = 编码/封装路径有 bug。
// 丢弃缓存而不是每轮重复报错：静止桌面下这里会以 ~125Hz 触发。
eprintln!("[capture] 静止重发时帧格式异常，丢弃缓存: {e}");
last_frame = None;
std::thread::sleep(Duration::from_millis(8));
}
}
last_sent = Instant::now();
} else {
std::thread::sleep(Duration::from_millis(8));
}
@@ -318,20 +370,46 @@ where
let (mut send, mut recv) = conn.accept_bi().await?;
let first = read_control(&mut recv).await?;
let code_ok = matches!(&first, ControlMsg::Auth { code: c } if *c == cfg.code);
if !code_ok {
write_control(&mut send, &ControlMsg::AuthFail("bad code".into())).await?;
// 先校验版本，再校验验证码：版本不一致时字段布局可能不同，验证码的解析结果不可信
let auth_err: Option<String> = match &first {
ControlMsg::Auth {
code: c,
proto_version,
} => {
if *proto_version != common::PROTO_VERSION {
Some(format!(
"protocol version mismatch: client={proto_version}, server={}",
common::PROTO_VERSION
))
} else if *c != cfg.code {
Some("bad code".into())
} else {
None
}
}
other => Some(format!("expected Auth, got {other:?}")),
};
if let Some(msg) = auth_err {
write_control(&mut send, &ControlMsg::AuthFail(msg.clone())).await?;
send.finish()?;
// 必须等对端读到 AuthFail 并挂断再退出：直接 return 会让进程带走 QUIC
// 缓冲区里尚未发出的数据，客户端只会看到 "connection lost" 而不是 AuthFail
let _ = tokio::time::timeout(Duration::from_secs(5), conn.closed()).await;
anyhow::bail!("auth failed");
anyhow::bail!(msg);
}
// 采集后端先建一次：既拿桌面尺寸回给主控端，也避免白建一个 duplication 只为读尺寸
let first_cap = capture_factory()?;
let (w, h) = first_cap.size();
write_control(&mut send, &ControlMsg::AuthOk { width: w, height: h }).await?;
write_control(
&mut send,
&ControlMsg::AuthOk {
width: w,
height: h,
proto_version: common::PROTO_VERSION,
},
)
.await?;
println!("[agent] auth ok, desktop {w}x{h}");
// 流 2：键鼠输入 + 控制上行（与视频分离，避免被视频积压阻塞）
agent/Cargo.toml
+4 -1
View File
@@ -12,7 +12,10 @@ rustls.workspace = true
rcgen.workspace = true
lz4_flex.workspace = true
openh264 = { version = "0.9.9", features = ["source"] }
nvenc = "0.1.0"
# 上游 nvenc 0.1.0 的 encode_picture 只设 picture_type=IDR、不置 FORCEIDR 标志，
# 导致运行期 KeyFrameReq 被编码器静默忽略（实测输出 NAL type=1 而非 5）。
# vendor 版本补上了该标志，见 vendor/nvenc/src/safe/encoder.rs 的「本项目补丁」注释。
nvenc = { path = "../vendor/nvenc" }
windows = { workspace = true, features = [
"Win32_Foundation",
"Win32_Graphics_Direct3D",
agent/src/bin/dump_stream.rs
+316
View File
@@ -0,0 +1,316 @@
//! 临时诊断工具：抓 agent 下发的码流，逐帧打印 NAL 结构并试解码。
//!
//! 目的：定位「首帧能解、之后连续 decode failed」的根因，不靠猜。
//!
//! 用法：
//! cargo run --release -p agent --bin dump_stream -- --port 47805 --code t
//! cargo run --release -p agent --bin dump_stream -- --port 47805 --code t --skip-at 2
//!
//! `--skip-at N`：模拟主控端的「积压跳帧」行为——第 N 帧读出但不喂给解码器，
//! 用来验证「跳帧是否直接导致后续所有 P 帧解不出来」。
use anyhow::{bail, Result};
use common::{read_control, read_video_frame, write_control, ControlMsg};
use openh264::decoder::Decoder;
use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier};
use rustls::pki_types::{CertificateDer, ServerName, UnixTime};
use rustls::{DigitallySignedStruct, SignatureScheme};
use std::net::ToSocketAddrs;
use std::sync::Arc;
use std::time::Duration;
#[derive(Debug)]
struct SkipVerify;
impl ServerCertVerifier for SkipVerify {
fn verify_server_cert(
&self,
_e: &CertificateDer<'_>,
_i: &[CertificateDer<'_>],
_s: &ServerName<'_>,
_o: &[u8],
_n: UnixTime,
) -> Result<ServerCertVerified, rustls::Error> {
Ok(ServerCertVerified::assertion())
}
fn verify_tls12_signature(
&self,
_m: &[u8],
_c: &CertificateDer<'_>,
_d: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
Ok(HandshakeSignatureValid::assertion())
}
fn verify_tls13_signature(
&self,
_m: &[u8],
_c: &CertificateDer<'_>,
_d: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
Ok(HandshakeSignatureValid::assertion())
}
fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
vec![
SignatureScheme::RSA_PKCS1_SHA256,
SignatureScheme::ECDSA_NISTP256_SHA256,
SignatureScheme::ED25519,
SignatureScheme::RSA_PSS_SHA256,
]
}
}
/// 扫 Annex-B 起始码，返回 NAL 类型列表（如 [7, 8, 5]）
fn nal_types(payload: &[u8]) -> Vec<u8> {
let mut out = Vec::new();
let mut i = 0usize;
while i + 3 < payload.len() {
let sc = if payload[i..].starts_with(&[0, 0, 0, 1]) {
4
} else if payload[i..].starts_with(&[0, 0, 1]) {
3
} else {
i += 1;
continue;
};
match payload.get(i + sc) {
Some(&b) => {
out.push(b & 0x1F);
i += sc + 1;
}
None => break,
}
}
out
}
fn describe(types: &[u8]) -> String {
let has = |t: u8| types.contains(&t);
let mut s = String::new();
if has(7) {
s.push_str("SPS ");
}
if has(8) {
s.push_str("PPS ");
}
if has(5) {
s.push_str("IDR ");
}
if has(1) {
s.push_str("P ");
}
if has(6) {
s.push_str("SEI ");
}
if s.is_empty() {
s.push_str("(未知)");
}
format!("{}[{}]", s.trim(), types.len())
}
fn arg(name: &str) -> Option<String> {
let a: Vec<String> = std::env::args().collect();
a.iter()
.position(|x| x == name)
.and_then(|i| a.get(i + 1))
.cloned()
}
#[tokio::main]
async fn main() -> Result<()> {
rustls::crypto::ring::default_provider()
.install_default()
.ok();
let port: u16 = arg("--port")
.map(|s| s.parse())
.transpose()?
.unwrap_or(47800);
let code = arg("--code").unwrap_or_else(|| "123456".into());
let skip_at: Option<usize> = arg("--skip-at").map(|s| s.parse()).transpose()?;
let total: usize = arg("--frames")
.map(|s| s.parse())
.transpose()?
.unwrap_or(30);
// --save <路径>：把**裸 H.264 码流**（各帧载荷按到达顺序直接拼接）
// 写成 Annex-B 文件，供安卓 MediaCodec 等外部解码器验证兼容性。
// 注意存的是载荷而非完整帧（不含自定义帧头），因为外部解码器只认标准 Annex-B。
let save_path = arg("--save");
let tls = rustls::ClientConfig::builder()
.dangerous()
.with_custom_certificate_verifier(Arc::new(SkipVerify))
.with_no_client_auth();
let qc: quinn::crypto::rustls::QuicClientConfig = Arc::new(tls).try_into()?;
let mut transport = quinn::TransportConfig::default();
transport.receive_window(quinn::VarInt::from_u32(16 * 1024 * 1024));
transport.stream_receive_window(quinn::VarInt::from_u32(8 * 1024 * 1024));
let mut client_config = quinn::ClientConfig::new(Arc::new(qc));
client_config.transport_config(Arc::new(transport));
let mut endpoint = quinn::Endpoint::client("0.0.0.0:0".parse()?)?;
endpoint.set_default_client_config(client_config);
let addr = ("127.0.0.1", port)
.to_socket_addrs()?
.next()
.ok_or_else(|| anyhow::anyhow!("解析失败"))?;
let conn = endpoint.connect(addr, "localhost")?.await?;
println!("[dump] connected");
let (mut send, mut recv) = conn.open_bi().await?;
write_control(
&mut send,
&ControlMsg::Auth {
code,
proto_version: common::PROTO_VERSION,
},
)
.await?;
match read_control(&mut recv).await? {
ControlMsg::AuthOk {
width,
height,
proto_version,
} => println!("[dump] auth ok {width}x{height} proto={proto_version}"),
other => bail!("认证失败: {other:?}"),
}
let (mut isend, _irecv) = conn.open_bi().await?;
// 入会先请求一次 IDR，和 viewer 行为一致。
// 注意：KeyFrameReq 必须走**流 2**（isend），agent 的读循环只监听流 2。
write_control(&mut isend, &ControlMsg::KeyFrameReq).await?;
let mut decoder = Decoder::new()?;
let mut sent_kfreq = false;
// 累积裸码流，用于 --save
let mut saved: Vec<u8> = Vec::new();
println!("\n序号 seq kf 载荷字节 NAL 结构 解码结果");
println!("{}", "-".repeat(72));
for i in 0..total {
let (hdr, payload) = match tokio::time::timeout(
Duration::from_secs(10),
read_video_frame(&mut recv),
)
.await
{
Ok(r) => r?,
Err(_) => {
println!("[{i}] 读帧超时（桌面可能完全静止）");
break;
}
};
let types = nal_types(&payload);
let is_idr = types.contains(&5);
// 需要在 save 里也保留这一帧时，先记录到缓冲
if save_path.is_some() {
saved.extend_from_slice(&payload);
}
// 交叉校验：帧头 keyframe 标志必须与码流实际 NAL 类型一致。
// 这条不一致意味着时间戳/序号契约或编码器标志有 bug，比画面花屏更难排查。
if hdr.keyframe != is_idr {
println!(
" ⚠️ 帧头 keyframe={} 但码流实际 IDR={}（seq={}）",
hdr.keyframe, is_idr, hdr.seq
);
}
// 模拟主控端积压跳帧：读出但不解码
if Some(i) == skip_at {
println!(
"[{i:2}] {:3} {:<5} {:8} {:<14} ⏭ 跳过解码（模拟积压跳帧）",
hdr.seq,
hdr.keyframe,
payload.len(),
describe(&types)
);
continue;
}
let res = match decoder.decode(&payload) {
Ok(Some(_)) => "✅ 解出画面".to_string(),
Ok(None) => "○ 无输出（缓冲中）".to_string(),
Err(e) => format!("❌ {e}"),
};
println!(
"[{i:2}] {:3} {:<5} {:8} {:<14} {}",
hdr.seq,
hdr.keyframe,
payload.len(),
describe(&types),
res
);
// 和 viewer 一样：解码失败就请求关键帧
if res.starts_with('❌') && !sent_kfreq {
sent_kfreq = true;
write_control(&mut isend, &ControlMsg::KeyFrameReq).await?;
println!(" ↑ 已发 KeyFrameReq");
}
}
// 诊断：静止桌面下再请求一次关键帧，看是否真能拿到 IDR
println!("\n[dump] 再发一次 KeyFrameReq，观察 10 帧…");
write_control(&mut isend, &ControlMsg::KeyFrameReq).await?;
for i in 0..10 {
match tokio::time::timeout(Duration::from_secs(5), read_video_frame(&mut recv)).await {
Ok(Ok((hdr, payload))) => {
let types = nal_types(&payload);
println!(
"[k{i:2}] seq={:3} kf={:<5} {:6}B {}",
hdr.seq,
hdr.keyframe,
payload.len(),
describe(&types)
);
if types.contains(&5) {
println!(" ↑ 收到 IDR");
}
}
_ => {
println!("[k{i:2}] 超时/流关闭");
break;
}
}
}
// 导出裸码流文件（Annex-B 拼接），供外部解码器验证
if let Some(p) = save_path.as_ref() {
std::fs::write(p, &saved)?;
let idr_cnt = {
// 粗略统计 IDR 数量：扫起始码后的 NAL 类型
let mut n = 0;
let mut i = 0usize;
while i + 3 < saved.len() {
let sc = if saved[i..].starts_with(&[0, 0, 0, 1]) {
4
} else if saved[i..].starts_with(&[0, 0, 1]) {
3
} else {
i += 1;
continue;
};
match saved.get(i + sc) {
Some(&b) if b & 0x1F == 5 => {
n += 1;
i += sc + 1;
}
Some(_) => i += sc + 1,
None => break,
}
}
n
};
println!(
"\n[dump] 已保存裸码流: {p}（{} 字节，含 {idr_cnt} 个 IDR）",
saved.len()
);
}
let _ = conn.close(0u32.into(), b"done");
Ok(())
}
agent/src/capture.rs
+127 -20
View File
@@ -4,7 +4,7 @@
//! 支持主控端 KeyFrameReq 强制关键帧（解码异常恢复）。
use anyhow::{anyhow, Result};
use common::{encode_video_frame, VideoFrameHeader};
use common::{encode_video_frame, h264_contains_idr, VideoFrameHeader};
use nvenc::bitstream::BitStream;
use nvenc::encoder::RegisteredResource;
use nvenc::session::{InitParams, Session};
@@ -39,6 +39,17 @@ use windows::Win32::Graphics::Dxgi::{
use std::time::{Duration, Instant};
/// 周期 IDR 的帧间隔（兜底恢复点）。
///
/// 两个编码器共用同一常量，避免 NVENC 与软编的恢复窗口不一致——此前分别是
/// 240 帧@60fps=4s 与 300 帧@60fps=5s，而 Linux 端又是 300 帧@30fps=10s。
///
/// 取值理由：**主恢复路径是 KeyFrameReq，不是自动 IDR**（协议 v2 后已确认请求
/// 真正生效，见 vendor/nvenc 的 FORCEIDR 补丁）。自动 IDR 只用于兜底
/// 「请求连续丢失」这种极端情况。60 帧 @60fps = 1 秒，既足够兜底，
/// 又不会因过密的整帧内编码抬高码率。
pub const INTRA_PERIOD_FRAMES: u32 = 60;
pub struct Duplication {
pub device: ID3D11Device,
context: ID3D11DeviceContext,
@@ -201,8 +212,15 @@ impl NvencEncoder {
// CBR 桌面实时档：码率恒定，延迟可预测（对标 UU 3.2Mbps@60fps）
config.preset_cfg.rc_params.rate_control_mode = NVencParamsRcMode::CBR;
config.preset_cfg.rc_params.average_bit_rate = bitrate_bps;
config.preset_cfg.gop_len = 240; // 周期 IDR（60fps 下 4s）：网络异常后的自愈恢复点，
// 配合 KeyFrameReq 即时恢复，避免长时间冻结
// 周期 IDR 是**兜底恢复点**，不是主恢复路径——主路径是 KeyFrameReq
// （协议 v2 后已确认真正生效，见 vendor/nvenc 的 FORCEIDR 补丁）。
//
// 取值权衡：GOP 越短，丢帧后自动恢复越快，但每次 IDR 都是整帧内编码，
// 码率开销显著（1080p 桌面 IDR 通常数倍于 P 帧）。
// 60 帧 @60fps = 1s：既覆盖「KeyFrameReq 连续丢失」这种极端情况，
// 又把兜底窗口压到 1 秒以内。原先 240 帧 = 4s，在请求已可靠的前提下
// 只会让受害时长无谓地长 4 倍。
config.preset_cfg.gop_len = INTRA_PERIOD_FRAMES;
config.preset_cfg.frame_interval_p = 1; // 无 B 帧
let init = InitParams {
@@ -312,7 +330,9 @@ pub fn run(tx: Sender<Option<Vec<u8>>>, keyreq: Arc<AtomicBool>, congestion: Arc
.max_frame_rate(FrameRate::from_hz(60.0))
.rate_control_mode(RateControlMode::Bitrate)
.usage_type(UsageType::ScreenContentRealTime)
.intra_frame_period(IntraFramePeriod::from_num_frames(300))
.intra_frame_period(IntraFramePeriod::from_num_frames(
INTRA_PERIOD_FRAMES,
))
.num_threads(4),
) {
Ok(e) => {
@@ -327,41 +347,63 @@ pub fn run(tx: Sender<Option<Vec<u8>>>, keyreq: Arc<AtomicBool>, congestion: Arc
}
};
let mut last_frame: Option<Vec<u8>> = None; // 软编静止重发
// 最近一帧的**裸载荷**与关键帧标志：静止重发时要用它们重建帧头，
// 不能直接重发整帧（那样 seq/ts_ms 会停在上次发送时刻）
let mut last_payload: Option<Vec<u8>> = None;
let mut last_kf = false;
let mut last_sent = Instant::now() - Duration::from_secs(10);
let min_frame = Duration::from_millis(16); // 60fps 上限
let mut scratch: Vec<u8> = Vec::new(); // 软编：行距不连续时的复用缓冲
// 帧序号：每发一帧 +1（含静止桌面的重复帧），供主控端做丢帧检测
let mut seq: u64 = 0;
// 编码耗时统计（每 120 帧输出一次均值，ms）
let mut stat_frames = 0u64;
let mut stat_convert_ms = 0u64;
let mut stat_encode_ms = 0u64;
let mut stat_skipped = 0u64; // 拥塞跳帧计数
// `$kf` 由调用点显式给出：NVENC 路径知道 `force_key`，软编路径同理。
// 不要在这里猜——首帧和 KeyFrameReq 都会产生 IDR，猜错会让解码端错过重同步点。
macro_rules! emit {
($payload:expr) => {{
($payload:expr, $kf:expr) => {{
let header = VideoFrameHeader {
width: dup.width,
height: dup.height,
ts_ms: start.elapsed().as_millis() as u64,
seq,
keyframe: $kf,
};
let frame = encode_video_frame(&header, &$payload);
let payload_vec: Vec<u8> = $payload;
let frame = encode_video_frame(&header, &payload_vec);
if tx.send(Some(frame.clone())).is_err() {
return; // 会话结束
}
last_frame = Some(frame);
last_payload = Some(payload_vec);
last_kf = $kf;
last_sent = Instant::now();
seq = seq.wrapping_add(1);
stat_frames += 1;
}};
}
loop {
// 会话结束守卫：发送端已 drop 接收器时立即退出。
// 只靠 `tx.send` 失败退出是不够的——桌面完全静止时这里根本不发送，
// 线程会永远持有 duplication，导致下一个会话的 DuplicateOutput 失败
// （DXGI 每个输出仅允许一个活动实例）。
if tx.receiver_count() == 0 {
eprintln!("[capture] no receiver, stop");
return;
}
let iter_start = Instant::now();
match dup.acquire(80) {
Ok(true) => {
// 主控端请求关键帧（解码异常恢复）
let force_key = keyreq.swap(false, Ordering::Relaxed);
// 拥塞反馈：发送侧排队深 → 跳过本帧（NVENC 少输入一帧，参考链连续；
// 解码端收到的帧序 = 编码端实际输出序，无花屏风险）
//
// ⚠️ 必须放在消费 keyreq **之前**：先前顺序是「先 swap 再判断拥塞」，
// 一旦本帧因拥塞被跳过，已消费的 KeyFrameReq 就随 continue 一起丢失，
// 解码端等不到 IDR（与下方静止分支同源的缺陷）。
if congestion.swap(false, Ordering::Relaxed) {
stat_skipped += 1;
let el = iter_start.elapsed();
@@ -370,13 +412,19 @@ pub fn run(tx: Sender<Option<Vec<u8>>>, keyreq: Arc<AtomicBool>, congestion: Arc
}
continue;
}
// 主控端请求关键帧（解码异常恢复）。走到这里一定会编码，
// 因此可以安全地消费标志。
let force_key = keyreq.swap(false, Ordering::Relaxed);
match &mut encoder {
EncoderKind::Nvenc(enc) => {
let enc_start = Instant::now();
let r = enc.encode(force_key, start.elapsed().as_millis() as u64);
stat_encode_ms += enc_start.elapsed().as_millis() as u64;
match r {
Ok(payload) => emit!(payload),
Ok(payload) => {
let kf = h264_contains_idr(&payload);
emit!(payload, kf)
}
Err(e) => eprintln!("[capture] nvenc encode failed: {e}"),
}
}
@@ -410,7 +458,11 @@ pub fn run(tx: Sender<Option<Vec<u8>>>, keyreq: Arc<AtomicBool>, congestion: Arc
let r = enc.encode(&yuv);
stat_encode_ms += enc_start.elapsed().as_millis() as u64;
match r {
Ok(ebs) => emit!(ebs.to_vec()),
Ok(ebs) => {
let payload = ebs.to_vec();
let kf = h264_contains_idr(&payload);
emit!(payload, kf)
}
Err(e) => {
eprintln!("[capture] encode failed: {e}");
}
@@ -442,25 +494,80 @@ pub fn run(tx: Sender<Option<Vec<u8>>>, keyreq: Arc<AtomicBool>, congestion: Arc
}
Ok(false) => {
// 桌面静止：低频重发（丢包后恢复 + 会话保活）。
// 关键：KeyFrameReq 在桌面静止时也必须响应 —— 此前只在新帧路径
// 消费该标志，桌面完全静止时解码端请求 IDR 永远得不到响应 → 黑屏。
//
// ⚠️ 这里曾有三个缺陷，全部围绕「KeyFrameReq 被静默丢弃」：
// 1. `swap` 写在时间闸**外面**：请求若落在距上次发送 500ms 之内，
// 标志已被消费但分支不进入，该次请求直接丢失（手册 §6.1 第 3 条）。
// 2. 软编分支只重发旧码流、不看 force_key：永远吐不出 IDR（手册 §6.1 第 2 条）。
// 3. NVENC 分支虽然把 force_key 传给了编码器，但受第 1 条限制。
//
// 现在的语义：**关键帧请求绕过 500ms 节流**——它是错误恢复路径，
// 延迟一次 IDR 的代价是解码端持续黑屏，远高于多发一帧的带宽成本。
let force_key = keyreq.swap(false, Ordering::Relaxed);
if last_sent.elapsed() > Duration::from_millis(500) {
let due = force_key || last_sent.elapsed() > Duration::from_millis(500);
if due {
match &mut encoder {
EncoderKind::Nvenc(enc) => {
// 重编当前 GPU 帧（内容未变，输出极小；force_key 时为 IDR）
let r = enc.encode(force_key, start.elapsed().as_millis() as u64);
match r {
Ok(payload) => emit!(payload),
Ok(payload) => {
let kf = h264_contains_idr(&payload);
emit!(payload, kf)
}
Err(e) => {
eprintln!("[capture] nvenc idle encode failed: {e}")
}
}
}
EncoderKind::Soft(_) => {
// 软编：直接重发最后一帧码流副本
if let Some(f) = last_frame.as_ref() {
let _ = tx.send(Some(f.clone()));
EncoderKind::Soft(enc) => {
// 软编：静止桌面没有新像素可编，只能重发最近一帧。
// 但 force_key 时必须真的重编出 IDR —— 重发旧 P 帧对
// 已经解不动的解码端毫无帮助。
//
// 裸像素来源：staging 纹理里仍是上一帧内容（DXGI 只在
// 有新帧时覆盖），重新 Map 一次即可，不必每帧额外拷 14MB。
if force_key {
enc.force_intra_frame();
let dims = (dup.width as usize, dup.height as usize);
let conv = dup.with_mapped(|src, pitch| {
if pitch == dims.0 * 4 {
YUVBuffer::from_bgra8_source(BgraSliceU8::new(
src, dims,
))
} else {
scratch.resize(dims.0 * dims.1 * 4, 0);
for (row, dst) in
scratch.chunks_exact_mut(dims.0 * 4).enumerate()
{
dst.copy_from_slice(
&src[row * pitch..row * pitch + dims.0 * 4],
);
}
YUVBuffer::from_bgra8_source(BgraSliceU8::new(
&scratch, dims,
))
}
});
match conv {
Ok(yuv) => match enc.encode(&yuv) {
Ok(ebs) => {
let payload = ebs.to_vec();
let kf = h264_contains_idr(&payload);
emit!(payload, kf)
}
Err(e) => eprintln!(
"[capture] idle keyframe encode failed: {e}"
),
},
Err(e) => {
eprintln!("[capture] idle keyframe convert failed: {e}")
}
}
} else if let Some(p) = last_payload.as_ref() {
// 非关键帧的保活重发：复用载荷但重建帧头（seq 必须递增）
let kf = last_kf;
emit!(p.clone(), kf)
}
}
}
agent/src/input.rs
+8 -2
View File
@@ -76,9 +76,15 @@ pub fn virtual_desktop() -> (i32, i32, i32, i32) {
pub fn handle(ev: InputEvent) {
match ev {
InputEvent::MouseMove { x, y } => {
// 主控端已按虚拟桌面归一化，这里直接透传 0..65535
// 归一化坐标系 = **被采集的那个输出**（capture 用 EnumOutputs(0)，
// 即主屏；AuthOk 的 width/height 也是它）。
// MOUSEEVENTF_ABSOLUTE 不带 VIRTUALDESK 时，0..=65535 恰好映射到
// 主显示器——与采集空间一致。
// 曾经带 VIRTUALDESK：多显示器机器上注入空间变成整个虚拟桌面
// （比主屏宽），手机画面与指针位置系统性错位（2026-10-30 实测）。
// 单显示器时两者重合，所以桌面端从未暴露此问题。
send_mouse(
MOUSEEVENTF_ABSOLUTE.0 | MOUSEEVENTF_VIRTUALDESK.0 | MOUSEEVENTF_MOVE.0,
MOUSEEVENTF_ABSOLUTE.0 | MOUSEEVENTF_MOVE.0,
x as i32,
y as i32,
0,
agent/src/main.rs
+85 -14
View File
@@ -1,6 +1,8 @@
//! 被控端 Agent：DXGI 桌面复制采集 → LZ4 压缩 → QUIC 下行视频；上行接收输入事件注入。
//! 被控端 Agent：DXGI 桌面复制采集 → H.264 编码 → QUIC 下行视频；上行接收输入事件注入。
//! P0 优化：视频帧经 watch 单槽通道（latest-wins，网络慢时只发最新帧，不积压）；
//! 键鼠/控制走独立 QUIC 双向流，不与视频互相阻塞。
//! 会话模型：**串行多会话**——accept 循环一次只服务一个主控，会话结束后回到 accept
//! 等下一个连接（进程不退出）。改成并发多会话需要先解决 DXGI 单实例独占，暂不做。
mod capture;
mod input;
@@ -10,6 +12,7 @@ use common::{read_control, write_control, ControlMsg};
use rustls::pki_types::PrivatePkcs8KeyDer;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::Duration;
use tokio::io::AsyncWriteExt;
use tokio::sync::{mpsc, watch};
@@ -50,6 +53,14 @@ async fn main() -> Result<()> {
transport.receive_window(quinn::VarInt::from_u32(16 * 1024 * 1024));
transport.stream_receive_window(quinn::VarInt::from_u32(8 * 1024 * 1024));
transport.send_window(64 * 1024);
// 空闲超时：主控端被强杀（非正常 close）时收不到它的 Ping，连接会一直占住
// 串行 accept，下一个连接要等几十秒才能建立。10s = 20 个 Ping 周期，
// 正常会话（Ping 每 500ms）永不触发，只用来回收死连接。
transport.max_idle_timeout(Some(
Duration::from_secs(10)
.try_into()
.expect("10s fits in VarInt"),
));
let mut server_config = quinn::ServerConfig::with_single_cert(vec![cert], key.into())?;
server_config.transport_config(Arc::new(transport));
@@ -59,26 +70,74 @@ async fn main() -> Result<()> {
)?;
println!("[agent] listening 0.0.0.0:{port}, verify code = {code}");
let conn = endpoint
.accept()
.await
.ok_or_else(|| anyhow::anyhow!("endpoint closed"))?
.await?;
println!("[agent] peer connected: {}", conn.remote_address());
// 串行多会话：一次只服务一个主控，结束后回到 accept。
// 不做并发多会话——DXGI Desktop Duplication 每个输出仅允许一个活动实例，
// 第二个 DuplicateOutput 会让第一个收到 ACCESS_LOST。
// 会话失败（含认证失败）只记日志，不能让进程退出——否则每次连接都要重启被控端。
loop {
let Some(incoming) = endpoint.accept().await else {
println!("[agent] endpoint closed, exit");
return Ok(());
};
match incoming.await {
Ok(conn) => {
println!("[agent] peer connected: {}", conn.remote_address());
if let Err(e) = serve_session(conn, &code).await {
println!("[agent] session ended: {e:#}");
}
println!("[agent] waiting for next connection");
}
Err(e) => println!("[agent] handshake failed: {e}"),
}
}
}
/// 单次会话：认证 → 采集/发送/输入 → 会话结束。
/// 返回 Err 表示会话异常结束（含认证失败），调用方继续 accept。
async fn serve_session(conn: quinn::Connection, code: &str) -> Result<()> {
// 流 1：认证 + 视频下行
let (mut send, mut recv) = conn.accept_bi().await?;
// 认证
// 认证：先校验版本，再校验验证码。
// 顺序刻意如此——版本不一致时字段布局可能不同，验证码的解析结果不可信。
let first = read_control(&mut recv).await?;
let code_ok = matches!(&first, ControlMsg::Auth { code: c } if *c == code);
if !code_ok {
write_control(&mut send, &ControlMsg::AuthFail("bad code".into())).await?;
let auth_err: Option<String> = match &first {
ControlMsg::Auth {
code: c,
proto_version,
} => {
if *proto_version != common::PROTO_VERSION {
Some(format!(
"protocol version mismatch: client={proto_version}, server={}",
common::PROTO_VERSION
))
} else if *c != code {
Some("bad code".into())
} else {
None
}
}
other => Some(format!("expected Auth, got {other:?}")),
};
if let Some(msg) = auth_err {
write_control(&mut send, &ControlMsg::AuthFail(msg.clone())).await?;
send.finish()?;
anyhow::bail!("auth failed");
// 必须等对端读到 AuthFail 再关连接：直接返回会让进程带走 QUIC 缓冲区里
// 尚未发出的数据，客户端只会看到 "connection lost" 而不是 AuthFail。
// （agent-core 已有同款修复，此处同源补齐——手册 §6.1 第 1 条）
let _ = tokio::time::timeout(Duration::from_secs(5), conn.closed()).await;
anyhow::bail!(msg);
}
let (w, h) = capture::desktop_size()?;
write_control(&mut send, &ControlMsg::AuthOk { width: w, height: h }).await?;
write_control(
&mut send,
&ControlMsg::AuthOk {
width: w,
height: h,
proto_version: common::PROTO_VERSION,
},
)
.await?;
println!("[agent] auth ok, desktop {w}x{h}");
// 流 2：键鼠输入 + 控制上行（与视频分离，避免被视频积压阻塞）
@@ -91,7 +150,14 @@ async fn main() -> Result<()> {
// 拥塞反馈标志：发送侧 write 超过单帧预算(20ms) → 置位，采集线程跳帧降注入率
let congestion = Arc::new(AtomicBool::new(false));
let congestion_capture = congestion.clone();
std::thread::spawn(move || capture::run(tx, keyreq_capture, congestion_capture));
// 采集线程退出信号：会话结束必须等它真正退出才能 accept 下一个连接——
// DXGI Desktop Duplication 每个输出只允许一个活动实例，旧线程还持有 duplication
// 时新线程的 DuplicateOutput 会让旧实例收到 ACCESS_LOST（表现为下一个会话黑屏）。
let (done_tx, done_rx) = std::sync::mpsc::channel::<()>();
std::thread::spawn(move || {
capture::run(tx, keyreq_capture, congestion_capture);
let _ = done_tx.send(());
});
// 上行控制消息（Pong）经通道交给主循环写流 2，避免多任务争抢
let (ctl_tx, mut ctl_rx) = mpsc::unbounded_channel::<ControlMsg>();
@@ -156,6 +222,11 @@ async fn main() -> Result<()> {
}
}
reader.abort();
// 关掉流并丢弃接收端，让采集线程下一次 send 失败而退出
let _ = send.finish();
drop(frames_rx);
// 等采集线程真正退出后再返回，调用方才会 accept 下一个连接（DXGI 独占，见上）
let _ = tokio::task::spawn_blocking(move || done_rx.recv_timeout(Duration::from_secs(2))).await;
println!("[agent] session closed");
Ok(())
}
android-ffi/Cargo.toml
+20
View File
@@ -0,0 +1,20 @@
[package]
name = "android-ffi"
version = "0.1.0"
edition = "2021"
# 安卓主控的 JNI 薄层：把 viewer-core 的异步 Session 包成 Kotlin 可同步调用的句柄。
# 2026-10-29 裁决变更：原方案 A2 选 UniFFI，但其实训运行时不在 Maven Central /
# Mozilla maven（查证 404），需自编译发布，成本超过收益。改为裸 JNI——阶段 0
# （D:\working\android\phase0）已验证该链路真机可跑。接口刻意保持窄（连接/取帧/
# 发输入/销毁），将来若引入 UniFFI 也只是替换本层。
[lib]
name = "androidffi"
crate-type = ["cdylib"]
[dependencies]
viewer-core = { path = "../viewer-core" }
common = { path = "../common" }
anyhow.workspace = true
tokio.workspace = true
jni = "0.21"
android-ffi/src/lib.rs
+316
View File
@@ -0,0 +1,316 @@
//! android-ffi：viewer-core 的 JNI 薄层（阶段 1）。
//!
//! 线程模型（决定接口形态的原因）：
//! - Kotlin 解码线程调用 `nativeNextFrame` 同步取帧（内部 block_on，一次一帧）；
//! - Kotlin UI 线程调用 `nativeSend*` 发输入——tokio unbounded channel 入队不阻塞；
//! - 共用一个 current-thread runtime：`block_on(next_frame)` 期间 runtime 仍在驱动
//! QUIC 收发与 Ping；输入入队只是 channel send，不需要 runtime 参与。
//!
//! **锁纪律**：`nativeNextFrame` 在等帧期间**不得持有**会话锁——桌面静止时帧间隔
//! 可达 500ms，持锁会让 UI 线程的输入调用同步卡住。做法：短暂持锁克隆
//! `Arc<Session>` 与 `&'static Runtime` 引用后立即释放，再 block_on。
//! 代价：block_on 期间 destroy 会 use-after-free——阶段 1 由 Kotlin 侧
//! 「先停解码线程、后 destroy」的顺序纪律保证（登记在方案风险清单）。
//!
//! 句柄约定：`nativeConnect` 返回 Box 裸指针（永不为 0）作为 jlong；
//! `nativeDestroy` 释放。句柄释放后不得再使用，Kotlin 侧用置 0 保护。
//!
//! 帧返回约定：byte[]，布局 = [1 标志][8 seq LE][8 ts LE][4 w LE][4 h LE][payload]。
//! 定长头 25 字节；Kotlin 读 buf[0] 判断 keyframe（标志位复用为：0=P 帧, 1=IDR 帧，
//! 会话结束则整个返回 null）。
use common::InputEvent;
use jni::objects::{JClass, JString};
use jni::sys::{jbyteArray, jint, jlong, jstring};
use jni::JNIEnv;
use std::sync::{Arc, Mutex};
use viewer_core::{ConnectConfig, Session};
/// 帧头长度：keyframe 标志(1) + seq(8) + ts(8) + w(4) + h(4)
const FRAME_HEADER_LEN: usize = 25;
struct SessionBox {
session: Arc<Session>,
runtime: tokio::runtime::Runtime,
}
/// 连接被控端。成功返回句柄（>0），失败返回 0；错误消息经 `nativeLastError()` 取。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeConnect<'local>(
mut env: JNIEnv<'local>,
_class: JClass<'local>,
host: JString<'local>,
port: jint,
code: JString<'local>,
) -> jlong {
let host_s = match env.get_string(&host) {
Ok(s) => s.to_string_lossy().into_owned(),
Err(_) => {
set_last_error("host 参数不是合法字符串");
return 0;
}
};
let code_s = match env.get_string(&code) {
Ok(s) => s.to_string_lossy().into_owned(),
Err(_) => {
set_last_error("code 参数不是合法字符串");
return 0;
}
};
let built = std::panic::catch_unwind(|| {
let runtime = match tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
{
Ok(r) => r,
Err(e) => return Err(format!("runtime init: {e}")),
};
let cfg = ConnectConfig::direct(host_s, port as u16, code_s);
match runtime.block_on(Session::connect(cfg)) {
Ok(session) => Ok(SessionBox {
session: Arc::new(session),
runtime,
}),
Err(e) => Err(format!("{e:#}")),
}
});
match built {
Ok(Ok(boxed)) => {
set_last_error("");
Box::into_raw(Box::new(Mutex::new(boxed))) as jlong
}
Ok(Err(msg)) => {
set_last_error(&msg);
0
}
Err(_) => {
set_last_error("panic in nativeConnect");
0
}
}
}
fn set_last_error(msg: &str) {
let mut slot = LAST_ERROR.lock().unwrap();
if msg.is_empty() {
*slot = None;
} else {
*slot = Some(msg.to_string());
}
}
static LAST_ERROR: Mutex<Option<String>> = Mutex::new(None);
/// 取最近一次 nativeConnect 的错误消息（无错误返回 null）。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeLastError<'local>(
env: JNIEnv<'local>,
_class: JClass<'local>,
) -> jstring {
let msg = LAST_ERROR.lock().unwrap().take();
match msg {
Some(m) => env
.new_string(m)
.map(|s| s.into_raw())
.unwrap_or(std::ptr::null_mut()),
None => std::ptr::null_mut(),
}
}
/// 取下一帧（阻塞直到有帧或会话结束）。返回 null = 会话结束。
/// 仅允许解码线程调用；destroy 前必须先停止本线程（见文件头锁纪律）。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeNextFrame<'local>(
env: JNIEnv<'local>,
_class: JClass<'local>,
handle: jlong,
) -> jbyteArray {
// 短暂持锁：克隆 Arc 与拿 runtime 引用，随后立即释放——等帧不能挡输入。
let (session, runtime) = {
let Some(boxed) = handle_to_box(handle) else {
return std::ptr::null_mut();
};
let guard = lock_box(boxed);
(guard.session.clone(), &guard.runtime as *const tokio::runtime::Runtime)
};
// SAFETY: 见文件头「锁纪律」。Kotlin 保证解码线程存活期间不 destroy。
let rt = unsafe { &*runtime };
match rt.block_on(session.next_frame()) {
None => std::ptr::null_mut(),
Some(f) => {
let mut buf = vec![0u8; FRAME_HEADER_LEN + f.payload.len()];
buf[0] = f.keyframe as u8;
buf[1..9].copy_from_slice(&f.seq.to_le_bytes());
buf[9..17].copy_from_slice(&f.ts_ms.to_le_bytes());
buf[17..21].copy_from_slice(&f.width.to_le_bytes());
buf[21..25].copy_from_slice(&f.height.to_le_bytes());
buf[FRAME_HEADER_LEN..].copy_from_slice(&f.payload);
match env.byte_array_from_slice(&buf) {
Ok(a) => a.into_raw(),
Err(_) => std::ptr::null_mut(),
}
}
}
}
/// 会话统计打包：高 32 位 rtt_ms，低 32 位 frames_in 低 32 位。-1 = 句柄无效。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeStats(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) -> jlong {
let Some(boxed) = handle_to_box(handle) else {
return -1;
};
let guard = lock_box(boxed);
let st = guard.session.stats();
// 打包进 i64：高 32 位 rtt_ms、低 32 位 frames_in，Kotlin 侧移位还原
(((st.rtt_ms & 0xFFFF_FFFF) << 32) | (st.frames_in & 0xFFFF_FFFF)) as jlong
}
/// 被控端桌面尺寸，打包成 i64：高 32 位宽、低 32 位高。0 = 句柄无效。
/// Kotlin 侧必须在建解码器之前调用，用返回值定 SurfaceView 宽高比。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeDesktopSize(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) -> jlong {
let Some(boxed) = handle_to_box(handle) else {
return 0;
};
let guard = lock_box(boxed);
let (w, h) = guard.session.desktop_size();
(((w as u64 & 0xFFFF_FFFF) << 32) | (h as u64 & 0xFFFF_FFFF)) as jlong
}
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeSendMouseMove(
_env: JNIEnv,
_class: JClass,
handle: jlong,
x: jint,
y: jint,
) {
with_session(handle, |s| {
s.send_input(InputEvent::MouseMove {
x: x.clamp(0, 65535) as u16,
y: y.clamp(0, 65535) as u16,
})
});
}
/// button: 0=左 1=右 其他=中；down: 1=按下 0=抬起
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeSendMouseButton(
_env: JNIEnv,
_class: JClass,
handle: jlong,
button: jint,
down: jint,
) {
use common::MouseButton;
let b = match button {
0 => MouseButton::Left,
1 => MouseButton::Right,
_ => MouseButton::Middle,
};
with_session(handle, |s| {
s.send_input(if down != 0 {
InputEvent::MouseDown(b)
} else {
InputEvent::MouseUp(b)
})
});
}
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeSendWheel(
_env: JNIEnv,
_class: JClass,
handle: jlong,
dy: jint,
) {
with_session(handle, |s| s.send_input(InputEvent::Wheel { dy }));
}
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeSendKey(
_env: JNIEnv,
_class: JClass,
handle: jlong,
scancode: jint,
extended: jint,
down: jint,
) {
with_session(handle, |s| {
s.send_input(if down != 0 {
InputEvent::KeyDown {
scancode: scancode as u16,
extended: extended != 0,
}
} else {
InputEvent::KeyUp {
scancode: scancode as u16,
extended: extended != 0,
}
})
});
}
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeRequestKeyframe(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
with_session(handle, |s| s.request_keyframe());
}
/// 释放会话。Kotlin 侧调用后必须把句柄置 0，且先停解码线程。
#[no_mangle]
pub extern "system" fn Java_com_verify_remote_RustBridge_nativeDestroy(
_env: JNIEnv,
_class: JClass,
handle: jlong,
) {
if handle == 0 {
return;
}
let real = handle as *mut Mutex<SessionBox>;
unsafe {
let _ = Box::from_raw(real);
}
}
fn handle_to_box(handle: jlong) -> Option<&'static Mutex<SessionBox>> {
if handle == 0 {
return None;
}
let real = handle as *mut Mutex<SessionBox>;
// 悬垂指针无法在 Rust 侧检测：阶段 1 由 Kotlin「destroy 后置 0 + 先停线程」
// 的顺序纪律保证。违反即 UB，登记在方案风险清单（阶段 2 可加代际表消除）。
// NonNull::new 是安全函数；解引用才是 unsafe 点
std::ptr::NonNull::new(real).map(|p| unsafe { p.as_ref() })
}
/// 锁中毒容忍：SessionBox 内部无跨线程共享可变状态（Session 各方法内部自带
/// 并发原语），中毒后继续用是安全的。
fn lock_box(b: &Mutex<SessionBox>) -> std::sync::MutexGuard<'_, SessionBox> {
match b.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
}
}
fn with_session(handle: jlong, f: impl FnOnce(&Session)) {
let Some(boxed) = handle_to_box(handle) else {
return;
};
let guard = lock_box(boxed);
f(&guard.session);
}
apps/android-viewer/.gitignore
+15
View File
@@ -0,0 +1,15 @@
# Gradle / Android 构建产物
.gradle/
build/
app/build/
local.properties
# Rust 交叉编译产物（由 build-android.ps1 生成到 app/libs）
app/libs/
# 验证截图
evidence/
# IDE
.idea/
*.iml
apps/android-viewer/app/build.gradle.kts
+56
View File
@@ -0,0 +1,56 @@
plugins {
id("com.android.application")
id("org.jetbrains.kotlin.android")
}
android {
namespace = "com.verify.remote"
compileSdk = 34
defaultConfig {
applicationId = "com.verify.remote"
minSdk = 26
targetSdk = 34
versionCode = 1
versionName = "0.1-phase1"
// 阶段 1 只测真机 arm64；x86_64 留给模拟器回归
ndk {
abiFilters += listOf("arm64-v8a")
}
}
buildTypes {
release {
isMinifyEnabled = false
}
}
compileOptions {
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
}
kotlinOptions {
jvmTarget = "11"
}
// .so 由外部脚本（build-android.ps1）预先拷到 app/libs/arm64-v8a/，
// 不让 Gradle 调 cargo——构建职责分离：Rust 侧失败与 Gradle 侧失败可立刻区分。
sourceSets {
getByName("main") {
jniLibs.srcDirs("libs")
}
}
packaging {
jniLibs {
// AGP 默认已按 useLegacyPackaging 处理；保持默认压缩+extractNativeLibs=false
// 由 AGP 保证 STORED+对齐，手动链的坑（lessons 记录）在 Gradle 下不存在
useLegacyPackaging = false
}
}
}
dependencies {
// 阶段 1 刻意零第三方 UI 依赖：纯 View 代码布局，减少变量。
// AndroidX 在阶段 2 做正式 UI 时再引入。
}
apps/android-viewer/app/src/main/AndroidManifest.xml
+21
View File
@@ -0,0 +1,21 @@
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
<!-- 阶段 1：局域网连 agent.exe 需要网络 -->
<uses-permission android:name="android.permission.INTERNET" />
<application
android:label="Remote Viewer"
android:hasCode="true">
<activity
android:name=".MainActivity"
android:exported="true"
android:keepScreenOn="true"
android:configChanges="orientation|screenSize|keyboardHidden">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
</manifest>
apps/android-viewer/app/src/main/kotlin/com/verify/remote/H264Decoder.kt
+159
View File
@@ -0,0 +1,159 @@
package com.verify.remote
import android.media.MediaCodec
import android.media.MediaFormat
import android.util.Log
import android.view.Surface
import java.nio.ByteBuffer
/**
* MediaCodec H.264 解码器封装（阶段 1：同步模式 + Surface 直出）。
*
* 两条实测教训直接写进实现（来源见个人层 lessons / baseline）：
* 1. Exynos 等设备带 requires-allocate-on-input/output-ports Quirk，
* 必须用 getInputBuffer(i)/getOutputBuffer(i)，不用已废弃的数组版 API。
* 2. 参考帧链断裂后 P 帧必然解不出：解码线程维护 awaitingIdr 状态，
* 只喂 IDR 之后的帧；错误后 flush + requestKeyframe 并等 IDR（R-3）。
*
* MediaFormat 不设 csd：码流每个 IDR 自带 SPS/PPS（dump_stream 实测
* "SPS PPS IDR" 结构），解码器从码流内联参数集自举。
*/
class H264Decoder(private val surface: Surface) {
@Volatile
var stopped = false
private set
private var codec: MediaCodec? = null
private val bufferInfo = MediaCodec.BufferInfo()
/** 等待 IDR 标志：true 时 P 帧一律不喂解码器 */
private var awaitingIdr = true
private var configuredW = 0
private var configuredH = 0
/** 解码一帧。返回 false = 该帧被丢弃（等 IDR 中或缓冲满），不视为错误。 */
fun queue(frame: RustBridge.Frame): Boolean {
// 等待 IDR：参考链未建立/已断裂时，P 帧喂进去只会产生错误
if (awaitingIdr && !frame.keyframe) return false
if (frame.keyframe) awaitingIdr = false
val c = ensureCodec(frame) ?: return false
return try {
val idx = c.dequeueInputBuffer(INPUT_TIMEOUT_US)
if (idx < 0) return false // 输入满：阶段 1 直接丢，由消费方决定
val inBuf: ByteBuffer = c.getInputBuffer(idx) ?: return false
inBuf.clear()
inBuf.put(frame.payload)
c.queueInputBuffer(idx, 0, frame.payload.size, frame.tsMs * 1000, 0)
drainOnce(c)
true
} catch (e: MediaCodec.CodecException) {
Log.e(TAG, "codec error isRecoverable=${e.isRecoverable}", e)
invalidate()
false
} catch (e: IllegalStateException) {
Log.e(TAG, "codec illegal state", e)
invalidate()
false
}
}
/** 解码错误/跳帧后调用：flush 复位解码器（三级恢复第一级），
* 置等待标志；调用方负责 requestKeyframe。 */
fun invalidate() {
awaitingIdr = true
try {
codec?.flush()
} catch (e: Exception) {
// flush 失败说明解码器不可恢复：销毁，等下一个 IDR 重建
Log.w(TAG, "flush failed, releasing codec", e)
releaseCodec()
}
}
/** 确保解码器就绪。只在 keyframe 到来时才配置/重建（首配与重配同路径）。 */
private fun ensureCodec(frame: RustBridge.Frame): MediaCodec? {
val c = codec
if (c != null && configuredW == frame.width && configuredH == frame.height) {
return c
}
if (c != null) {
Log.i(TAG, "size changed ${configuredW}x$configuredH -> ${frame.width}x${frame.height}, reconfigure")
releaseCodec()
}
if (codec == null && frame.keyframe) {
val name = pickDecoderName()
val nc = if (name.isEmpty()) MediaCodec.createDecoderByType(MIME)
else MediaCodec.createByCodecName(name)
val fmt = MediaFormat.createVideoFormat(MIME, frame.width, frame.height)
nc.configure(fmt, surface, null, 0)
nc.start()
codec = nc
configuredW = frame.width
configuredH = frame.height
Log.i(TAG, "decoder configured ${frame.width}x${frame.height} name=${nc.name}")
}
return codec
}
/** 优先硬解；查不到就返回空串让系统自选（避免选中软件解码器）。 */
private fun pickDecoderName(): String {
return try {
android.media.MediaCodecList(android.media.MediaCodecList.REGULAR_CODECS)
.codecInfos
.firstOrNull { info ->
!info.isEncoder &&
info.supportedTypes.any { t -> t.equals(MIME, true) } &&
!info.name.startsWith("OMX.google.") &&
!info.name.startsWith("c2.android.")
}?.name ?: ""
} catch (e: Exception) {
""
}
}
private fun drainOnce(c: MediaCodec) {
while (!stopped) {
val idx = c.dequeueOutputBuffer(bufferInfo, 0)
when {
idx == MediaCodec.INFO_TRY_AGAIN_LATER -> return
idx == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED ->
Log.i(TAG, "output format: ${c.outputFormat}")
idx >= 0 -> {
// Surface 直出：渲染由解码器完成，不取图像数据
c.releaseOutputBuffer(idx, true)
}
else -> return
}
}
}
fun release() {
stopped = true
releaseCodec()
}
private fun releaseCodec() {
try {
codec?.stop()
} catch (e: Exception) {
Log.w(TAG, "stop ignored", e)
}
try {
codec?.release()
} catch (e: Exception) {
Log.w(TAG, "release ignored", e)
}
codec = null
configuredW = 0
configuredH = 0
}
companion object {
private const val TAG = "H264Decoder"
private const val MIME = "video/avc"
private const val INPUT_TIMEOUT_US = 4_000L
}
}
apps/android-viewer/app/src/main/kotlin/com/verify/remote/MainActivity.kt
+236
View File
@@ -0,0 +1,236 @@
package com.verify.remote
import android.app.Activity
import android.os.Bundle
import android.util.Log
import android.view.KeyEvent
import android.view.SurfaceHolder
import android.widget.TextView
import kotlin.concurrent.thread
/**
* 安卓主控端（阶段 2a）：连接被控端 → MediaCodec 解码 → SurfaceView 显示 + 触摸操作。
*
* 线程模型（与 android-ffi 的锁纪律对应）：
* - UI 线程：触摸/按键 → nativeSend*（入队不阻塞）；连接/断开
* - 解码线程：nativeNextFrame 阻塞取帧 → H264Decoder.queue
* - 断开顺序：**先置 stopDecoding 并 join 解码线程，再 destroy**
* （Rust 侧明确：解码线程存活期间 destroy 是 UB）
*
* 布局：SurfaceView 按视频宽高比 letterbox 居中（RemoteSurfaceView），
* 触摸坐标只在画面区域内产生，与远端归一化坐标一一对应。
*/
class MainActivity : Activity(), RemoteSurfaceView.Listener {
private lateinit var surfaceView: RemoteSurfaceView
private lateinit var statusView: TextView
@Volatile
private var handle = 0L
@Volatile
private var stopDecoding = false
private var decodeThread: Thread? = null
@Volatile
private var remoteW = 0
@Volatile
private var remoteH = 0
/** 最近一次触摸事件（诊断用：logcat 被限流，状态栏是唯一可靠通道） */
@Volatile
private var lastTouchDbg = "-"
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
Log.i(TAG, "onCreate start")
surfaceView = RemoteSurfaceView(this).apply {
listener = this@MainActivity
onDebug = { msg -> lastTouchDbg = msg; refreshStatus() }
holder.addCallback(object : SurfaceHolder.Callback {
override fun surfaceCreated(holder: SurfaceHolder) {}
override fun surfaceChanged(h: SurfaceHolder, f: Int, w: Int, ht: Int) {}
override fun surfaceDestroyed(holder: SurfaceHolder) {
// Surface 没了解码器输出会失效：断开，重进 App 重连
disconnect()
}
})
}
statusView = TextView(this).apply {
textSize = 12f
setPadding(20, 12, 20, 12)
setBackgroundColor(0xCC000000.toInt())
setTextColor(0xFFFFFFFF.toInt())
}
// 画面居中、黑边填充。
// 用 FrameLayout + MATCH_PARENT（不是 WRAP_CONTENT）：FrameLayout 会把
// 子视图的 WRAP_CONTENT 当 MATCH_PARENT 处理，onMeasure 里拿到的
// MeasureSpec 就成了 EXACTLY 全尺寸，letterbox 比例收缩被直接忽略
// （2026-10-30 实测：视图被撑成 1080x2001，触摸按 2001 归一化 → 坐标全错）。
// MATCH_PARENT + gravity=center 时父容器传 AT_MOST，onMeasure 的
// setMeasuredDimension 才真正决定尺寸，再被 gravity 居中摆放。
val stage = android.widget.FrameLayout(this).apply {
setBackgroundColor(0xFF000000.toInt())
addView(
surfaceView,
android.widget.FrameLayout.LayoutParams(
android.widget.FrameLayout.LayoutParams.MATCH_PARENT,
android.widget.FrameLayout.LayoutParams.MATCH_PARENT,
android.view.Gravity.CENTER
)
)
}
val root = android.widget.LinearLayout(this).apply {
orientation = android.widget.LinearLayout.VERTICAL
addView(stage, android.widget.LinearLayout.LayoutParams(
android.widget.LinearLayout.LayoutParams.MATCH_PARENT, 0, 1f))
// 固定高度而非 WRAP_CONTENT：与 weight=1 的兄弟共存时，
// WRAP_CONTENT 可能被压成 0（实测 bounds=[0,0][0,0]，状态栏不可见）
addView(statusView, android.widget.LinearLayout.LayoutParams(
android.widget.LinearLayout.LayoutParams.MATCH_PARENT,
(48 * resources.displayMetrics.density).toInt()))
}
setContentView(root)
refreshStatus()
intent?.getStringExtra("host")?.let { host ->
connect(host, intent.getIntExtra("port", 47800), intent.getStringExtra("code") ?: "123456")
}
}
/** 状态栏统一刷新：连接态 + 帧数/RTT + 最近触摸诊断（logcat 被限流，这是唯一通道） */
@Volatile
private var lastFrames = 0L
@Volatile
private var lastRtt = 0L
private fun refreshStatus() {
if (handle == 0L) {
statusView.text = "未连接｜am start ... --es host <IP> --ei port 47800 --es code t"
return
}
statusView.text = buildString {
append("已连接 ${remoteW}x$remoteH 帧=$lastFrames RTT=${lastRtt}ms\n")
append("触摸: $lastTouchDbg")
}
}
// ---- RemoteSurfaceView.Listener ----
override fun sendMouseMove(x: Int, y: Int) {
val h = handle
if (h != 0L) RustBridge.nativeSendMouseMove(h, x, y)
}
override fun sendMouseButton(button: Int, down: Boolean) {
val h = handle
if (h != 0L) RustBridge.nativeSendMouseButton(h, button, if (down) 1 else 0)
}
override fun sendWheel(dy: Int) {
val h = handle
if (h != 0L) RustBridge.nativeSendWheel(h, dy)
}
/** 返回键 = 断开（不退出 App，便于重连） */
override fun onKeyDown(keyCode: Int, event: KeyEvent?): Boolean {
if (keyCode == KeyEvent.KEYCODE_BACK && handle != 0L) {
disconnect()
return true
}
return super.onKeyDown(keyCode, event)
}
// ---- 连接 / 解码 ----
private fun connect(host: String, port: Int, code: String) {
if (handle != 0L) return
statusView.text = "连接 $host:$port …"
thread(name = "connect") {
try {
val h = RustBridge.connect(host, port, code)
// 尺寸必须在解码前拿到：用它定 SurfaceView 宽高比。
// 若等首帧再改，视图会重新测量，把刚建好的解码器 surface 打掉。
val (w, ht) = RustBridge.desktopSize(h)
handle = h
remoteW = w
remoteH = ht
runOnUiThread {
if (w > 0 && ht > 0) {
surfaceView.videoAspectRatio = w.toFloat() / ht
}
refreshStatus()
startDecodeLoop(h)
}
} catch (e: Exception) {
Log.e(TAG, "connect failed", e)
runOnUiThread { statusView.text = "连接失败: ${e.message}" }
}
}
}
private fun startDecodeLoop(h: Long) {
stopDecoding = false
decodeThread = thread(name = "decode") {
val surface = surfaceView.holder.surface
var tries = 0
while (!surface.isValid && tries++ < 100) Thread.sleep(50)
if (!surface.isValid) {
Log.e(TAG, "surface never became valid")
return@thread
}
val dec = H264Decoder(surface)
var frames = 0L
var lastKeyReq = 0L
while (!stopDecoding) {
val f = try {
RustBridge.nextFrame(h)
} catch (e: Exception) {
Log.e(TAG, "nextFrame threw", e)
null
} ?: break
val fed = dec.queue(f)
if (!fed && !f.keyframe && System.currentTimeMillis() - lastKeyReq > 200) {
RustBridge.nativeRequestKeyframe(h) // 等 IDR 中：限频请求
lastKeyReq = System.currentTimeMillis()
}
frames++
if (frames % 120 == 0L) {
lastFrames = frames
lastRtt = RustBridge.rttMs(h)
runOnUiThread { refreshStatus() }
}
}
dec.release()
Log.i(TAG, "decode loop ended, frames=$frames")
runOnUiThread { refreshStatus() }
}
}
private fun disconnect() {
if (handle == 0L) return
stopDecoding = true
// 顺序纪律：等解码线程退出后才能 destroy（见 android-ffi 文件头）。
// 卡帧风险：agent 静止时 500ms 一帧、QUIC idle 超时后流必关，6s 足够覆盖。
decodeThread?.join(6000)
decodeThread = null
val h = handle
handle = 0L
remoteW = 0
remoteH = 0
lastFrames = 0
lastRtt = 0
RustBridge.destroy(h)
refreshStatus()
}
override fun onDestroy() {
super.onDestroy()
disconnect()
}
companion object {
private const val TAG = "MainActivity"
}
}
apps/android-viewer/app/src/main/kotlin/com/verify/remote/RemoteSurfaceView.kt
+237
View File
@@ -0,0 +1,237 @@
package com.verify.remote
import android.content.Context
import android.view.MotionEvent
import android.view.SurfaceView
import kotlin.math.abs
import kotlin.math.hypot
/**
* 保持视频宽高比的 SurfaceView（letterbox），并把触摸手势映射为远端鼠标事件。
*
* 为什么视图尺寸必须等于画面尺寸：触摸坐标按视图宽高归一化到远端 0..=65535。
* 若视图被拉伸（阶段 1 用 MATCH_PARENT），画面与视图不再一一对应，手指位置与
* 远端指针位置就会错位；letterbox 后黑边不属于本视图，触摸天然只在画面内产生事件。
*
* 手势映射（桌面远控的通行方案，参考 Moonlight / Scrcpy 的交互形态，
* 实现为本项目独立编写）：
* - 单指轻点 = 左键单击
* - 单指按下并移动 = 左键拖拽（窗口拖动、文字选中）
* - 双指轻点 = 右键
* - 双指上下滑动 = 滚轮
*
* 刻意不做「手指即光标」的绝对定位：手指会完全挡住目标，桌面精细操作不可用。
*/
class RemoteSurfaceView(context: Context) : SurfaceView(context) {
/** 由 Activity 实现，转发到 RustBridge */
interface Listener {
fun sendMouseMove(x: Int, y: Int)
fun sendMouseButton(button: Int, down: Boolean)
fun sendWheel(dy: Int)
}
var listener: Listener? = null
/** 视频宽高比（w/h）。>0 时视图按此比例测量。 */
var videoAspectRatio = 0f
set(value) {
if (value > 0f && field != value) {
field = value
requestLayout()
}
}
private enum class Mode { IDLE, TAP, DRAG, TWO_FINGER }
private var mode = Mode.IDLE
private var downX = 0f
private var downY = 0f
private var lastX = 0f
private var lastY = 0f
private var downTime = 0L
private var twoFingerStartY = 0f
private var twoFingerDownTime = 0L
private var twoFingerMoved = false
private var wheelAccum = 0f
/** 判定「拖拽 vs 点击」的位移阈值：12dp */
private val dragSlopPx = resources.displayMetrics.density * 12f
init {
isFocusable = true
}
/**
* letterbox 测量：在父容器给出的可用空间内按视频宽高比取最大内接矩形。
*
* 关键：父容器给 WRAP_CONTENT 时 MeasureSpec 是 AT_MOST，getSize() 是
* **上限**而非实际可用尺寸。据此算出的尺寸必须自己 setMeasuredDimension，
* 否则父容器会把视图撑满（2026-10-30 实测：视图变成 1080x2001 而非
* 1080x607，触摸按 2001 归一化 → 坐标全错，点中心落到画面外）。
*/
override fun onMeasure(widthSpec: Int, heightSpec: Int) {
val ar = videoAspectRatio
if (ar <= 0f) {
super.onMeasure(widthSpec, heightSpec)
return
}
val availW = MeasureSpec.getSize(widthSpec)
val availH = MeasureSpec.getSize(heightSpec)
if (availW <= 0 || availH <= 0) {
super.onMeasure(widthSpec, heightSpec)
return
}
var w = availW
var h = (availW / ar).toInt()
if (h > availH) {
h = availH
w = (availH * ar).toInt()
}
setMeasuredDimension(w.coerceAtLeast(1), h.coerceAtLeast(1))
}
private fun normX(x: Float) = (x / width * 65535).toInt().coerceIn(0, 65535)
private fun normY(y: Float) = (y / height * 65535).toInt().coerceIn(0, 65535)
private fun moved(fromX: Float, fromY: Float, toX: Float, toY: Float) =
hypot((toX - fromX).toDouble(), (toY - fromY).toDouble()) > dragSlopPx
private fun midX(e: MotionEvent) =
if (e.pointerCount >= 2) (e.getX(0) + e.getX(1)) / 2f else e.x
private fun midY(e: MotionEvent) =
if (e.pointerCount >= 2) (e.getY(0) + e.getY(1)) / 2f else e.y
/** 诊断回调：把触摸情况显示到状态栏（本机 logcat 被系统限流，日志不可靠） */
var onDebug: ((String) -> Unit)? = null
override fun onTouchEvent(event: MotionEvent): Boolean {
val l = listener ?: run {
onDebug?.invoke("touch: listener=null")
return false
}
if (width == 0 || height == 0) return false
val act = when (event.actionMasked) {
MotionEvent.ACTION_DOWN -> "DOWN"
MotionEvent.ACTION_MOVE -> "MOVE"
MotionEvent.ACTION_UP -> "UP"
MotionEvent.ACTION_POINTER_DOWN -> "P2DOWN"
MotionEvent.ACTION_POINTER_UP -> "P2UP"
MotionEvent.ACTION_CANCEL -> "CANCEL"
else -> event.actionMasked.toString()
}
onDebug?.invoke(
"$act view=(${event.x.toInt()},${event.y.toInt()}) " +
"norm=(${normX(event.x)},${normY(event.y)}) ${width}x$height"
)
when (event.actionMasked) {
MotionEvent.ACTION_DOWN -> {
downX = event.x; downY = event.y
lastX = event.x; lastY = event.y
downTime = System.currentTimeMillis()
mode = Mode.TAP
// 先把远端指针移到位，点击/拖拽都以此为基准
l.sendMouseMove(normX(event.x), normY(event.y))
}
MotionEvent.ACTION_POINTER_DOWN -> {
// 第二指落下 = 双指手势开始。若已进入拖拽（远端鼠标按着），
// 必须先抬起，否则远端会留下一个被按住未释放的左键。
if (mode == Mode.DRAG) l.sendMouseButton(BTN_LEFT, false)
mode = Mode.TWO_FINGER
twoFingerMoved = false
wheelAccum = 0f
twoFingerStartY = midY(event)
lastX = midX(event); lastY = midY(event)
twoFingerDownTime = System.currentTimeMillis()
}
MotionEvent.ACTION_MOVE -> when (mode) {
Mode.TAP -> if (moved(downX, downY, event.x, event.y)) {
mode = Mode.DRAG
l.sendMouseButton(BTN_LEFT, true)
lastX = event.x; lastY = event.y
l.sendMouseMove(normX(event.x), normY(event.y))
}
Mode.DRAG -> {
lastX = event.x; lastY = event.y
l.sendMouseMove(normX(event.x), normY(event.y))
}
Mode.TWO_FINGER -> {
val cx = midX(event); val cy = midY(event)
if (abs(cy - twoFingerStartY) > dragSlopPx || moved(lastX, lastY, cx, cy)) {
twoFingerMoved = true
}
// 手指下移（cy 增大）= 内容向上翻 = Windows 滚轮向下 → dy 取负
wheelAccum += (cy - lastY)
val notches = (wheelAccum / WHEEL_PX_PER_NOTCH).toInt()
if (notches != 0) {
l.sendWheel(-notches * WHEEL_DELTA)
wheelAccum -= notches * WHEEL_PX_PER_NOTCH
}
lastX = cx; lastY = cy
}
Mode.IDLE -> {}
}
MotionEvent.ACTION_POINTER_UP -> {
// 双指之一抬起：先判定右键，再把剩余手指的移动视为已结束手势
if (mode == Mode.TWO_FINGER) {
maybeRightClick(l)
mode = Mode.IDLE
}
}
MotionEvent.ACTION_UP -> {
when (mode) {
Mode.TAP -> {
// 轻点 = 左键单击（位置已在 DOWN 时发送）
l.sendMouseButton(BTN_LEFT, true)
l.sendMouseButton(BTN_LEFT, false)
performClick()
}
Mode.DRAG -> l.sendMouseButton(BTN_LEFT, false)
Mode.TWO_FINGER -> maybeRightClick(l)
Mode.IDLE -> {}
}
mode = Mode.IDLE
}
MotionEvent.ACTION_CANCEL -> {
if (mode == Mode.DRAG) l.sendMouseButton(BTN_LEFT, false)
mode = Mode.IDLE
}
}
return true
}
/** 双指未移动且按住时间短 → 右键 */
private fun maybeRightClick(l: Listener) {
if (twoFingerMoved) return
if (System.currentTimeMillis() - twoFingerDownTime > TWO_FINGER_TAP_MAX_MS) return
l.sendMouseMove(normX(lastX), normY(lastY))
l.sendMouseButton(BTN_RIGHT, true)
l.sendMouseButton(BTN_RIGHT, false)
performClick()
}
@Suppress("ClickableViewAccessibility")
override fun performClick(): Boolean {
super.performClick()
return true
}
companion object {
const val BTN_LEFT = 0
const val BTN_RIGHT = 1
const val BTN_MIDDLE = 2
private const val WHEEL_DELTA = 120
private const val WHEEL_PX_PER_NOTCH = 60f
private const val TWO_FINGER_TAP_MAX_MS = 500L
}
}
apps/android-viewer/app/src/main/kotlin/com/verify/remote/RustBridge.kt
+103
View File
@@ -0,0 +1,103 @@
package com.verify.remote
/**
* viewer-core（Rust）的 JNI 声明。符号约定：Java_com_verify_remote_RustBridge_*，
* 与 android-ffi/src/lib.rs 一一对应。
*
* 句柄纪律（Rust 侧文件头有完整说明）：
* - nativeConnect 返回 >0 句柄；0 表示失败，错误消息取 [lastError]
* - destroy 后必须置 0；**destroy 前必须先停掉解码线程**（否则 use-after-free）
*/
object RustBridge {
private const val FRAME_HEADER_LEN = 25
init {
System.loadLibrary("androidffi")
}
private external fun nativeConnect(host: String, port: Int, code: String): Long
external fun nativeLastError(): String?
/** 返回 null = 会话结束；否则 [FRAME_HEADER_LEN] 字节头 + Annex-B 载荷 */
external fun nativeNextFrame(handle: Long): ByteArray?
external fun nativeStats(handle: Long): Long
external fun nativeDesktopSize(handle: Long): Long
external fun nativeSendMouseMove(handle: Long, x: Int, y: Int)
external fun nativeSendMouseButton(handle: Long, button: Int, down: Int)
external fun nativeSendWheel(handle: Long, dy: Int)
external fun nativeSendKey(handle: Long, scancode: Int, extended: Int, down: Int)
external fun nativeRequestKeyframe(handle: Long)
external fun nativeDestroy(handle: Long)
/** 连接。失败抛 IllegalStateException（带 Rust 侧错误消息）。 */
fun connect(host: String, port: Int, code: String): Long {
val h = nativeConnect(host, port, code)
if (h == 0L) {
throw IllegalStateException(nativeLastError() ?: "connect failed")
}
return h
}
fun destroy(handle: Long) {
if (handle != 0L) nativeDestroy(handle)
}
/** 解出的帧（头字段已拆，payload 可直接喂 MediaCodec） */
data class Frame(
val keyframe: Boolean,
val seq: Long,
val tsMs: Long,
val width: Int,
val height: Int,
val payload: ByteArray,
)
/** 取帧 + 拆头。null = 会话结束。 */
fun nextFrame(handle: Long): Frame? {
val raw = nativeNextFrame(handle) ?: return null
if (raw.size < FRAME_HEADER_LEN) return null
return Frame(
keyframe = raw[0] != 0.toByte(),
seq = readLong(raw, 1),
tsMs = readLong(raw, 9),
width = readInt(raw, 17),
height = readInt(raw, 21),
payload = raw.copyOfRange(FRAME_HEADER_LEN, raw.size),
)
}
fun rttMs(handle: Long): Long {
val packed = nativeStats(handle)
return if (packed < 0) 0 else (packed ushr 32)
}
fun framesIn(handle: Long): Long {
val packed = nativeStats(handle)
return if (packed < 0) 0 else packed and 0xFFFF_FFFF
}
/**
* 被控端桌面尺寸 (w, h)。连接成功后、建解码器之前调用。
* 返回 (0,0) 表示句柄无效。
*/
fun desktopSize(handle: Long): Pair<Int, Int> {
val packed = nativeDesktopSize(handle)
if (packed == 0L) return 0 to 0
val w = ((packed ushr 32) and 0xFFFF_FFFF).toInt()
val h = (packed and 0xFFFF_FFFF).toInt()
return w to h
}
private fun readLong(b: ByteArray, off: Int): Long {
var v = 0L
for (i in 0 until 8) v = v or ((b[off + i].toLong() and 0xFF) shl (8 * i))
return v
}
private fun readInt(b: ByteArray, off: Int): Int {
return (b[off].toInt() and 0xFF) or
((b[off + 1].toInt() and 0xFF) shl 8) or
((b[off + 2].toInt() and 0xFF) shl 16) or
((b[off + 3].toInt() and 0xFF) shl 24)
}
}
apps/android-viewer/build-android.ps1
+85
View File
@@ -0,0 +1,85 @@
# Phase 1 build chain: Rust .so -> jniLibs -> Gradle debug APK -> install -> launch
#
# Keep this file ASCII-only (PS 5.1 reads BOM-less .ps1 as GBK; see lessons).
#
# Usage (from apps/android-viewer/):
# powershell -ExecutionPolicy Bypass -File build-android.ps1 [-Host_ 192.168.1.5] [-Port 47800] [-Code t] [-SkipRust]
# (param is Host_ because $Host is a PowerShell automatic variable)
#
# Prereqs (Windows dev box):
# - Rust + Android targets + cargo-ndk + NDK (see personal docs/baseline.md toolchain table)
# - JDK 17 + Gradle 8.14 (paths in $vars below, adjust to your machine)
# - local.properties with sdk.dir (gitignored; template: sdk.dir=<android sdk path>)
param(
[string]$Host_ = "192.168.1.5",
[int]$Port = 47800,
[string]$Code = "t",
[switch]$SkipRust
)
$ErrorActionPreference = "Stop"
# repo root = two levels up from this script (apps/android-viewer -> repo)
$proj = $PSScriptRoot
$repo = (Resolve-Path "$proj\..\..").Path
$env_ = "D:\working\environment"
$gradle = "D:\working\android\gradle\gradle-8.14.3\bin\gradle.bat"
$adb = "D:\working\android\sdk\platform-tools\adb.exe"
$env:RUSTUP_HOME = "$env_\rust"
$env:CARGO_HOME = "$env_\cargo"
$env:PATH = "$env_\mingw64\bin;$env_\cargo\bin;$env:PATH"
$env:ANDROID_HOME = "D:\working\android\sdk"
$env:ANDROID_NDK_HOME = "D:\working\android\sdk\ndk\30.0.16248370"
$env:GRADLE_USER_HOME = "D:\working\android\gradle-home"
$env:JAVA_HOME = "D:\working\java\jdk17"
if (-not (Test-Path "$proj\local.properties")) {
Set-Content "$proj\local.properties" 'sdk.dir=D\:\\working\\android\\sdk' -Encoding ascii
Write-Host "generated local.properties (sdk.dir)"
}
if (-not $SkipRust) {
Write-Host "=== 1/4 cargo ndk: android-ffi (arm64 release) ===" -ForegroundColor Cyan
Push-Location $repo
# cargo writes progress to stderr; PS 5.1 + ErrorActionPreference=Stop turns
# every stderr line into a terminating error. Relax for native calls and
# judge by exit code (same for gradle below).
$ErrorActionPreference = "Continue"
cargo ndk -t arm64-v8a build -p android-ffi --release 2>&1 |
Select-String "error|Finished|Building"
if ($LASTEXITCODE -ne 0) { throw "cargo ndk build failed (exit $LASTEXITCODE)" }
$ErrorActionPreference = "Stop"
Pop-Location
}
Write-Host "=== 2/4 copy .so into app/libs ===" -ForegroundColor Cyan
$so = "$repo\target\aarch64-linux-android\release\libandroidffi.so"
if (-not (Test-Path $so)) { throw "missing $so (run without -SkipRust)" }
New-Item -ItemType Directory -Force -Path "$proj\app\libs\arm64-v8a" | Out-Null
Copy-Item $so "$proj\app\libs\arm64-v8a\libandroidffi.so" -Force
Write-Host (" libandroidffi.so: {0:N0} KB" -f ((Get-Item $so).Length/1KB))
Write-Host "=== 3/4 gradle assembleDebug ===" -ForegroundColor Cyan
Push-Location $proj
$ErrorActionPreference = "Continue"
& $gradle :app:assembleDebug --console=plain 2>&1 | Select-Object -Last 25
$code = $LASTEXITCODE
$ErrorActionPreference = "Stop"
Pop-Location
if ($code -ne 0) { throw "gradle build failed (exit $code)" }
$apk = "$proj\app\build\outputs\apk\debug\app-debug.apk"
if (-not (Test-Path $apk)) { throw "apk not found: $apk" }
Write-Host (" APK: {0:N0} KB" -f ((Get-Item $apk).Length/1KB))
Write-Host "=== 4/4 install + launch ===" -ForegroundColor Cyan
$ErrorActionPreference = "Continue"
& $adb install -r $apk
if ($LASTEXITCODE -ne 0) { throw "install failed (phone must allow USB install)" }
& $adb logcat -c | Out-Null
& $adb shell am start -n com.verify.remote/.MainActivity `
--es host $Host_ --ei port $Port --es code $Code | Out-Null
$ErrorActionPreference = "Stop"
Write-Host "launched. watch: adb logcat -s MainActivity:H264Decoder RustBridge:* viewer-core:* AndroidRuntime:E"
Write-Host "after 10s capture screen: adb exec-out screencap -p > screen.png"
apps/android-viewer/build.gradle.kts
+4
View File
@@ -0,0 +1,4 @@
plugins {
id("com.android.application") version "8.9.2" apply false
id("org.jetbrains.kotlin.android") version "2.1.20" apply false
}
apps/android-viewer/gradle.properties
+4
View File
@@ -0,0 +1,4 @@
# 固定 JDK 17（不动全局 JAVA_HOME，遵守 2026-10-29 环境约束）
org.gradle.java.home=D\:\\working\\java\\jdk17
android.useAndroidX=false
org.gradle.jvmargs=-Xmx2g
apps/android-viewer/settings.gradle.kts
+17
View File
@@ -0,0 +1,17 @@
pluginManagement {
repositories {
google()
mavenCentral()
gradlePluginPortal()
}
}
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
repositories {
google()
mavenCentral()
}
}
rootProject.name = "remote-viewer"
include(":app")
common/src/lib.rs
+314 -7
View File
@@ -7,12 +7,29 @@ use anyhow::Result;
use serde::{Deserialize, Serialize};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
/// 媒体/控制协议版本。
///
/// **改动规则**：任何对 `ControlMsg` 或 `VideoFrameHeader` 字段的增删改都必须递增此值。
/// 理由：线格式是 bincode（按位置编码，无字段名），新增字段会让旧端解析失败或错位，
/// 且失败方式可能是静默的错误数据而非明确报错。递增后由 `Auth` 握手显式拒绝旧端，
/// 把"连上了但行为诡异"变成"连不上且提示升级"。
///
/// - v1：初版（无版本字段，隐式）
/// - v2：`Auth`/`AuthOk` 携带版本；帧头增加 `seq`/`keyframe`
pub const PROTO_VERSION: u16 = 2;
/// 控制通道消息（双向）
#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum ControlMsg {
/// 主控端认证（临时验证码，MVP 明文走 QUIC-TLS 加密通道）
Auth { code: String },
AuthOk { width: u32, height: u32 },
/// 主控端认证（临时验证码，MVP 明文走 QUIC-TLS 加密通道）。
/// `proto_version` 必须等于 [`PROTO_VERSION`]，否则被控端回 `AuthFail` 并拒绝。
Auth { code: String, proto_version: u16 },
/// 认证通过。`proto_version` 回显被控端的版本，供主控端确认双方一致。
AuthOk {
width: u32,
height: u32,
proto_version: u16,
},
AuthFail(String),
KeyFrameReq,
Ping(u64),
@@ -42,12 +59,22 @@ pub enum InputEvent {
KeyUp { scancode: u16, extended: bool },
}
/// 视频帧头（载荷为 LZ4 压缩的 BGRA 像素，按行紧凑排列）
/// 视频帧头。载荷为 H.264 Annex-B 码流（含 00 00 00 01 起始码）。
///
/// 字段语义：
/// - `width` / `height`：被控端桌面尺寸（逻辑尺寸，非编码对齐后的尺寸）
/// - `ts_ms`：被控端采集时刻（被控时钟），配合 Ping/Pong 估计的时钟偏移算端到端延迟
/// - `seq`：帧序号，从 0 单调递增，每发一帧 +1（含静止桌面的重复帧）。
/// 用于丢帧检测与弱网统计；注意它统计的是"发送序"，不是"显示序"
/// - `keyframe`：本帧是否 IDR。解码端（尤其 MediaCodec）需要它决定是否重置解码器，
/// 不必再自己扫 NAL 起始码判断
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct VideoFrameHeader {
pub width: u32,
pub height: u32,
pub ts_ms: u64,
pub seq: u64,
pub keyframe: bool,
}
async fn read_u32le<R: AsyncReadExt + Unpin>(r: &mut R) -> Result<u32> {
@@ -81,7 +108,9 @@ pub async fn read_control<R: AsyncReadExt + Unpin>(r: &mut R) -> Result<ControlM
Ok(bincode::deserialize(&buf)?)
}
/// 编码视频帧：[总长 u32][头长 u32][bincode 头][LZ4 载荷]
/// 编码视频帧：[总长 u32][头长 u32][bincode 头][H.264 载荷]
///
/// `总长` 不含自身这 4 字节，等于 `4 + 头长 + 载荷长`。
pub fn encode_video_frame(header: &VideoFrameHeader, payload: &[u8]) -> Vec<u8> {
let hb = bincode::serialize(header).expect("serialize header");
let total = 4 + hb.len() + payload.len();
@@ -93,14 +122,292 @@ pub fn encode_video_frame(header: &VideoFrameHeader, payload: &[u8]) -> Vec<u8>
out
}
/// 单帧总长上限（256 MiB）。超过即视为协议错误或恶意输入。
const MAX_VIDEO_FRAME: usize = 256 << 20;
/// 判断一段 Annex-B H.264 码流里是否含 IDR 片（NAL 类型 5）。
///
/// 用途：填充 [`VideoFrameHeader::keyframe`]。
/// 为什么不问编码器要这个标志——NVENC 与 openh264 的 API 都能间接推出，
/// 但两者规则不同（首帧强制、`force_intra_frame`、GOP 自动 IDR 交织），
/// 而**实际输出**是唯一可信来源。直接看码流，两种编码器一套逻辑。
///
/// 只识别 3/4 字节起始码；`start-code emulation prevention` 由编码器保证，
/// 因此不会在 SPS/PPS 载荷里误判出起始码。
pub fn h264_contains_idr(payload: &[u8]) -> bool {
let mut i = 0usize;
while i + 3 < payload.len() {
let sc = if payload[i..].starts_with(&[0, 0, 0, 1]) {
4
} else if payload[i..].starts_with(&[0, 0, 1]) {
3
} else {
i += 1;
continue;
};
match payload.get(i + sc) {
// nal_unit_type 占首字节低 5 位
Some(&b) if b & 0x1F == 5 => return true,
Some(_) => i += sc + 1,
None => break,
}
}
false
}
/// 读一个视频帧。
///
/// 对端数据一律视为不可信输入：`total` 与 `hlen` 都必须做**上下界**校验，
/// 否则构造一个 `total < 4` 的帧就能让接收端在切片处 panic（进程级拒绝服务）。
pub async fn read_video_frame<R: AsyncReadExt + Unpin>(
r: &mut R,
) -> Result<(VideoFrameHeader, Vec<u8>)> {
let total = read_u32le(r).await? as usize;
anyhow::ensure!(total < 256 << 20, "video frame too large: {total}");
// 下界：至少要容得下「头长字段(4) + 一个非空的 bincode 头」。
// 上界：防止对端声明超大长度导致巨量分配。
anyhow::ensure!(
(4..MAX_VIDEO_FRAME).contains(&total),
"video frame length out of range: {total}"
);
let mut buf = vec![0u8; total];
r.read_exact(&mut buf).await?;
let hlen = u32::from_le_bytes(buf[0..4].try_into().unwrap()) as usize;
let hlen = u32::from_le_bytes(buf[0..4].try_into().expect("已校验 total >= 4")) as usize;
// hlen 必须落在 [1, total-4] 内：0 无法反序列化，超出则切片越界
anyhow::ensure!(
hlen > 0 && hlen <= total - 4,
"video header length invalid: hlen={hlen}, total={total}"
);
let header: VideoFrameHeader = bincode::deserialize(&buf[4..4 + hlen])?;
Ok((header, buf[4 + hlen..].to_vec()))
}
/// 把一个已编码的完整帧拆回 `(帧头, 载荷)`。
///
/// 用途：静止桌面重发时，被控端需要复用上一帧的**载荷**但重建**帧头**
/// （`seq` 必须递增、`ts_ms` 必须是本次发送时刻）。直接 `clone` 整个帧会让
/// 这两个字段停在上一次发送的时刻，破坏 `seq` 单调递增契约。
///
/// 只接受本模块 [`encode_video_frame`] 产出的字节；格式不符返回 `Err`。
pub fn split_video_frame(frame: &[u8]) -> Result<(VideoFrameHeader, &[u8])> {
anyhow::ensure!(frame.len() >= 8, "frame too short: {}", frame.len());
let hlen = u32::from_le_bytes(frame[4..8].try_into().expect("已校验 len >= 8")) as usize;
anyhow::ensure!(
hlen > 0 && hlen <= frame.len() - 8,
"header length invalid: hlen={hlen}, len={}",
frame.len()
);
let header: VideoFrameHeader = bincode::deserialize(&frame[8..8 + hlen])?;
Ok((header, &frame[8 + hlen..]))
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::io::duplex;
fn hdr(seq: u64, keyframe: bool) -> VideoFrameHeader {
VideoFrameHeader {
width: 1920,
height: 1080,
ts_ms: 12345,
seq,
keyframe,
}
}
// ---- 控制消息 ----
#[test]
fn control_roundtrip_preserves_version() {
let m = ControlMsg::Auth {
code: "123456".into(),
proto_version: PROTO_VERSION,
};
let bytes = encode_control(&m);
// 长度前缀 + 载荷
let len = u32::from_le_bytes(bytes[0..4].try_into().unwrap()) as usize;
assert_eq!(len, bytes.len() - 4);
let back: ControlMsg = bincode::deserialize(&bytes[4..]).unwrap();
match back {
ControlMsg::Auth { code, proto_version } => {
assert_eq!(code, "123456");
assert_eq!(proto_version, PROTO_VERSION);
}
other => panic!("期望 Auth，实际 {other:?}"),
}
}
#[tokio::test]
async fn control_write_read_roundtrip() {
let (mut a, mut b) = duplex(64 * 1024);
let msgs = [
ControlMsg::Auth {
code: "x".into(),
proto_version: PROTO_VERSION,
},
ControlMsg::AuthOk {
width: 800,
height: 600,
proto_version: PROTO_VERSION,
},
ControlMsg::KeyFrameReq,
ControlMsg::Ping(42),
ControlMsg::Pong {
t: 1,
agent_ts: 2,
lost: 3,
sent: 4,
},
];
for m in &msgs {
write_control(&mut a, m).await.unwrap();
let got = read_control(&mut b).await.unwrap();
assert_eq!(format!("{got:?}"), format!("{m:?}"));
}
}
// ---- 视频帧头 ----
#[test]
fn video_header_roundtrip_keeps_seq_and_keyframe() {
for (seq, kf) in [(0u64, true), (1, false), (u64::MAX, false)] {
let h = hdr(seq, kf);
let payload = b"\x00\x00\x00\x01\x65fake-nal";
let bytes = encode_video_frame(&h, payload);
let total = u32::from_le_bytes(bytes[0..4].try_into().unwrap()) as usize;
assert_eq!(total, bytes.len() - 4, "总长字段应不含自身");
let hlen = u32::from_le_bytes(bytes[4..8].try_into().unwrap()) as usize;
let back: VideoFrameHeader = bincode::deserialize(&bytes[8..8 + hlen]).unwrap();
assert_eq!(back.seq, seq);
assert_eq!(back.keyframe, kf);
assert_eq!(&bytes[8 + hlen..], payload);
}
}
#[tokio::test]
async fn video_frame_write_read_roundtrip() {
let (mut a, mut b) = duplex(256 * 1024);
let h = hdr(7, true);
let payload = vec![0xABu8; 4096];
let bytes = encode_video_frame(&h, &payload);
tokio::io::AsyncWriteExt::write_all(&mut a, &bytes).await.unwrap();
let (h2, p2) = read_video_frame(&mut b).await.unwrap();
assert_eq!(h2.seq, 7);
assert!(h2.keyframe);
assert_eq!((h2.width, h2.height), (1920, 1080));
assert_eq!(p2, payload);
}
// ---- 不可信输入（P3 的核心回归） ----
/// `total < 4` 必须报错而不是 panic：旧实现在这里 `buf[0..4]` 会越界 panic。
#[tokio::test]
async fn rejects_total_below_minimum() {
for total in [0u32, 1, 3] {
let (mut a, mut b) = duplex(64);
tokio::io::AsyncWriteExt::write_all(&mut a, &total.to_le_bytes())
.await
.unwrap();
let err = read_video_frame(&mut b).await.unwrap_err().to_string();
assert!(
err.contains("out of range"),
"total={total} 应被拒绝，实际错误: {err}"
);
}
}
/// `total == 4` 但没有头数据：hlen 读出来是 0，必须拒绝。
#[tokio::test]
async fn rejects_zero_header_length() {
let (mut a, mut b) = duplex(64);
let mut bytes = 4u32.to_le_bytes().to_vec();
bytes.extend_from_slice(&0u32.to_le_bytes());
tokio::io::AsyncWriteExt::write_all(&mut a, &bytes)
.await
.unwrap();
let err = read_video_frame(&mut b).await.unwrap_err().to_string();
assert!(err.contains("header length invalid"), "实际错误: {err}");
}
/// hlen 声明超过 total-4：旧实现会切片越界 panic。
#[tokio::test]
async fn rejects_header_length_beyond_frame() {
let (mut a, mut b) = duplex(64);
// total = 8 → body 必须正好 8 字节（hlen 字段 4 + 4 字节填充）
let mut bytes = 8u32.to_le_bytes().to_vec();
bytes.extend_from_slice(&999u32.to_le_bytes()); // hlen = 999 > total-4 = 4
bytes.extend_from_slice(&[0u8; 4]);
tokio::io::AsyncWriteExt::write_all(&mut a, &bytes)
.await
.unwrap();
let err = read_video_frame(&mut b).await.unwrap_err().to_string();
assert!(err.contains("header length invalid"), "实际错误: {err}");
}
/// 超过 256 MiB 上限必须在**分配前**拒绝，不能先 vec! 再检查。
#[tokio::test]
async fn rejects_oversized_frame_without_allocating() {
let (mut a, mut b) = duplex(64);
let total = (MAX_VIDEO_FRAME as u32) + 1;
tokio::io::AsyncWriteExt::write_all(&mut a, &total.to_le_bytes())
.await
.unwrap();
let err = read_video_frame(&mut b).await.unwrap_err().to_string();
assert!(err.contains("out of range"), "实际错误: {err}");
}
/// 头部字节损坏（非合法 bincode）应返回 Err 而非 panic。
#[tokio::test]
async fn rejects_corrupt_header_bytes() {
let (mut a, mut b) = duplex(64);
let garbage = [0xFFu8; 8];
let total = (4 + garbage.len()) as u32;
let mut bytes = total.to_le_bytes().to_vec();
bytes.extend_from_slice(&(garbage.len() as u32).to_le_bytes());
bytes.extend_from_slice(&garbage);
tokio::io::AsyncWriteExt::write_all(&mut a, &bytes)
.await
.unwrap();
assert!(read_video_frame(&mut b).await.is_err());
}
// ---- IDR 识别（P2 的 keyframe 标志来源） ----
#[test]
fn detects_idr_in_annex_b() {
// SPS(7) + PPS(8) + IDR 片(5)，4 字节起始码
let idr = [
0, 0, 0, 1, 0x67, 0xAA, // SPS
0, 0, 0, 1, 0x68, 0xBB, // PPS
0, 0, 0, 1, 0x65, 0xCC, // IDR
];
assert!(h264_contains_idr(&idr));
// 3 字节起始码也应识别
let idr3 = [0, 0, 1, 0x65, 0xCC];
assert!(h264_contains_idr(&idr3));
}
#[test]
fn rejects_non_idr_payloads() {
// 纯 P 片（NAL 类型 1）
let p_frame = [0, 0, 0, 1, 0x41, 0xDD];
assert!(!h264_contains_idr(&p_frame));
// 只有 SPS/PPS，没有片（关键帧边界可能落在下一个包，此时不应谎报 keyframe）
let params_only = [
0, 0, 0, 1, 0x67, 0xAA,
0, 0, 0, 1, 0x68, 0xBB,
];
assert!(!h264_contains_idr(&params_only));
assert!(!h264_contains_idr(&[]));
assert!(!h264_contains_idr(&[0, 0, 1])); // 起始码后无 NAL 字节
}
}
linux-agent/src/input.rs
+3 -1
View File
@@ -242,8 +242,10 @@ mod tests {
#[test]
fn coordinate_scaling_covers_full_range() {
assert_eq!(scale(0, 1920), 0);
// 归一化上界必须映射到 extent-1，否则指针永远到不了屏幕最右/最下
assert_eq!(scale(65535, 1920), 1919);
assert_eq!(scale(32768, 1920), 960);
// 中点：32768 × 1919 ÷ 65535 = 959.49…，整数除法向下取整 → 959
assert_eq!(scale(32768, 1920), 959);
assert_eq!(scale(12345, 0), 0); // 退化尺寸不 panic
}
}
linux-agent/tests/smoke.rs
+8 -1
View File
@@ -289,11 +289,17 @@ async fn end_to_end_capture_encode_transport_input() -> Result<()> {
&mut send,
&ControlMsg::Auth {
code: "smoke-code".into(),
proto_version: common::PROTO_VERSION,
},
)
.await?;
match read_control(&mut recv).await? {
ControlMsg::AuthOk { width, height } => {
ControlMsg::AuthOk {
width,
height,
proto_version,
} => {
assert_eq!(proto_version, common::PROTO_VERSION, "被控端协议版本不符");
assert_eq!(
(width, height),
(sw, sh),
@@ -429,6 +435,7 @@ async fn rejects_wrong_verification_code() -> Result<()> {
&mut send,
&ControlMsg::Auth {
code: "wrong-code".into(),
proto_version: common::PROTO_VERSION,
},
)
.await?;
scripts/linux-smoke.sh
+4 -1
View File
@@ -43,6 +43,9 @@ if [ "$ready" -ne 1 ]; then
fi
echo "[smoke] X server 就绪：$(xdpyinfo -display "$DISPLAY_NUM" | grep -E 'dimensions' | head -1)"
# 两个测试共用同一块虚拟屏与指针，必须串行
# 先跑纯逻辑单测（键码映射 / 坐标换算），不需要 X server
cargo test -p linux-agent --bins
# 再跑端到端用例：两个用例共用同一块虚拟屏与指针，必须串行
cargo test -p linux-agent --test smoke -- --nocapture --test-threads=1
echo "[smoke] 通过"
vendor/nvenc/.gitignore
+2
View File
@@ -0,0 +1,2 @@
/target
/.vscode
vendor/nvenc/Cargo.toml
Vendored
+63
View File
@@ -0,0 +1,63 @@
# THIS FILE IS AUTOMATICALLY GENERATED BY CARGO
#
# When uploading crates to the registry Cargo will automatically
# "normalize" Cargo.toml files for maximal compatibility
# with all versions of Cargo and also rewrite `path` dependencies
# to registry (e.g., crates.io) dependencies.
#
# If you are reading this file be aware that the original Cargo.toml
# will likely look very different (and much more reasonable).
# See Cargo.toml.orig for the original contents.
[package]
edition = "2024"
rust-version = "1.85.1"
name = "nvenc"
version = "0.1.0"
authors = ["AlsoSylv"]
build = false
autolib = false
autobins = false
autoexamples = false
autotests = false
autobenches = false
description = "Wrapper for the nvidia NVENC API"
readme = "README.md"
keywords = [
"nvidia",
"graphics",
"media",
"video",
"ffi",
]
categories = [
"multimedia",
"external-ffi-bindings",
]
license = "MIT"
repository = "https://github.com/AlsoSylv/nvenc"
[lib]
name = "nvenc"
path = "src/lib.rs"
# 上游 README 的示例是已过时的 Linux GLX 用法（Session<NeedsConfig> 等类型已不存在），
# 直接跑 doctest 会失败。本项目只做 vendor 补丁、不维护文档示例，故关闭。
doctest = false
[[example]]
name = "simple_encode"
path = "examples/simple_encode.rs"
[dependencies.bitfields]
version = "1.0.2"
[dependencies.libloading]
version = "0.8.9"
[target."cfg(windows)".dependencies.windows]
version = "0.62"
features = [
"Win32_Graphics_Dxgi_Common",
"Win32_Graphics_Direct3D",
"Win32_Graphics_Direct3D11",
]
vendor/nvenc/Cargo.toml.orig
Generated Vendored
+24
View File
@@ -0,0 +1,24 @@
[package]
name = "nvenc"
version = "0.1.0"
edition = "2024"
authors = ["AlsoSylv"]
description = "Wrapper for the nvidia NVENC API"
rust-version = "1.85.1"
readme = "README.md"
repository = "https://github.com/AlsoSylv/nvenc"
license = "MIT"
keywords = ["nvidia", "graphics", "media", "video", "ffi"]
categories = ["multimedia", "external-ffi-bindings"]
[dependencies]
libloading = "0.8.9"
bitfields = "1.0.2"
[target.'cfg(windows)'.dependencies.windows]
version = "0.62"
features = [
"Win32_Graphics_Dxgi_Common",
"Win32_Graphics_Direct3D",
"Win32_Graphics_Direct3D11",
]
vendor/nvenc/LICENSE.txt
Vendored
+27
View File
@@ -0,0 +1,27 @@
Notice Regarding Standards. I do not provide a license or sublicense to
any Intellectual Property Rights relating to any standards, including but not
limited to any audio and/or video codec technologies such as MPEG-2, MPEG-4;
AVC/H.264; HEVC/H.265; AAC decode/FFMPEG; AAC encode/FFMPEG; VC-1; and MP3
(collectively, the "Media Technologies"). For clarity, you will pay any
royalties due for such third party technologies, which may include the Media
Technologies that are owed as a result of me providing the Software to you.
Copyright 2025 AlsoSylv
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
vendor/nvenc/README.md
Vendored
+34
View File
@@ -0,0 +1,34 @@
### If you like my work and want to support what I do, support me on Ko-Fi 💜!
[![Ko-Fi](https://img.shields.io/badge/Ko--fi-F16061?style=for-the-badge&logo=ko-fi&logoColor=white)](https://ko-fi.com/cyberite)
---
![Crates.io MSRV](https://img.shields.io/crates/msrv/nvenc?style=for-the-badge) ![Crates.io License](https://img.shields.io/crates/l/nvenc?style=for-the-badge) ![GitHub Repo stars](https://img.shields.io/github/stars/AlsoSylv/nvenc?style=for-the-badge)
The following example is for Linux GLX
```rust
use nvenc::{session::Session, encoder::Encoder};
fn main() {
// Setup GLX context
let session: Session<NeedsConfig> = Session::open_gl();
let (session, config): (Session<NeedsInit>, NVencPresetConfig)
= session.get_preset_config(
NV_ENC_CODEC_H264_GUID,
NV_ENC_PRESET_P3_GUID,
NVencTuningInfo::LowLatency
);
let init_params = nvenc::session::InitParams {
encode_guid: NV_ENC_H264_GUID,
preset_guid: NV_ENC_PRESET_P3_GUID,
resolution: [1920, 1080],
aspect_ratio: [16, 9],
frame_rate: [30, 1],
tuning_info: NVencTuningInfo::LowLatency,
buffer_format: NVencBufferFormat::ARGB,
encode_config: &mut config.preset_cfg,
enable_ptd: true,
};
let encoder: Encoder = session.init(init_params);
}
```
vendor/nvenc/examples/simple_encode.rs
Vendored
+203
View File
@@ -0,0 +1,203 @@
use std::fs::File;
use std::io::Write;
use std::time::Instant;
use nvenc::bitstream::BitStream;
use nvenc::session::InitParams;
use nvenc::{
session::Session,
sys::guids::{NV_ENC_CODEC_H264_GUID, NV_ENC_PRESET_P3_GUID},
};
#[cfg(windows)]
use windows::Win32::Graphics::Direct3D11::{ID3D11Device, ID3D11Texture2D};
#[cfg(windows)]
pub fn setup() -> ID3D11Device {
let dxgi_factory: windows::Win32::Graphics::Dxgi::IDXGIFactory =
unsafe { windows::Win32::Graphics::Dxgi::CreateDXGIFactory() }.unwrap();
let dxgi_adapter = unsafe { dxgi_factory.EnumAdapters(0) }.unwrap();
let desc = unsafe { dxgi_adapter.GetDesc() }.unwrap();
println!("{:?}", windows::core::HSTRING::from_wide(&desc.Description));
let mut device = None;
let mut device_context = None;
unsafe {
use windows::Win32::{
Foundation::HMODULE,
Graphics::{
Direct3D::{D3D_DRIVER_TYPE_UNKNOWN, D3D_FEATURE_LEVEL_11_0},
Direct3D11::{D3D11_CREATE_DEVICE_FLAG, D3D11_SDK_VERSION, D3D11CreateDevice},
},
};
D3D11CreateDevice(
&dxgi_adapter,
D3D_DRIVER_TYPE_UNKNOWN,
HMODULE(std::ptr::null_mut()),
D3D11_CREATE_DEVICE_FLAG(0),
Some(&[D3D_FEATURE_LEVEL_11_0]),
D3D11_SDK_VERSION,
Some(&raw mut device),
None,
Some(&raw mut device_context),
)
}
.unwrap();
let device = device.unwrap();
let _device_context = device_context.unwrap();
device
}
#[cfg(windows)]
fn image(device: &ID3D11Device) -> ID3D11Texture2D {
let data: [u8; 1920 * 4] = std::array::from_fn(|idx| match idx % 4 {
0 => 255,
1 => 0,
2 => 0,
3 => 255,
_ => unreachable!(),
});
let mut texture = None;
let desc = windows::Win32::Graphics::Direct3D11::D3D11_TEXTURE2D_DESC {
Width: 1920,
Height: 1080,
MipLevels: 1,
ArraySize: 1,
Format: windows::Win32::Graphics::Dxgi::Common::DXGI_FORMAT_R8G8B8A8_UNORM_SRGB,
SampleDesc: windows::Win32::Graphics::Dxgi::Common::DXGI_SAMPLE_DESC {
Count: 1,
Quality: 0,
},
Usage: windows::Win32::Graphics::Direct3D11::D3D11_USAGE_DEFAULT,
BindFlags: windows::Win32::Graphics::Direct3D11::D3D11_BIND_SHADER_RESOURCE.0 as u32,
CPUAccessFlags: windows::Win32::Graphics::Dxgi::Common::DXGI_CPU_ACCESS_NONE,
MiscFlags: 0,
};
let data = windows::Win32::Graphics::Direct3D11::D3D11_SUBRESOURCE_DATA {
pSysMem: data.as_ptr() as _,
SysMemPitch: 1920 * 4,
SysMemSlicePitch: 0,
};
unsafe {
device.CreateTexture2D(
&raw const desc,
Some(&raw const data),
Some(&raw mut texture),
)
}
.unwrap();
texture.unwrap()
}
fn main() {
let device = setup();
#[cfg(target_os = "linux")]
compile_error!("Linux is currently unsupported in this example");
#[cfg(windows)]
let session: Session<nvenc::session::NeedsConfig> = Session::open_dx(&device).unwrap();
assert!(
session
.get_encode_codecs()
.unwrap()
.contains(&NV_ENC_CODEC_H264_GUID)
);
assert!(
session
.get_encode_presets(NV_ENC_CODEC_H264_GUID)
.unwrap()
.contains(&NV_ENC_PRESET_P3_GUID)
);
let (session, mut config) = session
.get_encode_preset_config_ex(
NV_ENC_CODEC_H264_GUID,
NV_ENC_PRESET_P3_GUID,
nvenc::sys::enums::NVencTuningInfo::LowLatency,
)
.unwrap();
let texture = image(&device);
config.preset_cfg.rc_params.rate_control_mode = nvenc::sys::enums::NVencParamsRcMode::VBR;
config.preset_cfg.rc_params.average_bit_rate = 10_000_000;
config.preset_cfg.gop_len = 0xffffffff;
config.preset_cfg.frame_interval_p = 1;
println!("P-Frames: {}", config.preset_cfg.frame_interval_p);
println!("Bitrate: {}", config.preset_cfg.rc_params.average_bit_rate);
let init_params = InitParams {
encode_guid: NV_ENC_CODEC_H264_GUID,
preset_guid: NV_ENC_PRESET_P3_GUID,
aspect_ratio: [16, 9],
encode_config: &mut config.preset_cfg,
tuning_info: nvenc::sys::enums::NVencTuningInfo::LowLatency,
buffer_format: nvenc::sys::enums::NVencBufferFormat::ARGB,
frame_rate: [30, 1],
resolution: [1920, 1080],
enable_ptd: true,
max_encoder_resolution: [0, 0],
};
println!("FPS: 30");
println!("Len: 600 frames");
println!("Expected time: {}", 600 / 30);
println!("Codec: H264");
println!("Tuning Info {:?}:", init_params.tuning_info);
let encoder = session.init_encoder(init_params).unwrap();
let registered = encoder
.register_resource_dx11(&texture, nvenc::sys::enums::NVencBufferFormat::ARGB, 0)
.unwrap();
let (processed, to_use) = std::sync::mpsc::sync_channel::<BitStream>(2);
let (re_use, to_process) = std::sync::mpsc::sync_channel::<BitStream>(2);
processed
.send(encoder.create_bitstream_buffer().unwrap())
.unwrap();
processed
.send(encoder.create_bitstream_buffer().unwrap())
.unwrap();
let handle = std::thread::spawn({
move || {
let mut out = File::create("output.h264").unwrap();
while let Ok(output) = to_process.recv() {
let lock = output.try_lock(true).unwrap();
out.write_all(lock.as_slice()).unwrap();
drop(lock);
if let Err(_) = processed.send(output) {
break;
}
}
}
});
let instant = Instant::now();
let mut i = 0;
while i < 600 {
if let Ok(output) = to_use.recv() {
encoder
.encode_picture(
&registered,
&output,
i,
instant.duration_since(Instant::now()).as_millis() as u64,
nvenc::sys::enums::NVencBufferFormat::ARGB,
nvenc::sys::enums::NVencPicStruct::Frame,
nvenc::sys::enums::NVencPicType::P,
None,
)
.unwrap();
re_use.send(output).unwrap();
i += 1;
}
}
drop(to_use);
drop(re_use);
handle.join().unwrap();
}
vendor/nvenc/src/lib.rs
Vendored
+86
View File
@@ -0,0 +1,86 @@
#![doc = include_str!("../README.md")]
use std::{mem::MaybeUninit, sync::OnceLock};
use libloading::Library;
use crate::sys::{
function_table::NVencFunctionList,
result::{NVencError, NVencResult},
version::NVENC_API_LIST_VERSION,
};
mod safe;
pub use safe::*;
/// Contains the original C Types, functions, enums, and constants
pub mod sys;
#[cfg(all(target_arch = "x86_64", target_os = "windows"))]
pub const NVENC_DLL: &str = "nvEncodeAPI64.dll";
#[cfg(all(target_arch = "x86", target_os = "windows"))]
pub const NVENC_DLL: &str = "nvEncodeAPI.dll";
#[cfg(target_os = "linux")]
/// Platform specific DLL name, `libnvidia-encode.so.1` on Linux, `nvEncodeAPI64.dll` on Windows
pub const NVENC_DLL: &str = "libnvidia-encode.so.1";
#[cfg(all(target_arch = "x86", windows))]
#[macro_export]
macro_rules! stdcall {
(fn $args:tt $(-> $ret:tt)?) => { unsafe extern "stdcall" fn $args $(-> $ret)? };
}
#[doc(hidden)]
#[cfg(not(all(target_arch = "x86", windows)))]
#[macro_export]
macro_rules! stdcall {
(fn $args:tt $(-> $ret:ty)?) => { unsafe extern "C" fn $args $(-> $ret)? };
}
pub(crate) static LIBRARY: OnceLock<NVENCLibrary> = OnceLock::new();
/// Struct containing the library handle, init and get_max_version number functions
pub struct NVENCLibrary {
#[allow(unused)]
// Lib must stay alive for nvenc to be used
lib: Library,
init: stdcall!(fn(list: *mut MaybeUninit<NVencFunctionList>) -> NVencResult),
get_max_version: stdcall!(fn(version: *mut u32) -> NVencResult),
}
/// Attempt to init NVENC, returns an error if the library or loading functions cannot be found
pub fn nvenc_init() -> Result<&'static NVENCLibrary, libloading::Error> {
if let Some(lib) = LIBRARY.get() {
Ok(lib)
} else {
let lib = unsafe { Library::new(NVENC_DLL) }?;
let init: stdcall!(fn(function_list: *mut MaybeUninit<NVencFunctionList>) -> NVencResult) =
*unsafe { lib.get(b"NvEncodeAPICreateInstance") }?;
let get_max_version: stdcall!(fn(version: *mut u32) -> NVencResult) =
*unsafe { lib.get(b"NvEncodeAPIGetMaxSupportedVersion") }?;
Ok(LIBRARY.get_or_init(|| NVENCLibrary {
lib,
init,
get_max_version,
}))
}
}
impl NVENCLibrary {
/// Create a new instance of the NVencFunctionList,
pub fn create_instance(&self) -> Result<NVencFunctionList, NVencError> {
let mut list = MaybeUninit::zeroed();
unsafe { *(list.as_mut_ptr() as *mut u32) = NVENC_API_LIST_VERSION };
unsafe { (self.init)(&raw mut list) }.into_error().unwrap();
Ok(unsafe { list.assume_init() })
}
pub fn get_max_version(&self) -> Result<u32, NVencError> {
let mut version = 0;
unsafe { (self.get_max_version)(&raw mut version) }.into_error()?;
let minor = version & 0x0F;
let version = (version >> 4) | minor << 28;
Ok(version)
}
}
vendor/nvenc/src/safe/bitstream.rs
Vendored
+53
View File
@@ -0,0 +1,53 @@
use std::{ffi::c_void, sync::Arc};
use crate::{safe::encoder::EncoderInternal, sys::result::NVencError};
pub struct BitStream {
pub(crate) buffer: *mut c_void,
pub(crate) encoder: Arc<EncoderInternal>,
}
impl Drop for BitStream {
fn drop(&mut self) {
println!("Dropping bitstream buffer");
self.encoder.destroy_bitstream_buffer(self.buffer).unwrap();
}
}
unsafe impl Send for BitStream {}
impl BitStream {
/// Attempts to lock the bit stream, if `wait` is true it will wait
/// otherwise a `LockBusy` Error may be returned, in which case the
/// client should retry in a few milliseconds
pub fn try_lock(&self, wait: bool) -> Result<BitStreamLockGuard<'_>, NVencError> {
let lock = self.encoder.lock_bit_stream_buffer(self.buffer, wait)?;
Ok(BitStreamLockGuard {
buffer: self,
data_ptr: lock.bitstream_buffer,
data_len: lock.bitstream_size_in_bytes,
})
}
}
/// Holds a reference to the `BitStream` and holds the data and associated fields
pub struct BitStreamLockGuard<'a> {
buffer: &'a BitStream,
data_ptr: *mut c_void,
data_len: u32,
}
impl BitStreamLockGuard<'_> {
pub fn as_slice(&self) -> &[u8] {
unsafe { std::slice::from_raw_parts(self.data_ptr as _, self.data_len as _) }
}
}
impl<'a> Drop for BitStreamLockGuard<'a> {
fn drop(&mut self) {
self.buffer
.encoder
.unlock_bit_stream_buffer(self.buffer.buffer)
.unwrap();
}
}
vendor/nvenc/src/safe/encoder.rs
Vendored
+517
View File
@@ -0,0 +1,517 @@
use std::ffi::{CStr, c_void};
use std::marker::PhantomData;
use std::mem::ManuallyDrop;
use std::sync::Arc;
use std::{mem::MaybeUninit, ptr::NonNull};
#[cfg(windows)]
use windows::Win32::Graphics::Direct3D11::ID3D11Texture2D;
use crate::input_buffer::InputBuffer;
use crate::safe::bitstream::BitStream;
use crate::sys::enums::{
NVencBufferFormat, NVencBufferUsage, NVencInputResourceType, NVencMemoryHeap, NVencPicStruct,
NVencPicType,
};
use crate::sys::structs::{
Guid, NV_ENC_CREATE_INPUT_BUFFER_VER, NV_ENC_LOCK_INPUT_BUFFER_VER, NVencCodecPicParams,
NVencCreateInputBuffer, NVencLockInputBuffer, NVencPicParamsH264, NVencRegisterResource,
};
use crate::sys::{
enums::{NVencPicFlags, NVencTuningInfo},
function_table::NVencFunctionList,
result::NVencError,
structs::{
NV_ENC_CONFIG_VER, NV_ENC_CREATE_BITSTREAM_BUFFER_VER, NV_ENC_LOCK_BITSTREAM_VER,
NV_ENC_MAP_INPUT_RESOURCE_VER, NV_ENC_PIC_PARAMS_VER, NV_ENC_PRESET_CONFIG_VER,
NV_ENC_REGISTER_RESOURCE_VER, NVencCreateBitstreamBuffer, NVencInitializeParams,
NVencLockBitStream, NVencPicParams, NVencPresetConfig,
},
};
pub struct Encoder {
pub(crate) encoder: Arc<EncoderInternal>,
pub(crate) marker: PhantomData<*mut ()>,
}
pub trait EncoderInput {
#[doc(hidden)]
fn as_ptr(&self) -> *mut c_void;
fn pitch(&self) -> u32;
fn height(&self) -> u32;
fn width(&self) -> u32;
}
impl EncoderInput for RegisteredResource {
fn as_ptr(&self) -> *mut c_void {
self.mapped
}
fn pitch(&self) -> u32 {
self.pitch
}
fn height(&self) -> u32 {
self.height
}
fn width(&self) -> u32 {
self.width
}
}
impl EncoderInput for InputBuffer {
fn as_ptr(&self) -> *mut c_void {
self.buffer
}
fn pitch(&self) -> u32 {
self.width
}
fn height(&self) -> u32 {
self.height
}
fn width(&self) -> u32 {
self.width
}
}
impl Encoder {
#[cfg(windows)]
/// Registers a DX11 texture as an nvenc input resource
pub fn register_resource_dx11(
&self,
resource: &ID3D11Texture2D,
format: NVencBufferFormat,
pitch: u32,
) -> Result<RegisteredResource, NVencError> {
self.encoder.register_resource_dx(resource, format, pitch)
}
/// # Safety
/// `resource` must not be null
/// `ty` and `format` should match the original buffer
pub unsafe fn register_resource_raw(
&self,
resource: *mut c_void,
format: NVencBufferFormat,
usage: NVencBufferUsage,
ty: NVencInputResourceType,
resolution: [u32; 2],
pitch: u32,
) -> Result<*mut c_void, NVencError> {
unsafe {
self.encoder
.register_resource_raw(resource, format, usage, ty, resolution, pitch)
}
}
/// Takes an encoder inout, and outputs a bit stream that can be read back
pub fn encode_picture<I: EncoderInput>(
&self,
input: &I,
output: &BitStream,
frame_count: usize,
timestamp: u64,
format: NVencBufferFormat,
pic_struct: NVencPicStruct,
ty: NVencPicType,
codec_params: Option<CodecPicParams>,
) -> Result<(), NVencError> {
self.encoder.encode_picture(
input,
output,
frame_count,
timestamp,
format,
pic_struct,
ty,
codec_params,
)
}
/// Ends the encoder session
pub fn end_encode(&self) -> Result<(), NVencError> {
self.encoder.end_encode()
}
/// Creates a bitstream bfufer that can be used for output storage
pub fn create_bitstream_buffer(&self) -> Result<BitStream, NVencError> {
self.encoder.create_bitstream_buffer()
}
pub fn create_input_buffer(
&self,
width: u32,
height: u32,
memory_heap: NVencMemoryHeap,
format: NVencBufferFormat,
) -> Result<InputBuffer, NVencError> {
Ok(InputBuffer {
encoder: self.encoder.clone(),
buffer: self
.encoder
.create_input_buffer(width, height, memory_heap, format)?,
width,
height,
})
}
}
pub(crate) struct EncoderInternal {
pub(super) encoder: NonNull<c_void>,
pub(super) function_list: NVencFunctionList,
}
/// The Nvidia encoder is technically thread-safe
unsafe impl Send for EncoderInternal {}
/// The Nvidia encoder handles syncing via `lock_bit_stream` which is all that is allowed to be Send
unsafe impl Sync for EncoderInternal {}
impl Drop for EncoderInternal {
fn drop(&mut self) {
println!("Dropping encoder, this should happen last");
unsafe { (self.function_list.nvenc_destroy_encoder)(self.encoder.as_ptr()) };
}
}
impl EncoderInternal {
pub(crate) fn get_encode_preset_config_ex(
&self,
codec: Guid,
preset: Guid,
tuning_info: NVencTuningInfo,
) -> Result<NVencPresetConfig, NVencError> {
let mut config: std::mem::MaybeUninit<NVencPresetConfig> = std::mem::MaybeUninit::zeroed();
unsafe { &mut *config.as_mut_ptr() }.version = NV_ENC_PRESET_CONFIG_VER;
unsafe { &mut *config.as_mut_ptr() }.preset_cfg.version = NV_ENC_CONFIG_VER;
unsafe {
(self.function_list.nvenc_get_encode_preset_config_ex)(
self.encoder.as_ptr(),
codec,
preset,
tuning_info,
&raw mut config,
)
.into_error()?
};
Ok(unsafe { config.assume_init() })
}
pub(crate) fn init_encoder(
&self,
mut init_params: NVencInitializeParams,
) -> Result<(), NVencError> {
unsafe {
(self.function_list.nvenc_initialize_encoder)(
self.encoder.as_ptr(),
&raw mut init_params,
)
}
.into_error()?;
Ok(())
}
#[cfg(windows)]
pub fn register_resource_dx(
self: &Arc<EncoderInternal>,
resource: &ID3D11Texture2D,
format: NVencBufferFormat,
pitch: u32,
) -> Result<RegisteredResource, NVencError> {
use windows::Win32::Graphics::Direct3D11::D3D11_TEXTURE2D_DESC;
use windows::core::Interface;
let mut desc = D3D11_TEXTURE2D_DESC::default();
unsafe { resource.GetDesc(&raw mut desc) };
let registered = unsafe {
self.register_resource_raw(
resource.as_raw(),
NVencBufferFormat::Undefined,
NVencBufferUsage::Image,
NVencInputResourceType::DirectX,
[desc.Width, desc.Height],
pitch,
)?
};
let mapped = unsafe { self.map_input_resource_raw(registered, format)? };
Ok(RegisteredResource {
encoder: self.clone(),
pitch,
width: desc.Width,
height: desc.Height,
registered,
mapped,
})
}
pub unsafe fn map_input_resource_raw(
&self,
resource: *mut c_void,
format: NVencBufferFormat,
) -> Result<*mut c_void, NVencError> {
let mut map: crate::sys::structs::NVencMapInputResource = unsafe { std::mem::zeroed() };
map.version = NV_ENC_MAP_INPUT_RESOURCE_VER;
map.registered_resource = resource;
map.mapped_buffer_format = format;
unsafe {
(self.function_list.nvenc_map_input_resource)(self.encoder.as_ptr(), &raw mut map)
}
.into_error()?;
Ok(map.mapped_resource)
}
pub unsafe fn register_resource_raw(
&self,
resource: *mut c_void,
format: NVencBufferFormat,
usage: NVencBufferUsage,
ty: NVencInputResourceType,
resolution: [u32; 2],
pitch: u32,
) -> Result<*mut c_void, NVencError> {
let mut register_resource: NVencRegisterResource = unsafe { std::mem::zeroed() };
register_resource.buffer_format = format;
register_resource.buffer_usage = usage;
register_resource.resource_type = ty;
register_resource.version = NV_ENC_REGISTER_RESOURCE_VER;
register_resource.width = resolution[0];
register_resource.height = resolution[1];
register_resource.pitch = pitch;
register_resource.sub_resource_index = 0;
register_resource.resource_to_register = resource;
unsafe {
(self.function_list.nvenc_register_resource)(
self.encoder.as_ptr(),
&raw mut register_resource,
)
.into_error()
}?;
Ok(register_resource.registered_resource)
}
pub unsafe fn unmap_input_resource(
&self,
mapped_resource: *mut c_void,
) -> Result<(), NVencError> {
unsafe {
(self.function_list.nvenc_unmap_input_resource)(self.encoder.as_ptr(), mapped_resource)
}
.into_error()
}
pub unsafe fn unregister_resource(
&self,
registered_resource: *mut c_void,
) -> Result<(), NVencError> {
unsafe {
(self.function_list.nvenc_unregister_resource)(
self.encoder.as_ptr(),
registered_resource,
)
}
.into_error()
}
pub fn create_input_buffer(
&self,
width: u32,
height: u32,
memory_heap: NVencMemoryHeap,
format: NVencBufferFormat,
) -> Result<*mut c_void, NVencError> {
let mut input = NVencCreateInputBuffer {
version: NV_ENC_CREATE_INPUT_BUFFER_VER,
width: width,
height: height,
memory_heap,
buffer_fmt: format,
rsvd: 0,
input_buffer: std::ptr::null_mut(),
p_sys_mem_buffer: std::ptr::null_mut(),
rsvd1: unsafe { std::mem::zeroed() },
rsvd2: unsafe { std::mem::zeroed() },
};
unsafe {
(self.function_list.nvenc_create_input_buffer)(self.encoder.as_ptr(), &raw mut input)
.into_error()?
};
Ok(input.input_buffer)
}
pub fn destroy_input_buffer(&self, buffer: *mut c_void) -> Result<(), NVencError> {
unsafe { (self.function_list.nvenc_destroy_input_buffer)(self.encoder.as_ptr(), buffer) }
.into_error()
}
pub fn lock_input_buffer(&self, buffer: *mut c_void) -> Result<(*mut c_void, u32), NVencError> {
let mut lock = NVencLockInputBuffer {
version: NV_ENC_LOCK_INPUT_BUFFER_VER,
bitflags: 0,
input_buffer: buffer,
buffer_data_ptr: std::ptr::null_mut(),
pitch: 0,
rsvd1: unsafe { std::mem::zeroed() },
rsvd2: unsafe { std::mem::zeroed() },
};
unsafe {
(self.function_list.nvenc_lock_input_buffer)(self.encoder.as_ptr(), &raw mut lock)
.into_error()?;
}
Ok((lock.buffer_data_ptr, lock.pitch))
}
pub fn unlock_input_buffer(&self, locked_buffer: *mut c_void) -> Result<(), NVencError> {
unsafe {
(self.function_list.nvenc_unlock_input_buffer)(self.encoder.as_ptr(), locked_buffer)
.into_error()
}
}
pub fn encode_picture<I: EncoderInput>(
&self,
input: &I,
output: &BitStream,
frame_count: usize,
timestamp: u64,
format: NVencBufferFormat,
pic_struct: NVencPicStruct,
ty: NVencPicType,
codec_params: Option<CodecPicParams>,
) -> Result<(), NVencError> {
let mut params: MaybeUninit<NVencPicParams> = MaybeUninit::zeroed();
let sys_codec = match codec_params {
None => NVencCodecPicParams { rsvd: [0; 256] },
Some(CodecPicParams::H264(codec)) => NVencCodecPicParams {
h264_pic_params: ManuallyDrop::new(codec),
},
};
unsafe { &mut *params.as_mut_ptr() }.version = NV_ENC_PIC_PARAMS_VER;
unsafe { &mut *params.as_mut_ptr() }.input_width = input.width();
unsafe { &mut *params.as_mut_ptr() }.input_height = input.height();
unsafe { &mut *params.as_mut_ptr() }.input_pitch = input.pitch();
unsafe { &mut *params.as_mut_ptr() }.output_bitstream = output.buffer;
unsafe { &mut *params.as_mut_ptr() }.input_buffer = input.as_ptr();
unsafe { &mut *params.as_mut_ptr() }.buffer_format = format;
unsafe { &mut *params.as_mut_ptr() }.picture_struct = pic_struct;
// [本项目补丁] 只设 picture_type=IDR 不足以让 NVENC 真正输出 IDR 切片，
// 必须同时置 encode_pic_flags |= NV_ENC_PIC_FLAG_FORCEIDR。
//
// 上游 0.1.0 未设置该标志，导致运行期请求关键帧时编码器静默忽略、
// 继续输出 P 帧（实测：picture_type=IDR 但输出 NAL type=1）。
// 首帧之所以"看起来正常"，是因为 frame_idx==0 本来就强制 IDR，与本字段无关。
//
// 语义区分：IntraRefresh 走 ForceIntra，其余 IDR 类走 ForceIDR。
// 必须在 `picture_type = ty` 之前计算，否则 ty 已被移动。
let pic_flags = match &ty {
NVencPicType::IDR => NVencPicFlags::ForceIDR as u32,
NVencPicType::IntraRefresh => NVencPicFlags::ForceIntra as u32,
_ => 0,
};
unsafe { &mut *params.as_mut_ptr() }.picture_type = ty;
unsafe { &mut *params.as_mut_ptr() }.encode_pic_flags = pic_flags;
unsafe { &mut *params.as_mut_ptr() }.frame_idx = frame_count as u32;
unsafe { &mut *params.as_mut_ptr() }.input_time_stamp = timestamp;
unsafe { &mut *params.as_mut_ptr() }.codec_pic_params = sys_codec;
match unsafe {
(self.function_list.nvenc_encode_picture)(self.encoder.as_ptr(), &raw mut params)
.into_error()
} {
Err(err) => {
let ptr =
unsafe { (self.function_list.nvenc_get_last_error)(self.encoder.as_ptr()) };
println!("{:?}", unsafe { CStr::from_ptr(ptr) });
Err(err)
}
Ok(()) => Ok(()),
}
}
pub fn end_encode(&self) -> Result<(), NVencError> {
let mut params: MaybeUninit<NVencPicParams> = MaybeUninit::zeroed();
unsafe { &mut *params.as_mut_ptr() }.version = NV_ENC_PIC_PARAMS_VER;
unsafe { &mut *params.as_mut_ptr() }.encode_pic_flags = NVencPicFlags::Eos as u32;
unsafe {
(self.function_list.nvenc_encode_picture)(self.encoder.as_ptr(), &raw mut params)
.into_error()
}?;
Ok(())
}
pub(crate) fn lock_bit_stream_buffer(
&self,
buffer: *mut c_void,
wait: bool,
) -> Result<NVencLockBitStream, NVencError> {
let mut lock: MaybeUninit<NVencLockBitStream> = MaybeUninit::zeroed();
unsafe { &mut *lock.as_mut_ptr() }.version = NV_ENC_LOCK_BITSTREAM_VER;
unsafe { &mut *lock.as_mut_ptr() }.output_bit_stream = buffer;
unsafe { &mut *lock.as_mut_ptr() }
.bit_fields
.set_do_not_wait(!wait);
unsafe { (self.function_list.nvenc_lock_bit_stream)(self.encoder.as_ptr(), &raw mut lock) }
.into_error()?;
Ok(unsafe { lock.assume_init() })
}
pub(crate) fn unlock_bit_stream_buffer(&self, buffer: *mut c_void) -> Result<(), NVencError> {
unsafe { (self.function_list.nvenc_unlock_bit_stream)(self.encoder.as_ptr(), buffer) }
.into_error()
}
pub fn create_bitstream_buffer(self: &Arc<Self>) -> Result<BitStream, NVencError> {
let mut params: NVencCreateBitstreamBuffer = unsafe { std::mem::zeroed() };
params.version = NV_ENC_CREATE_BITSTREAM_BUFFER_VER;
unsafe {
(self.function_list.nvenc_create_bit_stream_buffer)(
self.encoder.as_ptr(),
&raw mut params,
)
}
.into_error()?;
Ok(BitStream {
buffer: params.bitstream_buffer,
encoder: self.clone(),
})
}
pub(crate) fn destroy_bitstream_buffer(&self, buffer: *mut c_void) -> Result<(), NVencError> {
unsafe {
(self.function_list.nvenc_destory_bit_stream_buffer)(self.encoder.as_ptr(), buffer)
}
.into_error()
}
}
pub enum CodecPicParams {
H264(NVencPicParamsH264),
}
pub struct RegisteredResource {
registered: *mut c_void,
mapped: *mut c_void,
pitch: u32,
width: u32,
height: u32,
encoder: Arc<EncoderInternal>,
}
impl Drop for RegisteredResource {
fn drop(&mut self) {
unsafe { self.encoder.unmap_input_resource(self.mapped).unwrap() };
unsafe { self.encoder.unregister_resource(self.registered).unwrap() };
}
}
vendor/nvenc/src/safe/input_buffer.rs
Vendored
+60
View File
@@ -0,0 +1,60 @@
use std::{ffi::c_void, sync::Arc};
use crate::{encoder::EncoderInternal, sys::result::NVencError};
pub struct InputBuffer {
pub(crate) encoder: Arc<EncoderInternal>,
pub(crate) buffer: *mut c_void,
pub(crate) width: u32,
pub(crate) height: u32,
}
impl InputBuffer {
pub fn lock(&self) -> Result<InputBufferLock<'_>, NVencError> {
let (lock_ptr, pitch) = self.encoder.lock_input_buffer(self.buffer)?;
Ok(InputBufferLock {
input_buffer: self,
lock_ptr,
pitch,
})
}
}
impl Drop for InputBuffer {
fn drop(&mut self) {
self.encoder.destroy_input_buffer(self.buffer).unwrap()
}
}
pub struct InputBufferLock<'a> {
input_buffer: &'a InputBuffer,
lock_ptr: *mut c_void,
pitch: u32,
}
impl InputBufferLock<'_> {
pub unsafe fn data_ptr(&self) -> *mut u8 {
self.lock_ptr as _
}
pub fn width(&self) -> u32 {
self.input_buffer.width
}
pub fn height(&self) -> u32 {
self.input_buffer.height
}
pub fn pitch(&self) -> u32 {
self.pitch
}
}
impl Drop for InputBufferLock<'_> {
fn drop(&mut self) {
self.input_buffer
.encoder
.unlock_input_buffer(self.lock_ptr)
.unwrap();
}
}
vendor/nvenc/src/safe/mod.rs
Vendored
+8
View File
@@ -0,0 +1,8 @@
/// Contains the type for the safe BitStream buffer wrapper
pub mod bitstream;
/// Wrapper for the NVEncoder after creating an active session
pub mod encoder;
/// Wrapper around an NVenc input buffer
pub mod input_buffer;
/// Type-state wrapper around session initialization
pub mod session;
vendor/nvenc/src/safe/session.rs
Vendored
+198
View File
@@ -0,0 +1,198 @@
use std::{marker::PhantomData, sync::Arc};
use crate::{
encoder::{Encoder, EncoderInternal},
sys::{
enums::{NVencBufferFormat, NVencDeviceType, NVencTuningInfo},
result::NVencError,
structs::{
Guid, NV_ENC_INITIALIZE_PARAMS_VER, NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER,
NVencConfig, NVencInitializeParams, NVencOpenEncodeSessionExParams, NVencPresetConfig,
},
version::NVENC_API_VERSION,
},
};
pub struct Session<T> {
encoder: Arc<EncoderInternal>,
p: PhantomData<T>,
}
pub struct NeedsConfig;
pub struct NeedsInit;
#[cfg(windows)]
use windows::core::Interface;
impl Session<NeedsConfig> {
#[cfg(windows)]
pub fn open_dx(device: &impl Interface) -> Result<Self, NVencError> {
let lib = crate::nvenc_init().unwrap();
let function_list = lib.create_instance()?;
let mut session_params: NVencOpenEncodeSessionExParams = unsafe { std::mem::zeroed() };
session_params.version = NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER;
session_params.device_type = NVencDeviceType::DirectX;
session_params.api_version = NVENC_API_VERSION;
session_params.device = device.as_raw();
let mut encoder = std::ptr::null_mut();
unsafe {
(function_list.nvenc_open_encode_session_ex)(&raw mut session_params, &raw mut encoder)
.into_error()
}?;
assert!(!encoder.is_null());
let encoder = EncoderInternal {
encoder: unsafe { std::ptr::NonNull::new_unchecked(encoder) },
function_list,
};
Ok(Self {
encoder: Arc::new(encoder),
p: PhantomData,
})
}
pub fn open_gl() -> Result<Self, NVencError> {
let lib = crate::nvenc_init().unwrap();
let function_list = lib.create_instance()?;
let mut session_params: NVencOpenEncodeSessionExParams = unsafe { std::mem::zeroed() };
session_params.version = NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER;
session_params.device_type = NVencDeviceType::OpenGL;
session_params.api_version = NVENC_API_VERSION;
session_params.device = std::ptr::null_mut();
let mut encoder = std::ptr::null_mut();
unsafe {
(function_list.nvenc_open_encode_session_ex)(&raw mut session_params, &raw mut encoder)
.into_error()
}?;
assert!(!encoder.is_null());
let encoder = EncoderInternal {
encoder: unsafe { std::ptr::NonNull::new_unchecked(encoder) },
function_list,
};
Ok(Self {
encoder: Arc::new(encoder),
p: PhantomData,
})
}
pub fn get_encode_codecs(&self) -> Result<Box<[Guid]>, NVencError> {
let mut count = 0;
unsafe {
(self.encoder.function_list.nvenc_get_encoder_guid_count)(
self.encoder.encoder.as_ptr(),
&raw mut count,
)
.into_error()
}?;
let mut array = vec![Guid::default(); count as usize];
unsafe {
(self.encoder.function_list.nvenc_get_encoder_guids)(
self.encoder.encoder.as_ptr(),
array.as_mut_ptr(),
count,
&raw mut count,
)
}
.into_error()?;
array.truncate(count as usize);
Ok(array.into())
}
pub fn get_encode_presets(&self, codec: Guid) -> Result<Box<[Guid]>, NVencError> {
let mut count = 0;
unsafe {
(self.encoder.function_list.nvenc_get_encode_preset_count)(
self.encoder.encoder.as_ptr(),
codec.clone(),
&raw mut count,
)
.into_error()
}?;
let mut array = vec![Guid::default(); count as usize];
unsafe {
(self.encoder.function_list.nvenc_get_encode_preset_guids)(
self.encoder.encoder.as_ptr(),
codec,
array.as_mut_ptr(),
count,
&raw mut count,
)
}
.into_error()?;
array.truncate(count as usize);
Ok(array.into())
}
pub fn get_encode_preset_config_ex(
self,
codec: Guid,
preset: Guid,
tuning_info: NVencTuningInfo,
) -> Result<(Session<NeedsInit>, NVencPresetConfig), NVencError> {
self.encoder
.get_encode_preset_config_ex(codec, preset, tuning_info)
.map(|config| {
(
Session {
encoder: self.encoder,
p: PhantomData,
},
config,
)
})
}
}
pub struct InitParams<'a> {
pub encode_guid: Guid,
pub preset_guid: Guid,
pub resolution: [u32; 2],
pub aspect_ratio: [u32; 2],
pub frame_rate: [u32; 2],
pub tuning_info: NVencTuningInfo,
pub buffer_format: NVencBufferFormat,
pub encode_config: &'a mut NVencConfig,
pub enable_ptd: bool,
pub max_encoder_resolution: [u32; 2],
// TODO: Support for async encoding and bit fields
}
impl Session<NeedsInit> {
pub fn init_encoder(self, init_params: InitParams) -> Result<Encoder, NVencError> {
// This sucks, lol
let raw_init_params = NVencInitializeParams {
encode_guid: init_params.encode_guid,
preset_guid: init_params.preset_guid,
encode_width: init_params.resolution[0],
encode_height: init_params.resolution[1],
dar_width: init_params.aspect_ratio[0],
dar_height: init_params.aspect_ratio[1],
frame_rate_num: init_params.frame_rate[0],
frame_rate_den: init_params.frame_rate[1],
tuning_info: init_params.tuning_info,
buffer_format: init_params.buffer_format,
encode_config: init_params.encode_config as _,
enable_ptd: init_params.enable_ptd as u32,
version: NV_ENC_INITIALIZE_PARAMS_VER,
enable_encode_async: 0,
bit_flags: crate::sys::structs::NVencInitializeParamsBitfields::new(),
priv_data_size: 0,
rsvd: 0,
priv_data: std::ptr::null_mut(),
max_encode_width: init_params.max_encoder_resolution[0],
max_encode_height: init_params.max_encoder_resolution[1],
max_me_hint_counts_per_block: [Default::default(), Default::default()],
num_state_buffers: 0,
output_stats_level: crate::sys::enums::NVencOutputStatsLevel::None,
rsvd1: unsafe { std::mem::zeroed() },
rsvd2: unsafe { std::mem::zeroed() },
};
self.encoder.init_encoder(raw_init_params)?;
Ok(Encoder {
encoder: self.encoder,
marker: PhantomData,
})
}
}
vendor/nvenc/src/sys/enums.rs
Vendored
+481
View File
@@ -0,0 +1,481 @@
#[repr(C)]
pub enum NVencParamsFrameFieldMode {
Frame = 0x01,
Field = 0x02,
MBAFF = 0x03,
}
#[repr(C)]
pub enum NVencParamsRcMode {
ConstQP = 0x00,
VBR = 0x01,
CBR = 0x02,
}
#[repr(C)]
pub enum NVencMultiPass {
Disabled = 0x00,
TwoPassQuarterResolution = 0x01,
TwoPassFullResolution = 0x02,
}
#[repr(C)]
pub enum NVencStateRestoreType {
Full = 0x01,
RateControl = 0x02,
Encode = 0x03,
}
#[repr(C)]
pub enum NVencOutputStatsLevel {
None = 0,
BlockLevel = 1,
RowLevel = 2,
}
#[repr(C)]
pub enum NVencEmphasisMapLevel {
Level0 = 0x0,
Level1 = 0x1,
Level2 = 0x2,
Level3 = 0x3,
Level4 = 0x4,
Level5 = 0x5,
}
#[repr(C)]
pub enum NVencQPMapMode {
Disabled = 0x0,
Emphasis = 0x1,
Delta = 0x2,
/// Currently unsupported as of NVENC 13.0
Map = 0x3,
}
#[repr(C)]
pub enum NVencPicStruct {
Frame = 0x01,
FieldTopBottom = 0x02,
FieldBottomTop = 0x03,
}
#[repr(C)]
pub enum NVencDisplayPicStruct {
Frame = 0x0,
FieldTopBottom = 0x1,
FieldBottomTop = 0x2,
Doubling = 0x3,
Tripling = 0x4,
}
#[repr(C)]
pub enum NVencPicType {
P = 0x00,
B = 0x01,
IDR = 0x03,
BI = 0x04,
Skipped = 0x05,
IntraRefresh = 0x06,
NonRefP = 0x07,
Switch = 0x08,
UNKNOWN = 0xFF,
}
#[repr(C)]
pub enum NVencMVPrecision {
Default = 0x00,
FullPel = 0x01,
HalfPel = 0x02,
QuarterPel = 0x03,
}
#[repr(C)]
pub enum NVencBufferFormat {
Undefined = 0x00000000,
NV12 = 0x00000001,
YV12 = 0x00000010,
IYUV = 0x00000100,
YUV444 = 0x00001000,
YUV420_10Bit = 0x00010000,
YUV444_10Bit = 0x00100000,
ARGB = 0x01000000,
ARGB10 = 0x02000000,
AYUV = 0x04000000,
ABGR = 0x10000000,
ABGR10 = 0x20000000,
U8 = 0x40000000,
NV16 = 0x40000001,
P210 = 0x40000002,
}
#[repr(C)]
pub enum NvencLevel {
AutoSelect = 0,
}
// TODO: Translate level macro
#[repr(C)]
pub enum H264Level {
H264_1 = 10,
H264_1b = 9,
H264_11 = 11,
H264_12 = 12,
H264_13 = 13,
H264_2 = 20,
H264_21 = 21,
H264_22 = 22,
H264_3 = 30,
H264_31 = 31,
H264_32 = 32,
H264_4 = 40,
H264_41 = 41,
H264_42 = 42,
H264_5 = 50,
H264_51 = 51,
H264_52 = 52,
H264_60 = 60,
H264_61 = 61,
H264_62 = 62,
}
#[repr(C)]
pub enum HEVCLevel {
HEVC1 = 30,
HEVC2 = 60,
HEVC21 = 63,
HEVC3 = 90,
HEVC31 = 93,
HEVC4 = 120,
HEVC41 = 123,
HEVC5 = 150,
HEVC51 = 153,
HEVC52 = 156,
HEVC6 = 180,
HEVC61 = 183,
HEVC62 = 186,
HEVCMain = 0,
HEVCHigh = 1,
}
#[repr(C)]
pub enum AV1Level {
// Also AV1_0
AV1_2 = 0,
// Also AV1_1
AV1_21 = 1,
AV1_22 = 2,
AV1_23 = 3,
AV1_3 = 4,
AV1_31 = 5,
AV1_32 = 6,
AV1_33 = 7,
AV1_4 = 8,
AV1_41 = 9,
AV1_42 = 10,
AV1_43 = 11,
AV1_5 = 12,
AV1_51 = 13,
AV1_52 = 14,
AV1_53 = 15,
AV1_6 = 16,
AV1_61 = 17,
AV1_62 = 18,
AV1_63 = 19,
AV1_7 = 20,
AV1_71 = 21,
AV1_72 = 22,
AV1_73 = 23,
AV1AutoSelect = 24,
}
#[repr(C)]
pub enum NVencPicFlags {
ForceIntra = 0x1,
ForceIDR = 0x2,
OutputSpspps = 0x4,
Eos = 0x8,
DisableEncStateAdvance = 0x10,
OuputReconFrame = 0x20,
}
#[repr(C)]
pub enum NVencMemoryHeap {
AutoSelect = 0,
VID = 1,
SystemCached = 2,
SystemUncached = 3,
}
#[repr(C)]
pub enum NVencBFrameRefMode {
Disabled = 0x0,
Each = 0x1,
Middle = 0x2,
}
#[repr(C)]
pub enum NVencH264EntropyCodingMode {
AutoSelect = 0x0,
Cabac = 0x1,
Cavlc = 0x2,
}
#[repr(C)]
pub enum NVencH264BDirectMode {
Autoselect = 0x0,
Disable = 0x1,
Temporal = 0x2,
Spatial = 0x3,
}
#[repr(C)]
pub enum NVencH264FMOMode {
AutoSelect = 0x0,
Enable = 0x1,
Disable = 0x2,
}
#[repr(C)]
pub enum NVencH264AdaptiveTransformMode {
AutoSelect = 0x0,
Disable = 0x1,
Enable = 0x2,
}
#[repr(C)]
pub enum NVencStereoPackingMode {
None = 0x0,
Checkerboard = 0x1,
ColInterleave = 0x2,
RowInterleave = 0x3,
SideBySide = 0x4,
TopBottm = 0x5,
FrameSeq = 0x6,
}
#[repr(C)]
pub enum NVencInputResourceType {
DirectX = 0x0,
CudaDeivcePtr = 0x1,
CudaArray = 0x2,
OpenGLTex = 0x3,
}
#[repr(C)]
pub enum NVencBufferUsage {
Image = 0x0,
MotionVector = 0x1,
BitStream = 0x2,
OutputRecon = 0x4,
}
#[repr(C)]
#[derive(Clone)]
pub enum NVencDeviceType {
DirectX = 0x0,
Cuda = 0x1,
OpenGL = 0x2,
}
#[repr(C)]
pub enum NVencNumRefFrames {
AutoSelect = 0x0,
Frames1 = 0x1,
Frames2 = 0x2,
Frames3 = 0x3,
Frames4 = 0x4,
Frames5 = 0x5,
Frames6 = 0x6,
Frames7 = 0x7,
}
#[repr(C)]
pub enum NVencTemporalFilterLevel {
Level0 = 0,
Level4 = 4,
}
#[repr(C)]
pub enum NVencCaps {
NumMaxBFrames,
SupportedRateControlModes,
FieldEncoding,
MonoChrome,
FMO,
QPELMV,
BDirectMode,
CABAC,
AdaptiveTransform,
StereoMVC,
MaxTemporalLayers,
HierarchicalBFrames,
LevelMax,
LevelMin,
SeperateColorPlane,
WidthMax,
HeightMax,
TemporalSVC,
DynResChnage,
DynBitrateChange,
DynForceConstQP,
DynRcModeChange,
SubframeReadBack,
ConstrainedEncoding,
IntraRefresh,
CustomVBVBufSize,
DynamicSliceMode,
Invalidation,
PreProcSupport,
AsyncEncodeSupport,
MbNumMax,
MbPerSecMax,
YUV444Encode,
LoselessEncode,
SupportSao,
MeOnlyMode,
LookAhead,
TemporalAQ,
MaxLtrFrames,
SupportWeightedPrediction,
DynamicQueryEncoderCapacity,
BFrameRefMode,
EmphasisLevelMap,
WidthMin,
HeightMin,
MultipleRefFrames,
AlphaLayerEncoding,
NumEncoderEngines,
SingleSliceIntraRefresh,
DisableEncStateAdvance,
OutputReconSurface,
OutputBlockStats,
SupportTemporalFilter,
LookAheadLevel,
UndirectionalB,
MVHEVCEncode,
SupportYUV422Encode,
ExposedCount,
}
#[repr(C)]
pub enum NVencHEVCCuSize {
AutoSelect = 0,
Size8x8 = 1,
Size16x16 = 2,
Size32x32 = 3,
Size64x64 = 4,
}
#[repr(C)]
pub enum NVencAV1PartSize {
AutoSelect = 0,
Size4x4 = 1,
Size8x8 = 2,
Size16x16 = 3,
Size32x32 = 4,
Size64x64 = 5,
}
#[repr(C)]
pub enum NVencVUIVideoFormat {
Component = 0,
Pal = 1,
NTSC = 2,
SeCam = 3,
Mac = 4,
Unspecified = 5,
}
#[repr(C)]
pub enum NVencVUIColorPrimaries {
Undefined = 0,
BT709 = 1,
Unspecified = 2,
Reserved = 3,
BT470M = 4,
BT470BG = 5,
SMPTE170M = 6,
SMPTE240M = 7,
FIlm = 8,
BT2020 = 9,
SMPTE428 = 10,
SMPTE431 = 11,
SMPTE432 = 12,
JEDECP22 = 22,
}
#[repr(C)]
pub enum NVencVUITransCharacteristics {
Undefined = 0,
BT709 = 1,
Unspecified = 2,
Reserved = 3,
BT470M = 4,
BT470BG = 5,
SMPTE170M = 6,
SMPTE240M = 7,
Linear = 8,
Log = 9,
LogSqrt = 10,
IEC61966_2_4 = 11,
SRGB = 13,
BT2020_10 = 14,
BT2020_12 = 15,
SMPTE2084 = 16,
SMPTE428 = 17,
AribStdB67 = 18,
}
#[repr(C)]
pub enum NVencVUIMatrixCoeffs {
RGB = 0,
BT709 = 1,
Unspecified = 2,
Reserved = 3,
FCC = 4,
BT470BG = 5,
SMPTE170M,
SMPTE240M,
YCGCO = 8,
BT2020NCL = 9,
BT2020CL = 10,
SMPTE2085 = 11,
}
#[repr(C)]
pub enum NVencLookAheadLevel {
Level0 = 0,
Level1 = 1,
Level2 = 2,
Level3 = 3,
AutoSelect = 15,
}
#[repr(C)]
pub enum NVencBitDepth {
Invalid = 0,
Depth8 = 8,
Depth10 = 10,
}
#[derive(Debug)]
#[repr(C)]
pub enum NVencTuningInfo {
Undefined = 0,
HighQuality = 1,
LowLatency = 2,
UltraLowLatency = 3,
Lossless = 4,
UltraHighQuality = 5,
Count,
}
#[repr(C)]
pub enum NVencSplitEncodeMode {
Audo = 0,
AutoForced = 1,
TwoForced = 2,
ThreeForced = 3,
FourForced = 4,
Disable = 15,
}
vendor/nvenc/src/sys/function_table.rs
Vendored
+214
View File
@@ -0,0 +1,214 @@
use std::{ffi::c_char, mem::MaybeUninit, os::raw::c_void};
use crate::{
stdcall,
sys::{
enums::{NVencBufferFormat, NVencTuningInfo},
result::NVencResult,
structs::{
Guid, NVencCapsParam, NVencCreateBitstreamBuffer, NVencCreateInputBuffer,
NVencInitializeParams, NVencLockBitStream, NVencLockInputBuffer, NVencMapInputResource,
NVencOpenEncodeSessionExParams, NVencPicParams, NVencPresetConfig,
NVencRegisterResource,
},
},
};
#[repr(C)]
#[derive(Clone)]
pub struct NVencFunctionList {
pub version: u32,
rsvd: u32,
#[deprecated]
nvenc_open_encode_session: NvencOpenEncodeSession,
pub nvenc_get_encoder_guid_count: NvencGetEncodeGuidCount,
nvenc_get_encoder_profile_guid_count: NvencGetEncodeProfileGuidCount,
nvenc_get_encoder_profile_guids: NvencGetEncodeProfileGuids,
pub nvenc_get_encoder_guids: NvencGetEncodeGuids,
nvenc_get_input_format_count: NvencGetInputFormatCount,
nvenc_get_input_formats: NvencGetInputFormats,
nvenc_get_encode_caps: NvencGetEncodeCaps,
pub nvenc_get_encode_preset_count: NvencGetEncodePresetCount,
pub nvenc_get_encode_preset_guids: NvencGetEncodePresetGuids,
nvenc_get_encode_preset_config: NvencGetEncodePresetConfig,
pub nvenc_initialize_encoder: NvencInitializeEncoder,
pub nvenc_create_input_buffer: NvencCreateInputBuffer,
pub nvenc_destroy_input_buffer: NvencDestroyInputBuffer,
pub nvenc_create_bit_stream_buffer: NvencCreateBitStreamBuffer,
pub nvenc_destory_bit_stream_buffer: NvencDestroyBitStreamBuffer,
pub nvenc_encode_picture: NvencEncodePicture,
pub nvenc_lock_bit_stream: NvencLockBitStream,
pub nvenc_unlock_bit_stream: NvencUnlockBitStream,
pub nvenc_lock_input_buffer: NvencLockInputBuffer,
pub nvenc_unlock_input_buffer: NvencUnlockInputBuffer,
nvenc_get_encode_stats: NvencGetEncodeStats,
nvenc_get_sequence_params: NvencGetSequenceParams,
nvenc_register_async_event: NvencRegisterAsyncEvent,
nvenc_unregister_async_event: NvencUnregisterAsyncEvent,
pub nvenc_map_input_resource: NvencMapInputResource,
pub nvenc_unmap_input_resource: NvencUnmapInputResource,
pub nvenc_destroy_encoder: NvencDestroyEncoder,
nvenc_invalidate_ref_frames: NvencInvalidateRefFrames,
pub nvenc_open_encode_session_ex: NvencOpenEncodeSessionEx,
pub nvenc_register_resource: NvencRegisterResource,
pub nvenc_unregister_resource: NvencUnRegisterResource,
nvenc_reconfigure_encoder: NvencReconfigureEncoder,
rsvd1: *mut c_void,
nvenc_create_mv_buffer: NvencCreateMVBuffer,
nvenc_destory_mv_buffer: NvencDestoryMVBuffer,
nvenc_run_motion_estimation_only: NvencRunMotionEstimationOnly,
pub nvenc_get_last_error: NvencGetLastError,
nvenc_set_io_cuda_streams: NvencSetIOCudaStreams,
pub nvenc_get_encode_preset_config_ex: NvencGetEncodePresetConfigEx,
nvenc_get_sequence_params_ex: NvencGetSequenceParamsEx,
nvenc_store_encoder_state: NvencStoreEncoderState,
nvenc_look_ahead_picture: NvencLookAheadPicture,
rsvd2: [*mut c_void; 275],
}
type NvencOpenEncodeSession =
stdcall!(fn(device: *mut c_void, device_type: i32, encoder: *mut *mut c_void) -> NVencResult);
type NvencGetEncodeGuidCount =
stdcall!(fn(encoder: *mut c_void, encoder_guid_count: *mut u32) -> NVencResult);
type NvencGetEncodeGuids = stdcall!(
fn(
encoder: *mut c_void,
guids: *mut Guid,
guid_array_size: u32,
guid_count: *mut u32,
) -> NVencResult
);
type NvencGetEncodeProfileGuidCount = stdcall!(
fn(
encoder: *mut c_void,
encode_guid: Guid,
encoder_profile_guid_count: *mut u32,
) -> NVencResult
);
type NvencGetEncodeProfileGuids = stdcall!(
fn(
encoder: *mut c_void,
profile_guids: *mut Guid,
guid_array_size: u32,
guid_count: *mut u32,
) -> NVencResult
);
type NvencGetInputFormatCount = stdcall!(
fn(encoder: *mut c_void, encoder_guid: Guid, input_fmt_count: *mut u32) -> NVencResult
);
type NvencGetInputFormats = stdcall!(
fn(
encoder: *mut c_void,
input_formats: *mut NVencBufferFormat,
input_format_array_size: u32,
input_format_count: *mut u32,
) -> NVencResult
);
type NvencGetEncodeCaps = stdcall!(
fn(
encoder: *mut c_void,
encoder_guid: Guid,
caps_param: *mut NVencCapsParam,
caps_val: *mut i32,
) -> NVencResult
);
type NvencGetEncodePresetCount = stdcall!(
fn(
encoder: *mut c_void,
encoder_guid: Guid,
encoder_preset_guid_count: *mut u32,
) -> NVencResult
);
type NvencGetEncodePresetGuids = stdcall!(
fn(
encoder: *mut c_void,
encode_guid: Guid,
preset_guids: *mut Guid,
guid_array_size: u32,
encoder_preset_guid_count: *mut u32,
) -> NVencResult
);
type NvencGetEncodePresetConfig = stdcall!(
fn(
encoder: *mut c_void,
encode_guid: Guid,
preset_guid: Guid,
preset_config: *mut NVencPresetConfig,
) -> NVencResult
);
type NvencGetEncodePresetConfigEx = stdcall!(
fn(
encoder: *mut c_void,
encode_guid: Guid,
preset_guid: Guid,
tuning_info: NVencTuningInfo,
preset_config: *mut MaybeUninit<NVencPresetConfig>,
) -> NVencResult
);
type NvencInitializeEncoder = stdcall!(
fn(encoder: *mut c_void, create_encode_params: *mut NVencInitializeParams) -> NVencResult
);
type NvencCreateInputBuffer = stdcall!(
fn(
encoder: *mut c_void,
create_input_buffer_params: *mut NVencCreateInputBuffer,
) -> NVencResult
);
type NvencDestroyInputBuffer =
stdcall!(fn(encoder: *mut c_void, input_buffer: *mut c_void) -> NVencResult);
type NvencCreateBitStreamBuffer = stdcall!(
fn(
encoder: *mut c_void,
create_bit_stream_buffer_params: *mut NVencCreateBitstreamBuffer,
) -> NVencResult
);
type NvencDestroyBitStreamBuffer =
stdcall!(fn(encoder: *mut c_void, bitstream_buffer: *mut c_void) -> NVencResult);
type NvencEncodePicture = stdcall!(
fn(encoder: *mut c_void, encode_pic_params: *mut MaybeUninit<NVencPicParams>) -> NVencResult
);
type NvencLockBitStream = stdcall!(
fn(
encoder: *mut c_void,
lock_bitstream_buffer_params: *mut MaybeUninit<NVencLockBitStream>,
) -> NVencResult
);
type NvencUnlockBitStream =
stdcall!(fn(encoder: *mut c_void, bitstream_buffer: *mut c_void) -> NVencResult);
type NvencLockInputBuffer = stdcall!(
fn(encoder: *mut c_void, lock_input_buffer_params: *mut NVencLockInputBuffer) -> NVencResult
);
type NvencUnlockInputBuffer =
stdcall!(fn(encoder: *mut c_void, input_buffer: *mut c_void) -> NVencResult);
type NvencGetEncodeStats = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencGetSequenceParams = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencRegisterAsyncEvent = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencUnregisterAsyncEvent = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencMapInputResource = stdcall!(
fn(encoder: *mut c_void, map_input_resource_params: *mut NVencMapInputResource) -> NVencResult
);
type NvencUnmapInputResource =
stdcall!(fn(encoder: *mut c_void, mapped_input_buffer: *mut c_void) -> NVencResult);
type NvencDestroyEncoder = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencInvalidateRefFrames = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencOpenEncodeSessionEx = stdcall!(
fn(
open_session_ex_params: *mut NVencOpenEncodeSessionExParams,
encoder: *mut *mut c_void,
) -> NVencResult
);
type NvencRegisterResource = stdcall!(
fn(encoder: *mut c_void, register_res_params: *mut NVencRegisterResource) -> NVencResult
);
type NvencUnRegisterResource =
stdcall!(fn(encoder: *mut c_void, registered_res: *mut c_void) -> NVencResult);
type NvencReconfigureEncoder = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencCreateMVBuffer = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencDestoryMVBuffer = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencRunMotionEstimationOnly = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencGetLastError = stdcall!(fn(encoder: *mut c_void) -> *const c_char);
type NvencSetIOCudaStreams = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencGetSequenceParamsEx = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencStoreEncoderState = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
type NvencLookAheadPicture = stdcall!(fn(encoder: *mut c_void) -> NVencResult);
vendor/nvenc/src/sys/guids.rs
Vendored
+146
View File
@@ -0,0 +1,146 @@
use crate::sys::structs::Guid;
pub const NV_ENC_CODEC_H264_GUID: Guid = Guid::from_values(
0x6bc82762,
0x4e63,
0x4ca4,
[0xaa, 0x85, 0x1e, 0x50, 0xf3, 0x21, 0xf6, 0xbf],
);
pub const NV_ENC_CODEC_HEVC_GUID: Guid = Guid::from_values(
0x790cdc88,
0x4522,
0x4d7b,
[0x94, 0x25, 0xbd, 0xa9, 0x97, 0x5f, 0x76, 0x3],
);
pub const NV_ENC_CODEC_AV1_GUID: Guid = Guid::from_values(
0x0a352289,
0x0aa7,
0x4759,
[0x86, 0x2d, 0x5d, 0x15, 0xcd, 0x16, 0xd2, 0x54],
);
pub const NV_ENC_CODEC_PROFILE_AUTOSELECT_GUID: Guid = Guid::from_values(
0xbfd6f8e7,
0x233c,
0x4341,
[0x8b, 0x3e, 0x48, 0x18, 0x52, 0x38, 0x3, 0xf4],
);
pub const NV_ENC_H264_PROFILE_BASELINE_GUID: Guid = Guid::from_values(
0x727bcaa,
0x78c4,
0x4c83,
[0x8c, 0x2f, 0xef, 0x3d, 0xff, 0x26, 0x7c, 0x6a],
);
pub const NV_ENC_H264_PROFILE_MAIN_GUID: Guid = Guid::from_values(
0x60b5c1d4,
0x67fe,
0x4790,
[0x94, 0xd5, 0xc4, 0x72, 0x6d, 0x7b, 0x6e, 0x6d],
);
pub const NV_ENC_H264_PROFILE_HIGH_GUID: Guid = Guid::from_values(
0xe7cbc309,
0x4f7a,
0x4b89,
[0xaf, 0x2a, 0xd5, 0x37, 0xc9, 0x2b, 0xe3, 0x10],
);
pub const NV_ENC_H264_PROFILE_HIGH_10_GUID: Guid = Guid::from_values(
0x8f0c337e,
0x186c,
0x48e9,
[0xa6, 0x9d, 0x7a, 0x83, 0x34, 0x08, 0x97, 0x5],
);
pub const NV_ENC_H264_PROFILE_HIGH_422_GUID: Guid = Guid::from_values(
0xff3242e9,
0x613c,
0x4295,
[0xa1, 0xe8, 0x2a, 0x7f, 0xe9, 0x4d, 0x81, 0x33],
);
pub const NV_ENC_H264_PROFILE_HIGH_444_GUID: Guid = Guid::from_values(
0x7ac663cb,
0xa598,
0x4960,
[0xb8, 0x44, 0x33, 0x9b, 0x26, 0x1a, 0x7d, 0x52],
);
pub const NV_ENC_H264_PROFILE_STEREO_GUID: Guid = Guid::from_values(
0x40847bf5,
0x33f7,
0x4601,
[0x90, 0x84, 0xe8, 0xfe, 0x3c, 0x1d, 0xb8, 0xb7],
);
pub const NV_ENC_H264_PROFILE_PROGRESSIVE_HIGH_GUID: Guid = Guid::from_values(
0xb405afac,
0xf32b,
0x417b,
[0x89, 0xc4, 0x9a, 0xbe, 0xed, 0x3e, 0x59, 0x78],
);
pub const NV_ENC_H264_PROFILE_CONSTRAINED_HIGH_GUID: Guid = Guid::from_values(
0xaec1bd87,
0xe85b,
0x48f2,
[0x84, 0xc3, 0x98, 0xbc, 0xa6, 0x28, 0x50, 0x72],
);
pub const NV_ENC_HEVC_PROFILE_MAIN_GUID: Guid = Guid::from_values(
0xb514c39a,
0xb55b,
0x40fa,
[0x87, 0x8f, 0xf1, 0x25, 0x3b, 0x4d, 0xfd, 0xec],
);
pub const NV_ENC_HEVC_PROFILE_MAIN10_GUID: Guid = Guid::from_values(
0xfa4d2b6c,
0x3a5b,
0x411a,
[0x80, 0x18, 0x0a, 0x3f, 0x5e, 0x3c, 0x9b, 0xe5],
);
pub const NV_ENC_HEVC_PROFILE_FREXT_GUID: Guid = Guid::from_values(
0x51ec32b5,
0x1b4c,
0x453c,
[0x9c, 0xbd, 0xb6, 0x16, 0xbd, 0x62, 0x13, 0x41],
);
pub const NV_ENC_AV1_PROFILE_MAIN_GUID: Guid = Guid::from_values(
0x5f2a39f5,
0xf14e,
0x4f95,
[0x9a, 0x9e, 0xb7, 0x6d, 0x56, 0x8f, 0xcf, 0x97],
);
pub const NV_ENC_PRESET_P1_GUID: Guid = Guid::from_values(
0xfc0a8d3e,
0x45f8,
0x4cf8,
[0x80, 0xc7, 0x29, 0x88, 0x71, 0x59, 0xe, 0xbf],
);
pub const NV_ENC_PRESET_P2_GUID: Guid = Guid::from_values(
0xf581cfb8,
0x88d6,
0x4381,
[0x93, 0xf0, 0xdf, 0x13, 0xf9, 0xc2, 0x7d, 0xab],
);
pub const NV_ENC_PRESET_P3_GUID: Guid = Guid::from_values(
0x36850110,
0x3a07,
0x441f,
[0x94, 0xd5, 0x36, 0x70, 0x63, 0x1f, 0x91, 0xf6],
);
pub const NV_ENC_PRESET_P4_GUID: Guid = Guid::from_values(
0x90a7b826,
0xdf06,
0x4862,
[0xb9, 0xd2, 0xcd, 0x6d, 0x73, 0xa0, 0x86, 0x81],
);
pub const NV_ENC_PRESET_P5_GUID: Guid = Guid::from_values(
0x21c6e6b4,
0x297a,
0x4cba,
[0x99, 0x8f, 0xb6, 0xcb, 0xde, 0x72, 0xad, 0xe3],
);
pub const NV_ENC_PRESET_P6_GUID: Guid = Guid::from_values(
0x8e75c279,
0x6299,
0x4ab6,
[0x83, 0x2, 0xb, 0x21, 0x5a, 0x33, 0x5c, 0xf5],
);
pub const NV_ENC_PRESET_P7_GUID: Guid = Guid::from_values(
0x84848c12,
0x6f71,
0x4c13,
[0x93, 0x1b, 0x53, 0xe2, 0x83, 0xf5, 0x79, 0x74],
);
vendor/nvenc/src/sys/mod.rs
Vendored
+6
View File
@@ -0,0 +1,6 @@
pub mod enums;
pub mod function_table;
pub mod guids;
pub mod result;
pub mod structs;
pub mod version;
vendor/nvenc/src/sys/result.rs
Vendored
+79
View File
@@ -0,0 +1,79 @@
#[derive(Debug, PartialEq, Eq)]
#[repr(C)]
pub enum NVencResult {
Success,
NoDevice,
UnsupportedDevice,
InvalidEncoderDevice,
InvalidDevice,
DeviceNotExist,
InvalidPtr,
InvalidParam,
InvalidEvent,
InvalidCall,
OutOfMemory,
NotInitialized,
UnsupportedParam,
LockBusy,
NotEnoughBuffer,
InvalidVersion,
MapFailed,
NeedMoreInput,
EncoderBusy,
EventNotRegisterd,
Generic,
IncompatibleClientKey,
Unimplemented,
ResourceRegisterFailed,
ResourceNotRegistered,
ResourceNotMapped,
NeedMoreOutput,
}
impl NVencResult {
pub fn into_error(self) -> Result<(), NVencError> {
NVencError::from_result(self)
}
}
impl NVencError {
pub fn from_result(value: NVencResult) -> Result<(), Self> {
if value == NVencResult::Success {
Ok(())
} else {
// Safety: They are one case different, so they always align
unsafe { Err(std::mem::transmute::<NVencResult, NVencError>(value)) }
}
}
}
#[derive(Debug, PartialEq, Eq)]
#[repr(C)]
pub enum NVencError {
NoDevice = NVencResult::NoDevice as isize,
UnsupportedDevice,
InvalidEncoderDevice,
InvalidDevice,
DeviceNotExist,
InvalidPtr,
InvalidEvent,
InvalidParam,
InvalidCall,
OutOfMemory,
NotInitialized,
UnsupportedParam,
LockBusy,
NotEnoughBuffer,
InvalidVersion,
MapFailed,
NeedMoreInput,
EncoderBusy,
EventNotRegisterd,
Generic,
IncompatibleClientKey,
Unimplemented,
ResourceRegisterFailed,
ResourceNotRegistered,
ResourceNotMapped,
NeedMoreOutput,
}
vendor/nvenc/src/sys/structs.rs
Vendored
+944
View File
@@ -0,0 +1,944 @@
use std::{ffi::c_void, mem::ManuallyDrop};
use crate::sys::{
enums::{
NVencAV1PartSize, NVencBFrameRefMode, NVencBitDepth, NVencBufferFormat, NVencBufferUsage,
NVencCaps, NVencDeviceType, NVencDisplayPicStruct, NVencH264AdaptiveTransformMode,
NVencH264BDirectMode, NVencH264EntropyCodingMode, NVencH264FMOMode, NVencHEVCCuSize,
NVencInputResourceType, NVencLookAheadLevel, NVencMVPrecision, NVencMemoryHeap,
NVencMultiPass, NVencNumRefFrames, NVencOutputStatsLevel, NVencParamsFrameFieldMode,
NVencParamsRcMode, NVencPicStruct, NVencPicType, NVencQPMapMode, NVencStateRestoreType,
NVencStereoPackingMode, NVencTemporalFilterLevel, NVencTuningInfo, NVencVUIColorPrimaries,
NVencVUIMatrixCoeffs, NVencVUITransCharacteristics, NVencVUIVideoFormat,
},
version::struct_version,
};
#[derive(Default, Clone, PartialEq, Eq)]
#[repr(C)]
pub struct Guid {
pub data1: u32,
pub data2: u16,
pub data3: u16,
pub data4: [u8; 8],
}
impl Guid {
pub const fn from_values(data1: u32, data2: u16, data3: u16, data4: [u8; 8]) -> Self {
Self {
data1,
data2,
data3,
data4,
}
}
pub const fn from_u128(uuid: u128) -> Self {
Self {
data1: (uuid >> 96) as u32,
data2: ((uuid >> 80) & 0xffff) as u16,
data3: ((uuid >> 64) & 0xffff) as u16,
data4: (uuid as u64).to_be_bytes(),
}
}
}
#[repr(C)]
pub struct NVencCapsParam {
version: u32,
caps_to_query: NVencCaps,
rsvd: [u32; 62],
}
pub const NV_ENC_CAPS_PARAM_VERSION: u32 = struct_version(1);
#[repr(C)]
pub struct NVencRestoreEncoderStateParams {
version: u32,
buffer: u32,
state: NVencStateRestoreType,
rsvd: u32,
output_bit_stream: *mut c_void,
completion_event: *mut c_void,
rsvd1: [u32; 64],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_RESTORE_STATE_PARAMS_VER: u32 = struct_version(2);
#[repr(C)]
pub struct NVencOutputStatsBlock {
version: u32,
qp: u8,
rsvd: [u8; 3],
bit_count: u32,
satd_cost: u32,
rsvd1: [u32; 12],
}
pub const NV_ENC_OUTPUT_STATS_BLOCK_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencOutputStatsRow {
version: u32,
qp: u8,
rsvd: [u8; 3],
bit_count: u32,
satd_cost: u32,
rsvd1: [u32; 12],
}
pub const NV_ENC_OUTPUT_STATS_ROW_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencEncodeOutParams {
version: u32,
bitstream_size_in_bytes: u32,
rsvd: [u32; 62],
}
pub const NV_ENC_ENCODE_OUT_PARAMS_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencLookAheadPicParams {
version: u32,
rsvd: u32,
input_buffer: *mut c_void,
picture_type: NVencPicType,
rsvd1: [u32; 63],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_LOOKAHEAD_PIC_PARAMS_VER: u32 = struct_version(2);
#[repr(C)]
pub struct NVencCreateInputBuffer {
pub version: u32,
pub width: u32,
pub height: u32,
pub memory_heap: NVencMemoryHeap,
pub buffer_fmt: NVencBufferFormat,
pub rsvd: u32,
pub input_buffer: *mut c_void,
pub p_sys_mem_buffer: *mut c_void,
pub rsvd1: [u32; 58],
pub rsvd2: [*mut c_void; 63],
}
pub const NV_ENC_CREATE_INPUT_BUFFER_VER: u32 = struct_version(2);
#[repr(C)]
pub struct NVencCreateBitstreamBuffer {
pub version: u32,
#[deprecated]
size: u32,
#[deprecated]
memory_heap: NVencMemoryHeap,
rsvd: u32,
pub bitstream_buffer: *mut c_void,
bitstream_buffer_ptr: *mut c_void,
rsvd1: [u32; 58],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_CREATE_BITSTREAM_BUFFER_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencMVector {
mvx: i16,
mvy: i16,
}
#[repr(C)]
pub struct NVencH264MVData {
mv: [NVencMVector; 4],
mb_type: u8,
partition_type: u8,
rsvd: u16,
mb_cost: u32,
}
#[repr(C)]
pub struct NVencHEVCMVData {
mv: [NVencMVector; 4],
cu_type: u8,
cu_size: u8,
partition_mode: u8,
last_cu_int_ctb: u8,
}
#[repr(C)]
pub struct NVencCreateMVBuffer {
version: u32,
rsvd: u32,
mv_buffer: *mut c_void,
rsvd1: [u32; 254],
rsvd2: [*mut c_void; 63],
}
pub const NV_ENC_CREATE_MV_BUFFER_VER: u32 = struct_version(2);
#[repr(C)]
pub struct NVencQP {
qp_inter_p: u32,
qp_inter_b: u32,
qp_intra: u32,
}
pub const MAX_NUM_VIEWS_MINUS_1: usize = 7;
#[repr(C)]
pub struct NVencRcParams {
version: u32,
pub rate_control_mode: NVencParamsRcMode,
const_qp: NVencQP,
pub average_bit_rate: u32,
max_bit_rate: u32,
vbv_buffer_size: u32,
vbv_initial_delay: u32,
// TODO: bit fields
bit_fields: NVencRcParamsBitFlags,
min_qp: NVencQP,
max_qp: NVencQP,
initial_rcqp: NVencQP,
temporal_layer_idx_mask: u32,
temporal_layer_qp: [u8; 8],
target_quality: u8,
target_quality_lsb: u8,
pub look_ahead_depth: u16,
low_delay_key_frame_scale: u8,
y_dc_qp_index_offset: i8,
u_dc_qp_index_offset: i8,
v_dc_qp_index_offset: i8,
qp_map_mode: NVencQPMapMode,
multi_pass: NVencMultiPass,
alpha_layer_bitrate_ratio: u32,
cb_qp_index_offset: i8,
cr_qp_index_offset: i8,
rsvd2: u16,
lookahead_level: NVencLookAheadLevel,
view_bitrate_ratios: [u8; MAX_NUM_VIEWS_MINUS_1],
rsvd3: u8,
rsvd1: u32,
}
#[repr(transparent)]
pub struct NVencRcParamsBitFlags(u32);
impl NVencRcParamsBitFlags {}
pub const NV_ENC_RC_PARAMS_VER: u32 = struct_version(1);
pub const MAX_NUM_CLOCK_TS: usize = 3;
#[repr(C)]
pub struct NVencClockTimeStampSet {
pub bitflags: u32,
pub time_offset: u32,
}
impl NVencClockTimeStampSet {}
#[repr(C)]
pub struct NVencTimeCode {
pub display_pic_struct: NVencDisplayPicStruct,
pub clock_timestamp: [NVencClockTimeStampSet; MAX_NUM_CLOCK_TS],
pub skip_clock_timestamp_insertion: u32,
}
pub const MULTIVIEW_MAX_NUM_REF_DISPLAY: usize = 32;
#[repr(C)]
pub struct HEVC3DReferenceDisplayInfo {
// TODO
pub bitflags: u32,
pub prec_ref_display_width: i32,
pub prec_ref_viewing_dist: i32,
pub num_ref_displays_minus1: i32,
pub left_view_id: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub right_view_id: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub exponent_ref_display_width: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub mantissa_ref_display_width: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub exponent_ref_viewing_distance: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub mantissa_ref_viewing_distance: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub num_sample_shift_plus512: [i32; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub additional_shift_present_flag: [u8; MULTIVIEW_MAX_NUM_REF_DISPLAY],
pub rsvd2: [u32; 4],
}
#[repr(C)]
pub struct ChromaPoints {
x: u16,
y: u16,
}
#[repr(C)]
pub struct MasteringDisplayInfo {
pub g: ChromaPoints,
pub b: ChromaPoints,
pub r: ChromaPoints,
pub white_point: ChromaPoints,
pub max_luma: u32,
pub min_luma: u32,
}
#[repr(C)]
pub struct ContentLightLevel {
pub max_content_light_level: u16,
pub max_pic_average_light_level: u16,
}
#[repr(C)]
pub struct NVencConfigH264VUIParameters {
overscan_info_present_flag: u32,
overscan_info: u32,
video_signal_type_present_flag: u32,
video_format: NVencVUIVideoFormat,
video_full_range_flag: u32,
color_description_present_flag: u32,
color_primaries: NVencVUIColorPrimaries,
transfer_characteristics: NVencVUITransCharacteristics,
color_matrix: NVencVUIMatrixCoeffs,
chroma_sample_location_flag: u32,
chroma_sample_location_top: u32,
chroma_sample_location_bot: u32,
bitstream_restriction_flag: u32,
timing_info_present_flag: u32,
num_unit_in_ticks: u32,
time_scale: u32,
rsvd: [u32; 12],
}
pub type NVencConfigHEVCVUIParamaters = NVencConfigH264VUIParameters;
#[repr(C)]
#[derive(Default)]
pub struct NVencExternalMeHintCountsPerBlockType {
pub bitflags: u32,
pub rsvd1: [u32; 3],
}
#[repr(transparent)]
pub struct NVencExternalMeHint(i32);
#[repr(C)]
pub struct NVencExternalMeSbHint {
flags1: i16,
flags2: i16,
flags3: i16,
}
#[repr(C)]
pub struct NVencConfigH264 {
bitflags: u32,
level: u32,
idr_period: u32,
seperate_color_plane_flag: u32,
disable_deblocking_filter_idc: u32,
num_termpoal_layers: u32,
sps_id: u32,
pps_id: u32,
adaptive_transform_mode: NVencH264AdaptiveTransformMode,
fmo_mode: NVencH264FMOMode,
bdirect_mode: NVencH264BDirectMode,
entropy_coding_mode: NVencH264EntropyCodingMode,
stereo_mode: NVencStereoPackingMode,
intra_refresh_period: u32,
intra_refresh_count: u32,
max_num_ref_frames: u32,
slice_mode: u32,
slice_mode_data: u32,
h264_vui_parameters: NVencConfigH264VUIParameters,
ltr_num_frames: u32,
ltr_trust_mode: u32,
chroma_format_idc: u32,
max_termpoal_layers: u32,
use_b_frames_as_ref: NVencBFrameRefMode,
num_ref_l0: NVencNumRefFrames,
num_ref_l1: NVencNumRefFrames,
output_bit_depth: NVencBitDepth,
input_bit_depth: NVencBitDepth,
tf_level: NVencTemporalFilterLevel,
rsvd1: [u32; 264],
rsvd: [*mut c_void; 64],
}
#[repr(C)]
pub struct NVencConfigHEVC {
level: u32,
tier: u32,
min_cu_size: NVencHEVCCuSize,
max_cu_size: NVencHEVCCuSize,
bitflags: u32,
idr_period: u32,
intra_refresh_cnt: u32,
max_num_ref_frames_in_dpb: u32,
ltr_num_frames: u32,
vps_id: u32,
sps_id: u32,
pps_id: u32,
slice_mode: u32,
slice_mode_data: u32,
max_temporal_layers_minus_1: u32,
hevc_vui_paramters: NVencConfigHEVCVUIParamaters,
ltr_trust_mode: u32,
use_b_frames_as_ref: NVencBFrameRefMode,
tf_level: NVencTemporalFilterLevel,
disable_deblocking_filter_idc: u32,
output_bit_depth: NVencBitDepth,
input_bit_depth: NVencBitDepth,
num_temporal_layers: u32,
num_views: u32,
rsvd1: [u32; 208],
rsvd2: [*mut c_void; 64],
}
pub const NV_MAX_TILE_COLS_AV1: usize = 64;
pub const NV_MAX_TILE_ROWS_AV1: usize = 64;
#[repr(C)]
pub struct NVencFilmGrainParamsAV1 {
bitfields: u32,
point_y_value: [u8; 14],
point_y_scaling: [u8; 14],
point_cb_value: [u8; 10],
point_cb_scaling: [u8; 10],
point_cr_value: [u8; 10],
point_cr_scaling: [u8; 10],
ar_coeffs_y_plus128: [u8; 24],
ar_coeffs_cb_plus128: [u8; 25],
ar_coeffs_cr_plus128: [u8; 25],
rsvd2: [u8; 2],
cb_mult: u8,
cb_luma_mult: u8,
cb_offset: u16,
cr_mult: u8,
cr_luma_mult: u8,
cr_offset: u16,
}
#[repr(C)]
pub struct NVencConfigAV1 {
level: u32,
tier: u32,
min_part_size: NVencAV1PartSize,
max_part_size: NVencAV1PartSize,
bitflags: u32,
idr_period: u32,
intra_refresh_period: u32,
intra_refresh_count: u32,
max_num_ref_frames_in_dpb: u32,
num_tiles_columns: u32,
num_tile_rows: u32,
rsvd2: u32,
tile_widths: *mut u32,
tile_heights: *mut u32,
max_temporal_layers_minus_1: u32,
color_primaries: NVencVUIColorPrimaries,
transfer_characteristics: NVencVUITransCharacteristics,
matrix_coefficients: NVencVUIMatrixCoeffs,
color_range: u32,
chroma_sample_position: u32,
use_b_frames_as_ref: NVencBFrameRefMode,
film_grain_params: NVencFilmGrainParamsAV1,
num_fwd_refs: NVencNumRefFrames,
num_bwd_refs: NVencNumRefFrames,
input_bit_depth: NVencBitDepth,
ltr_num_frames: u32,
num_temporal_layers: u32,
tf_level: NVencTemporalFilterLevel,
rsvd1: [u32; 230],
rsvd3: [*mut c_void; 62],
}
#[repr(C)]
pub struct NVencConfigH264MeOnly {
bitflags: u32,
rsvd1: [u32; 255],
rsvd2: [*mut c_void; 64],
}
#[repr(C)]
pub struct NVencConfigHEVCMeOnly {
rsvd: [u32; 256],
rsvd1: [*mut c_void; 64],
}
#[repr(C)]
pub union NVencCodecConfig {
h264: ManuallyDrop<NVencConfigH264>,
hevc: ManuallyDrop<NVencConfigHEVC>,
av1: ManuallyDrop<NVencConfigAV1>,
h264_meonly: ManuallyDrop<NVencConfigH264MeOnly>,
hevc_meonly: ManuallyDrop<NVencConfigHEVCMeOnly>,
rsvd: [u32; 320],
}
#[repr(C)]
pub struct NVencConfig {
pub version: u32,
pub profile_guid: Guid,
pub gop_len: u32,
pub frame_interval_p: i32,
mono_chrome_encoding: u32,
frame_field_mode: NVencParamsFrameFieldMode,
mv_precision: NVencMVPrecision,
pub rc_params: NVencRcParams,
encode_codec_config: NVencCodecConfig,
rsvd: [u32; 278],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_CONFIG_VER: u32 = struct_version(9) | (1 << 31);
#[bitfields::bitfield(u32)]
pub struct NVencInitializeParamsBitfields {
#[bits(1)]
pub report_slice_offsets: bool,
#[bits(1)]
pub enable_sub_frame_write: bool,
#[bits(1)]
pub enable_external_me_hints: bool,
#[bits(1)]
pub enable_me_only_mode: bool,
#[bits(1)]
pub enable_weighted_prediction: bool,
#[bits(4)]
pub split_encode_mode: u8,
#[bits(1)]
pub enable_output_in_vid_mem: bool,
#[bits(1)]
pub enable_recon_frame_output: bool,
#[bits(1)]
pub enable_output_stats: bool,
#[bits(1)]
pub enable_uni_directional_b: bool,
#[bits(19)]
pub rsvd: u32,
}
#[repr(C)]
pub struct NVencInitializeParams {
pub version: u32,
pub encode_guid: Guid,
pub preset_guid: Guid,
pub encode_width: u32,
pub encode_height: u32,
pub dar_width: u32,
pub dar_height: u32,
pub frame_rate_num: u32,
pub frame_rate_den: u32,
pub enable_encode_async: u32,
pub enable_ptd: u32,
pub bit_flags: NVencInitializeParamsBitfields,
pub(crate) priv_data_size: u32,
pub(crate) rsvd: u32,
pub(crate) priv_data: *mut c_void,
pub encode_config: *mut NVencConfig,
pub max_encode_width: u32,
pub max_encode_height: u32,
pub max_me_hint_counts_per_block: [NVencExternalMeHintCountsPerBlockType; 2],
pub tuning_info: NVencTuningInfo,
pub buffer_format: NVencBufferFormat,
pub num_state_buffers: u32,
pub output_stats_level: NVencOutputStatsLevel,
pub(crate) rsvd1: [u32; 284],
pub(crate) rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_INITIALIZE_PARAMS_VER: u32 = struct_version(7) | (1 << 31);
#[repr(C)]
pub struct NVencReconfigureParams {
version: u32,
rsvd: u32,
re_init_encode_params: NVencInitializeParams,
bitflags: u32,
rsvd2: u32,
}
pub const NV_ENC_RECONFIGURE_PARAMS_VER: u32 = struct_version(2) | (1 << 31);
#[repr(C)]
pub struct NVencPresetConfig {
pub version: u32,
rsvd: u32,
pub preset_cfg: NVencConfig,
rsvd1: [u32; 256],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_PRESET_CONFIG_VER: u32 = struct_version(5) | (1 << 31);
#[repr(C)]
pub struct NVencPicParamsMVC {
version: u32,
view_id: u32,
temporal_id: u32,
priority_id: u32,
rsvd1: [u32; 12],
rsvd2: [*mut c_void; 8],
}
pub const NV_ENC_PIC_PARAMS_MVC_VER: u32 = struct_version(1);
#[repr(C)]
pub union NVencPicParamsH264EXT {
pub mvc_pic_params: ManuallyDrop<NVencPicParamsMVC>,
pub rsvd1: [u32; 32],
}
#[repr(C)]
pub struct NVencSeiPayload {
pub payload_size: u32,
pub payload_type: u32,
pub payload: *mut u8,
}
pub type NVencH264SeiPayload = NVencSeiPayload;
#[repr(C)]
pub struct NVencPicParamsH264 {
pub display_poc_syntax: u32,
pub rsvd3: u32,
pub ref_pic_flag: u32,
pub color_plane_id: u32,
pub force_intra_refresh_with_frame_count: u32,
pub bitflags: u32,
pub slice_type_data: *mut u8,
pub slice_type_array_count: u32,
pub sei_payload_array_count: u32,
pub sei_payload_array: *mut NVencSeiPayload,
pub slice_mode: u32,
pub slice_mode_data: u32,
pub ltr_mark_framce_idx: u32,
pub ltr_use_frame_bitmap: u32,
pub ltr_usage_mode: u32,
pub force_intra_slice_count: u32,
pub force_intra_slice_idx: *mut u32,
pub h264_ext_pic_params: NVencPicParamsH264EXT,
pub time_code: NVencTimeCode,
pub rsvd: [u32; 202],
pub rsvd2: [*mut c_void; 61],
}
#[repr(C)]
pub struct NVencPicParamsHEVC {
pub display_poc_syntax: u32,
pub ref_pic_flag: u32,
pub temporal_id: u32,
pub force_intra_refresh_with_frame_count: u32,
pub bitflags: u32,
pub rsvd1: u32,
pub slice_type_data: *mut u8,
pub slice_type_array_count: u32,
pub slice_mode: u32,
pub slice_mode_data: u32,
pub ltr_mark_frame_idx: u32,
pub ltr_usage_frame_bitmap: u32,
pub ltr_usage_mode: u32,
pub sei_payload_array_count: u32,
pub rsvd: u32,
pub sei_payload_array: *mut NVencSeiPayload,
pub time_code: NVencTimeCode,
pub num_temporal_layers: u32,
pub view_id: u32,
pub _3d_reference_display_info: *mut HEVC3DReferenceDisplayInfo,
pub max_cll: *mut ContentLightLevel,
pub mastering_display: *mut MasteringDisplayInfo,
pub rsvd2: [u32; 234],
pub rsvd3: [*mut c_void; 58],
}
pub type NVencAV1OBUPayload = NVencSeiPayload;
#[repr(C)]
pub struct NVencPicParamsAV1 {
display_poc_syntax: u32,
ref_pic_flag: u32,
temporal_id: u32,
force_intra_refresh_with_frame_count: u32,
bitflags: u32,
num_tile_columns: u32,
num_tile_rows: u32,
rsvd: u32,
tile_widths: *mut u32,
tile_heights: *mut u32,
obu_payload_array_count: u32,
rsvd1: u32,
obu_payload_array: *mut NVencAV1OBUPayload,
file_grain_params: *mut NVencFilmGrainParamsAV1,
ltr_mark_frame_idx: u32,
ltr_use_frame_bitmap: u32,
num_temporal_layers: u32,
rsvd4: u32,
max_cll: *mut ContentLightLevel,
mastering_display: *mut MasteringDisplayInfo,
rsvd2: [u32; 242],
rsvd3: [*mut c_void; 59],
}
#[repr(C)]
pub union NVencCodecPicParams {
pub h264_pic_params: ManuallyDrop<NVencPicParamsH264>,
pub hevc_pic_params: ManuallyDrop<NVencPicParamsHEVC>,
av1_pic_params: ManuallyDrop<NVencPicParamsAV1>,
pub rsvd: [u32; 256],
}
#[repr(C)]
pub struct NVencPicParams {
pub version: u32,
pub input_width: u32,
pub input_height: u32,
pub input_pitch: u32,
pub encode_pic_flags: u32,
pub frame_idx: u32,
pub input_time_stamp: u64,
input_duration: u64,
pub input_buffer: *mut c_void,
pub output_bitstream: *mut c_void,
// TODO: Windows only, needed for async
completion_event: *mut c_void,
pub buffer_format: NVencBufferFormat,
pub picture_struct: NVencPicStruct,
pub picture_type: NVencPicType,
pub codec_pic_params: NVencCodecPicParams,
me_hint_counts_per_block: [NVencExternalMeHintCountsPerBlockType; 2],
me_external_hints: *mut NVencExternalMeHint,
rsvd2: [u32; 7],
rsvd5: [*mut c_void; 2],
qp_delta_map: *mut i8,
qp_deltra_map_size: u32,
rsvd_bit_fields: u32,
me_hint_ref_pic_dist: [u16; 2],
rsvd4: u32,
alpha_buffer: *mut c_void,
me_external_sb_hints: *mut NVencExternalMeSbHint,
me_sb_hints_count: u32,
state_buffer_idx: u32,
output_recon_buffer: *mut c_void,
rsvd3: [u32; 284],
rsvd6: [*mut c_void; 57],
}
pub const NV_ENC_PIC_PARAMS_VER: u32 = struct_version(7) | (1 << 31);
#[repr(C)]
pub struct NVencMeOnlyParams {
version: u32,
input_width: u32,
input_height: u32,
rsvd: u32,
input_buffer: *mut c_void,
reference_frame: *mut c_void,
mv_buffer: *mut c_void,
rsvd2: u32,
buffer_formtat: NVencBufferFormat,
view_id: u32,
me_hint_counts_per_block: [NVencExternalMeHintCountsPerBlockType; 2],
me_external_hints: NVencExternalMeHint,
rsvd1: [u32; 241],
rsvd3: [*mut c_void; 59],
}
pub const NV_ENC_MEONLY_PARAMS_VER: u32 = struct_version(4);
#[bitfields::bitfield(u32)]
pub struct NVencLockBitStreamBitFields {
#[bits(1)]
pub do_not_wait: bool,
#[bits(1)]
pub ltr_frame: bool,
#[bits(1)]
pub get_rc_stats: bool,
#[bits(29)]
rsvd: u32,
}
#[repr(C)]
pub struct NVencLockBitStream {
pub version: u32,
pub bit_fields: NVencLockBitStreamBitFields,
pub output_bit_stream: *mut c_void,
slice_offsets: *mut u32,
frame_idx: u32,
hw_encode_status: u32,
num_slices: u32,
pub bitstream_size_in_bytes: u32,
output_time_stamp: u64,
pub output_duration: u64,
pub bitstream_buffer: *mut c_void,
picture_type: NVencPicType,
picture_struct: NVencPicStruct,
framge_avg_qp: u32,
frame_satd: u32,
ltr_frame_idx: u32,
ltr_frame_bitmap: u32,
temporal_id: u32,
inter_mb_count: u32,
average_mvx: i32,
average_mvy: i32,
alpha_layer_size_in_bytes: u32,
output_stats_ptr_size: u32,
rsvd: u32,
output_stats_ptr: *mut c_void,
frame_idx_display: u32,
rsvd1: [u32; 219],
rsvd2: [*mut c_void; 63],
rsvd_interanl: [u32; 8],
}
pub const NV_ENC_LOCK_BITSTREAM_VER: u32 = struct_version(2) | (1 << 31);
#[repr(C)]
pub struct NVencLockInputBuffer {
pub version: u32,
pub bitflags: u32,
pub input_buffer: *mut c_void,
pub buffer_data_ptr: *mut c_void,
pub pitch: u32,
pub rsvd1: [u32; 251],
pub rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_LOCK_INPUT_BUFFER_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencMapInputResource {
pub version: u32,
sub_resource_idx: u32,
input_resource: *mut c_void,
pub registered_resource: *mut c_void,
pub mapped_resource: *mut c_void,
pub mapped_buffer_format: NVencBufferFormat,
rsvd1: [u32; 251],
rsvd: [*mut c_void; 63],
}
pub const NV_ENC_MAP_INPUT_RESOURCE_VER: u32 = struct_version(4);
#[repr(C)]
pub struct NVencInputResourceOpenGLTex {
texture: u32,
target: u32,
}
#[repr(C)]
pub struct NVencFencePointD3D12 {
version: u32,
rsvd: u32,
p_fence: *mut c_void,
wait_value: u64,
signal_value: u64,
bit_flags: u32,
rsvd1: [u32; 7],
}
pub const NV_ENC_FENCE_POINT_D3D12_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencInputResourceD3D12 {
version: u32,
rsvd: u32,
input_buffer: *mut c_void,
input_fence_point: NVencFencePointD3D12,
rsvd1: [u32; 16],
rsvd2: [*mut c_void; 16],
}
pub const NV_ENC_INPUT_RESOURCE_D3D12_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencOutputResourceD3D12 {
version: u32,
rsvd: u32,
output_buffer: *mut c_void,
output_fence_point: NVencFencePointD3D12,
rsvd1: [u32; 16],
rsvd2: [*mut c_void; 16],
}
pub const NV_ENC_OUTPUT_RESOURCE_D3D12_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencRegisterResource {
pub version: u32,
pub resource_type: NVencInputResourceType,
pub width: u32,
pub height: u32,
pub pitch: u32,
pub sub_resource_index: u32,
pub resource_to_register: *mut c_void,
pub registered_resource: *mut c_void,
pub buffer_format: NVencBufferFormat,
pub buffer_usage: NVencBufferUsage,
pub input_fence_point: *mut NVencFencePointD3D12,
pub chroma_offset: [u32; 2],
pub chroma_offset_in: [u32; 2],
pub(crate) rsvd1: [u32; 244],
pub(crate) rsvd2: [*mut c_void; 61],
}
pub const NV_ENC_REGISTER_RESOURCE_VER: u32 = struct_version(5);
#[repr(C)]
pub struct NVencStat {
version: u32,
rsvd: u32,
output_bit_stream: *mut c_void,
bit_stream_size: u32,
pic_type: u32,
last_valid_byte_offset: u32,
slice_offsets: [u32; 16],
pic_idx: u32,
frame_avg_qp: u32,
bit_flags: u32,
ltr_frame_idx: u32,
intra_mb_count: u32,
inter_mb_count: u32,
average_mv_x: i32,
average_mv_y: i32,
rsvd1: [u32; 227],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_STAT_VER: u32 = struct_version(2);
#[repr(C)]
pub struct NVencSequenceParamPayload {
version: u32,
in_buffer_size: u32,
sps_id: u32,
pps_id: u32,
spspps_buffer: *mut c_void,
output_spspps_payload_size: *mut u32,
rsvd: [u32; 250],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_SEQUENCE_PARAM_PAYLOAD_VER: u32 = struct_version(1);
#[repr(C)]
pub struct NVencEventParams {
version: u32,
rsvd: u32,
completion_event: *mut c_void,
rsvd1: [u32; 254],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_EVENT_PARAMS_VER: u32 = struct_version(2);
#[repr(C)]
pub struct NVencOpenEncodeSessionExParams {
pub version: u32,
pub device_type: NVencDeviceType,
pub device: *mut c_void,
rsvd: *mut c_void,
pub api_version: u32,
rsvd1: [u32; 253],
rsvd2: [*mut c_void; 64],
}
pub const NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER: u32 = struct_version(1);
vendor/nvenc/src/sys/version.rs
Vendored
+12
View File
@@ -0,0 +1,12 @@
//! Contains the nvenc version numbers, encoded as they are in C, for the version this crate was built against
pub const NVENC_MAJOR_VERSION: u16 = 13;
pub const NVENC_MINOR_VERSION: u8 = 0;
pub const NVENC_API_VERSION: u32 = NVENC_MAJOR_VERSION as u32 | (NVENC_MINOR_VERSION as u32) << 24;
pub const NVENC_API_LIST_VERSION: u32 = struct_version(2);
pub const fn struct_version(nvenc_api_version: u8) -> u32 {
NVENC_API_VERSION | (nvenc_api_version as u32) << 16 | 0x7 << 28
}
viewer-core/Cargo.toml
+22
View File
@@ -0,0 +1,22 @@
[package]
name = "viewer-core"
version = "0.1.0"
edition = "2021"
# 阶段 1 先做纯 Rust 库并在桌面 headless 验证；
# UniFFI / cdylib 在下一步（Android 接入）再加，隔离未知项。
[lib]
name = "viewer_core"
crate-type = ["lib"]
[dependencies]
common = { path = "../common" }
anyhow.workspace = true
tokio.workspace = true
quinn.workspace = true
rustls.workspace = true
[dev-dependencies]
# headless_check 用软解交叉验证收到的码流合法（库本身刻意不依赖 openh264，
# 解码在 Android 侧由 MediaCodec 承担）
openh264 = { version = "0.9.9", features = ["source"] }
viewer-core/examples/headless_check.rs
+149
View File
@@ -0,0 +1,149 @@
//! viewer-core 的桌面 headless 验证工具（阶段 1 验收用，不依赖 Android）。
//!
//! 证明三件事，逐条打印：
//! 1. viewer-core 能完成 QUIC 连接 + 协议 v2 认证握手；
//! 2. 帧流按契约交付（seq 单调递增、keyframe 标志与码流实际 NAL 一致）；
//! 3. 收到的码流能被 openh264 解出画面（即：Android 侧 MediaCodec 将拿到同样
//! 合法的 Annex-B——解码器不同，但码流合法性在这里就证明了）。
//!
//! 用法：
//! 终端1: target\release\agent.exe --port 47900 --code t
//! 终端2: cargo run -p viewer-core --bin headless_check -- --port 47900 --code t --frames 60
use anyhow::Result;
use std::time::Duration;
use viewer_core::{ConnectConfig, Session};
fn arg(name: &str) -> Option<String> {
let a: Vec<String> = std::env::args().collect();
a.iter()
.position(|x| x == name)
.and_then(|i| a.get(i + 1))
.cloned()
}
/// 与 dump_stream 同款：扫 Annex-B 起始码拿 NAL 类型，交叉校验 keyframe 标志
fn has_idr(payload: &[u8]) -> bool {
let mut i = 0usize;
while i + 3 < payload.len() {
let sc = if payload[i..].starts_with(&[0, 0, 0, 1]) {
4
} else if payload[i..].starts_with(&[0, 0, 1]) {
3
} else {
i += 1;
continue;
};
match payload.get(i + sc) {
Some(&b) if b & 0x1F == 5 => return true,
Some(_) => i += sc + 1,
None => break,
}
}
false
}
#[tokio::main]
async fn main() -> Result<()> {
let port: u16 = arg("--port")
.map(|s| s.parse())
.transpose()?
.unwrap_or(47800);
let code = arg("--code").unwrap_or_else(|| "123456".into());
// --host：默认 127.0.0.1；填局域网 IP 可顺带验证防火墙入站路径
let host = arg("--host").unwrap_or_else(|| "127.0.0.1".into());
let frames_wanted: u64 = arg("--frames")
.map(|s| s.parse())
.transpose()?
.unwrap_or(60);
let mut session = Session::connect(ConnectConfig::direct(&host, port, code)).await?;
println!("[check] 连接成功 host={host} peer={}", session.peer_addr());
let mut decoder = openh264::decoder::Decoder::new()?;
let mut last_seq: Option<u64> = None;
let mut got = 0u64;
let mut seq_violations = 0u64;
let mut kf_mismatches = 0u64;
let mut decoded = 0u64;
let mut decode_failed_before_idr = 0u64;
let mut awaiting_idr = true; // 与 Android 侧相同的状态机
let mut rtt_shown = false;
let deadline = std::time::Instant::now() + Duration::from_secs(30);
while got < frames_wanted && std::time::Instant::now() < deadline {
let Some(f) = tokio::time::timeout(Duration::from_secs(10), session.next_frame())
.await
.ok()
.flatten()
else {
break;
};
got += 1;
// 契约 2a：seq 单调递增
if let Some(prev) = last_seq {
if f.seq <= prev {
seq_violations += 1;
println!("[check] ⚠️ seq 回退: {prev} -> {}", f.seq);
}
}
last_seq = Some(f.seq);
// 契约 2b：keyframe 标志与码流一致
let real_kf = has_idr(&f.payload);
if f.keyframe != real_kf {
kf_mismatches += 1;
println!(
"[check] ⚠️ 帧头 keyframe={} 但码流实际 IDR={}（seq={}）",
f.keyframe, real_kf, f.seq
);
}
// 契约 3：与 Android 相同的状态机下解码必须成功
if awaiting_idr && !f.keyframe {
continue; // 等 IDR，P 帧不喂（这是 R-3 教训的实现）
}
if f.keyframe {
awaiting_idr = false;
}
match decoder.decode(&f.payload) {
Ok(_) => decoded += 1,
Err(e) => {
decode_failed_before_idr += 1;
println!("[check] ❌ seq={} 解码失败: {e}", f.seq);
awaiting_idr = true;
session.request_keyframe();
}
}
// RTT 出现即记录一次
if !rtt_shown {
let s = session.stats();
if s.rtt_ms > 0 {
println!("[check] RTT={}ms frames_in={} bytes_in={}", s.rtt_ms, s.frames_in, s.bytes_in);
rtt_shown = true;
}
}
}
let s = session.stats();
println!("\n========== 阶段 1 headless 验收 ==========");
println!("收到帧: {got}（目标 {frames_wanted}）");
println!("seq 违例: {seq_violations}");
println!("keyframe 标志不符: {kf_mismatches}");
println!("解码成功: {decoded} / 解码失败: {decode_failed_before_idr}");
println!("统计: rtt={}ms frames_in={} bytes_in={}", s.rtt_ms, s.frames_in, s.bytes_in);
let ok = got >= frames_wanted.min(30)
&& seq_violations == 0
&& kf_mismatches == 0
&& decoded > 0
&& decode_failed_before_idr == 0;
println!(
"结论: {}",
if ok { "✅ viewer-core 收帧链路通过" } else { "❌ 存在未满足项" }
);
session.close().await;
Ok(())
}
viewer-core/src/lib.rs
+431
View File
@@ -0,0 +1,431 @@
//! viewer-core：安卓主控的连接 / 协议 / 收帧层（纯 Rust，平台无关）。
//!
//! 设计边界（对齐《多平台架构设计》与 2026-10-29 方案裁决）：
//! - **只做「怎么连 + 收发什么」**：QUIC 会话、认证握手、视频帧流、输入上行、
//! Ping/Pong、KeyFrameReq。解码（MediaCodec）、渲染（SurfaceView）、触摸映射
//! 全在 Kotlin 侧，本 crate 不碰像素，也不引入 openh264。
//! - **连接层是唯一的可替换点**：`PeerTarget::Direct` 现在只支持 `IP:端口` 局域网
//! 直连；将来设备码 + 信令 + 打洞只换 `connect()` 内部。**不定义 Relay 变体**
//! （中继已被团队裁决排除，见《多平台架构设计》§6.1）。
//! - 帧经 mpsc 通道交付，**不丢帧**：消费方（Kotlin）按 `keyframe` 标志自行决定
//! 跳帧策略，且跳帧后必须 `request_keyframe()` 并等 IDR（协议 v2 提供该字段）。
//! 丢帧决策放在这里会让 Rust 层承担它看不到的解码器状态。
//!
//! 安全现状：`insecure_skip_verify` 默认 true（与桌面 viewer 的 SkipVerify 一致，
//! 仅限局域网测试）。产品化前必须换成信令下发的证书指纹钉扎（`signaling.rs`
//! 已有 `CertFingerprint`），该项登记在方案阶段 4。
use anyhow::{bail, Context, Result};
use common::{
read_control, read_video_frame, write_control, ControlMsg, InputEvent, VideoFrameHeader,
PROTO_VERSION,
};
use rustls::client::danger::{
HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier,
};
use rustls::pki_types::{CertificateDer, ServerName, UnixTime};
use rustls::{DigitallySignedStruct, SignatureScheme};
use std::net::ToSocketAddrs;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::sync::mpsc;
/// 连接目标。当前只有局域网直连；设备码 + 打洞在后续阶段替换 `connect()` 内部实现。
#[derive(Debug, Clone)]
pub enum PeerTarget {
/// 局域网 / 可直连地址
Direct { host: String, port: u16 },
}
/// 一次会话的参数
#[derive(Debug, Clone)]
pub struct ConnectConfig {
pub target: PeerTarget,
/// 被控端验证码
pub code: String,
/// 仅测试用：跳过服务端证书校验（与桌面 viewer 行为一致）。
/// 公网/产品形态必须为 false 并启用证书钉扎。
pub insecure_skip_verify: bool,
}
impl ConnectConfig {
/// 局域网测试的便捷构造
pub fn direct(host: impl Into<String>, port: u16, code: impl Into<String>) -> Self {
Self {
target: PeerTarget::Direct {
host: host.into(),
port,
},
code: code.into(),
insecure_skip_verify: true,
}
}
}
/// 交付给消费方的一帧视频。字段语义与 `common::VideoFrameHeader` 一致，
/// payload 为标准 Annex-B H.264（可直接喂 MediaCodec）。
#[derive(Debug, Clone)]
pub struct Frame {
pub width: u32,
pub height: u32,
pub ts_ms: u64,
pub seq: u64,
pub keyframe: bool,
pub payload: Vec<u8>,
}
impl From<(VideoFrameHeader, Vec<u8>)> for Frame {
fn from((h, payload): (VideoFrameHeader, Vec<u8>)) -> Self {
Self {
width: h.width,
height: h.height,
ts_ms: h.ts_ms,
seq: h.seq,
keyframe: h.keyframe,
payload,
}
}
}
/// 会话状态快照（供 UI 显示，不做 EMA——那属于 Kotlin 的 HUD 层）
#[derive(Debug, Clone, Copy, Default)]
pub struct SessionStats {
/// 最近一次 Ping 往返（ms），0 = 还没测出
pub rtt_ms: u64,
/// 收到的帧数
pub frames_in: u64,
/// 收到的字节数（视频载荷累计）
pub bytes_in: u64,
}
/// MVP：跳过服务端证书校验（自签证书，仅局域网）。
#[derive(Debug)]
struct SkipVerify;
impl ServerCertVerifier for SkipVerify {
fn verify_server_cert(
&self,
_end_entity: &CertificateDer<'_>,
_intermediates: &[CertificateDer<'_>],
_server_name: &ServerName<'_>,
_ocsp_response: &[u8],
_now: UnixTime,
) -> std::result::Result<ServerCertVerified, rustls::Error> {
Ok(ServerCertVerified::assertion())
}
fn verify_tls12_signature(
&self,
_message: &[u8],
_certificate: &CertificateDer<'_>,
_dss: &DigitallySignedStruct,
) -> std::result::Result<HandshakeSignatureValid, rustls::Error> {
Ok(HandshakeSignatureValid::assertion())
}
fn verify_tls13_signature(
&self,
_message: &[u8],
_certificate: &CertificateDer<'_>,
_dss: &DigitallySignedStruct,
) -> std::result::Result<HandshakeSignatureValid, rustls::Error> {
Ok(HandshakeSignatureValid::assertion())
}
fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
vec![
SignatureScheme::RSA_PKCS1_SHA256,
SignatureScheme::ECDSA_NISTP256_SHA256,
SignatureScheme::ED25519,
SignatureScheme::RSA_PSS_SHA256,
]
}
}
/// 输入/控制命令（写入流 2）
enum Command {
Input(InputEvent),
KeyFrameReq,
Ping(u64),
}
/// 已建立的会话。Drop 时后台任务随连接关闭自然退出；显式 `close()` 更干净。
pub struct Session {
tx: mpsc::UnboundedSender<Command>,
/// 帧接收端。通道关闭 = 会话结束（被控端断开或流错误）。
/// Mutex 包一层是为了让 `next_frame` 取 `&self`：JNI 层里解码线程独占 recv，
/// 输入线程并发调 `send_input`（走无锁的 unbounded tx），两者互不阻塞。
frames: Arc<tokio::sync::Mutex<mpsc::Receiver<Frame>>>,
stats: Arc<std::sync::Mutex<SessionStats>>,
/// 被控端桌面尺寸（AuthOk 携带）。主控端据此在解码前定好视图宽高比——
/// 若等首帧再改尺寸会触发 SurfaceView 重新测量，把刚创建的解码器 surface 打掉。
desktop: (u32, u32),
/// 保持 Endpoint 存活（drop 会销毁连接）
_endpoint: quinn::Endpoint,
conn: quinn::Connection,
}
impl Session {
/// 建立连接并完成认证握手 + 首帧关键帧请求。
pub async fn connect(cfg: ConnectConfig) -> Result<Self> {
rustls::crypto::ring::default_provider()
.install_default()
.ok();
let tls = if cfg.insecure_skip_verify {
rustls::ClientConfig::builder()
.dangerous()
.with_custom_certificate_verifier(Arc::new(SkipVerify))
.with_no_client_auth()
} else {
// 证书钉扎尚未接入（方案阶段 4）。这里明确报错而不是静默放行。
bail!("insecure_skip_verify=false 需要证书钉扎，尚未实现（见方案阶段 4）");
};
let qc: quinn::crypto::rustls::QuicClientConfig = Arc::new(tls)
.try_into()
.context("build QUIC client config")?;
// 接收窗口放大：视频下行大流量，quinn 默认 512KB 流窗口会卡吞吐
// （与桌面 viewer / agent 的实测调优一致）
let mut transport = quinn::TransportConfig::default();
transport.receive_window(quinn::VarInt::from_u32(16 * 1024 * 1024));
transport.stream_receive_window(quinn::VarInt::from_u32(8 * 1024 * 1024));
let mut client_config = quinn::ClientConfig::new(Arc::new(qc));
client_config.transport_config(Arc::new(transport));
let mut endpoint = quinn::Endpoint::client("0.0.0.0:0".parse()?)?;
endpoint.set_default_client_config(client_config);
let PeerTarget::Direct { host, port } = &cfg.target;
let addr = (host.as_str(), *port)
.to_socket_addrs()?
.next()
.with_context(|| format!("resolve {host}:{port}"))?;
let conn = endpoint
.connect(addr, "localhost")?
.await
.with_context(|| format!("QUIC connect {addr}"))?;
// ---- 流 1：认证 + 视频下行 ----
let (mut v_send, mut v_recv) = conn.open_bi().await?;
write_control(
&mut v_send,
&ControlMsg::Auth {
code: cfg.code.clone(),
proto_version: PROTO_VERSION,
},
)
.await
.context("send Auth")?;
let desktop: (u32, u32);
match read_control(&mut v_recv).await.context("read AuthOk")? {
ControlMsg::AuthOk {
width,
height,
proto_version,
} => {
if proto_version != PROTO_VERSION {
bail!(
"protocol mismatch: agent={proto_version}, viewer-core={PROTO_VERSION}"
);
}
desktop = (width, height);
log(&format!("auth ok, remote desktop {width}x{height}"));
}
ControlMsg::AuthFail(e) => bail!("auth failed: {e}"),
m => bail!("unexpected first message: {m:?}"),
}
// ---- 流 2：输入 + 控制 + Ping/Pong ----
let (mut ctl_send, mut ctl_recv) = conn.open_bi().await?;
// 入会即请求 IDR：从 GOP 中间接入时裸 P 帧解不出来（桌面端同款教训）
write_control(&mut ctl_send, &ControlMsg::KeyFrameReq).await?;
let stats = Arc::new(std::sync::Mutex::new(SessionStats::default()));
// 帧通道：容量 2 足够——消费方（Kotlin）每帧都要喂解码器，
// 积压超过 2 帧说明该跳帧的是消费方的决策，不是这里的。
let (frame_tx, frames) = mpsc::channel::<Frame>(2);
{
let stats = stats.clone();
tokio::spawn(async move {
loop {
match read_video_frame(&mut v_recv).await {
Ok((h, payload)) => {
{
let mut s = stats.lock().unwrap();
s.frames_in += 1;
s.bytes_in += payload.len() as u64;
}
let frame: Frame = (h, payload).into();
if frame_tx.send(frame).await.is_err() {
break; // 消费方已退出
}
}
Err(e) => {
log(&format!("video stream ended: {e}"));
break;
}
}
}
});
}
// 命令通道 → 流 2 写任务
let (cmd_tx, mut cmd_rx) = mpsc::unbounded_channel::<Command>();
tokio::spawn(async move {
loop {
let Some(cmd) = cmd_rx.recv().await else {
break;
};
let msg = match cmd {
Command::Input(ev) => ControlMsg::Input(ev),
Command::KeyFrameReq => ControlMsg::KeyFrameReq,
Command::Ping(t) => ControlMsg::Ping(t),
};
if write_control(&mut ctl_send, &msg).await.is_err() {
break;
}
}
});
// 流 2 读任务：Pong → RTT
{
let stats = stats.clone();
tokio::spawn(async move {
loop {
match read_control(&mut ctl_recv).await {
Ok(ControlMsg::Pong { t, .. }) => {
let rtt = monotonic_ms().saturating_sub(t);
stats.lock().unwrap().rtt_ms = rtt;
}
Ok(_) => {}
Err(e) => {
log(&format!("control stream ended: {e}"));
break;
}
}
}
});
}
// Ping 任务：500ms 一次（与桌面 viewer 一致）
{
let cmd_tx_ping = cmd_tx.clone();
tokio::spawn(async move {
loop {
tokio::time::sleep(Duration::from_millis(500)).await;
if cmd_tx_ping.send(Command::Ping(monotonic_ms())).is_err() {
break;
}
}
});
}
Ok(Self {
tx: cmd_tx,
frames: Arc::new(tokio::sync::Mutex::new(frames)),
stats,
desktop,
_endpoint: endpoint,
conn,
})
}
/// 被控端桌面尺寸 `(width, height)`，来自 AuthOk，解码前即可用。
pub fn desktop_size(&self) -> (u32, u32) {
self.desktop
}
/// 取下一帧。`None` = 会话结束。
/// `&self`：JNI 层里解码线程独占 recv，输入线程可并发 `send_input`。
pub async fn next_frame(&self) -> Option<Frame> {
self.frames.lock().await.recv().await
}
/// 发送一个输入事件（非阻塞入队，由写任务串行发出）。
pub fn send_input(&self, ev: InputEvent) {
let _ = self.tx.send(Command::Input(ev));
}
/// 请求被控端强制关键帧（解码器错误/跳帧后的恢复路径）。
pub fn request_keyframe(&self) {
let _ = self.tx.send(Command::KeyFrameReq);
}
/// 当前会话统计快照。
pub fn stats(&self) -> SessionStats {
*self.stats.lock().unwrap()
}
/// 被控端地址（日志/UI 用）
pub fn peer_addr(&self) -> String {
self.conn.remote_address().to_string()
}
/// 主动结束会话。
pub async fn close(self) {
let _ = self.conn.close(0u32.into(), b"viewer closing");
}
}
/// 单调毫秒时钟（进程启动起算）。Ping/Pong 两端共用同一口径即可测 RTT。
fn monotonic_ms() -> u64 {
use std::sync::OnceLock;
static T0: OnceLock<Instant> = OnceLock::new();
T0.get_or_init(Instant::now).elapsed().as_millis() as u64
}
/// 轻量日志出口。阶段 2 接 UniFFI 时替换为 Kotlin 侧回调。
fn log(msg: &str) {
eprintln!("[viewer-core] {msg}");
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn frame_from_header_preserves_contract_fields() {
let h = VideoFrameHeader {
width: 1920,
height: 1080,
ts_ms: 42,
seq: 7,
keyframe: true,
};
let payload = vec![0u8; 16];
let f: Frame = (h.clone(), payload.clone()).into();
assert_eq!((f.width, f.height), (1920, 1080));
assert_eq!(f.seq, 7);
assert!(f.keyframe);
assert_eq!(f.payload, payload);
}
#[test]
fn direct_config_defaults_to_insecure_lan_mode() {
let c = ConnectConfig::direct("192.168.1.5", 47800, "abc");
assert!(c.insecure_skip_verify);
match c.target {
PeerTarget::Direct { host, port } => {
assert_eq!(host, "192.168.1.5");
assert_eq!(port, 47800);
}
}
}
#[tokio::test]
async fn strict_verify_mode_is_explicit_unimplemented_error() {
// 安全红线不能静默：显式要求校验时必须是明确错误，而不是悄悄放行。
// 不用 unwrap_err()：它要求 Ok 类型（Session）实现 Debug，而连接对象
// 的 Debug 输出没有验证价值。
let cfg = ConnectConfig {
target: PeerTarget::Direct {
host: "127.0.0.1".into(),
port: 1,
},
code: "x".into(),
insecure_skip_verify: false,
};
match Session::connect(cfg).await {
Err(e) => assert!(e.to_string().contains("证书钉扎"), "实际错误: {e}"),
Ok(_) => panic!("insecure_skip_verify=false 必须明确报错，不得静默放行"),
}
}
}
viewer/src/main.rs
+69 -4
View File
@@ -110,6 +110,8 @@ struct Stats {
agent_sent: AtomicU64,
/// H.264 解码失败计数（关键帧自愈验证指标）
dec_fail: AtomicU64,
/// 当前是否处于「参考链断裂、等待 IDR」状态（HUD 显示 RECOVER）
awaiting_idr: AtomicBool,
/// 帧到达间隔环形缓冲
gaps: std::sync::Mutex<GapRing>,
}
@@ -129,6 +131,7 @@ impl Default for Stats {
agent_lost: AtomicU64::new(0),
agent_sent: AtomicU64::new(0),
dec_fail: AtomicU64::new(0),
awaiting_idr: AtomicBool::new(true),
gaps: std::sync::Mutex::new(GapRing {
last: None,
ring: [0; 64],
@@ -218,9 +221,27 @@ async fn run(addr: String, port: u16, code: String) -> Result<()> {
let (mut send, mut recv) = conn.open_bi().await?;
write_control(&mut send, &ControlMsg::Auth { code }).await?;
write_control(
&mut send,
&ControlMsg::Auth {
code,
proto_version: common::PROTO_VERSION,
},
)
.await?;
match read_control(&mut recv).await? {
ControlMsg::AuthOk { width, height } => {
ControlMsg::AuthOk {
width,
height,
proto_version,
} => {
// 版本已在 Auth 里声明，这里再核对一次：防止被控端是旧版却"接受"了认证
if proto_version != common::PROTO_VERSION {
bail!(
"protocol version mismatch: server={proto_version}, client={}",
common::PROTO_VERSION
);
}
println!("[viewer] auth ok, remote desktop {width}x{height}");
// 流 2：键鼠输入 + Ping/Pong 独立通道，不与视频流竞争
let (isend, irecv) = conn.open_bi().await?;
@@ -257,6 +278,15 @@ fn session(
let mut last_kfreq = Instant::now();
let mut skip_stale = 0u64;
let mut last_arr: Option<Instant> = None;
// 「等待 IDR」状态：参考链一旦断裂（跳帧或解码失败），后续 P 帧**必然**
// 解不出来。此时必须停止喂解码器，只等 IDR 重新同步。
//
// 之前没有这个状态，后果是：跳掉一帧后，每个 P 帧都被送进 decoder →
// 每个都失败 → 每个都打日志 → 每个都触发一次 KeyFrameReq。
// 表现为「decode failed」刷屏，且解码器反复在错误状态里挣扎。
//
// 放在 Stats 里（而非局部变量）以便 HUD 显示 RECOVER 状态。
// 初始 true：入会时已发 KeyFrameReq，拿到第一个 IDR 前不该硬解。
tokio::spawn(async move {
let mut size_logged = false;
loop {
@@ -304,7 +334,10 @@ fn session(
"[viewer] backlog: stale burst frame skipped (total {skip_stale})"
);
}
// 限频请求关键帧：积压排空后从 IDR 重新衔接画面
// 跳帧直接切断 H.264 参考链：之后所有 P 帧都解不出来。
// 立刻转入「等待 IDR」并请求关键帧，绝不能让后续 P 帧
// 继续喂给解码器（那是必然失败的徒劳尝试）。
stats.awaiting_idr.store(true, Ordering::Relaxed);
if last_kfreq.elapsed() >= Duration::from_millis(200) {
last_kfreq = Instant::now();
if let Ok(mut slot) = latest.lock() {
@@ -313,6 +346,22 @@ fn session(
}
continue;
}
// 参考链未恢复时，P 帧一律跳过——此为正确性要求，不是优化。
if stats.awaiting_idr.load(Ordering::Relaxed) && !hdr.keyframe {
skip_stale += 1;
// 补发请求，防止首次 KeyFrameReq 丢失（时间闸限频）
if last_kfreq.elapsed() >= Duration::from_millis(200) {
last_kfreq = Instant::now();
if let Ok(mut slot) = latest.lock() {
*slot = Some((0, 0, Vec::new()));
}
}
continue;
}
// 拿到 IDR 才能重新开始解码
if hdr.keyframe {
stats.awaiting_idr.store(false, Ordering::Relaxed);
}
match decoder.decode(&payload) {
Ok(Some(yuv)) => {
let (dw, dh) = yuv.dimensions();
@@ -334,7 +383,8 @@ fn session(
}
Ok(None) => {}
Err(e) => {
// 解码异常：请求被控端强制关键帧
// 解码异常：参考链已不可信，转入等待 IDR
stats.awaiting_idr.store(true, Ordering::Relaxed);
stats.dec_fail.fetch_add(1, Ordering::Relaxed);
eprintln!("[viewer] decode failed: {e}, request keyframe");
if let Ok(mut slot) = latest.lock() {
@@ -577,6 +627,21 @@ fn session(
("P95", &hud_p95.map(|p| format!("{p}ms")).unwrap_or("--".into()), p95_c),
("LOSS", &loss_str, loss_c),
("DECFAIL", &format!("{hud_dec}"), if hud_dec == 0 { C_GRAY } else { C_RED }),
(
// 参考链断裂、正在等 IDR 的显式指示：此时画面冻结是**预期行为**，
// 不是故障。没有这个指示，用户会以为软件卡死。
"STATE",
&if stats.awaiting_idr.load(Ordering::Relaxed) {
"RECOVER".to_string()
} else {
"OK".to_string()
},
if stats.awaiting_idr.load(Ordering::Relaxed) {
C_AMBER
} else {
C_GREEN
},
),
("RATE", &format!("{:.0}MBP", hud_mbps), C_GRAY),
("RES", &format!("{w}x{h}"), C_GRAY),
],
多平台架构设计.md
+53 -8
View File
@@ -189,7 +189,9 @@ pub enum Path { Direct { addr: SocketAddr }, Relay { endpoint: EndpointId } }
---
## 6. ⚠️ 必须现在拍板：手机主控 vs 不做中继
## 6. ✅ 已裁决：不做中继（2026-10-28）
> **裁决结论**：公司不提供中继服务器，中继不做。以下保留原始分析以免后人重蹈讨论。
**手机在蜂窝网络下几乎 100% 处于运营商级 NAT（CGNAT），打洞打不通。**
这不是实现问题，是网络事实——`100.64.0.0/10` 那段地址根本没给你可映射的公网端口。
@@ -197,16 +199,56 @@ pub enum Path { Direct { addr: SocketAddr }, Relay { endpoint: EndpointId } }
后果：**"主控端支持安卓/iOS" + "不做中继" = 手机只能在 WiFi 下用。**
地铁上、外面、用流量，一律连不上。
三条出路，必须选一条：
三条出路：
| 方案 | 代价 | 说明 |
|---|---|---|
| **A. 只靠 IPv6** | 部分用户不可用 | 国内三大运营商 v6 覆盖率高，手机 v6 可用性远好于 v4 公网。**推荐先做，能覆盖大多数** |
| **B. 补一个只为移动端服务的中继** | 视频带宽成本 | 可用率 100%，但账单跟着视频流量走 |
| **C. 接受"手机只能 WiFi"** | 产品力受限 | 零成本，但要写进产品说明，别让用户以为坏了 |
| **A. 只靠 IPv6** | 部分用户不可用 | 国内三大运营商 v6 覆盖率高，手机 v6 可用性远好于 v4 公网 |
| B. 补一个只为移动端服务的中继 | 视频带宽成本 | **已排除：公司不提供额外服务器** |
| C. 接受"手机只能 WiFi" | 产品力受限 | 零成本，但要写进产品说明 |
**我的建议：A 先做（架构里 v6 是一等公民），B 的接口留在 `PathProvider` 里不给实现，等真要推移动端再决定。**
不要现在就把路堵死——留接口的成本几乎为零，推倒重来成本极高。
### 6.1 裁决内容与边界（2026-10-28）
| 项 | 裁决 |
|---|---|
| 中继 / TURN / relay 服务器 | **不做**。公司不提供额外服务器，且无预算计划 |
| IPv6 | **一等公民**。候选地址排序 v6 优先，`signaling.rs` 已实现 |
| 中继接口预留 | **不预留**。不写 `PathProvider` trait、不写 `Path::Relay` 变体 |
| 连接层边界 | **仍要做**。理由见 §6.2，与中继无关 |
**为什么不留中继接口**：留接口的收益是「将来补中继不用动会话层」，但既然中继已被排除，
`Path::Relay` 这类变体永远不会被构造，只会产生 `dead_code` 警告，要么 `#[allow]` 要么日后删除。
**零收益的预留就是纯负担。**
### 6.2 连接层边界与中继接口是两件事
这两件事常被混为一谈，必须分开：
| | 内容 | 现在做不做 |
|---|---|---|
| **连接层边界** | 把"怎么连"（信令/打洞/直连）与"连上干什么"（收流/解码/渲染）分开 | ✅ **要做** |
| **中继接口** | `enum Path { Direct, Relay }` 这类类型预留 | ❌ **不做** |
连接层边界本身就有独立价值，与中继无关：
1. **连接方式确定会变**：局域网 `IP:端口` → 将来设备码 + 打洞，这是已定的路线；
2. **安卓主控要复用**：`viewer-core` 收流解码渲染不应知道对面是怎么连上的；
3. **可测性**：解码逻辑能脱离真连接单独测试。
做法是把它抽成一个函数边界（输入目标描述、返回 `quinn::Connection`），
**不是**定义 trait 或枚举。
### 6.3 对产品的影响（必须写进对外说明）
| 网络环境 | 能否连接 |
|---|---|
| 双方任一侧无公网 IPv6，且处于 CGNAT 后 | ❌ **无法连接，无解** |
| 双方都有公网 IPv6 | ✅ 可直连 |
| 局域网 / 双方都有可打洞的公网 IPv4 | ✅ 可直连 |
| 对称 NAT（双方） | ❌ 通常不通 |
**这不是 bug，是已知边界。** 用户在外网连不上时，UI 必须给出可执行提示
（`PunchFailReason::user_hint()` 已提供文案），不能停留在"连接中"。
---
@@ -263,7 +305,10 @@ pub enum Path { Direct { addr: SocketAddr }, Relay { endpoint: EndpointId } }
## 10. 开放问题（需要拍板）
1. 中继：确认走 §6 的 A 方案（v6 优先、留接口不实现）？
> §10 第 1 条已于 2026-10-28 裁决，见 §6。
1. ~~中继：确认走 §6 的 A 方案（v6 优先、留接口不实现）？~~
**已裁决：不做中继、不留接口、v6 优先、但保留连接层边界。详见 §6.1 / §6.2。**
2. 移动端：先 Android 还是直接砍 iOS？
3. UI 框架：确认 Tauri v2，还是先原生桌面 + 后面再说移动？
4. 账号体系：现在是"输验证码"，要支持设备列表就得有账号/设备 ID 体系——什么时候做？
打洞与信令设计.md
+3 -1
View File
@@ -1,7 +1,9 @@
# 打洞与信令设计（无中继方案）
> 状态：协议契约已落在 `common/src/signaling.rs`，待 A 过审后合入 main。
> 状态：协议契约已落在 `common/src/signaling.rs`，已合入 main（`703eb64`）。
> 决策：**不做中继（无 TURN / coturn）**，服务器只保留信令 + STUN 职能。
> **裁决确认（2026-10-28）**：公司不提供中继服务器，本决策为最终结论，
> 同时**不预留中继接口**（理由见《多平台架构设计.md》§6.1 / §6.2）。
> 关联：`构建运行手册.md` §7 第 4 项、`Git协作规范.md` 二·B 第 ③ 项。
## 1. 结论先行：信令不是中继
构建运行手册.md
+22 -13
View File
@@ -20,31 +20,34 @@
- `rustup default stable-x86_64-pc-windows-gnu`，装到 `%USERPROFILE%\.cargo\bin`
- MinGW-w64：`D:\mingw64\bin`（gcc 16.2.0）
- 构建时 PATH 需同时包含上面两个目录（缺 MinGW 会在链接 openh264 的 C++ 时失败）
- **cargo 源**：国内直连 crates.io 会超时（cargo 不读系统代理），需配置镜像。已在本机
`%USERPROFILE%\.cargo\config.toml` 写入 USTC 源：
- **cargo 网络**：国内直连 crates.io 会超时，且 **cargo 不读 Windows 系统代理**，必须显式告诉它。
已在本机 `%USERPROFILE%\.cargo\config.toml` 写入（Clash 之类混合端口，按实际改）：
```toml
[source.crates-io]
replace-with = "ustc"
[source.ustc]
registry = "sparse+https://mirrors.ustc.edu.cn/crates.io-index/"
[http]
proxy = "http://127.0.0.1:7897"
```
⚠️ **不要用国内镜像做源替换**：实测 USTC 镜像的索引 `cksum` 与其实际下发的 `.crate`
字节**不一致**（`nvenc 0.1.0`：索引写 `3227c3b0…`，包体实际 `ab6fde0e…`），
会报 `failed to verify the checksum of nvenc`；且源替换会改变 registry 身份，
使 `Cargo.lock` 里已有的 checksum 与全部缓存失配。**走官方源 + 代理最稳。**
> 注：旧版本手册写的 `D:\wordkb\...\uurev\tools\rust` 路径已不存在，按上面重新准备工具链即可。
## 3. 构建
```bash
export RUSTUP_HOME="D:\\wordkb\\2026-10-22-15-52-47\\uurev\\tools\\rust\\rustup"
export CARGO_HOME="D:\\wordkb\\2026-10-22-15-52-47\\uurev\\tools\\rust\\cargo"
export PATH="/d/wordkb/2026-10-22-15-52-47/uurev/tools/rust/cargo/bin:/d/wordkb/2026-10-22-15-52-47/uurev/tools/mingw64/bin:$PATH"
cd /d/wordkb/2026-10-22-15-52-47/mvp-remote
# Git Bash 写法：PATH 需含 rustup 的 cargo 与 MinGW-w64
export PATH="$USERPROFILE/.cargo/bin:/d/mingw64/bin:$PATH"
cd /d/code/remote # 换成你自己的工作目录 / worktree
cargo build --release
```
产物：
- 被控端 `target/release/agent.exe`
- 主控端 `target/release/viewer.exe`
- 编码基准（可选）`target/release/encbench.exe`
## 4. 运行（两台机器，或同机双开验证）
@@ -164,9 +167,15 @@ viewer.exe --addr <被控机IP> --port 47800 --code 123456
bash scripts/linux-smoke.sh
```
脚本自建 Xvfb 虚拟屏后跑端到端测试，覆盖：X11 采集 → openh264 编码 → QUIC 下行解帧、
验证码认证（含**错误验证码被拒**）、Ping/Pong（含下行丢包统计字段）、静止桌面下的关键帧
恢复，以及**输入注入实证**——注入鼠标移动后，从另一条 X11 连接读回指针真实坐标。
脚本自建 Xvfb 虚拟屏后跑**单测 + 端到端**两层：
- 单测（不需要 X server）：键码映射、坐标换算等纯逻辑；
- 端到端：X11 采集 → openh264 编码 → QUIC 下行解帧、验证码认证（含**错误验证码被拒**）、
Ping/Pong（含下行丢包统计字段）、静止桌面下的关键帧恢复，以及**输入注入实证**——
注入鼠标移动后，从另一条 X11 连接读回指针真实坐标。
> 单测那层别省：坐标换算的取整边界就是靠它才发现的（集成测试有 ±2 像素容差，
> 期望值写错也照样绿）。
> 静止关键帧那条用例做过**变异测试**：把修复退回会失败、恢复后通过，确认断言不是摆设。
> 注意扰动量必须小（画 64x64 小方块）——整屏刷白会触发编码器场景切换自动 IDR，