功能：上线 GPU 监控看板并核实 IOMMU 配置
变更详情：GPU 数据展示看板上线（http://192.168.1.51:8090，systemd 自启）；修正 GPU/IOMMU 实测事实 - tools/gpu_dashboard.py：宿主机端采集（lspci/sysfs/qm），直通前显示驱动/解绑/IOMMU 状态，直通后经 guest agent 采 nvidia-smi 实时利用率/显存/温度/NVENC 会话 - 实测修正：GPU 为 CLEVO 准系统 GM204M/965M（非笔记本误判）；IOMMU 已启用（组 2 = 00:01.0 + 01:00.0 整组直通）；Maxwell 需 R580 legacy 驱动 - docs/gpu-passthrough-20261030.md：明日直通操作单
This commit is contained in:
parent
85f64df1a5
commit
73a635c4cf
@ -54,7 +54,9 @@
|
||||
- 完整方案：`docs/web-console-mvp-plan-20261028.md`。
|
||||
|
||||
## 尚未完成
|
||||
- 服务端 GPU 直通（宿主机 starhe 有 GTX 965M / IOMMU Group 2 独占，Intel 核显留给主机显示）
|
||||
- 服务端 GPU 直通：GPU 为 CLEVO 准系统上的 GM204M/965M（10de:13d9，宿主被 nouveau 占用需解绑）；
|
||||
**IOMMU 已启用**（组 2 = root port 00:01.0 + GPU 01:00.0，整组直通）；VM 内需装 R580 legacy 驱动。
|
||||
操作单与 GPU 看板见 `docs/gpu-passthrough-20261030.md`，看板 `http://192.168.1.51:8090/`（systemd 自启）
|
||||
- RK3528 VPU 硬解（h264_v4l2m2m 等缺可用设备，需 MPP FFmpeg 补丁或 V4L2 request 驱动）
|
||||
- 2K 分辨率链路验证（虚拟显示器 + 盒子显示模式需同步调整）
|
||||
- Weston/DRM/mppvideodec
|
||||
|
||||
46
docs/gpu-passthrough-20261030.md
Normal file
46
docs/gpu-passthrough-20261030.md
Normal file
@ -0,0 +1,46 @@
|
||||
# GPU 直通前置核查与数据看板（2026-10-30）
|
||||
|
||||
## 硬件事实（SSH 到宿主机实测，修正旧文档误判）
|
||||
|
||||
宿主机 `starhe`（192.168.1.51，pve-manager/9.2.21，内核 7.0.2-6-pve）：
|
||||
|
||||
```text
|
||||
00:02.0 VGA : Intel 4th Gen Core (Haswell) iGPU [8086:0416] 驱动 i915（宿主显示用，保留）
|
||||
01:00.0 3D : NVIDIA GM204M [GeForce GTX 965M] [10de:13d9] 驱动 nouveau（直通前需解绑）
|
||||
两者 Subsystem 均为 CLEVO/KAPOK [1558:6557]
|
||||
```
|
||||
|
||||
- 机身是 CLEVO 准系统（外形像台式机箱），GPU 是移动版 GM204M/965M——
|
||||
此前"笔记本"与"IOMMU Group 2 独占"的记载部分有误，以本页为准。
|
||||
- HDMI 音频功能（01:00.1）不存在，无需处理。
|
||||
- **IOMMU 实测已启用**（/sys/kernel/iommu_groups 有 15 组，DMAR 生效；
|
||||
内核 6.8+ 默认开启，不需要显式 intel_iommu=on）。
|
||||
- GPU 位于 **IOMMU 组 2**，组内设备：`0000:00:01.0`（root port）+ `0000:01:00.0`（GPU），
|
||||
直通需整组传入。
|
||||
- VM 100：q35 + SeaBIOS、4 核 8G、win10，未配置 hostpci。
|
||||
|
||||
## 明日直通操作单（按序执行）
|
||||
|
||||
1. 约窗口：宿主机重启会让全部 VM（100/101/102）停机。
|
||||
2. 解绑 nouveau：`/etc/modprobe.d/blacklist.conf` 加 `blacklist nouveau`，
|
||||
`update-initramfs -u`，重启宿主机（重启后 `lsmod | grep nouveau` 应为空）。
|
||||
3. 绑 vfio：`/etc/modprobe.d/vfio.conf` 写 `options vfio-pci ids=10de:13d9`
|
||||
（含 root port 的话用 QEMU 参数整组传），重启后确认
|
||||
`lspci -nnk -s 01:00.0` 显示 `Kernel driver in use: vfio-pci`（看板会自动变绿）。
|
||||
4. 挂载：`qm set 100 -hostpci0 00:01.0,pcie=1`（按组传；若启动报隔离不满足，
|
||||
用 `-hostpci0 0000:00:01.0;0000:01:00.0` 形式补传 root port）。
|
||||
vBIOS ROM 先不挂，Code 43 在 R465+ 驱动已移除检测。
|
||||
5. VM 内装驱动：Maxwell 已进 legacy 分支，装 **R580 系**驱动（提前下载好放 VM）。
|
||||
6. 验证：VM 内 `nvidia-smi` 出卡 → 看板"VM 内实时数据"卡片自动出现利用率/显存/温度。
|
||||
7. Sunshine 切 NVENC（配置 encoder=nvenc 或按其自动探测），1080p60 复测帧率。
|
||||
|
||||
## GPU 数据看板（已上线）
|
||||
|
||||
- 地址：`http://192.168.1.51:8090/`（自动刷新 4 秒）
|
||||
- 源码：`tools/gpu_dashboard.py`（宿主机 `/root/gpu_dashboard.py`，
|
||||
systemd 单元 `gpu-dashboard.service` 已设开机自启）
|
||||
- 直通前显示：型号/驱动/直通阶段/IOMMU 状态与组内设备/VM 配置；
|
||||
直通后自动经 `qm guest exec 100` 采集 nvidia-smi：利用率、显存、温度、
|
||||
驱动版本、功耗、NVENC 编码会话数。
|
||||
- 手动重启：`systemctl restart gpu-dashboard`。
|
||||
- 后续：接管理平台后，该数据源可换成平台 GPU 池接口（Web 控制台监控模块复用）。
|
||||
249
tools/gpu_dashboard.py
Normal file
249
tools/gpu_dashboard.py
Normal file
@ -0,0 +1,249 @@
|
||||
#!/usr/bin/env python3
|
||||
"""服务器 GPU 状态看板（部署在 PVE 宿主机上，仅依赖 python3 标准库）。
|
||||
|
||||
数据源：
|
||||
- lspci / /sys / lsmod：GPU 型号、当前驱动（nouveau/vfio-pci/nvidia）、IOMMU 状态
|
||||
- qm config/status：VM 100 的直通配置（hostpci0）与运行状态
|
||||
- 直通生效后：qm guest exec 在 VM 100 内执行 nvidia-smi 采集实时数据
|
||||
（利用率 / 显存 / 温度 / 驱动版本 / NVENC 编码会话）
|
||||
|
||||
用法：python3 gpu_dashboard.py [端口] # 默认 8090，浏览器访问 http://<宿主机IP>:8090/
|
||||
"""
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
VMID = 100
|
||||
CACHE_TTL = 2.0
|
||||
GUEST_TIMEOUT = 25
|
||||
|
||||
_lock = threading.Lock()
|
||||
_cache = {"t": 0.0, "data": None}
|
||||
|
||||
|
||||
def run(cmd, timeout=15):
|
||||
try:
|
||||
r = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)
|
||||
return (r.stdout or "").strip()
|
||||
except Exception as e:
|
||||
return "ERROR: %s" % e
|
||||
|
||||
|
||||
def read(path):
|
||||
try:
|
||||
with open(path) as f:
|
||||
return f.read().strip()
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def gpu_identity():
|
||||
out = run("lspci -nnk -s 01:00.0")
|
||||
name = ""
|
||||
driver = ""
|
||||
for line in out.splitlines():
|
||||
if "NVIDIA" in line and not name:
|
||||
idx = line.find("NVIDIA")
|
||||
name = line[idx:].strip()
|
||||
if "Kernel driver in use" in line:
|
||||
driver = line.split(":", 1)[-1].strip()
|
||||
audio = ""
|
||||
for line in run("lspci -nnk -s 01:00.1").splitlines():
|
||||
if "Kernel driver in use" in line:
|
||||
audio = line.split(":", 1)[-1].strip()
|
||||
return {
|
||||
"name": name or "NVIDIA GPU (0000:01:00.0)",
|
||||
"driver": driver or "（无，未绑定）",
|
||||
"audio_driver": audio or "（无）",
|
||||
}
|
||||
|
||||
|
||||
def iommu_state():
|
||||
# 新内核（6.8+）默认启用 Intel DMAR，无需显式 intel_iommu=on；
|
||||
# 以 iommu_groups 是否存在且 GPU 已入组为准，内核参数仅作展示。
|
||||
cmdline = read("/proc/cmdline")
|
||||
flag = "intel_iommu=on" in cmdline or "iommu=on" in cmdline
|
||||
groups = run("find /sys/kernel/iommu_groups -maxdepth 1 -mindepth 1 -type d 2>/dev/null | wc -l")
|
||||
n = int(groups) if groups.isdigit() else 0
|
||||
gpu_group = run("for d in /sys/kernel/iommu_groups/*/devices/0000:01:00.0; do "
|
||||
"[ -e \"$d\" ] && dirname \"$d\" | sed 's|.*iommu_groups/||;s|/devices||'; done")
|
||||
members = run("ls /sys/kernel/iommu_groups/%s/devices/ 2>/dev/null" % gpu_group) if gpu_group else ""
|
||||
if n == 0:
|
||||
state, ok = "未启用（iommu 组数为 0，需 BIOS 开 VT-d + 内核参数 intel_iommu=on）", False
|
||||
elif not gpu_group:
|
||||
state, ok = "已启用，但 GPU 不在任何组内（异常）", False
|
||||
else:
|
||||
state, ok = "已启用（DMAR 生效）", True
|
||||
return {"state": state, "ok": ok, "kernel_flag": flag, "groups": n,
|
||||
"gpu_group": gpu_group, "group_members": members}
|
||||
|
||||
|
||||
def passthrough_state():
|
||||
cfg = run("qm config %d 2>/dev/null" % VMID)
|
||||
hostpci = ""
|
||||
for line in cfg.splitlines():
|
||||
if line.startswith("hostpci"):
|
||||
hostpci = line
|
||||
break
|
||||
status = run("qm status %d 2>/dev/null" % VMID).replace("status: ", "").replace("status:", "").strip()
|
||||
return {"hostpci": hostpci or "（未配置）", "vm_status": status or "unknown"}
|
||||
|
||||
|
||||
def guest_gpu():
|
||||
"""直通后在 VM 内经 guest agent 执行 nvidia-smi；未直通/无驱动时返回 None。"""
|
||||
q = ("nvidia-smi --query-gpu=name,utilization.gpu,memory.used,memory.total,"
|
||||
"temperature.gpu,driver_version,power.draw --format=csv,noheader")
|
||||
out = run("qm guest exec %d -- powershell -NoProfile -Command \"%s\"" % (VMID, q),
|
||||
timeout=GUEST_TIMEOUT)
|
||||
try:
|
||||
d = json.loads(out)
|
||||
except Exception:
|
||||
return None
|
||||
if d.get("returncode") != 0:
|
||||
return None
|
||||
text = (d.get("out-data") or "").strip()
|
||||
if not text or "nvidia-smi" in (d.get("err-data") or ""):
|
||||
return None
|
||||
parts = [p.strip() for p in text.split(",")]
|
||||
if len(parts) < 6:
|
||||
return None
|
||||
info = {
|
||||
"name": parts[0],
|
||||
"utilization": parts[1],
|
||||
"vram_used": parts[2],
|
||||
"vram_total": parts[3],
|
||||
"temperature": parts[4],
|
||||
"driver": parts[5],
|
||||
"power": parts[6] if len(parts) > 6 else "",
|
||||
}
|
||||
enc = run("qm guest exec %d -- powershell -NoProfile -Command \""
|
||||
"nvidia-smi --query-gpu=encoder.stats.sessionCount --format=csv,noheader\"" % VMID,
|
||||
timeout=GUEST_TIMEOUT)
|
||||
try:
|
||||
d2 = json.loads(enc)
|
||||
v = (d2.get("out-data") or "").strip()
|
||||
if d2.get("returncode") == 0 and v:
|
||||
info["enc_sessions"] = v
|
||||
except Exception:
|
||||
pass
|
||||
return info
|
||||
|
||||
|
||||
def collect():
|
||||
ident = gpu_identity()
|
||||
driver = ident["driver"]
|
||||
if "vfio-pci" in driver:
|
||||
stage = "vfio 已接管（可直通/已解绑 nouveau）"
|
||||
elif "nouveau" in driver:
|
||||
stage = "未直通（GPU 由宿主 nouveau 占用，直通前需解绑）"
|
||||
elif "nvidia" in driver:
|
||||
stage = "未直通（GPU 由宿主 nvidia 驱动占用）"
|
||||
else:
|
||||
stage = "GPU 未绑定驱动"
|
||||
data = {
|
||||
"time": time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"gpu": ident,
|
||||
"stage": stage,
|
||||
"iommu": iommu_state(),
|
||||
"pt": passthrough_state(),
|
||||
"guest": guest_gpu() if "hostpci" in passthrough_state()["hostpci"] else None,
|
||||
}
|
||||
return data
|
||||
|
||||
|
||||
def get_data():
|
||||
with _lock:
|
||||
now = time.time()
|
||||
if _cache["data"] is not None and now - _cache["t"] < CACHE_TTL:
|
||||
return _cache["data"]
|
||||
data = collect()
|
||||
_cache["data"] = data
|
||||
_cache["t"] = now
|
||||
return data
|
||||
|
||||
|
||||
HTML = """<!DOCTYPE html>
|
||||
<html lang="zh-CN"><head><meta charset="utf-8">
|
||||
<title>服务器 GPU 状态</title>
|
||||
<style>
|
||||
body { background:#0f1420; color:#e6e9ef; font-family:"Microsoft YaHei",sans-serif;
|
||||
margin:0; padding:32px; }
|
||||
h1 { font-size:22px; margin:0 0 4px; }
|
||||
.sub { color:#8a94a6; font-size:12px; margin-bottom:24px; }
|
||||
.grid { display:grid; grid-template-columns:repeat(auto-fit,minmax(280px,1fr)); gap:14px; max-width:1100px; }
|
||||
.card { background:#1a2233; border:1px solid #2b3654; border-radius:10px; padding:16px 18px; }
|
||||
.card h3 { margin:0 0 10px; font-size:13px; color:#8a94a6; font-weight:normal; }
|
||||
.card .v { font-size:15px; line-height:1.7; word-break:break-all; }
|
||||
.ok { color:#3fb950; } .bad { color:#e05561; } .warn { color:#d29922; }
|
||||
.big { font-size:30px; font-weight:bold; }
|
||||
</style></head><body>
|
||||
<h1>服务器 GPU 状态</h1>
|
||||
<div class="sub" id="time">加载中…（每 4 秒自动刷新）</div>
|
||||
<div class="grid">
|
||||
<div class="card"><h3>显卡</h3><div class="v" id="gpu">-</div></div>
|
||||
<div class="card"><h3>直通阶段</h3><div class="v" id="stage">-</div></div>
|
||||
<div class="card"><h3>IOMMU / VT-d</h3><div class="v" id="iommu">-</div></div>
|
||||
<div class="card"><h3>VM %d 直通配置</h3><div class="v" id="pt">-</div></div>
|
||||
<div class="card"><h3>VM 内实时数据（nvidia-smi）</h3><div class="v" id="guest">直通生效后显示</div></div>
|
||||
</div>
|
||||
<script>
|
||||
const $ = id => document.getElementById(id);
|
||||
function esc(s){ return String(s).replace(/</g,"<"); }
|
||||
function render(d){
|
||||
$("time").textContent = "更新于 " + d.time + "（每 4 秒自动刷新）";
|
||||
$("gpu").innerHTML = "<b>" + esc(d.gpu.name) + "</b><br>当前驱动：" + esc(d.gpu.driver) +
|
||||
"<br>HDMI 音频驱动：" + esc(d.gpu.audio_driver);
|
||||
const stColor = d.stage.includes("未直通") ? "warn" : "ok";
|
||||
$("stage").innerHTML = "<span class='" + stColor + "'>" + esc(d.stage) + "</span>";
|
||||
$("iommu").innerHTML = (d.iommu.ok ? "<span class='ok'>" : "<span class='bad'>") +
|
||||
esc(d.iommu.state) + "</span><br>内核参数 intel_iommu=on：" +
|
||||
(d.iommu.kernel_flag ? "已设置" : "未设置（新内核默认开启，非必须）") +
|
||||
(d.iommu.gpu_group ? "<br>GPU 所在组：" + esc(d.iommu.gpu_group) +
|
||||
(d.iommu.group_members ? "<br>组内设备：" + esc(d.iommu.group_members) : "") : "");
|
||||
$("pt").innerHTML = "运行状态：" + esc(d.pt.vm_status) + "<br>hostpci：" + esc(d.pt.hostpci);
|
||||
if (d.guest) {
|
||||
$("guest").innerHTML =
|
||||
"<span class='big'>" + esc(d.guest.utilization) + "</span> 利用率<br>" +
|
||||
"显存：" + esc(d.guest.vram_used) + " / " + esc(d.guest.vram_total) + "<br>" +
|
||||
"温度：" + esc(d.guest.temperature) +
|
||||
(d.guest.power ? " 功耗：" + esc(d.guest.power) : "") + "<br>" +
|
||||
"驱动：" + esc(d.guest.driver) +
|
||||
(d.guest.enc_sessions !== undefined ? "<br>NVENC 编码会话：" + esc(d.guest.enc_sessions) : "");
|
||||
} else {
|
||||
$("guest").innerHTML = "<span class='warn'>直通生效并在 VM 内装好驱动后，这里显示实时利用率/显存/温度/NVENC 会话</span>";
|
||||
}
|
||||
}
|
||||
function tick(){ fetch("/api").then(r=>r.json()).then(render).catch(()=>{}); }
|
||||
tick(); setInterval(tick, 4000);
|
||||
</script></body></html>""" % VMID
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def _send(self, code, body, ctype):
|
||||
raw = body.encode("utf-8")
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", ctype + "; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(raw)))
|
||||
self.end_headers()
|
||||
self.wfile.write(raw)
|
||||
|
||||
def do_GET(self):
|
||||
if self.path.startswith("/api"):
|
||||
self._send(200, json.dumps(get_data(), ensure_ascii=False), "application/json")
|
||||
elif self.path.startswith("/healthz"):
|
||||
self._send(200, "ok", "text/plain")
|
||||
else:
|
||||
self._send(200, HTML, "text/html")
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
port = int(sys.argv[1]) if len(sys.argv) > 1 else 8090
|
||||
srv = ThreadingHTTPServer(("0.0.0.0", port), Handler)
|
||||
print("GPU dashboard on http://0.0.0.0:%d/ (VM %d)" % (port, VMID), flush=True)
|
||||
srv.serve_forever()
|
||||
Loading…
Reference in New Issue
Block a user