Add configurable GPU monitoring and nightly power schedule

This commit is contained in:
张帅 2026-11-09 18:38:13 +08:00
parent cedae7ab92
commit 9f16609595
13 changed files with 252 additions and 16 deletions

View File

@ -1,6 +1,6 @@
# Hardware Dashboard # Hardware Dashboard
这是运行在 192.168.1.37 上的局域网设备监控与维护面板。页面提供 CPU、内存、双 P100 显卡、磁盘 SMART、性能历史曲线、硬件问题记录和定时任务管理。后端只使用 Python 标准库;前端使用原生 HTML、CSS 和 JavaScript,不依赖外部 CDN。 这是运行在 192.168.1.37 上的局域网设备监控与维护面板。页面提供 CPU、内存、NVIDIA 显卡、磁盘 SMART、性能历史曲线、硬件问题记录和定时任务管理。支持在网页选择使用 1 张或 2 张显卡,支持夜间关机与 RTC 定时唤醒。后端只使用 Python 标准库;前端使用原生 HTML、CSS 和 JavaScript,不依赖外部 CDN。
当前访问地址是 http://192.168.1.37:8090。定时任务可以执行服务器命令,必须使用独立的管理员账号登录。该账号与 SSH root 密码相互独立。 当前访问地址是 http://192.168.1.37:8090。定时任务可以执行服务器命令,必须使用独立的管理员账号登录。该账号与 SSH root 密码相互独立。
@ -48,11 +48,11 @@ sudo cp config/hardware-watch.example.json /etc/hardware-watch.json
sudo nano /etc/hardware-watch.json sudo nano /etc/hardware-watch.json
``` ```
把 expected_gpu_count、GPU UUID 和磁盘名改成当前服务器的实际值。修改后重启监控采集服务。定时任务管理服务首次初始化会生成独立管理员密码文件;不要把生成的密码、任务数据库、日志或系统配置哈希提交到仓库。 把 expected_gpu_count 和磁盘名改成当前服务器的实际值。若经常更换显卡,将 expected_gpu_uuids 设为空列表;也可在“计算显卡”卡片选择数量并保存。网页设置通过现有管理员登录验证,保存后同步到硬件采集器,重启后仍有效。定时任务管理服务首次初始化会生成独立管理员密码文件;不要把生成的密码、任务数据库、日志或系统配置哈希提交到仓库。
## 安全边界 ## 安全边界
- 监控页面本身是只读的,可查看状态、曲线和日志。 - 状态、曲线和日志无需登录;修改显卡数量需要管理员登录。
- 定时任务能以 root 身份执行命令,新增、编辑、启用和立即执行均需要任务管理员登录。 - 定时任务能以 root 身份执行命令,新增、编辑、启用和立即执行均需要任务管理员登录。
- 不要把这个服务直接暴露到公网。当前设计只适合可信局域网。 - 不要把这个服务直接暴露到公网。当前设计只适合可信局域网。
- 远程访问应通过可信 VPN 或带 HTTPS 与额外认证的反向代理,而不是直接端口映射。 - 远程访问应通过可信 VPN 或带 HTTPS 与额外认证的反向代理,而不是直接端口映射。

View File

@ -1,9 +1,6 @@
{ {
"expected_gpu_count": 2, "expected_gpu_count": 2,
"expected_gpu_uuids": [ "expected_gpu_uuids": [],
"GPU-00000000-0000-0000-0000-000000000001",
"GPU-00000000-0000-0000-0000-000000000002"
],
"expected_disks": ["nvme0n1"], "expected_disks": ["nvme0n1"],
"gpu_warn_c": 85, "gpu_warn_c": 85,
"history_since": "7 days ago", "history_since": "7 days ago",

View File

@ -1,6 +1,6 @@
# 192.168.1.37 监控与定时任务技术说明 # 192.168.1.37 监控与定时任务技术说明
版本:v1.2,2026 年 11 月 4 日。 版本:v1.3,2026 年 11 月 9 日。新增显卡数量设置和夜间节电计划,详见第 12 节。
## 1. 访问与登录 ## 1. 访问与登录
@ -261,3 +261,49 @@ systemctl restart hardware-taskd.service
| 未登录、错误来源、错误 CSRF | 分别被 HTTP 401/403 拒绝。 | | 未登录、错误来源、错误 CSRF | 分别被 HTTP 401/403 拒绝。 |
这些测试仅打印信息或短暂等待,没有执行重启、关机、证书续签或修改现有业务配置。测试计划在交付前移除,历史中保留有“部署验证”字样的执行记录。 这些测试仅打印信息或短暂等待,没有执行重启、关机、证书续签或修改现有业务配置。测试计划在交付前移除,历史中保留有“部署验证”字样的执行记录。
## 12. 显卡数量与夜间节电(2026 年 11 月 9 日)
### 显卡数量
在“设备总览”的“计算显卡”卡片中,选择“1 张显卡”或“2 张显卡”,再点击“保存”。使用已有任务管理员账号登录后,设置保存到 `/etc/hardware-watch.json`,刷新网页、重启服务和重启服务器后仍有效。多个浏览器共用服务器上的设置。
保存会更新 `expected_gpu_count` 并清空 `expected_gpu_uuids`。监控随后按 NVIDIA 显卡数量检查,不再要求原来的 P100 UUID;可更换为 V100 等受当前驱动支持的 NVIDIA 显卡。温度、供电、驱动异常、PCIe 通信错误继续检测。选择 2 张而实际仅检测到 1 张时,仍会报缺卡。
页面通过 `/api/tasks/gpu-count` 调用已有 root 任务代理,沿用管理员会话、CSRF 和来源检查。普通网页进程仍无权写入 `/etc`。配置变更记录到任务管理审计表,同时触发一次硬件采集;页面每 10 秒刷新。
配置变更不会删除历史告警。旧缺卡和旧 UUID 告警随采集恢复保留在历史中,不应再作为当前故障显示。
### 每天 00:00 关机、06:00 开机
页面“定时任务”中使用名称“夜间节电:00:00 关机,06:00 开机”。执行周期为北京时间每天 `00:00`,命令为:
```bash
/usr/bin/python3 /opt/hardware-dashboard/scripts/night-power-cycle.py
```
脚本先用 `rtcwake --utc --mode no` 设置当天北京时间 06:00 的 RTC 唤醒,并回读 `/sys/class/rtc/rtc0/wakealarm` 核对时间。确认成功后调用 `systemctl --no-block poweroff` 正常关机。服务器的硬件时钟采用 UTC;不要求存在 `/etc/adjtime`。
脚本仅允许在北京时间 00:00 至 00:04 执行关机,避免白天点击“执行一次”时意外关机。若缺少 RTC 接口、设置闹钟失败或回读时间不符,本轮任务报错并保持开机。每天的任务均重新设置闹钟。系统定时器使用 `Persistent=false`,开机后不会补执行错过的关机任务。
仅检查环境和下次唤醒时间:
```bash
/usr/bin/python3 /opt/hardware-dashboard/scripts/night-power-cycle.py --check
```
这个检查不设置闹钟,也不执行关机。真正开机由主板完成,需要 BIOS 支持并启用关机状态 RTC 唤醒,且电源持续接通。00:00 正在运行的计算任务会中断;关机期间无法访问监控网页。RTC 闹钟设置成功不能证明主板一定能从关机状态唤醒,完整效果应在首次夜间周期后确认。
需要全天运行时,在页面停用这项计划。停用会取消后续关机调度,已设置的 RTC 闹钟仍可能保留。新增任务的“夜间节电”模板默认停用;如已有启用的同名任务,应编辑已有任务,避免重复添加。
本次在 192.168.1.37 已创建并启用一项夜间节电任务。首次关机计划为北京时间 `2026-11-10 00:00`,RTC 闹钟已设置并回读确认为 `2026-11-10 06:00`。未在部署时执行实际关机。原代码、原显卡配置和任务数据库的备份位于 `/root/hardware-dashboard-backups/20261109-183339/`。
### 新增文件与验证
- `scripts/night-power-cycle.py`:RTC 设置、时间回读、关机及只读检查。
- `tests/test_power_cycle.py`:验证时区、跨日、白天禁止关机、闹钟失败或不匹配时不关机。
- `tests/test_tasks.py`:增加显卡数量输入校验、登录与 CSRF、其他配置保留及关机任务识别检查。
技术栈仍为 Python 标准库、HTML、CSS、原生 JavaScript、SQLite 和 systemd;RTC 唤醒使用系统已有的 util-linux `rtcwake`。部署时需同步新增的 `scripts/` 目录,无需安装前端构建工具或第三方 Python 包。
本次本地和服务器端均通过 22 项自动化测试,前端脚本语法检查通过。实际网页已验证单卡显示 `1/1`、无缺卡告警、仅显示一张显卡卡片;390 像素手机视口无横向溢出。登录后保存 1/2 张显卡及夜间模板填写流程使用浏览器模拟接口验证,没有执行真实关机。RTC 闹钟已在真实设备上设置并回读,完整关机唤醒周期尚待首次计划执行后验证。

View File

@ -0,0 +1,52 @@
#!/usr/bin/python3
"""At midnight, arm a 06:00 RTC alarm and request an orderly shutdown."""
import argparse
import datetime as dt
from pathlib import Path
import shutil
import subprocess
TZ = dt.timezone(dt.timedelta(hours=8), 'Asia/Shanghai')
ALARM = Path('/sys/class/rtc/rtc0/wakealarm')
def wake_time(now):
target = now.astimezone(TZ).replace(hour=6, minute=0, second=0, microsecond=0)
return target if target > now else target + dt.timedelta(days=1)
def power_cycle(now, check=False):
target = wake_time(now)
print(f'当前时间:{now.astimezone(TZ).isoformat(timespec="seconds")}', flush=True)
print(f'计划唤醒:{target.isoformat(timespec="seconds")}', flush=True)
rtcwake = shutil.which('rtcwake')
systemctl = shutil.which('systemctl')
if not rtcwake or not systemctl or not ALARM.exists():
raise RuntimeError('未发现 RTC 唤醒接口或必要命令,本次不执行关机。')
if check:
print('检查完成;没有设置闹钟,也没有执行关机。', flush=True)
return
local = now.astimezone(TZ)
if local.hour != 0 or local.minute >= 5:
raise RuntimeError('此任务仅允许在北京时间 00:00–00:04 执行;本次不关机。')
# This server stores its hardware clock in UTC (timedatectl LocalRTC=no).
subprocess.run([rtcwake, '--utc', '--device', '/dev/rtc0', '--mode', 'no',
'--time', str(int(target.timestamp()))], check=True, timeout=20)
alarm = int(ALARM.read_text().strip() or '0')
if abs(alarm - target.timestamp()) > 2:
raise RuntimeError('RTC 唤醒时间回读不一致,本次不执行关机。')
print('RTC 闹钟已确认,开始正常关机。正在运行的计算任务将停止。', flush=True)
subprocess.run([systemctl, '--no-block', 'poweroff'], check=True, timeout=20)
def main():
parser = argparse.ArgumentParser(description='每天 00:00 关机,设置当天 06:00 RTC 唤醒。')
parser.add_argument('--check', action='store_true', help='仅检查环境和时间,不设置闹钟、不关机')
args = parser.parse_args()
try:
power_cycle(dt.datetime.now(TZ), args.check)
except (RuntimeError, OSError, ValueError, subprocess.SubprocessError) as exc:
parser.exit(1, f'夜间节电计划未完成:{exc}\n')
if __name__ == '__main__':
main()

View File

@ -1,5 +1,5 @@
#!/usr/bin/python3 #!/usr/bin/python3
"""LAN-only, read-only hardware dashboard. Python 3.12 standard library.""" """LAN hardware dashboard with authenticated management. Python standard library."""
import argparse import argparse
import csv import csv
from contextlib import contextmanager from contextlib import contextmanager
@ -312,6 +312,10 @@ def sample(previous):
faults.append(issue('journal', '硬件日志采集状态超过 3 分钟未更新,或系统时间异常。')) faults.append(issue('journal', '硬件日志采集状态超过 3 分钟未更新,或系统时间异常。'))
else: else:
for value in hw.get('active', {}).values(): for value in hw.get('active', {}).values():
# Live GPU sampling uses the current configuration; the minute-level
# collector may still contain an alarm from before a settings change.
if value.get('category') in ('gpu_missing','gpu_identity'):
continue
matched = next((x for x in faults if x['category'] == value.get('category')), None) matched = next((x for x in faults if x['category'] == value.get('category')), None)
if matched: if matched:
if value.get('severity') == 'ERROR': if value.get('severity') == 'ERROR':
@ -332,7 +336,8 @@ def sample(previous):
'cpu_threads': os.cpu_count(), 'cpu_sockets': cpu_sockets, 'cpu_temp': cpu_temp, 'load': list(os.getloadavg()), 'cpu_threads': os.cpu_count(), 'cpu_sockets': cpu_sockets, 'cpu_temp': cpu_temp, 'load': list(os.getloadavg()),
'memory': (1 - ram_available / ram_total) * 100, 'memory_total_gib': ram_total / 1073741824, 'memory': (1 - ram_available / ram_total) * 100, 'memory_total_gib': ram_total / 1073741824,
'memory_available_gib': ram_available / 1073741824, 'gpus': gpu, 'memory_available_gib': ram_available / 1073741824, 'gpus': gpu,
'expected_gpu_count': expected, 'gpu_warn_c': cfg.get('gpu_warn_c',85), 'expected_gpu_count': expected, 'gpu_count_only': not bool(cfg.get('expected_gpu_uuids')),
'gpu_warn_c': cfg.get('gpu_warn_c',85),
'gpu_error': gpu_error, 'disk_temp': disk_temp, 'disks': disks, 'rates': rates, 'gpu_error': gpu_error, 'disk_temp': disk_temp, 'disks': disks, 'rates': rates,
'device_map': device_map(gpu), 'device_map': device_map(gpu),
'disk_used_gib': used / 1073741824, 'disk_total_gib': capacity / 1073741824, 'disk_used_gib': used / 1073741824, 'disk_total_gib': capacity / 1073741824,
@ -695,7 +700,8 @@ class Handler(BaseHTTPRequestHandler):
parsed=urlparse(self.path) parsed=urlparse(self.path)
actions={'/api/tasks/login':'auth.login','/api/tasks/logout':'auth.logout','/api/tasks/password':'auth.password', actions={'/api/tasks/login':'auth.login','/api/tasks/logout':'auth.logout','/api/tasks/password':'auth.password',
'/api/tasks/preview':'tasks.preview','/api/tasks/save':'tasks.save','/api/tasks/enable':'tasks.enable', '/api/tasks/preview':'tasks.preview','/api/tasks/save':'tasks.save','/api/tasks/enable':'tasks.enable',
'/api/tasks/delete':'tasks.delete','/api/tasks/run':'tasks.run','/api/tasks/stop':'tasks.stop'} '/api/tasks/delete':'tasks.delete','/api/tasks/run':'tasks.run','/api/tasks/stop':'tasks.stop',
'/api/tasks/gpu-count':'monitor.gpu_count'}
if parsed.path not in actions:return self.send('Not found','text/plain',404) if parsed.path not in actions:return self.send('Not found','text/plain',404)
origin=self.headers.get('Origin','') origin=self.headers.get('Origin','')
host=self.headers.get('Host','') host=self.headers.get('Host','')

View File

@ -34,6 +34,11 @@ function renderStatus(data){
metric('cpu-value',data.cpu,'%',1);meter('cpu-bar',data.cpu);$('cpu-threads').textContent=(data.cpu_sockets?data.cpu_sockets+' 路 CPU · ':'')+(data.cpu_threads||'—')+' 线程';$('cpu-temp').textContent=num(data.cpu_temp)+' °C';$('cpu-temp').title=data.cpu_name; metric('cpu-value',data.cpu,'%',1);meter('cpu-bar',data.cpu);$('cpu-threads').textContent=(data.cpu_sockets?data.cpu_sockets+' 路 CPU · ':'')+(data.cpu_threads||'—')+' 线程';$('cpu-temp').textContent=num(data.cpu_temp)+' °C';$('cpu-temp').title=data.cpu_name;
metric('memory-value',data.memory,'%');meter('memory-bar',data.memory);$('memory-used').textContent=`${num(data.memory_total_gib-data.memory_available_gib,1)} / ${num(data.memory_total_gib,1)} GiB`;$('memory-free').textContent=num(data.memory_available_gib,1)+' GiB 可用'; metric('memory-value',data.memory,'%');meter('memory-bar',data.memory);$('memory-used').textContent=`${num(data.memory_total_gib-data.memory_available_gib,1)} / ${num(data.memory_total_gib,1)} GiB`;$('memory-free').textContent=num(data.memory_available_gib,1)+' GiB 可用';
metric('gpu-count',data.gpus.length,`/ ${data.expected_gpu_count} 在线`);$('gpu-online-text').replaceChildren(el('i','dot'),document.createTextNode(data.gpus.length>=data.expected_gpu_count?'全部预期显卡可用':'显卡数量少于预期'));$('gpu-online-text').firstChild.style.background=data.gpus.length>=data.expected_gpu_count?colors.green:colors.red;$('gpu-type').textContent=data.gpus[0]?.name?.includes('P100')?'Tesla P100':'NVIDIA'; metric('gpu-count',data.gpus.length,`/ ${data.expected_gpu_count} 在线`);$('gpu-online-text').replaceChildren(el('i','dot'),document.createTextNode(data.gpus.length>=data.expected_gpu_count?'全部预期显卡可用':'显卡数量少于预期'));$('gpu-online-text').firstChild.style.background=data.gpus.length>=data.expected_gpu_count?colors.green:colors.red;$('gpu-type').textContent=data.gpus[0]?.name?.includes('P100')?'Tesla P100':'NVIDIA';
const gpuSelect=$('gpu-expected-count');
if(gpuSelect.dataset.pending===String(data.expected_gpu_count)){delete gpuSelect.dataset.pending;delete gpuSelect.dataset.dirty;}
if(!gpuSelect.dataset.dirty&&!gpuSelect.dataset.saving)gpuSelect.value=String(data.expected_gpu_count);
gpuSelect.disabled=!!gpuSelect.dataset.saving;$('gpu-config-save').disabled=!!gpuSelect.dataset.saving;
$('gpu-check-mode').textContent=data.gpu_count_only?'按所选数量检查,支持更换显卡':'当前按显卡身份检查';
const disks=Object.values(data.smart||{}),diskBad=!data.hardware_stale&&disks.some(x=>x.passed===false||x.critical_warning),diskKnown=!data.hardware_stale&&disks.length&&disks.every(x=>x.available&&x.passed===true); const disks=Object.values(data.smart||{}),diskBad=!data.hardware_stale&&disks.some(x=>x.passed===false||x.critical_warning),diskKnown=!data.hardware_stale&&disks.length&&disks.every(x=>x.available&&x.passed===true);
$('disk-health').textContent=diskBad?'需要检查':diskKnown?'健康正常':'状态未知';$('disk-health').style.color=diskBad?colors.red:diskKnown?colors.green:colors.amber; $('disk-health').textContent=diskBad?'需要检查':diskKnown?'健康正常':'状态未知';$('disk-health').style.color=diskBad?colors.red:diskKnown?colors.green:colors.amber;
$('disk-health-detail').textContent=diskKnown?'SMART 检查通过':diskBad?'请展开磁盘相关日志':'尚未取得完整 SMART 数据';$('disk-space').textContent=`已用 ${num(data.disk_used_gib,0)} / ${num(data.disk_total_gib,0)} GiB`;$('disk-temp').textContent=num(data.disk_temp)+' °C'; $('disk-health-detail').textContent=diskKnown?'SMART 检查通过':diskBad?'请展开磁盘相关日志':'尚未取得完整 SMART 数据';$('disk-space').textContent=`已用 ${num(data.disk_used_gib,0)} / ${num(data.disk_total_gib,0)} GiB`;$('disk-temp').textContent=num(data.disk_temp)+' °C';
@ -45,6 +50,7 @@ function renderStatus(data){
} }
function renderGPUs(data){ function renderGPUs(data){
const holder=$('gpu-grid');holder.replaceChildren(); const holder=$('gpu-grid');holder.replaceChildren();
holder.classList.toggle('single',Math.max(data.gpus.length,data.expected_gpu_count)===1);
const gpus=[...data.gpus].sort((a,b)=>a.uuid.localeCompare(b.uuid)); const gpus=[...data.gpus].sort((a,b)=>a.uuid.localeCompare(b.uuid));
gpus.forEach((gpu,index)=>{ gpus.forEach((gpu,index)=>{
const card=el('article','gpu-card');card.style.setProperty('--accent',index%2?colors.amber:colors.blue); const card=el('article','gpu-card');card.style.setProperty('--accent',index%2?colors.amber:colors.blue);

File diff suppressed because one or more lines are too long

View File

@ -84,6 +84,12 @@ h1{font-size:28px;letter-spacing:.3px;font-weight:600;margin:7px 0 5px;line-heig
.metric-bottom{display:flex;justify-content:space-between;align-items:center;margin-top:15px;gap:5px;white-space:nowrap;font-size:10px;color:#8195a3} .metric-bottom{display:flex;justify-content:space-between;align-items:center;margin-top:15px;gap:5px;white-space:nowrap;font-size:10px;color:#8195a3}
.metric-bottom strong{color:#b9c6cd;font-size:10px;font-weight:400} .metric-bottom strong{color:#b9c6cd;font-size:10px;font-weight:400}
.gpu-online-label,.disk-health-detail{font-size:11px;color:#a4b5bf;display:flex;gap:6px;align-items:center;height:4px;line-height:16px;margin-top:17px;margin-bottom:5px} .gpu-online-label,.disk-health-detail{font-size:11px;color:#a4b5bf;display:flex;gap:6px;align-items:center;height:4px;line-height:16px;margin-top:17px;margin-bottom:5px}
.gpu-config-controls{display:flex;align-items:center;flex-wrap:wrap;gap:7px;margin-top:18px;font-size:11px;color:var(--muted)}
.gpu-config-controls select,.gpu-config-controls button{font:inherit;color:#d9e7ed;background:#0f1b22;border:1px solid #38515f;border-radius:5px;padding:6px 8px;min-height:32px}
.gpu-config-controls button{color:var(--green);cursor:pointer}
.gpu-config-controls select:focus-visible,.gpu-config-controls button:focus-visible{outline:2px solid var(--green);outline-offset:2px}
.gpu-config-controls :disabled{opacity:.5;cursor:wait}
.gpu-config-hint{font-size:10px;line-height:1.6;color:var(--muted);margin:9px 0 0;overflow-wrap:anywhere}
.disk-health-detail{height:10px;margin-top:5px} .disk-health-detail{height:10px;margin-top:5px}
.section-heading{display:flex;align-items:center;justify-content:space-between;gap:18px;margin:34px 0 16px;scroll-margin-top:24px} .section-heading{display:flex;align-items:center;justify-content:space-between;gap:18px;margin:34px 0 16px;scroll-margin-top:24px}
.section-heading h2{font-size:19px;font-weight:600;letter-spacing:.4px;margin:5px 0} .section-heading h2{font-size:19px;font-weight:600;letter-spacing:.4px;margin:5px 0}
@ -94,6 +100,7 @@ h1{font-size:28px;letter-spacing:.3px;font-weight:600;margin:7px 0 5px;line-heig
.compact span{font-size:10px;color:var(--muted)} .compact span{font-size:10px;color:var(--muted)}
.subtle-pill{border:1px solid var(--line);padding:3px 9px;border-radius:4px;white-space:nowrap;font:10px var(--mono)!important;color:#91a6b3!important} .subtle-pill{border:1px solid var(--line);padding:3px 9px;border-radius:4px;white-space:nowrap;font:10px var(--mono)!important;color:#91a6b3!important}
.gpu-grid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:14px} .gpu-grid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:14px}
.gpu-grid.single{grid-template-columns:minmax(0,1fr)}
.gpu-card{border:1px solid var(--line);border-radius:9px;background:var(--panel);padding:19px 21px;position:relative;overflow:hidden;--accent:var(--blue)} .gpu-card{border:1px solid var(--line);border-radius:9px;background:var(--panel);padding:19px 21px;position:relative;overflow:hidden;--accent:var(--blue)}
.gpu-card::before{content:"";position:absolute;top:19px;bottom:19px;left:0;width:3px;background:var(--accent);border-radius:0 4px 4px 0} .gpu-card::before{content:"";position:absolute;top:19px;bottom:19px;left:0;width:3px;background:var(--accent);border-radius:0 4px 4px 0}
.gpu-card .gpu-head{display:flex;gap:10px;align-items:center} .gpu-card .gpu-head{display:flex;gap:10px;align-items:center}

View File

@ -2,6 +2,7 @@
(()=>{ (()=>{
'use strict'; 'use strict';
let auth={logged_in:false},tasks=[],listBusy=false,listRefreshPending=false,selectedRun=null,runsTask=null,confirmResolve=null,runReadSequence=0; let auth={logged_in:false},tasks=[],listBusy=false,listRefreshPending=false,selectedRun=null,runsTask=null,confirmResolve=null,runReadSequence=0;
let pendingGpuSave=false;
const states={running:['执行中','running'],success:['执行成功','success'],failed:['执行失败','failed'],timeout:['超时已停止','failed'],cancelled:['已停止','neutral'],interrupted:['执行被中断','warning']}; const states={running:['执行中','running'],success:['执行成功','success'],failed:['执行失败','failed'],timeout:['超时已停止','failed'],cancelled:['已停止','neutral'],interrupted:['执行被中断','warning']};
function taskError(id,message){$(id).textContent=message;$(id).hidden=!message;} function taskError(id,message){$(id).textContent=message;$(id).hidden=!message;}
async function request(path,body){ async function request(path,body){
@ -32,9 +33,19 @@ function openLogin(){taskError('task-login-error','');$('task-login-password').v
$('task-login-open').addEventListener('click',openLogin); $('task-login-open').addEventListener('click',openLogin);
$('task-login-form').addEventListener('submit',async event=>{ $('task-login-form').addEventListener('submit',async event=>{
event.preventDefault();const button=event.submitter;button.disabled=true;taskError('task-login-error',''); event.preventDefault();const button=event.submitter;button.disabled=true;taskError('task-login-error','');
try{const result=await request('/login',{username:$('task-login-user').value.trim(),password:$('task-login-password').value});setAuth(result);$('task-login-password').value='';$('task-login-dialog').close();await refreshList();toast('已登录定时任务管理。');} try{const result=await request('/login',{username:$('task-login-user').value.trim(),password:$('task-login-password').value});setAuth(result);$('task-login-password').value='';$('task-login-dialog').close();await refreshList();if(pendingGpuSave){pendingGpuSave=false;await saveGpuCount();}else toast('管理员已登录。');}
catch(error){taskError('task-login-error',error.message);}finally{button.disabled=false;} catch(error){taskError('task-login-error',error.message);}finally{button.disabled=false;}
}); });
$('gpu-expected-count').addEventListener('change',()=>{$('gpu-expected-count').dataset.dirty='true';$('gpu-config-hint').textContent='尚未保存。保存后按所选数量检查,不再绑定原显卡。';});
async function saveGpuCount(){
if(!auth.logged_in){pendingGpuSave=true;openLogin();$('gpu-config-hint').textContent='请使用现有管理员账号登录,登录后会保存本次选择。';return;}
const select=$('gpu-expected-count');select.dataset.saving='true';select.disabled=true;$('gpu-config-save').disabled=true;
try{const result=await request('/gpu-count',{expected_gpu_count:Number(select.value)});select.dataset.pending=String(result.expected_gpu_count);select.dataset.dirty='true';$('gpu-config-hint').textContent=result.message+' 最迟下次采样更新。';toast(result.message);await loadStatus();setTimeout(()=>{loadStatus();loadEvents();},1500);}
catch(error){$('gpu-config-hint').textContent=error.message;if(error.status===401){pendingGpuSave=true;openLogin();}else toast(error.message);}
finally{delete select.dataset.saving;select.disabled=false;$('gpu-config-save').disabled=false;}
}
$('gpu-config-save').addEventListener('click',saveGpuCount);
$('task-login-dialog').addEventListener('close',()=>{if(!auth.logged_in)pendingGpuSave=false;});
$('task-logout').addEventListener('click',async()=>{try{await request('/logout',{});setAuth({logged_in:false});document.querySelectorAll('.task-dialog[open]').forEach(dialog=>dialog.close());$('task-body').value='';$('task-run-output').textContent='';toast('已退出任务管理。');}catch(error){toast(error.message);}}); $('task-logout').addEventListener('click',async()=>{try{await request('/logout',{});setAuth({logged_in:false});document.querySelectorAll('.task-dialog[open]').forEach(dialog=>dialog.close());$('task-body').value='';$('task-run-output').textContent='';toast('已退出任务管理。');}catch(error){toast(error.message);}});
document.querySelectorAll('[data-close-dialog]').forEach(button=>button.addEventListener('click',()=>$(button.dataset.closeDialog).close())); document.querySelectorAll('[data-close-dialog]').forEach(button=>button.addEventListener('click',()=>$(button.dataset.closeDialog).close()));
function confirmAction(title,message,code=''){ function confirmAction(title,message,code=''){
@ -136,7 +147,7 @@ $('task-create').addEventListener('click',()=>openEditor().catch(error=>toast(er
$('task-preview').addEventListener('click',async()=>{if(!$('task-editor-form').reportValidity())return;const button=$('task-preview');button.disabled=true;try{const result=await request('/preview',formDefinition());$('task-preview-text').textContent=result.schedule_label+';预计下次:'+(result.next_run?when(result.next_run):'没有可计算的将来时间');}catch(error){$('task-preview-text').textContent=error.message;}finally{button.disabled=false;}}); $('task-preview').addEventListener('click',async()=>{if(!$('task-editor-form').reportValidity())return;const button=$('task-preview');button.disabled=true;try{const result=await request('/preview',formDefinition());$('task-preview-text').textContent=result.schedule_label+';预计下次:'+(result.next_run?when(result.next_run):'没有可计算的将来时间');}catch(error){$('task-preview-text').textContent=error.message;}finally{button.disabled=false;}});
$('task-editor-form').addEventListener('submit',async event=>{ $('task-editor-form').addEventListener('submit',async event=>{
event.preventDefault();taskError('task-editor-error','');const definition=formDefinition(); event.preventDefault();taskError('task-editor-error','');const definition=formDefinition();
const dangerous=/\b(reboot|shutdown|poweroff|halt)\b|\binit\s+[06]\b/i.test(definition.body+' '+definition.path); const dangerous=/\b(reboot|shutdown|poweroff|halt)\b|\binit\s+[06]\b|night-power-cycle\.py/i.test(definition.body+' '+definition.path);
if(definition.enabled&&!await confirmAction(dangerous?'保存并启用重启/关机类任务?':'保存并启用任务?',dangerous?'此任务将在计划时间中断服务器业务。请确认脚本和执行时段已核对。':'保存后会按设置的周期执行,不会立即运行。',definition.mode==='file'?definition.path:definition.body.slice(0,800)))return; if(definition.enabled&&!await confirmAction(dangerous?'保存并启用重启/关机类任务?':'保存并启用任务?',dangerous?'此任务将在计划时间中断服务器业务。请确认脚本和执行时段已核对。':'保存后会按设置的周期执行,不会立即运行。',definition.mode==='file'?definition.path:definition.body.slice(0,800)))return;
$('task-save').disabled=true; $('task-save').disabled=true;
try{const result=await request('/save',definition);$('task-editor-dialog').close();toast(result.message);await refreshList();} try{const result=await request('/save',definition);$('task-editor-dialog').close();toast(result.message);await refreshList();}
@ -146,6 +157,7 @@ $('task-template').addEventListener('change',()=>{
const type=$('task-template').value;$('task-enabled').checked=false;$('task-template-hint').hidden=type==='custom'; const type=$('task-template').value;$('task-enabled').checked=false;$('task-template-hint').hidden=type==='custom';
if(type==='hardware'){$('task-name').value='检查显卡状态';$('task-mode').value='command';$('task-body').value='/usr/bin/nvidia-smi';$('task-cycle').value='daily';$('task-time').value='08:00';$('task-template-hint').textContent='只读取显卡状态,结果会出现在执行日志。可先保存为停用,再执行一次测试。';} if(type==='hardware'){$('task-name').value='检查显卡状态';$('task-mode').value='command';$('task-body').value='/usr/bin/nvidia-smi';$('task-cycle').value='daily';$('task-time').value='08:00';$('task-template-hint').textContent='只读取显卡状态,结果会出现在执行日志。可先保存为停用,再执行一次测试。';}
if(type==='reboot'){$('task-name').value='服务器定时重启';$('task-mode').value='command';$('task-body').value='/usr/bin/systemctl reboot';$('task-cycle').value='weekly';$('task-time').value='03:00';$('task-weekdays').querySelectorAll('input').forEach(input=>input.checked=input.value==='6');$('task-template-hint').textContent='这条命令会重启整台服务器,正在运行的 GPU 任务会中断。模板默认停用,请先确认允许重启的时间。';} if(type==='reboot'){$('task-name').value='服务器定时重启';$('task-mode').value='command';$('task-body').value='/usr/bin/systemctl reboot';$('task-cycle').value='weekly';$('task-time').value='03:00';$('task-weekdays').querySelectorAll('input').forEach(input=>input.checked=input.value==='6');$('task-template-hint').textContent='这条命令会重启整台服务器,正在运行的 GPU 任务会中断。模板默认停用,请先确认允许重启的时间。';}
if(type==='night_poweroff'){$('task-name').value='夜间节电:00:00 关机,06:00 开机';$('task-mode').value='command';$('task-body').value='/usr/bin/python3 /opt/hardware-dashboard/scripts/night-power-cycle.py';$('task-cycle').value='daily';$('task-time').value='00:00';$('task-notes').value='北京时间。先设置并回读当天 06:00 RTC 闹钟,再正常关机。需要主板支持关机状态 RTC 唤醒,并保持供电。';$('task-template-hint').textContent='每天 00:00 中断计算任务并关机,06:00 由主板唤醒。脚本只允许在 00:00–00:04 关机;检查环境请在命令末尾加 --check。若已有同名计划,请编辑已有计划,避免重复添加。模板默认停用。';}
if(type==='certificate'){$('task-name').value='SSL 证书续签';$('task-mode').value='script';$('task-interpreter').value='bash';$('task-cycle').value='daily';$('task-time').value='03:10';$('task-body').value='#!/usr/bin/env bash\nset -euo pipefail\n# 先确认已有域名和续签配置,且没有其他重复的自动续签计划。\nif command -v certbot >/dev/null 2>&1; then\n certbot renew --quiet\nelif [ -x /root/.acme.sh/acme.sh ]; then\n /root/.acme.sh/acme.sh --cron --home /root/.acme.sh\nelse\n echo "未检测到 Certbot 或 acme.sh,请先配置证书续签工具。" >&2\n exit 1\nfi\n# 如需重新加载 Web 服务,请按实际服务另行配置 deploy-hook。';$('task-template-hint').textContent='这是续签示例,不会安装或配置域名。请先确认 Certbot/acme.sh 的现有配置,并检查是否已有续签任务。';} if(type==='certificate'){$('task-name').value='SSL 证书续签';$('task-mode').value='script';$('task-interpreter').value='bash';$('task-cycle').value='daily';$('task-time').value='03:10';$('task-body').value='#!/usr/bin/env bash\nset -euo pipefail\n# 先确认已有域名和续签配置,且没有其他重复的自动续签计划。\nif command -v certbot >/dev/null 2>&1; then\n certbot renew --quiet\nelif [ -x /root/.acme.sh/acme.sh ]; then\n /root/.acme.sh/acme.sh --cron --home /root/.acme.sh\nelse\n echo "未检测到 Certbot 或 acme.sh,请先配置证书续签工具。" >&2\n exit 1\nfi\n# 如需重新加载 Web 服务,请按实际服务另行配置 deploy-hook。';$('task-template-hint').textContent='这是续签示例,不会安装或配置域名。请先确认 Certbot/acme.sh 的现有配置,并检查是否已有续签任务。';}
updateEditorFields(); updateEditorFields();
}); });

View File

@ -189,7 +189,7 @@ def validate_definition(payload,check_paths=True):
notes=str(payload.get('notes','')).strip()[:400] notes=str(payload.get('notes','')).strip()[:400]
timeout=number(payload.get('timeout',1800),10,86400,'最长执行秒数') timeout=number(payload.get('timeout',1800),10,86400,'最长执行秒数')
if not isinstance(payload.get('enabled',False),bool):raise TaskError('启用状态必须为是或否。') if not isinstance(payload.get('enabled',False),bool):raise TaskError('启用状态必须为是或否。')
dangerous=bool(re.search(r'\b(reboot|shutdown|poweroff|halt)\b|\binit\s+[06]\b',body+' '+path,re.I)) dangerous=bool(re.search(r'\b(reboot|shutdown|poweroff|halt)\b|\binit\s+[06]\b|night-power-cycle\.py',body+' '+path,re.I))
return {'name':name,'mode':mode,'interpreter':interpreter,'body':body,'path':path,'arguments':arguments, return {'name':name,'mode':mode,'interpreter':interpreter,'body':body,'path':path,'arguments':arguments,
'cwd':cwd,'schedule':payload.get('schedule',{}),'schedule_lines':lines,'schedule_label':label, 'cwd':cwd,'schedule':payload.get('schedule',{}),'schedule_lines':lines,'schedule_label':label,
'timeout':timeout,'notes':notes,'enabled':bool(payload.get('enabled',False)), 'timeout':timeout,'notes':notes,'enabled':bool(payload.get('enabled',False)),

View File

@ -20,6 +20,7 @@ import time
import task_core as core import task_core as core
ADMIN=Path('/etc/hardware-task-admin.json') ADMIN=Path('/etc/hardware-task-admin.json')
MONITOR_CONFIG=Path('/etc/hardware-watch.json')
SOCKET=Path('/run/hardware-taskd/control.sock') SOCKET=Path('/run/hardware-taskd/control.sock')
SESSIONS={};ATTEMPTS={};GUARD=threading.RLock();MUTATE=threading.RLock() SESSIONS={};ATTEMPTS={};GUARD=threading.RLock();MUTATE=threading.RLock()
ALLOWED_UID=None ALLOWED_UID=None
@ -110,6 +111,7 @@ def dispatch(request):
next_times=[time.time()+int(definition['schedule_lines'][0].split('=')[1][:-1])] next_times=[time.time()+int(definition['schedule_lines'][0].split('=')[1][:-1])]
return {'schedule_label':definition['schedule_label'],'next_run':min(next_times) if next_times else None,'dangerous':definition['dangerous']} return {'schedule_label':definition['schedule_label'],'next_run':min(next_times) if next_times else None,'dangerous':definition['dangerous']}
with MUTATE: with MUTATE:
if action=='monitor.gpu_count':return save_gpu_count(payload,client)
if action=='tasks.save':return core.save_task(payload,client) if action=='tasks.save':return core.save_task(payload,client)
if action=='tasks.enable': if action=='tasks.enable':
if not isinstance(payload.get('enabled'),bool):raise core.TaskError('启用状态格式不正确。') if not isinstance(payload.get('enabled'),bool):raise core.TaskError('启用状态格式不正确。')
@ -119,6 +121,26 @@ def dispatch(request):
if action=='tasks.stop':return core.stop_task(payload.get('id'),client) if action=='tasks.stop':return core.stop_task(payload.get('id'),client)
raise core.TaskError('不支持这个任务操作。',404) raise core.TaskError('不支持这个任务操作。',404)
def save_gpu_count(payload,client=''):
count=payload.get('expected_gpu_count')
if type(count) is not int or count not in (1,2):
raise core.TaskError('请选择使用 1 张或 2 张显卡。')
config=json.loads(MONITOR_CONFIG.read_text())
if not isinstance(config,dict):raise core.TaskError('硬件监控配置格式不正确。',500)
config.update(expected_gpu_count=count,expected_gpu_uuids=[])
mode=MONITOR_CONFIG.stat().st_mode & 0o777
core.atom_write(MONITOR_CONFIG,json.dumps(config,ensure_ascii=False,indent=2)+'\n',mode)
# Replacement drops any ACL on the old file. Grant the dashboard's service
# group read access while keeping configuration writes restricted to root.
if pwd is not None and os.geteuid()==0:
import grp
os.chown(MONITOR_CONFIG,0,grp.getgrnam('hardware-watch').gr_gid)
os.chmod(MONITOR_CONFIG,0o640)
core.audit('gpu_count_changed',client=client,detail=f'expected_gpu_count={count}; identity=count')
refresh=core.execute(['systemctl','start','--no-block','hardware-watch.service'],check=False)
return {'expected_gpu_count':count,'message':f'已设置使用 {count} 张显卡;按数量检查,允许更换显卡。',
'refresh_pending':refresh.returncode==0}
class Handler(socketserver.StreamRequestHandler): class Handler(socketserver.StreamRequestHandler):
def handle(self): def handle(self):
self.request.settimeout(25) self.request.settimeout(25)

57
tests/test_power_cycle.py Normal file
View File

@ -0,0 +1,57 @@
import datetime as dt
import importlib.util
from pathlib import Path
import subprocess
import unittest
from unittest.mock import patch
spec = importlib.util.spec_from_file_location('power_cycle', Path(__file__).resolve().parents[1] / 'scripts/night-power-cycle.py')
power = importlib.util.module_from_spec(spec)
spec.loader.exec_module(power)
class PowerCycleTests(unittest.TestCase):
def setUp(self):
self.now = dt.datetime(2026, 11, 10, 0, 0, 1, tzinfo=power.TZ)
def test_six_am_uses_shanghai_time_even_when_input_is_utc(self):
target = power.wake_time(self.now.astimezone(dt.timezone.utc))
self.assertEqual(target.isoformat(), '2026-11-10T06:00:00+08:00')
tomorrow = power.wake_time(self.now.replace(hour=18))
self.assertEqual(tomorrow.isoformat(), '2026-11-11T06:00:00+08:00')
def test_dry_check_and_daytime_run_never_shutdown(self):
with patch.object(power.shutil, 'which', side_effect=lambda name: '/usr/bin/' + name), \
patch.object(Path, 'exists', return_value=True), \
patch.object(power.subprocess, 'run') as run:
power.power_cycle(self.now, check=True)
run.assert_not_called()
with self.assertRaises(RuntimeError):
power.power_cycle(self.now.replace(hour=18))
run.assert_not_called()
def test_shutdown_only_after_alarm_is_armed_and_verified(self):
target = power.wake_time(self.now)
with patch.object(power.shutil, 'which', side_effect=lambda name: '/usr/bin/' + name), \
patch.object(Path, 'exists', return_value=True), \
patch.object(Path, 'read_text', return_value=str(int(target.timestamp()))), \
patch.object(power.subprocess, 'run') as run:
power.power_cycle(self.now)
self.assertEqual(run.call_count, 2)
self.assertIn('no', run.call_args_list[0].args[0])
self.assertIn(str(int(target.timestamp())), run.call_args_list[0].args[0])
self.assertEqual(run.call_args_list[1].args[0], ['/usr/bin/systemctl', '--no-block', 'poweroff'])
def test_failed_or_wrong_alarm_prevents_shutdown(self):
with patch.object(power.shutil, 'which', side_effect=lambda name: '/usr/bin/' + name), \
patch.object(Path, 'exists', return_value=True), \
patch.object(Path, 'read_text', return_value='0'), \
patch.object(power.subprocess, 'run') as run:
with self.assertRaises(RuntimeError):
power.power_cycle(self.now)
self.assertEqual(run.call_count, 1)
run.reset_mock()
run.side_effect = subprocess.CalledProcessError(1, 'rtcwake')
with self.assertRaises(subprocess.CalledProcessError):
power.power_cycle(self.now)
self.assertEqual(run.call_count, 1)

View File

@ -96,4 +96,35 @@ class TaskTests(unittest.TestCase):
self.assertTrue(result['tail_only']);self.assertEqual(len(result['output']),131072) self.assertTrue(result['tail_only']);self.assertEqual(len(result['output']),131072)
with self.assertRaises(core.TaskError):core.read_run('../password') with self.assertRaises(core.TaskError):core.read_run('../password')
def test_gpu_count_requires_login_csrf_and_preserves_other_config(self):
config=Path(self.temp.name)/'hardware-watch.json'
config.write_text(json.dumps({'expected_gpu_count':2,'expected_gpu_uuids':['old-p100'],
'expected_disks':['nvme0n1'],'gpu_warn_c':85}))
with patch.object(taskd,'MONITOR_CONFIG',config),patch.object(core,'execute') as execute:
execute.return_value.returncode=0
request={'action':'monitor.gpu_count','payload':{'expected_gpu_count':1}}
with self.assertRaises(core.TaskError) as error:taskd.dispatch(request)
self.assertEqual(error.exception.status,401)
login=taskd.login({'username':'admin','password':'only-a-test-password'},'qa')
request['token']=login['token']
with self.assertRaises(core.TaskError) as error:taskd.dispatch(request)
self.assertEqual(error.exception.status,403)
request['csrf']=login['csrf']
for bad in [True,0,3,'1',1.0]:
request['payload']={'expected_gpu_count':bad}
with self.subTest(value=bad),self.assertRaises(core.TaskError):taskd.dispatch(request)
for count in (1,2):
request['payload']={'expected_gpu_count':count}
taskd.dispatch(request)
saved=json.loads(config.read_text())
self.assertEqual(saved['expected_gpu_count'],count)
self.assertEqual(saved['expected_gpu_uuids'],[])
self.assertEqual(saved['expected_disks'],['nvme0n1'])
self.assertEqual(saved['gpu_warn_c'],85)
self.assertEqual(execute.call_count,2)
def test_night_cycle_is_identified_as_shutdown_task(self):
definition=core.validate_definition(self.definition(body='/usr/bin/python3 /opt/hardware-dashboard/scripts/night-power-cycle.py'),check_paths=False)
self.assertTrue(definition['dangerous'])
if __name__=='__main__':unittest.main(verbosity=2) if __name__=='__main__':unittest.main(verbosity=2)