feat(backup): mysqldump 定时备份/恢复演练/健康检查脚本与手册
- scripts/backup_full.sh：容器内 mysqldump --single-transaction，gzip 落盘 db/backups/，保留 30 天；凭据 source docker/.env 不硬编码；实测备份成功 - scripts/restore_drill.sh：恢复到临时库 payroll_drill_* 并逐表核对行数 （16 张关键表），默认不 DROP、--drop-exec 才删——绝不覆盖生产库； 实测 16 表全 ✓。规避嵌套 shell 吞反引号（mysqladmin create/drop、 双引号包裹表名） - scripts/health_check.sh：六项检查（容器健康×2/磁盘水位/备份新鲜度/ Redis 探活/24h 登录失败阈值），PASS/WARN/FAIL 报告，退出码供监控； Redis 密码经宿主机 .env 验证（容器内无该环境变量），实测六项全 PASS - docs/备份恢复手册.md：备份/演练/生产恢复七步/定时 schtasks 两则， db/backups/ 已入 .gitignore Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
liuwenxin
parent
927592b472
commit
b5e27810fd
70
docs/备份恢复手册.md
Normal file
70
docs/备份恢复手册.md
Normal file
@ -0,0 +1,70 @@
|
|||||||
|
# 备份恢复手册（⑨ 备份恢复与运维检查）
|
||||||
|
|
||||||
|
> 用户强调「这个备份恢复很重要」。**备份不演练 = 没有备份**——本手册的一页纸目标：任何运维/开发拿到都能完成「备份 → 恢复到临时库核对 → 删临时库」全链路。
|
||||||
|
|
||||||
|
## 一、备份（每日一次，建议凌晨低峰）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 项目根目录执行
|
||||||
|
bash scripts/backup_full.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
- 产物：`db/backups/payroll_YYYYMMDD_HHMMSS.sql.gz`（gzip 压缩）
|
||||||
|
- 保留策略：自动清理 30 天前的旧备份
|
||||||
|
- 凭据：从 `docker/.env` 读取（不入脚本、不入版本库）
|
||||||
|
|
||||||
|
**Windows 定时任务**（任务计划程序 / `schtasks`）：
|
||||||
|
|
||||||
|
```cmd
|
||||||
|
:: 每日 02:30 备份（经 Git Bash）
|
||||||
|
schtasks /create /tn "ys-payroll-backup" /tr "\"C:\Program Files\Git\bin\bash.exe\" -lc \"bash scripts/backup_full.sh\"" /sc daily /st 02:30 /ru SYSTEM
|
||||||
|
```
|
||||||
|
|
||||||
|
## 二、恢复演练（备份后顺手跑一次，验证备份真能用）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/restore_drill.sh # 恢复到临时库 + 行数核对，临时库默认保留
|
||||||
|
bash scripts/restore_drill.sh --drop-exec # 核对通过后再加此参数删除临时库
|
||||||
|
```
|
||||||
|
|
||||||
|
- **绝不直接覆盖生产库**——演练恢复到独立临时库 `payroll_drill_<时间戳>`
|
||||||
|
- 行数对比覆盖 16 张关键表（users/roles/param_*/business*/personnel/products/customers/shipments*/receipts/approval_*/system_settings）
|
||||||
|
- 全部 ✓ 即备份可恢复；出现 ✗ 时临时库保留供排查
|
||||||
|
|
||||||
|
## 三、生产恢复（真出事了，七步）
|
||||||
|
|
||||||
|
> ⚠️ 生产恢复是**高危操作**：先把容器停了再恢复，避免写冲突。下面假设恢复到**全新空数据卷**（最干净路径）。
|
||||||
|
|
||||||
|
1. **停容器保卷**：`docker compose -f docker/docker-compose.dev.yml down`
|
||||||
|
2. **删旧卷（谨慎！确认要全量重来才执行）**：`docker volume rm ys-payroll_mysql-dev-data`
|
||||||
|
- 若只想恢复到某个时间点**且不丢其它数据**，改用「恢复到临时库 → mysqldump 单表导出 → 手工回灌」的细粒度路径，勿整体覆盖。
|
||||||
|
3. **起空库**：`docker compose -f docker/docker-compose.dev.yml up -d`（首启自动跑 `db/init/init-dev.sql` 建表 + 种子）
|
||||||
|
4. **选一份备份**：`ls -t db/backups/payroll_*.sql.gz | head`
|
||||||
|
5. **导入该备份**：
|
||||||
|
```bash
|
||||||
|
gunzip -c db/backups/payroll_XXXXXXXX_XXXXXX.sql.gz | \
|
||||||
|
docker exec -i ys-mysql-dev sh -c 'mysql -uroot -p"$MYSQL_ROOT_PASSWORD" payroll'
|
||||||
|
```
|
||||||
|
6. **核对行数**：再跑一次 `bash scripts/restore_drill.sh`（此时源库=刚恢复的库，应全 ✓）
|
||||||
|
7. **重启后端**：`cd backend && uv run uvicorn app.main:app --reload --port 8000`，登录抽查几个页面确认数据回来
|
||||||
|
|
||||||
|
## 四、运维健康检查（每日一次）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/health_check.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
六项检查：容器健康 ×2 / 磁盘水位（80% 阈值）/ 备份新鲜度（24h 内）/ Redis 探活 / 24h 登录失败次数（暴力破解预警）。任一项 WARN/FAIL 时退出码非零，方便监控采集。
|
||||||
|
|
||||||
|
**Windows 定时**：
|
||||||
|
|
||||||
|
```cmd
|
||||||
|
:: 每日 08:50 检查（上班前出报告）
|
||||||
|
schtasks /create /tn "ys-payroll-health" /tr "\"C:\Program Files\Git\bin\bash.exe\" -lc \"bash scripts/health_check.sh >> db/backups/health.log 2>&1\"" /sc daily /st 08:50 /ru SYSTEM
|
||||||
|
```
|
||||||
|
|
||||||
|
## 五、注意事项
|
||||||
|
|
||||||
|
- 备份含**全部业务数据与参数种子**，属敏感文件——`db/backups/` 已加入 `.gitignore`，**不要**把它提交到版本库
|
||||||
|
- 恢复演练产生的 `payroll_drill_*` 库会占磁盘，核对完及时 `--drop-exec` 清理
|
||||||
|
- 备份脚本与恢复脚本都只依赖 Docker（无需宿主机装 MySQL 客户端）
|
||||||
58
scripts/backup_full.sh
Normal file
58
scripts/backup_full.sh
Normal file
@ -0,0 +1,58 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# 衍水工资系统 —— MySQL 全量备份（⑨ 备份恢复与运维检查）
|
||||||
|
#
|
||||||
|
# 思路：在跑中的容器内执行 mysqldump，免暴露 3306 端口给宿主机工具，
|
||||||
|
# 也避免宿主机装 MySQL 客户端。凭据从 docker/.env 读取（不入脚本、不入版本库）。
|
||||||
|
#
|
||||||
|
# 用法：在项目根目录执行 bash scripts/backup_full.sh
|
||||||
|
# 产物：db/backups/payroll_YYYYMMDD_HHMMSS.sql.gz，保留最近 30 天。
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
cd "$ROOT"
|
||||||
|
|
||||||
|
if [[ -f docker/.env ]]; then
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source docker/.env
|
||||||
|
else
|
||||||
|
echo "ERROR: docker/.env 不存在（请先创建 docker/.env 并填入 MySQL 凭据）" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
: "${MYSQL_DATABASE:?docker/.env 缺 MYSQL_DATABASE}"
|
||||||
|
: "${MYSQL_ROOT_PASSWORD:?docker/.env 缺 MYSQL_ROOT_PASSWORD}"
|
||||||
|
|
||||||
|
BACKUP_DIR="$ROOT/db/backups"
|
||||||
|
mkdir -p "$BACKUP_DIR"
|
||||||
|
STAMP="$(date +%Y%m%d_%H%M%S)"
|
||||||
|
OUT="$BACKUP_DIR/payroll_${STAMP}.sql.gz"
|
||||||
|
|
||||||
|
# 容器名固定（compose 显式 container_name），与 docker/docker-compose.dev.yml 保持一致
|
||||||
|
CONTAINER="ys-mysql-dev"
|
||||||
|
if ! docker inspect "$CONTAINER" >/dev/null 2>&1; then
|
||||||
|
echo "ERROR: 容器 $CONTAINER 未运行（先 docker compose up -d）" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "开始备份 $MYSQL_DATABASE → $(basename "$OUT")"
|
||||||
|
docker exec "$CONTAINER" sh -c \
|
||||||
|
"mysqldump -uroot -p\"\$MYSQL_ROOT_PASSWORD\" --single-transaction --routines --triggers \"$MYSQL_DATABASE\"" \
|
||||||
|
| gzip > "$OUT"
|
||||||
|
|
||||||
|
if [[ ! -s "$OUT" ]]; then
|
||||||
|
echo "ERROR: 备份文件为空（数据库名或凭据可能不对）" >&2
|
||||||
|
rm -f "$OUT"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
SIZE="$(du -h "$OUT" | cut -f1)"
|
||||||
|
echo "备份完成：$(basename "$OUT") ($SIZE)"
|
||||||
|
|
||||||
|
# 保留 30 天
|
||||||
|
find "$BACKUP_DIR" -maxdepth 1 -name 'payroll_*.sql.gz' -mtime +30 -print -delete
|
||||||
|
echo "已清理 30 天前的旧备份"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "最新备份："
|
||||||
|
ls -lh "$BACKUP_DIR"/*.sql.gz 2>/dev/null | tail -3
|
||||||
105
scripts/health_check.sh
Normal file
105
scripts/health_check.sh
Normal file
@ -0,0 +1,105 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# 运维健康检查 —— 容器/磁盘/备份新鲜度/登录异常（⑨ 输出一份可读报告）
|
||||||
|
#
|
||||||
|
# 用法：在项目根目录执行 bash scripts/health_check.sh
|
||||||
|
# 每项 PASS/WARN/FAIL，最后给出结论。WARN/FAIL 时退出码非零提示运维介入。
|
||||||
|
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
cd "$ROOT"
|
||||||
|
|
||||||
|
if [[ -f docker/.env ]]; then
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source docker/.env
|
||||||
|
fi
|
||||||
|
|
||||||
|
ISSUES=0
|
||||||
|
report() { printf "%-10s %s\n" "$1" "$2"; }
|
||||||
|
warn() { report "WARN" "$1"; ISSUES=$((ISSUES+1)); }
|
||||||
|
fail() { report "FAIL" "$1"; ISSUES=$((ISSUES+1)); }
|
||||||
|
pass() { report "PASS" "$1"; }
|
||||||
|
|
||||||
|
echo "======== 衍水工资系统 健康检查 $(date '+%Y-%m-%d %H:%M:%S') ========"
|
||||||
|
|
||||||
|
# 1. 容器健康
|
||||||
|
for C in ys-mysql-dev ys-redis-dev; do
|
||||||
|
if ! docker inspect "$C" >/dev/null 2>&1; then
|
||||||
|
fail "容器 $C 不存在"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
STATUS="$(docker inspect -f '{{.State.Health.Status}}' "$C" 2>/dev/null || echo none)"
|
||||||
|
RUNNING="$(docker inspect -f '{{.State.Running}}' "$C" 2>/dev/null)"
|
||||||
|
if [[ "$RUNNING" == "true" && "$STATUS" == "healthy" ]]; then
|
||||||
|
pass "容器 $C 运行中且健康"
|
||||||
|
elif [[ "$RUNNING" == "true" ]]; then
|
||||||
|
warn "容器 $C 运行中但健康检查=$STATUS"
|
||||||
|
else
|
||||||
|
fail "容器 $C 未运行"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# 2. 磁盘水位
|
||||||
|
DISK_PCT="$(df -h "$ROOT" | awk 'NR==2 {gsub(/%/,"",$5); print $5}')"
|
||||||
|
if [[ -z "$DISK_PCT" ]]; then
|
||||||
|
warn "磁盘信息读取失败"
|
||||||
|
elif (( DISK_PCT < 80 )); then
|
||||||
|
pass "磁盘使用率 ${DISK_PCT}%（阈值 80%）"
|
||||||
|
elif (( DISK_PCT < 90 )); then
|
||||||
|
warn "磁盘使用率 ${DISK_PCT}%（建议清理）"
|
||||||
|
else
|
||||||
|
fail "磁盘使用率 ${DISK_PCT}%（高危，立即清理）"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 3. 备份新鲜度
|
||||||
|
BACKUP_DIR="$ROOT/db/backups"
|
||||||
|
LATEST="$(ls -t "$BACKUP_DIR"/payroll_*.sql.gz 2>/dev/null | head -1)"
|
||||||
|
if [[ -z "$LATEST" ]]; then
|
||||||
|
warn "未找到任何备份（应每日备份）"
|
||||||
|
else
|
||||||
|
AGE_H=$(( ($(date +%s) - $(stat -c %Y "$LATEST")) / 3600 ))
|
||||||
|
if (( AGE_H < 24 )); then
|
||||||
|
pass "最新备份 $(basename "$LATEST")，${AGE_H} 小时前"
|
||||||
|
elif (( AGE_H < 48 )); then
|
||||||
|
warn "最新备份已 ${AGE_H} 小时前（超过 24 小时）"
|
||||||
|
else
|
||||||
|
fail "最新备份已 ${AGE_H} 小时前（备份任务可能已停）"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 4. Redis 存活（密码经宿主 docker/.env，不经命令行明文参数）
|
||||||
|
if [[ -n "${REDIS_PASSWORD:-}" ]]; then
|
||||||
|
if docker exec ys-redis-dev redis-cli --no-auth-warning -a "$REDIS_PASSWORD" ping 2>/dev/null | grep -q PONG; then
|
||||||
|
pass "Redis 响应 PONG"
|
||||||
|
else
|
||||||
|
warn "Redis ping 无响应（检查 REDIS_PASSWORD 或容器状态）"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
warn "docker/.env 缺 REDIS_PASSWORD，跳过 Redis 探活"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 5. 登录日志异常（24h 失败次数）
|
||||||
|
if [[ -n "${MYSQL_ROOT_PASSWORD:-}" ]] && docker inspect ys-mysql-dev >/dev/null 2>&1; then
|
||||||
|
FAILS=$(docker exec ys-mysql-dev sh -c \
|
||||||
|
"mysql -uroot -p\"\$MYSQL_ROOT_PASSWORD\" ${MYSQL_DATABASE:-payroll} -N -B -e \"
|
||||||
|
SELECT COUNT(*) FROM login_logs
|
||||||
|
WHERE success = 0 AND create_time > NOW() - INTERVAL 24 HOUR;\" 2>/dev/null" || echo "?")
|
||||||
|
if [[ "$FAILS" == "?" ]]; then
|
||||||
|
warn "登录日志查询失败"
|
||||||
|
elif (( FAILS < 10 )); then
|
||||||
|
pass "24h 登录失败 ${FAILS} 次（阈值 10）"
|
||||||
|
else
|
||||||
|
warn "24h 登录失败 ${FAILS} 次（可能有暴力破解，建议核查）"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
warn "跳过登录日志检查（无凭据或容器未运行）"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "=============================================================="
|
||||||
|
if (( ISSUES == 0 )); then
|
||||||
|
echo "结论：全部通过，系统健康。"
|
||||||
|
exit 0
|
||||||
|
else
|
||||||
|
echo "结论：发现 $ISSUES 项需关注（见上）。"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
88
scripts/restore_drill.sh
Normal file
88
scripts/restore_drill.sh
Normal file
@ -0,0 +1,88 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# 恢复演练 —— 把最近一份备份恢复到「临时库」并核对行数（⑨：备份不演练 = 没有备份）
|
||||||
|
#
|
||||||
|
# 演练绝不直接覆盖生产库；临时库名 payroll_drill_<时间戳>，核对完由
|
||||||
|
# 调用方决定是否删除（本脚本默认不 DROP，安全优先）。
|
||||||
|
#
|
||||||
|
# 用法：在项目根目录执行 bash scripts/restore_drill.sh
|
||||||
|
# 产物：临时库 + 终端行数对比报告；--drop-exec 参数在核对通过后删除临时库。
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
cd "$ROOT"
|
||||||
|
|
||||||
|
if [[ -f docker/.env ]]; then
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source docker/.env
|
||||||
|
else
|
||||||
|
echo "ERROR: docker/.env 不存在" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
: "${MYSQL_DATABASE:?docker/.env 缺 MYSQL_DATABASE}"
|
||||||
|
: "${MYSQL_ROOT_PASSWORD:?docker/.env 缺 MYSQL_ROOT_PASSWORD}"
|
||||||
|
|
||||||
|
CONTAINER="ys-mysql-dev"
|
||||||
|
if ! docker inspect "$CONTAINER" >/dev/null 2>&1; then
|
||||||
|
echo "ERROR: 容器 $CONTAINER 未运行" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
BACKUP_DIR="$ROOT/db/backups"
|
||||||
|
LATEST="$(ls -t "$BACKUP_DIR"/payroll_*.sql.gz 2>/dev/null | head -1)"
|
||||||
|
if [[ -z "$LATEST" ]]; then
|
||||||
|
echo "ERROR: 未找到备份文件（先 bash scripts/backup_full.sh）" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
DRILL_DB="payroll_drill_$(date +%Y%m%d_%H%M%S)"
|
||||||
|
echo "演练恢复：$(basename "$LATEST") → 临时库 $DRILL_DB"
|
||||||
|
|
||||||
|
# 1. 创建临时库（mysqladmin create 无反引号转义问题）
|
||||||
|
docker exec "$CONTAINER" sh -c \
|
||||||
|
"mysqladmin -uroot -p\"\$MYSQL_ROOT_PASSWORD\" create \"$DRILL_DB\""
|
||||||
|
|
||||||
|
# 2. 解压导入
|
||||||
|
gunzip -c "$LATEST" | docker exec -i "$CONTAINER" sh -c \
|
||||||
|
"mysql -uroot -p\"\$MYSQL_ROOT_PASSWORD\" \"$DRILL_DB\""
|
||||||
|
|
||||||
|
# 3. 行数核对：关键业务表
|
||||||
|
TABLES=(users roles user_roles param_definitions param_values businesses business_items personnel products customers shipments shipment_items receipts approval_orders approval_steps system_settings)
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "======== 行数对比（源库 vs 演练库）========"
|
||||||
|
printf "%-24s %12s %12s %6s\n" "表名" "源库" "演练库" "结果"
|
||||||
|
ALL_OK=1
|
||||||
|
for T in "${TABLES[@]}"; do
|
||||||
|
SRC=$(docker exec "$CONTAINER" sh -c \
|
||||||
|
"mysql -uroot -p\"\$MYSQL_ROOT_PASSWORD\" -N -B -e \"SELECT COUNT(*) FROM $MYSQL_DATABASE.$T;\" 2>/dev/null" || echo "?")
|
||||||
|
DST=$(docker exec "$CONTAINER" sh -c \
|
||||||
|
"mysql -uroot -p\"\$MYSQL_ROOT_PASSWORD\" -N -B -e \"SELECT COUNT(*) FROM $DRILL_DB.$T;\" 2>/dev/null" || echo "?")
|
||||||
|
if [[ "$SRC" == "$DST" ]]; then
|
||||||
|
MARK="✓"
|
||||||
|
else
|
||||||
|
MARK="✗"
|
||||||
|
ALL_OK=0
|
||||||
|
fi
|
||||||
|
printf "%-24s %12s %12s %6s\n" "$T" "$SRC" "$DST" "$MARK"
|
||||||
|
done
|
||||||
|
echo "=========================================="
|
||||||
|
|
||||||
|
if [[ "$ALL_OK" == "1" ]]; then
|
||||||
|
echo ""
|
||||||
|
echo "结果：全部表行数一致，备份可恢复。"
|
||||||
|
if [[ "${1:-}" == "--drop-exec" ]]; then
|
||||||
|
docker exec "$CONTAINER" sh -c \
|
||||||
|
"mysqladmin -uroot -p\"\$MYSQL_ROOT_PASSWORD\" -f drop \"$DRILL_DB\""
|
||||||
|
echo "已删除临时库 $DRILL_DB"
|
||||||
|
else
|
||||||
|
echo "临时库 $DRILL_DB 保留（核对用）；确认无误后重跑本脚本加 --drop-exec 删除。"
|
||||||
|
fi
|
||||||
|
exit 0
|
||||||
|
else
|
||||||
|
echo ""
|
||||||
|
echo "结果：存在不一致，请检查备份文件与数据库结构。" >&2
|
||||||
|
echo "临时库 $DRILL_DB 保留供排查。" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
Loading…
Reference in New Issue
Block a user