提升性能与兼容性 #8

Closed
da_mou_mou wants to merge 3 commits from da_mou_mou/GMapiServer:main into main
10 changed files with 1911 additions and 256 deletions

428
README.md
View File

@@ -1,82 +1,59 @@
GMapiServer
# GMapiServer
GMapiServer 是一个基于 Python 和 Flask 构建的高性能 API 服务器,支持通过 URL 输入文件并输出下载链接,提供同步和异步两种处理模式。内置缓存自动清理机制(默认保留 2 小时),支持多种工具接口调用,包括实时任务状态查询和视频帧处理功能。
🚀 项目简介
## 🚀 项目简介
GMapiServer 是一个轻量级、模块化的 API 服务框架,旨在为开发者提供以下核心功能:
URL 输入与输出:支持通过 URL 提交文件,返回生成文件的下载链接。
- **URL 输入与输出**:支持通过 URL 提交文件,返回生成文件的下载链接。
- **异步处理**:任务在后台异步执行,不阻塞主进程,提升服务器并发性能。
- **自动缓存清理**:内置定时清理机制,自动删除过期的缓存文件(包括输出文件),默认保留时间 2 小时。
- **多工具接口集成**:集成 FFmpeg 和 Sanjuuni 工具接口,支持在线调用。
- **实时任务状态**:提供任务状态查询接口,支持日志实时推送。
- **视频帧提取**:支持从视频中提取指定分辨率、帧率的图片帧,可打包为二进制格式。
- **硬件加速支持**:支持 NVIDIA CUDA、Intel QSV、VA-API、树莓派 v4l2m2m 等硬件加速方式。
• 异步处理:任务在后台异步执行,不阻塞主进程,提升服务器并发性能。
## 🛠️ 技术栈
• 自动缓存清理:内置定时清理机制,自动删除过期的缓存文件(包括输出文件),默认保留时间 2 小时。
- **编程语言**: Python 3.10+
- **Web 框架**: Flask
- **异步处理**: 基于 threading
- **缓存清理**: 定时任务
- **依赖管理**: pip
- **视频处理**: FFmpeg (支持硬件加速)
• 多工具接口集成:集成 FFmpeg 和 Sanjuuni 工具接口,支持在线调用。
## 🧩 支持的接口
• 实时任务状态:提供任务状态查询接口,支持日志实时推送。
### 1. FFmpeg 工具接口
• 视频帧提取:支持从视频中提取指定分辨率、帧率的图片帧,可打包为二进制格式
- **功能**: 在线调用 FFmpeg 工具进行视频/音频处理(如转码、裁剪、合并等)
- **接口文档**: <https://www.liulikeji.cn/archives/FFmpegApi>
- **同步接口**: POST /api/ffmpeg
- **异步接口**: POST /api/ffmpeg/async
🛠️ 技术栈
### 2. Sanjuuni 工具接口
• 编程语言: Python
- **功能**: 在线调用 <https://github.com/MCJack123/sanjuuni/tree/master具体功能需参考其官方文档)。>
- **接口文档**: <https://www.liulikeji.cn/archives/SanjuuniApi>
- **同步接口**: POST /api/sanjuuni
- **异步接口**: POST /api/sanjuuni/async
• Web 框架: Flask
### 3. 视频帧提取接口
• 异步处理: 基于 threading
- **功能**: 从视频中提取指定分辨率、帧率的图片帧支持强制分辨率调整和填充支持B站BV号。
- **异步接口**: POST /api/video\_frame/async
- **参数**:
- **video\_url**: 视频文件 URL (或使用 video\_bv 参数替代)
- **video\_bv**: B站视频BV号 (自动转换为URL)
- **w**: 输出宽度
- **h**: 输出高度
- **fps**: 输出帧率 (默认: 30)
- **force\_resolution**: 是否强制调整分辨率 (默认: false)
- **pad\_to\_target**: 是否填充到目标分辨率 (默认: false)
- **pix\_fmt**: 输出像素格式,可选 `pal8``rgb24`(默认: `pal8`
- **返回示例**:
• 缓存清理: 定时任务
• 依赖管理: pip
🧩 支持的接口
1. FFmpeg 工具接口
• 功能: 在线调用 FFmpeg 工具进行视频/音频处理(如转码、裁剪、合并等)。
• 接口文档: https://www.liulikeji.cn/archives/FFmpegApi
• 同步接口: POST /api/ffmpeg
• 异步接口: POST /api/ffmpeg/async
2. Sanjuuni 工具接口
• 功能: 在线调用 https://github.com/MCJack123/sanjuuni/tree/master具体功能需参考其官方文档
• 接口文档: https://www.liulikeji.cn/archives/SanjuuniApi
• 同步接口: POST /api/sanjuuni
• 异步接口: POST /api/sanjuuni/async
3. 视频帧提取接口
• 功能: 从视频中提取指定分辨率、帧率的图片帧支持强制分辨率调整和填充支持B站BV号。
• 异步接口: POST /api/video_frame/async
• 参数:
• video_url: 视频文件 URL (或使用 video_bv 参数替代)
• video_bv: B站视频BV号 (自动转换为URL)
• w: 输出宽度
• h: 输出高度
• fps: 输出帧率 (默认: 30)
• force_resolution: 是否强制调整分辨率 (默认: false)
• pad_to_target: 是否填充到目标分辨率 (默认: false)
• pix_fmt: 输出像素格式,可选 `pal8``rgb24`(默认: `pal8`
• 返回示例:
```json
{
"task_id": "abc12345",
@@ -85,7 +62,8 @@ GMapiServer 是一个轻量级、模块化的 API 服务框架,旨在为开发
}
```
实时任务状态响应(处理中):
- **实时任务状态响应(处理中)**:
```json
{
"task_id": "abc12345",
@@ -111,7 +89,8 @@ GMapiServer 是一个轻量级、模块化的 API 服务框架,旨在为开发
}
```
任务完成时结果:
- **任务完成时结果**:
```json
{
"task_id": "abc12345",
@@ -139,78 +118,122 @@ GMapiServer 是一个轻量级、模块化的 API 服务框架,旨在为开发
}
```
4. 异步任务状态查询
### 4. 异步任务状态查询
• 功能: 查询异步任务状态和进度,支持日志实时推送。
- **功能**: 查询异步任务状态和进度,支持日志实时推送。
- **接口**: GET /api/task/\<task\_id>
- **返回字段**:
- **status**: 任务状态pending/running/completed/error
- **progress**: 进度百分比
- **new\_logs**: 新增的日志条目
- **result**: 完成后的结果包含下载URL
- **error**: 错误信息如果状态为error
• 接口: GET /api/task/<task_id>
### 5. 帧打包接口
• 返回字段:
- **功能**: 将多个帧图片打包为二进制格式,便于批量下载。
- **接口**:
- `GET /api/framepack?<task_id>-<start>-<end>`(推荐)
- `POST /api/framepack`(兼容旧方式)
- **参数**:
- **新方式**:
- `task_id`: 视频帧任务ID
- `start`: 起始帧从1开始
- `end`: 结束帧(包含)
- **旧方式**:
- `urls`: 帧图片URL列表格式为 `/frames/<job_id>/<filename>.png`
- **返回**: 二进制流,格式为帧数(4字节) + 各帧数据
• status: 任务状态pending/running/completed/error
### 6. 文件下载接口
• progress: 进度百分比
- **功能**: 下载处理完成的文件。
- **接口**: GET /download/\<file\_id>/<filename>
• new_logs: 新增的日志条目
### 7. 任务列表查询
• result: 完成后的结果包含下载URL
- **功能**: 列出所有任务状态(调试用)。
- **接口**: GET /api/tasks
• error: 错误信息如果状态为error
### 8. 健康检查
5. 帧打包接口
- **功能**: 服务健康检查。
- **接口**: GET /health
• 功能: 将多个帧图片打包为二进制格式,便于批量下载。
## 📦 部署与使用
• 接口:
`GET /api/framepack?<task_id>-<start>-<end>`(推荐)
`POST /api/framepack`(兼容旧方式)
• 参数:
• 新方式:
`task_id`: 视频帧任务ID
`start`: 起始帧从1开始
`end`: 结束帧(包含)
• 旧方式:
`urls`: 帧图片URL列表格式为 `/frames/<job_id>/<filename>.png`
• 返回: 二进制流,格式为帧数(4字节) + 各帧数据
6. 文件下载接口
• 功能: 下载处理完成的文件。
• 接口: GET /download/<file_id>/<filename>
7. 任务列表查询
• 功能: 列出所有任务状态(调试用)。
• 接口: GET /api/tasks
8. 健康检查
• 功能: 服务健康检查。
• 接口: GET /health
📦 部署与使用
1. 安装依赖
### 1. 安装依赖
```bash
pip install -r requirements.txt
```
### 2. 配置文件
2. 启动服务器
编辑 `config.json` 配置各项参数:
```json
{
"streaming": {
"enabled": true,
"buffer_size": 8192,
"chunk_delay_ms": 0
},
"server": {
"port": 5000,
"debug": false,
"threaded": true
},
"cleanup": {
"interval_hours": 1,
"retention_hours": 2,
"download_cache_max_mb": 500,
"video_cache_max_mb": 2000,
"cache_retention_hours": 2
},
"logging": {
"level": "INFO",
"file_path": "data/server.log"
},
"hwaccel": {
"enabled": true,
"method": "auto"
},
"video_download": {
"limit_resolution": true,
"max_resolution": "720p",
"preferred_codec": "h264"
}
}
```
### 配置说明
| 配置项 | 说明 | 默认值 |
| --------------------------------- | --------------------------------------- | ---- |
| `hwaccel.enabled` | 是否启用硬件加速 | true |
| `hwaccel.method` | 硬件加速方式auto/cuda/vaapi/qsv/v4l2m2m/drm | auto |
| `video_download.limit_resolution` | 是否限制下载分辨率 | true |
| `video_download.max_resolution` | 最大下载分辨率480p/720p/1080p/best | 720p |
| `video_download.preferred_codec` | 首选视频编码h264/h265/av1/vp9/auto | h264 |
### 3. 启动服务器
```bash
# 方式1直接启动
python main.py
# 方式2使用启动脚本
python start_server.py
# 方式3指定端口
python main.py --port 5000
```
### 4. 异步任务使用示例
3. 异步任务使用示例
创建异步FFmpeg任务
**创建异步FFmpeg任务**
```bash
curl -X POST http://localhost:5000/api/ffmpeg/async \
-H "Content-Type: application/json" \
-d '{
@@ -218,20 +241,23 @@ curl -X POST http://localhost:5000/api/ffmpeg/async \
"output_format": "mp4",
"args": ["-c:v", "libx264", "-crf", "23"]
}'
```
**查询任务状态**
查询任务状态
```bash
curl http://localhost:5000/api/task/<task_id>
```
4. 指定公网地址
### 5. 指定公网地址
在相应工具模块中配置下载URL
```python
# sanjuuni_utils.py
return {
'status': 'success',
'download_url': f"http://ffmpeg.liulikeji.cn/download/{output_id}/{output_filename}", # 外部访问地址
'download_url': f"http://ffmpeg.liulikeji.cn/download/{output_id}/{output_filename}",
'file_id': output_id,
'temp_dir': temp_dir
}
@@ -239,122 +265,150 @@ return {
# ffmpeg_utils.py
return {
'status': 'success',
'download_url': f"http://ffmpeg.liulikeji.cn/download/{output_id}/{output_filename}", # 外部访问地址
'download_url': f"http://ffmpeg.liulikeji.cn/download/{output_id}/{output_filename}",
'file_id': output_id,
'temp_dir': temp_dir
}
```
## 🔧 硬件加速配置
⏰ 自动缓存清理配置
### 支持的硬件加速方式
• 默认保留时间: 2 小时
| 加速方式 | 适用平台 | 说明 |
| ---------------- | ---------------- | ---------------------- |
| **cuda** | NVIDIA GPU | 需要 NVIDIA 显卡和 CUDA 工具包 |
| **vaapi** | Intel/AMD Linux | 需要支持 VA-API 的显卡 |
| **qsv** | Intel Quick Sync | Intel 核显专用 |
| **videotoolbox** | macOS | Apple 芯片专用 |
| **v4l2m2m** | 树莓派/嵌入式 Linux | 支持 H.264 编码 |
| **drm** | Linux DRM/KMS | 树莓派官方系统推荐 |
| **mmal** | 树莓派(旧版) | 树莓派传统硬件加速 |
| **vdpau** | NVIDIA Linux | NVIDIA VDPAU 加速 |
• 自定义配置:
# 在 file_cleanup.py 中修改缓存清理时间
CLEANUP_INTERVAL = 600 # 清理任务执行间隔(秒)
FILE_EXPIRY = 7200 # 文件记录过期时间(秒)
TASK_EXPIRY = 7200 # 任务记录过期时间(秒)
### 验证硬件加速
🔄 最新缓存与恢复机制2026-05 更新)
```bash
# 检查可用编码器
ffmpeg -encoders | grep v4l2m2m
1. 持久化存储
# 测试硬件编码
ffmpeg -i input.mp4 -c:v h264_v4l2m2m output.mp4
• 任务与缓存索引会写入 `cache_state/state.json`
# 检查硬件加速方式
ffmpeg -hwaccels
```
• 服务重启后自动恢复任务、文件注册表和视频帧缓存索引。
## ⏰ 自动缓存清理配置
2. 视频帧任务去重(全参数匹配)
- **默认保留时间**: 2 小时
- **自定义配置**: 在 `config.json` 中修改
`/api/video_frame/async` 会对请求计算签名(包含 URL、分辨率、fps、pix_fmt、force_resolution、pad_to_target 等)。
## 🔄 缓存与恢复机制2026-05 更新)
• 若签名匹配到已有任务:
• 任务 `pending/running`:直接返回已有 `task_id`
• 任务 `completed` 且缓存有效:复用已有 `task_id` 并重置过期计时。
• 任务 `error` / 缓存失效 / 缓存目录缺失:自动新建 `task_id` 重新转换。
### 1. 持久化存储
3. 重启中断识别
- 任务与缓存索引会写入 `cache_state/state.json`
- 服务重启后自动恢复任务、文件注册表和视频帧缓存索引。
• 启动时会将上次 `pending/running` 的任务标记为“因服务器重启中断”。
### 2. 视频帧任务去重(全参数匹配)
• 这类任务不会被复用,同参数新请求会重新创建任务
- `/api/video_frame/async` 会对请求计算签名(包含 URL、分辨率、fps、pix\_fmt、force\_resolution、pad\_to\_target 等)
- 若签名匹配到已有任务:
- 任务 `pending/running`:直接返回已有 `task_id`
- 任务 `completed` 且缓存有效:复用已有 `task_id` 并重置过期计时。
- 任务 `error` / 缓存失效 / 缓存目录缺失:自动新建 `task_id` 重新转换。
4. 看门狗式缓存续期2 小时)
### 3. 重启中断识别
• 视频帧缓存采用看门狗语义:每次相关访问都会将过期时间重置为 `当前时间 + 2小时`
- 启动时会将上次 `pending/running` 的任务标记为“因服务器重启中断”
- 这类任务不会被复用,同参数新请求会重新创建任务。
• 相关访问包括:
`/api/video_frame/async` 命中已有任务
`/api/framepack` 请求了该 `job_id` 的帧
• 视频帧任务转换完成后的结果写回
### 4. 看门狗式缓存续期2 小时)
• 若超过 2 小时没有相关访问,则按清理策略过期
- 视频帧缓存采用看门狗语义:每次相关访问都会将过期时间重置为 `当前时间 + 2小时`
- 相关访问包括:
- `/api/video_frame/async` 命中已有任务
- `/api/framepack` 请求了该 `job_id` 的帧
- 视频帧任务转换完成后的结果写回
- 若超过 2 小时没有相关访问,则按清理策略过期。
5. NoDelete 保护
### 5. NoDelete 保护
清理线程会检测 `frames/<task_id>/NoDelete` 文件。
- 清理线程会检测 `frames/<task_id>/NoDelete` 文件。
- 若存在该文件:
- 跳过该目录清理
- 并重置该任务缓存计时器(`now + 2h`
• 若存在该文件:
• 跳过该目录清理
• 并重置该任务缓存计时器(`now + 2h`
### 6. 双语日志
6. 双语日志
- `ffmpeg``sanjuuni``video_frame` 的业务日志已统一为中英双语格式:`中文 | English`
`ffmpeg``sanjuuni``video_frame` 的业务日志已统一为中英双语格式:`中文 | English`
🔥 视频帧实时进度功能
## 🔥 视频帧实时进度功能
GMapiServer 的视频帧提取功能支持实时进度跟踪,让客户端可以在任务处理过程中实时获取已生成的帧并进行下载。
• 实时帧进度跟踪: 自动解析FFmpeg输出的进度信息实时更新当前帧数
• 智能进度计算: 基于帧数自动计算转换进度20-80%
• 音频预处理: 音频在视频处理前提取,客户端可提前下载音频文件
• 实时URL返回: 在处理过程中不断返回已生成的帧URL列表
## 优势特性
### 优势特性
1. **减少等待时间**: 客户端无需等待整个视频处理完成
2. **并行下载**: 可以在转换过程中并行下载已生成的帧
3. **实时反馈**: 用户可以看到实时处理进度和日志
4. **资源优化**: 避免了"堵车"效应,提高系统并发能力
## 使用建议
### 使用建议
客户端可以轮询 `/api/task/<task_id>` 接口获取最新状态
通过检查 `result.current_frames``result.total_frames` 判断转换进度
音频文件可以立即下载(音频处理优先级更高)
- 客户端可以轮询 `/api/task/<task_id>` 接口获取最新状态
- 通过检查 `result.current_frames``result.total_frames` 判断转换进度
- 音频文件可以立即下载(音频处理优先级更高)
## 🔧 异步任务处理流程
🔧 异步任务处理流程
1. **任务创建**: 客户端提交任务请求服务器返回任务ID和状态查询URL
2. **后台处理**: 任务在独立线程中执行,不阻塞主进程
3. **状态查询**: 客户端可轮询状态接口获取任务进度和实时日志
4. **结果获取**: 任务完成后可通过状态接口或文件下载接口获取结果
5. **自动清理**: 任务结果文件在指定时间后自动清理
1. 任务创建: 客户端提交任务请求服务器返回任务ID和状态查询URL
2. 后台处理: 任务在独立线程中执行,不阻塞主进程
3. 状态查询: 客户端可轮询状态接口获取任务进度和实时日志
4. 结果获取: 任务完成后可通过状态接口或文件下载接口获取结果
5. 自动清理: 任务结果文件在指定时间后自动清理
## ⚠️ 注意事项
⚠️ 注意事项
1. **文件合法性**: 请确保上传文件符合法律法规,不得用于非法用途。
2. **缓存文件**: 系统会自动清理过期文件,请及时下载生成的输出文件。
3. **异步任务**: 任务状态已支持持久化恢复;但重启前处于运行中的任务会被标记为中断,需要重新发起。
4. **资源限制**: 视频帧提取和转码操作可能消耗较多CPU和内存资源。
5. **树莓派注意**: 硬件解码可能不可用但硬件编码h264\_v4l2m2m可用性能提升约3倍。
1. 文件合法性: 请确保上传文件符合法律法规,不得用于非法用途。
2. 缓存文件: 系统会自动清理过期文件,请及时下载生成的输出文件。
3. 异步任务: 任务状态已支持持久化恢复;但重启前处于运行中的任务会被标记为中断,需要重新发起。
4. 资源限制: 视频帧提取和转码操作可能消耗较多CPU和内存资源。
## 📝 贡献与反馈
📝 贡献与反馈
欢迎提交 Issue 或 Pull Request
如有问题,请联系:[xingluo01@liulikeji.cn] 或 [qq:180877430]
📁 项目结构
欢迎提交 Issue 或 Pull Request\
如有问题,请联系:\[<xingluo01@liulikeji.cn>] 或 \[qq:180877430]
## 📁 项目结构
```
GMapiServer/
├── main.py # 主程序入口
├── start_server.py # 启动脚本(含运行库检查)
├── config.py # 配置管理
├── config.json # 配置文件
├── shared_utils.py # 共享工具函数
├── ffmpeg_utils.py # FFmpeg处理模块
├── sanjuuni_utils.py # Sanjuuni处理模块
├── video_frame_utils.py # 视频帧提取模块
├── file_cleanup.py # 文件清理模块
├── hwaccel_detect.py # 硬件加速检测模块
├── lib_downloader.py # 运行库下载器
├── logging_config.py # 日志配置
├── logo.py # Logo输出
├── requirements.txt # 依赖包列表
├── test/ # 测试工具目录
│ └── decode_performance_test.py # 性能测试脚本
├── lib/ # 本地运行库
│ ├── ffmpeg/
│ └── sanjuuni/
├── data/ # 数据目录
│ ├── ffmpeg/
│ ├── sanjuuni/
│ └── videoframe/
└── README.md # 项目说明文档
```

89
config.json Normal file
View File

@@ -0,0 +1,89 @@
{
// ==================== 转码模式配置 ====================
"streaming": {
// 转码模式(无法生效):
// true - 流式处理(边转码边发送,适合高性能服务器)
// false - 非流式处理(转码完成后再发送,适合低性能服务器避免超时)
"enabled": true,
// 流式传输缓冲区大小(字节)
"buffer_size": 8192,
// 块发送延迟(毫秒),用于控制发送速率
"chunk_delay_ms": 0
},
// ==================== 服务器配置 ====================
"server": {
// 服务监听端口
"port": 5000,
// 调试模式(开发环境使用)
"debug": false,
// 多线程模式
"threaded": true
},
// ==================== 自动清理配置 ====================
"cleanup": {
// 清理任务执行间隔(小时)
"interval_hours": 1,
// 下载临时文件保留时间(小时)
"retention_hours": 2,
// 下载缓存最大大小MB
"download_cache_max_mb": 500,
// 视频缓存最大大小MB
"video_cache_max_mb": 2000,
// 缓存文件保留时间(小时)
"cache_retention_hours": 2
},
// ==================== 日志配置 ====================
"logging": {
// 日志级别DEBUG, INFO, WARNING, ERROR
"level": "INFO",
// 日志文件路径
"file_path": "data/server.log"
},
// ==================== 硬件加速配置 ====================
"hwaccel": {
// 是否启用硬件加速
"enabled": true,
// 硬件加速方式:
// "auto" - 自动检测最佳方案
// "cuda" - NVIDIA GPU加速
// "vaapi" - Intel/AMD Linux加速
// "qsv" - Intel Quick Sync加速
// "videotoolbox" - macOS加速
// "mmal" - 树莓派MMAL硬件加速需自定义编译FFmpeg
// "v4l2m2m" - Linux V4L2 Memory-to-Memory加速适用于树莓派/嵌入式设备)
// "drm" - Linux DRM/KMS加速树莓派官方系统推荐
// "vdpau" - NVIDIA VDPAU加速
// "vulkan" - Vulkan通用GPU加速跨平台
"method": "auto"
},
// ==================== 视频下载配置 ====================
"video_download": {
// 是否限制下载分辨率(减轻转码压力)
"limit_resolution": true,
// 最大下载分辨率当limit_resolution为true时生效
// 可选值480p, 720p, 1080p, 1440p, 2160p, 4320p
// 设置为"best"则下载最佳质量(不限制)
"max_resolution": "720p",
// 首选视频编码(减轻解码压力)
// 可选值:"h264", "h265", "av1", "vp9", "auto"
// 建议树莓派使用"h264"以支持硬件加速
"preferred_codec": "h264"
}
}

174
config.py Normal file
View File

@@ -0,0 +1,174 @@
import os
import json
CONFIG_FILE = 'config.json'
DEFAULT_CONFIG = {
'streaming': {
'enabled': True,
'buffer_size': 8192,
'chunk_delay_ms': 0
},
'server': {
'port': 5000,
'debug': False,
'threaded': True
},
'cleanup': {
'interval_hours': 1,
'retention_hours': 2,
'download_cache_max_mb': 500,
'video_cache_max_mb': 2000,
'cache_retention_hours': 24
},
'logging': {
'level': 'INFO',
'file_path': 'data/server.log'
},
'hwaccel': {
'enabled': True,
'method': 'auto'
},
'video_download': {
'limit_resolution': True,
'max_resolution': '720p',
'preferred_codec': 'h264'
}
}
_config = None
def remove_comments(json_string):
"""移除JSON字符串中的//注释"""
lines = json_string.split('\n')
result = []
for line in lines:
if '//' in line:
in_string = False
string_char = ''
escape = False
for i, char in enumerate(line):
if escape:
escape = False
continue
if char == '\\' and in_string:
escape = True
continue
if char in ('"', "'"):
if in_string and char == string_char:
in_string = False
elif not in_string:
in_string = True
string_char = char
if not in_string and '//' in line[i:]:
line = line[:i]
break
stripped = line.strip()
if stripped or result:
result.append(line)
return '\n'.join(result)
def load_config():
global _config
if _config is not None:
return _config
if os.path.exists(CONFIG_FILE):
try:
with open(CONFIG_FILE, 'r', encoding='utf-8') as f:
content = f.read()
content = remove_comments(content)
_config = json.loads(content)
except Exception as e:
print(f"配置文件读取失败,使用默认配置: {e}")
_config = DEFAULT_CONFIG.copy()
else:
_config = DEFAULT_CONFIG.copy()
save_config()
return _config
def save_config():
if _config is not None:
os.makedirs(os.path.dirname(CONFIG_FILE) or '.', exist_ok=True)
with open(CONFIG_FILE, 'w', encoding='utf-8') as f:
json.dump(_config, f, ensure_ascii=False, indent=2)
def get_streaming_enabled():
config = load_config()
return config.get('streaming', {}).get('enabled', True)
def get_streaming_buffer_size():
config = load_config()
return config.get('streaming', {}).get('buffer_size', 8192)
def get_streaming_chunk_delay():
config = load_config()
return config.get('streaming', {}).get('chunk_delay_ms', 0)
def get_server_port():
config = load_config()
return config.get('server', {}).get('port', 5000)
def get_server_debug():
config = load_config()
return config.get('server', {}).get('debug', False)
def get_server_threaded():
config = load_config()
return config.get('server', {}).get('threaded', True)
def get_cleanup_interval():
config = load_config()
return config.get('cleanup', {}).get('interval_hours', 1)
def get_cleanup_retention():
config = load_config()
return config.get('cleanup', {}).get('retention_hours', 2)
def get_logging_level():
config = load_config()
return config.get('logging', {}).get('level', 'INFO')
def get_logging_file_path():
config = load_config()
return config.get('logging', {}).get('file_path', 'data/server.log')
def get_hwaccel_enabled():
config = load_config()
return config.get('hwaccel', {}).get('enabled', True)
def get_hwaccel_method():
config = load_config()
return config.get('hwaccel', {}).get('method', 'auto')
def get_download_cache_max_mb():
config = load_config()
return config.get('cleanup', {}).get('download_cache_max_mb', 500)
def get_video_cache_max_mb():
config = load_config()
return config.get('cleanup', {}).get('video_cache_max_mb', 2000)
def get_cache_retention_hours():
config = load_config()
return config.get('cleanup', {}).get('cache_retention_hours', 24)
def get_video_download_limit_resolution():
config = load_config()
return config.get('video_download', {}).get('limit_resolution', True)
def get_video_download_max_resolution():
config = load_config()
return config.get('video_download', {}).get('max_resolution', '720p')
def get_video_download_preferred_codec():
config = load_config()
return config.get('video_download', {}).get('preferred_codec', 'h264')

View File

@@ -9,6 +9,94 @@ import threading
import time
from contextlib import contextmanager
from shared_utils import add_task_log, save_persistent_state
from config import get_streaming_enabled, get_streaming_buffer_size, get_hwaccel_enabled, get_hwaccel_method
_hwaccel_args = None
_hwaccel_method = None
_ffmpeg_path = None
def find_ffmpeg():
"""查找FFmpeg可执行文件路径"""
global _ffmpeg_path
if _ffmpeg_path is not None:
return _ffmpeg_path
if sys.platform.startswith('win'):
local_path = 'lib/ffmpeg/bin/ffmpeg.exe'
exe_name = 'ffmpeg.exe'
else:
local_path = 'lib/ffmpeg/bin/ffmpeg'
exe_name = 'ffmpeg'
if os.path.exists(local_path):
_ffmpeg_path = local_path
return _ffmpeg_path
if sys.platform.startswith('linux'):
try:
result = subprocess.run(['which', exe_name], capture_output=True, text=True)
if result.returncode == 0:
_ffmpeg_path = result.stdout.strip()
logging.info(bi(f"使用系统FFmpeg: {_ffmpeg_path}", f"Using system FFmpeg: {_ffmpeg_path}"))
return _ffmpeg_path
except Exception as e:
logging.warning(bi(f"查找系统FFmpeg失败: {e}", f"Failed to find system FFmpeg: {e}"))
_ffmpeg_path = local_path
return _ffmpeg_path
def init_hwaccel():
"""初始化硬件加速"""
global _hwaccel_args, _hwaccel_method
if not get_hwaccel_enabled():
logging.info(bi("硬件加速已禁用", "Hardware acceleration disabled"))
_hwaccel_args = []
_hwaccel_method = None
return
try:
from hwaccel_detect import get_working_hwaccel, get_hwaccel_args, test_hwaccel, detect_hardware_acceleration
config_method = get_hwaccel_method()
available_methods = detect_hardware_acceleration()
if config_method != 'auto':
if config_method not in available_methods:
_hwaccel_args = []
_hwaccel_method = None
logging.warning(bi(f"配置指定的硬件加速方式 '{config_method}' 不在系统支持列表中。可用方式: {available_methods},使用软件解码", f"Configured hardware acceleration method '{config_method}' not in system support list. Available methods: {available_methods}, using software decoding"))
return
if test_hwaccel(config_method):
_hwaccel_method = config_method
_hwaccel_args = get_hwaccel_args(config_method)
logging.info(bi(f"使用配置指定的硬件加速方式: {_hwaccel_method}", f"Using configured hardware acceleration method: {_hwaccel_method}"))
else:
_hwaccel_args = []
_hwaccel_method = None
logging.warning(bi(f"配置指定的硬件加速方式 {config_method} 测试失败,使用软件解码", f"Configured hardware acceleration method {config_method} test failed, using software decoding"))
return
else:
_hwaccel_method = get_working_hwaccel()
if _hwaccel_method:
_hwaccel_args = get_hwaccel_args(_hwaccel_method)
logging.info(bi(f"自动检测到硬件加速方式: {_hwaccel_method}", f"Automatically detected hardware acceleration method: {_hwaccel_method}"))
else:
_hwaccel_args = []
logging.info(bi("未检测到可用的硬件加速,使用软件解码", "No available hardware acceleration detected, using software decoding"))
except Exception as e:
_hwaccel_args = []
_hwaccel_method = None
logging.error(bi(f"初始化硬件加速失败: {e}", f"Failed to initialize hardware acceleration: {e}"))
def get_hwaccel_args():
"""获取硬件加速参数"""
global _hwaccel_args
if _hwaccel_args is None:
init_hwaccel()
return _hwaccel_args
UPLOAD_FOLDER = os.path.join('data', 'ffmpeg')
os.makedirs(UPLOAD_FOLDER, exist_ok=True)
@@ -79,25 +167,25 @@ def execute_ffmpeg(input_path, output_path, ffmpeg_args, task_id=None, task_regi
filtered_args = [arg for arg in ffmpeg_args if not (arg.lower() in ['-i', '-input'] or
(ffmpeg_args.index(arg) > 0 and ffmpeg_args[ffmpeg_args.index(arg) - 1].lower() == '-i'))]
cmd = ['lib/ffmpeg/bin/ffmpeg', '-i', input_path] + filtered_args + [output_path]
ffmpeg_bin = find_ffmpeg()
hwaccel_args = get_hwaccel_args()
cmd = [ffmpeg_bin] + hwaccel_args + ['-i', input_path] + filtered_args + [output_path]
if task_id and task_registry and task_lock:
add_task_log(task_id, bi(f"执行FFmpeg命令: {' '.join(cmd)}", f"Execute FFmpeg command: {' '.join(cmd)}"), task_registry, task_lock)
logging.info(bi(f"执行FFmpeg命令: {' '.join(cmd)}", f"Execute FFmpeg command: {' '.join(cmd)}"))
# 使用Popen来实时捕获输出
process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, # 将stderr重定向到stdout
stderr=subprocess.STDOUT,
universal_newlines=True,
encoding='utf-8',
errors='replace',
bufsize=1
)
# 实时读取输出
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
@@ -106,14 +194,11 @@ def execute_ffmpeg(input_path, output_path, ffmpeg_args, task_id=None, task_regi
output = output.strip()
if output and task_id and task_registry and task_lock:
add_task_log(task_id, output, task_registry, task_lock)
# 简单的进度解析可以根据FFmpeg的输出格式进行优化
if 'time=' in output and 'bitrate=' in output:
# 这里可以添加更复杂的进度解析逻辑
if task_id and task_registry and task_lock:
with task_lock:
if task_id in task_registry:
# 处理进度从50%开始到100%
task_registry[task_id]['progress'] = 50 + 50 // 2 # 简单示例
task_registry[task_id]['progress'] = 50 + 50 // 2
returncode = process.poll()
if returncode != 0:
@@ -135,7 +220,78 @@ def execute_ffmpeg(input_path, output_path, ffmpeg_args, task_id=None, task_regi
logging.error(error_msg)
raise
def execute_ffmpeg_streaming(input_path, ffmpeg_args, task_id=None, task_registry=None, task_lock=None):
try:
filtered_args = [arg for arg in ffmpeg_args if not (arg.lower() in ['-i', '-input'] or
(ffmpeg_args.index(arg) > 0 and ffmpeg_args[ffmpeg_args.index(arg) - 1].lower() == '-i'))]
ffmpeg_bin = find_ffmpeg()
hwaccel_args = get_hwaccel_args()
cmd = [ffmpeg_bin] + hwaccel_args + ['-i', input_path] + filtered_args + ['-f', 'mp4', '-']
if task_id and task_registry and task_lock:
add_task_log(task_id, bi(f"执行FFmpeg流式命令: {' '.join(cmd)}", f"Execute FFmpeg streaming command: {' '.join(cmd)}"), task_registry, task_lock)
logging.info(bi(f"执行FFmpeg流式命令: {' '.join(cmd)}", f"Execute FFmpeg streaming command: {' '.join(cmd)}"))
process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
bufsize=get_streaming_buffer_size()
)
buffer = b''
while True:
chunk = process.stdout.read(get_streaming_buffer_size())
if chunk:
buffer += chunk
while len(buffer) >= get_streaming_buffer_size():
yield buffer[:get_streaming_buffer_size()]
buffer = buffer[get_streaming_buffer_size():]
if not chunk and process.poll() is not None:
break
stderr_line = process.stderr.readline()
if stderr_line:
stderr_line = stderr_line.decode('utf-8', errors='replace').strip()
if stderr_line and task_id and task_registry and task_lock:
add_task_log(task_id, stderr_line, task_registry, task_lock)
if 'time=' in stderr_line and 'bitrate=' in stderr_line and task_id and task_registry and task_lock:
with task_lock:
if task_id in task_registry:
task_registry[task_id]['progress'] = 50 + 50 // 2
if buffer:
yield buffer
returncode = process.poll()
if returncode != 0:
stderr_output = process.stderr.read().decode('utf-8', errors='replace') if process.stderr else ''
error_msg = bi(f"FFmpeg流式处理失败返回码: {returncode}", f"FFmpeg streaming processing failed, return code: {returncode}")
if stderr_output:
error_msg += f" - {stderr_output[:200]}"
if task_id and task_registry and task_lock:
add_task_log(task_id, error_msg, task_registry, task_lock)
logging.error(error_msg)
raise Exception(error_msg)
if task_id and task_registry and task_lock:
add_task_log(task_id, bi("FFmpeg流式处理成功完成", "FFmpeg streaming processing completed successfully"), task_registry, task_lock)
logging.info(bi("FFmpeg流式处理成功完成", "FFmpeg streaming processing completed successfully"))
except Exception as e:
error_msg = bi(f"执行FFmpeg流式处理时出错: {e}", f"Error while executing FFmpeg streaming: {e}")
if task_id and task_registry and task_lock:
add_task_log(task_id, error_msg, task_registry, task_lock)
logging.error(error_msg)
raise
def process_ffmpeg(data, file_registry, file_lock, task_id=None, task_registry=None, task_lock=None):
streaming = get_streaming_enabled()
try:
dir_name = task_id if task_id else str(uuid.uuid4())[:8]
temp_dir = os.path.join(UPLOAD_FOLDER, dir_name)
@@ -156,12 +312,29 @@ def process_ffmpeg(data, file_registry, file_lock, task_id=None, task_registry=N
output_path = os.path.join(temp_dir, output_filename)
if task_id and task_registry and task_lock:
add_task_log(task_id, bi(f"开始FFmpeg处理输出格式: {output_format}", f"Start FFmpeg processing, output format: {output_format}"), task_registry, task_lock)
mode_text = "流式" if streaming else "非流式"
add_task_log(task_id, bi(f"开始FFmpeg{mode_text}处理,输出格式: {output_format}", f"Start FFmpeg {mode_text} processing, output format: {output_format}"), task_registry, task_lock)
with task_lock:
if task_id in task_registry:
task_registry[task_id]['progress'] = 50 # 开始处理进度50%
task_registry[task_id]['progress'] = 50
execute_ffmpeg(input_path, output_path, ffmpeg_args, task_id, task_registry, task_lock)
if streaming:
if task_id and task_registry and task_lock:
add_task_log(task_id, bi("使用流式处理模式", "Using streaming mode"), task_registry, task_lock)
logging.info(bi("使用流式处理模式", "Using streaming mode"))
output_data = b''
for chunk in execute_ffmpeg_streaming(input_path, ffmpeg_args, task_id, task_registry, task_lock):
output_data += chunk
with open(output_path, 'wb') as f:
f.write(output_data)
else:
if task_id and task_registry and task_lock:
add_task_log(task_id, bi("使用非流式处理模式(转码完成后发送)", "Using non-streaming mode (send after completion)"), task_registry, task_lock)
logging.info(bi("使用非流式处理模式(转码完成后发送)", "Using non-streaming mode (send after completion)"))
execute_ffmpeg(input_path, output_path, ffmpeg_args, task_id, task_registry, task_lock)
with file_lock:
file_registry[output_id] = {
@@ -176,11 +349,10 @@ def process_ffmpeg(data, file_registry, file_lock, task_id=None, task_registry=N
add_task_log(task_id, bi(f"已注册新文件ID: {output_id}, 路径: {output_path}", f"Registered new file ID: {output_id}, path: {output_path}"), task_registry, task_lock)
with task_lock:
if task_id in task_registry:
task_registry[task_id]['progress'] = 100 # 处理完成进度100%
task_registry[task_id]['progress'] = 100
logging.info(bi(f"已注册新文件ID: {output_id}, 路径: {output_path}", f"Registered new file ID: {output_id}, path: {output_path}"))
# 返回临时目录路径以便在主函数中删除
return {
'status': 'success',
'download_url': f"http://newgmapi.liulikeji.cn/download/{output_id}/{output_filename}",

251
hwaccel_detect.py Normal file
View File

@@ -0,0 +1,251 @@
import subprocess
import os
import sys
import logging
def is_raspberry_pi():
try:
with open('/proc/cpuinfo', 'r') as f:
content = f.read()
if 'Raspberry Pi' in content or 'BCM2835' in content or 'BCM2711' in content:
return True
except Exception:
pass
return False
def find_v4l2_device():
for i in range(10, 16):
device_path = f'/dev/video{i}'
if os.path.exists(device_path):
try:
result = subprocess.run(
['v4l2-ctl', '--device', device_path, '--all'],
capture_output=True,
text=True,
timeout=5
)
if result.returncode == 0 and 'codec' in result.stdout.lower():
return device_path
except Exception:
pass
return device_path
for i in range(32):
device_path = f'/dev/video{i}'
if os.path.exists(device_path):
return device_path
return '/dev/video11'
def detect_hardware_acceleration():
hwaccel_methods = []
if sys.platform.startswith('win'):
ffmpeg_path = 'lib/ffmpeg/bin/ffmpeg.exe'
else:
ffmpeg_path = 'lib/ffmpeg/bin/ffmpeg'
if not os.path.exists(ffmpeg_path):
try:
result = subprocess.run(['which', 'ffmpeg'], capture_output=True, text=True)
if result.returncode == 0:
ffmpeg_path = result.stdout.strip()
else:
logging.warning(bi("FFmpeg未找到跳过硬件加速检测", "FFmpeg not found, skipping hardware acceleration detection"))
return []
except Exception:
logging.warning(bi("FFmpeg未找到跳过硬件加速检测", "FFmpeg not found, skipping hardware acceleration detection"))
return []
try:
result = subprocess.run(
[ffmpeg_path, '-hwaccels'],
capture_output=True,
text=True,
timeout=10
)
if result.returncode == 0:
output = result.stdout.strip()
for line in output.split('\n'):
line = line.strip()
if line and line != 'Hardware acceleration methods:':
hwaccel_methods.append(line)
logging.info(bi(f"检测到可用的硬件加速方式: {hwaccel_methods}", f"Detected available hardware acceleration methods: {hwaccel_methods}"))
except Exception as e:
logging.error(bi(f"检测硬件加速时出错: {e}", f"Error detecting hardware acceleration: {e}"))
if is_raspberry_pi():
result = subprocess.run(
[ffmpeg_path, '-hide_banner', '-encoders'],
capture_output=True,
text=True,
timeout=10
)
if result.returncode == 0:
if 'h264_v4l2m2m' in result.stdout:
if 'v4l2m2m' not in hwaccel_methods:
hwaccel_methods.append('v4l2m2m')
logging.info(bi("树莓派特殊处理:检测到 v4l2m2m 编码器,添加到可用硬件加速列表", "Raspberry Pi special handling: detected v4l2m2m encoder, added to available hardware acceleration list"))
return hwaccel_methods
def get_best_hwaccel():
available = detect_hardware_acceleration()
if is_raspberry_pi():
priority_order = ['v4l2m2m', 'drm', 'mmal', 'vulkan', 'vdpau', 'vaapi', 'cuda', 'qsv']
else:
priority_order = ['cuda', 'vaapi', 'qsv', 'videotoolbox', 'dxva2', 'd3d11va', 'amf', 'v4l2m2m', 'drm', 'vulkan', 'vdpau']
for method in priority_order:
if method in available:
logging.info(bi(f"选择硬件加速方式: {method}", f"Selected hardware acceleration method: {method}"))
return method
return None
def get_hwaccel_args(hwaccel_method=None):
if not hwaccel_method:
hwaccel_method = get_best_hwaccel()
if not hwaccel_method:
return []
args = []
if hwaccel_method == 'vaapi':
args.extend(['-hwaccel', 'vaapi', '-hwaccel_device', '/dev/dri/renderD128'])
elif hwaccel_method == 'qsv':
args.extend(['-hwaccel', 'qsv', '-hwaccel_device', 'auto'])
elif hwaccel_method == 'mmal':
args.extend(['-hwaccel', 'mmal', '-hwaccel_device', 'vc.smem'])
elif hwaccel_method == 'v4l2m2m':
device_path = find_v4l2_device()
args.extend(['-hwaccel_device', device_path])
elif hwaccel_method == 'drm':
args.extend(['-hwaccel', 'drm', '-hwaccel_device', '/dev/dri/renderD128'])
elif hwaccel_method == 'vdpau':
args.extend(['-hwaccel', 'vdpau', '-hwaccel_device', 'auto'])
elif hwaccel_method == 'vulkan':
args.extend(['-hwaccel', 'vulkan', '-hwaccel_device', 'auto'])
return args
def test_hwaccel(hwaccel_method):
if sys.platform.startswith('win'):
ffmpeg_path = 'lib/ffmpeg/bin/ffmpeg.exe'
else:
ffmpeg_path = 'lib/ffmpeg/bin/ffmpeg'
if not os.path.exists(ffmpeg_path):
try:
result = subprocess.run(['which', 'ffmpeg'], capture_output=True, text=True)
if result.returncode == 0:
ffmpeg_path = result.stdout.strip()
else:
logging.warning(bi(f"本地路径未找到FFmpeg跳过{hwaccel_method}测试", f"FFmpeg not found in local path, skipping test for {hwaccel_method}"))
return True
except Exception:
logging.warning(bi(f"无法找到FFmpeg跳过{hwaccel_method}测试", f"Cannot find FFmpeg, skipping test for {hwaccel_method}"))
return True
try:
if hwaccel_method == 'v4l2m2m':
device_path = find_v4l2_device()
cmd = [
ffmpeg_path,
'-f', 'lavfi',
'-i', 'color=c=black:s=1920x1080:r=1',
'-t', '1',
'-c:v', 'h264_v4l2m2m',
'-f', 'null',
'-'
]
elif hwaccel_method == 'vaapi':
cmd = [
ffmpeg_path,
'-hwaccel', 'vaapi',
'-hwaccel_device', '/dev/dri/renderD128',
'-f', 'lavfi',
'-i', 'color=c=black:s=1920x1080:r=1',
'-t', '1',
'-c:v', 'h264_vaapi',
'-f', 'null',
'-'
]
elif hwaccel_method == 'drm':
cmd = [
ffmpeg_path,
'-hwaccel', 'drm',
'-hwaccel_device', '/dev/dri/renderD128',
'-f', 'lavfi',
'-i', 'color=c=black:s=1920x1080:r=1',
'-t', '1',
'-c:v', 'h264',
'-f', 'null',
'-'
]
elif hwaccel_method == 'mmal':
cmd = [
ffmpeg_path,
'-hwaccel', 'mmal',
'-hwaccel_device', 'vc.smem',
'-f', 'lavfi',
'-i', 'color=c=black:s=1920x1080:r=1',
'-t', '1',
'-c:v', 'h264',
'-f', 'null',
'-'
]
else:
cmd = [
ffmpeg_path,
'-hwaccel', hwaccel_method,
'-f', 'lavfi',
'-i', 'color=c=black:s=1920x1080:r=1',
'-t', '1',
'-c:v', 'h264',
'-f', 'null',
'-'
]
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=20
)
if result.returncode == 0:
logging.info(bi(f"硬件加速 {hwaccel_method} 测试通过", f"Hardware acceleration {hwaccel_method} test passed"))
return True
error_stderr = result.stderr.lower()
if 'invalid device' in error_stderr or 'not found' in error_stderr or \
'no such device' in error_stderr or 'failed to open' in error_stderr or \
'could not find' in error_stderr or 'permission denied' in error_stderr:
logging.warning(bi(f"硬件加速 {hwaccel_method} 设备无效", f"Hardware acceleration {hwaccel_method} device invalid: {result.stderr.strip()[:100]}"))
return False
logging.warning(bi(f"硬件加速 {hwaccel_method} 测试返回码非零但非致命: {result.returncode}", f"Hardware acceleration {hwaccel_method} test returned non-zero but non-fatal: {result.returncode}"))
return True
except Exception as e:
logging.error(bi(f"测试硬件加速 {hwaccel_method} 失败: {e}", f"Failed to test hardware acceleration {hwaccel_method}: {e}"))
return False
def get_working_hwaccel():
available = detect_hardware_acceleration()
if is_raspberry_pi():
priority_order = ['v4l2m2m', 'drm', 'mmal', 'vulkan', 'vdpau', 'vaapi', 'cuda', 'qsv']
else:
priority_order = ['cuda', 'vaapi', 'qsv', 'videotoolbox', 'dxva2', 'd3d11va', 'amf', 'v4l2m2m', 'drm', 'vulkan', 'vdpau']
for method in priority_order:
if method in available and test_hwaccel(method):
return method
return None

644
lib_downloader.py Normal file
View File

@@ -0,0 +1,644 @@
import os
import sys
import platform
import tarfile
import zipfile
import logging
import hashlib
import time
import subprocess
import shutil
from pathlib import Path
from datetime import datetime
LIB_DIR = os.path.join(os.path.dirname(__file__), 'lib')
DOWNLOAD_CACHE = os.path.join(LIB_DIR, '.download_cache')
DATA_DIR = os.path.join(os.path.dirname(__file__), 'data')
def bi(zh, en):
return f"{zh} | {en}"
def get_directory_size(directory):
"""计算目录大小(字节)"""
total_size = 0
for dirpath, dirnames, filenames in os.walk(directory):
for f in filenames:
try:
fp = os.path.join(dirpath, f)
total_size += os.path.getsize(fp)
except Exception:
pass
return total_size
def format_size(bytes_size):
"""格式化文件大小"""
if bytes_size < 1024:
return f"{bytes_size} B"
elif bytes_size < 1024 * 1024:
return f"{bytes_size / 1024:.2f} KB"
elif bytes_size < 1024 * 1024 * 1024:
return f"{bytes_size / (1024 * 1024):.2f} MB"
else:
return f"{bytes_size / (1024 * 1024 * 1024):.2f} GB"
def cleanup_download_cache():
"""清理下载缓存(仅按时间清理)"""
if not os.path.exists(DOWNLOAD_CACHE):
return
try:
from config import get_cache_retention_hours
max_age_hours = get_cache_retention_hours()
except Exception:
max_age_hours = 24
current_size = get_directory_size(DOWNLOAD_CACHE)
print(f"\n🧹 清理下载缓存")
print(f" 当前缓存大小: {format_size(current_size)}")
print(f" 保留时间: {max_age_hours} 小时")
files_to_delete = []
for filename in os.listdir(DOWNLOAD_CACHE):
filepath = os.path.join(DOWNLOAD_CACHE, filename)
if os.path.isfile(filepath):
try:
mtime = os.path.getmtime(filepath)
age_hours = (time.time() - mtime) / 3600
files_to_delete.append((mtime, filepath, age_hours))
except Exception:
pass
files_to_delete.sort(key=lambda x: x[0])
deleted_size = 0
deleted_count = 0
for mtime, filepath, age_hours in files_to_delete:
if age_hours > max_age_hours:
try:
file_size = os.path.getsize(filepath)
os.remove(filepath)
deleted_size += file_size
deleted_count += 1
print(f" 删除: {os.path.basename(filepath)} ({format_size(file_size)})")
except Exception as e:
print(f" 删除失败: {os.path.basename(filepath)} - {e}")
if deleted_count > 0:
print(f" 已清理 {deleted_count} 个文件,释放 {format_size(deleted_size)} 空间")
else:
print(f" 无需清理")
logging.info(bi(f"下载缓存清理完成: 删除 {deleted_count} 个文件,释放 {format_size(deleted_size)}", f"Download cache cleanup completed: deleted {deleted_count} files, freed {format_size(deleted_size)}"))
def cleanup_video_cache():
"""清理视频帧缓存(按时间和大小清理)"""
try:
from config import get_cache_retention_hours, get_video_cache_max_mb
max_age_hours = get_cache_retention_hours()
max_size_mb = get_video_cache_max_mb()
except Exception:
max_age_hours = 24
max_size_mb = 2000
videoframe_dir = os.path.join(DATA_DIR, 'videoframe')
if not os.path.exists(videoframe_dir):
return
current_size = get_directory_size(videoframe_dir)
current_size_mb = current_size / (1024 * 1024)
print(f"\n🧹 清理视频帧缓存")
print(f" 当前缓存大小: {format_size(current_size)}")
print(f" 最大缓存: {max_size_mb} MB")
print(f" 保留时间: {max_age_hours} 小时")
dirs_to_delete = []
for item in os.listdir(videoframe_dir):
item_path = os.path.join(videoframe_dir, item)
if os.path.isdir(item_path):
try:
mtime = os.path.getmtime(item_path)
age_hours = (time.time() - mtime) / 3600
dir_size = get_directory_size(item_path)
dirs_to_delete.append((mtime, item_path, age_hours, dir_size))
except Exception:
pass
dirs_to_delete.sort(key=lambda x: x[0])
deleted_size = 0
deleted_count = 0
for mtime, dir_path, age_hours, dir_size in dirs_to_delete:
if age_hours > max_age_hours or current_size_mb > max_size_mb:
try:
shutil.rmtree(dir_path)
deleted_size += dir_size
deleted_count += 1
current_size_mb -= dir_size / (1024 * 1024)
print(f" 删除: {os.path.basename(dir_path)} ({format_size(dir_size)})")
if current_size_mb <= max_size_mb:
break
except Exception as e:
print(f" 删除失败: {os.path.basename(dir_path)} - {e}")
if deleted_count > 0:
print(f" 已清理 {deleted_count} 个任务目录,释放 {format_size(deleted_size)} 空间")
else:
print(f" 无需清理")
logging.info(bi(f"视频帧缓存清理完成: 删除 {deleted_count} 个任务,释放 {format_size(deleted_size)}", f"Video frame cache cleanup completed: deleted {deleted_count} tasks, freed {format_size(deleted_size)}"))
def detect_linux_distro():
"""检测Linux发行版"""
if os.path.exists('/etc/os-release'):
with open('/etc/os-release', 'r') as f:
for line in f:
if line.startswith('ID='):
distro = line.strip().split('=')[1].strip('"')
return distro.lower()
return 'unknown'
def run_command(cmd, description=""):
"""运行命令并返回结果"""
try:
print(f" [{description}] 执行命令: {' '.join(cmd)}")
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
print(f" [{description}] 命令执行成功")
return True, result.stdout
except subprocess.CalledProcessError as e:
print(f" [{description}] 命令执行失败: {e.stderr}")
return False, e.stderr
except Exception as e:
print(f" [{description}] 命令执行异常: {e}")
return False, str(e)
def install_with_package_manager(lib_name):
"""使用包管理器安装依赖"""
distro = detect_linux_distro()
print(f" [{lib_name}] 检测到发行版: {distro}")
package_commands = {
'debian': {
'update': ['apt-get', 'update'],
'ffmpeg': ['apt-get', 'install', '-y', 'ffmpeg'],
'sanjuuni': None
},
'ubuntu': {
'update': ['apt-get', 'update'],
'ffmpeg': ['apt-get', 'install', '-y', 'ffmpeg'],
'sanjuuni': None
},
'arch': {
'update': ['pacman', '-Syu', '--noconfirm'],
'ffmpeg': ['pacman', '-S', '--noconfirm', 'ffmpeg'],
'sanjuuni': ['pacman', '-S', '--noconfirm', 'sanjuuni']
},
'fedora': {
'update': ['dnf', 'check-update'],
'ffmpeg': ['dnf', 'install', '-y', 'ffmpeg'],
'sanjuuni': None
},
'centos': {
'update': ['yum', 'check-update'],
'ffmpeg': ['yum', 'install', '-y', 'ffmpeg'],
'sanjuuni': None
},
'opensuse': {
'update': ['zypper', 'refresh'],
'ffmpeg': ['zypper', 'install', '-y', 'ffmpeg'],
'sanjuuni': None
}
}
if distro not in package_commands:
print(f" [{lib_name}] 不支持的发行版: {distro},使用下载方式安装")
return False
commands = package_commands[distro]
cmd = commands.get(lib_name)
if cmd is None:
print(f" [{lib_name}] {distro} 不支持通过包管理器安装,使用下载方式")
return False
if os.geteuid() != 0:
print(f" [{lib_name}] 需要root权限使用sudo")
cmd = ['sudo'] + cmd
success, output = run_command(cmd, lib_name)
return success
ARCHITECTURE_MAP = {
('Windows', 'AMD64'): 'windows-x86_64',
('Windows', 'x86'): 'windows-x86',
('Linux', 'x86_64'): 'linux-x86_64',
('Linux', 'aarch64'): 'linux-aarch64',
('Linux', 'armv7l'): 'linux-armv7',
('Darwin', 'x86_64'): 'macos-x86_64',
('Darwin', 'arm64'): 'macos-arm64',
}
DEFAULT_DOWNLOAD_URLS = {
'ffmpeg': {
'windows-x86_64': 'https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-win64-gpl.zip',
'windows-x86': 'https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-win32-gpl.zip',
'linux-x86_64': 'https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linux64-gpl.tar.xz',
'linux-aarch64': 'https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linuxarm64-gpl.tar.xz',
'macos-x86_64': 'https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-macos64-gpl.tar.xz',
'macos-arm64': 'https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-macosarm64-gpl.tar.xz',
},
'sanjuuni': {
'windows-x86_64': 'https://github.com/MCJack123/sanjuuni/releases/latest/download/sanjuuni-Win64.zip',
}
}
def get_system_architecture():
system = platform.system()
machine = platform.machine()
if system == 'Linux' and machine == 'armv7l':
arch = 'armv7l'
else:
arch = machine
key = (system, arch)
return ARCHITECTURE_MAP.get(key, None)
def format_speed(bytes_per_second):
if bytes_per_second < 1024:
return f"{bytes_per_second} B/s"
elif bytes_per_second < 1024 * 1024:
return f"{bytes_per_second / 1024:.2f} KB/s"
else:
return f"{bytes_per_second / (1024 * 1024):.2f} MB/s"
def wait_for_input(prompt="按 Enter 键继续..."):
try:
input(prompt)
except EOFError:
pass
def is_library_installed(lib_name, target_arch):
if lib_name == 'ffmpeg':
bin_dir = os.path.join(LIB_DIR, 'ffmpeg', 'bin')
if sys.platform.startswith('win'):
exe_path = os.path.join(bin_dir, 'ffmpeg.exe')
else:
exe_path = os.path.join(bin_dir, 'ffmpeg')
if os.path.exists(exe_path):
return True
if sys.platform.startswith('linux'):
try:
result = subprocess.run(['which', 'ffmpeg'], capture_output=True)
if result.returncode == 0:
print(f" [FFmpeg] 在系统路径中找到: {result.stdout.decode().strip()}")
return True
except Exception:
pass
return False
elif lib_name == 'sanjuuni':
if sys.platform.startswith('win'):
exe_name = 'sanjuuni.exe'
else:
exe_name = 'sanjuuni'
lib_path = os.path.join(LIB_DIR, 'sanjuuni', exe_name)
if os.path.exists(lib_path):
return True
for path in os.environ.get('PATH', '').split(os.pathsep):
exe_path = os.path.join(path, exe_name)
if os.path.exists(exe_path):
return True
return False
return False
def download_file_with_retry(url, dest_path, lib_name="", max_retries=3, retry_delay=5):
import requests
for attempt in range(max_retries):
try:
logging.info(bi(f"[{lib_name}] 开始下载 (尝试 {attempt + 1}/{max_retries}): {url}", f"[{lib_name}] Starting download (attempt {attempt + 1}/{max_retries}): {url}"))
print(f" [{lib_name}] 正在下载... (尝试 {attempt + 1}/{max_retries})")
response = requests.get(url, stream=True, timeout=30)
response.raise_for_status()
total_size = int(response.headers.get('content-length', 0))
downloaded_size = 0
start_time = time.time()
last_time = start_time
last_downloaded = 0
progress_bar_length = 40
os.makedirs(os.path.dirname(dest_path), exist_ok=True)
if total_size > 0:
print(f" [{lib_name}] 文件大小: {format_size(total_size)}")
with open(dest_path, 'wb') as f:
for chunk in response.iter_content(chunk_size=8192):
if chunk:
f.write(chunk)
downloaded_size += len(chunk)
current_time = time.time()
if current_time - last_time >= 0.5 or downloaded_size == total_size:
elapsed_time = current_time - start_time
speed = downloaded_size / elapsed_time if elapsed_time > 0 else 0
if total_size > 0:
progress = downloaded_size / total_size
progress_percent = int(progress * 100)
filled_length = int(progress_bar_length * progress)
bar = '' * filled_length + '' * (progress_bar_length - filled_length)
eta_seconds = (total_size - downloaded_size) / speed if speed > 0 else 0
eta_minutes = int(eta_seconds // 60)
eta_remaining = int(eta_seconds % 60)
print(f"\r [{lib_name}] [{bar}] {progress_percent}% ({format_size(downloaded_size)}/{format_size(total_size)}) - {format_speed(speed)} - ETA: {eta_minutes:02d}:{eta_remaining:02d}", end='')
else:
print(f"\r [{lib_name}] 已下载: {format_size(downloaded_size)} - {format_speed(speed)}", end='')
last_time = current_time
last_downloaded = downloaded_size
elapsed_time = time.time() - start_time
avg_speed = downloaded_size / elapsed_time if elapsed_time > 0 else 0
print()
print(f" [{lib_name}] 下载完成!")
print(f" [{lib_name}] 文件保存到: {dest_path}")
print(f" [{lib_name}] 下载大小: {format_size(downloaded_size)}")
print(f" [{lib_name}] 平均速度: {format_speed(avg_speed)}")
print(f" [{lib_name}] 耗时: {int(elapsed_time // 60)}{int(elapsed_time % 60)}")
logging.info(bi(f"[{lib_name}] 下载完成: {dest_path} ({format_size(downloaded_size)})", f"[{lib_name}] Download completed: {dest_path} ({format_size(downloaded_size)})"))
return dest_path
except Exception as e:
error_msg = f"[{lib_name}] 下载失败 (尝试 {attempt + 1}/{max_retries}): {e}"
print(f"\n [{lib_name}] ❌ {error_msg}")
print(f" [{lib_name}] 错误类型: {type(e).__name__}")
logging.error(error_msg)
if os.path.exists(dest_path):
os.remove(dest_path)
if attempt < max_retries - 1:
print(f" [{lib_name}] 将在 {retry_delay} 秒后重试...")
time.sleep(retry_delay)
else:
print(f" [{lib_name}] 已达到最大重试次数 ({max_retries}次)")
wait_for_input("按 Enter 键继续...")
raise
def download_file(url, dest_path, lib_name=""):
return download_file_with_retry(url, dest_path, lib_name, max_retries=3, retry_delay=5)
def extract_archive(file_path, dest_dir, lib_name=""):
try:
file_size = os.path.getsize(file_path)
print(f" [{lib_name}] 开始解压...")
print(f" [{lib_name}] 压缩包大小: {format_size(file_size)}")
logging.info(bi(f"[{lib_name}] 开始解压: {file_path}", f"[{lib_name}] Starting extraction: {file_path}"))
start_time = time.time()
if file_path.endswith('.zip'):
with zipfile.ZipFile(file_path, 'r') as zip_ref:
file_count = len(zip_ref.namelist())
print(f" [{lib_name}] 包含 {file_count} 个文件")
zip_ref.extractall(dest_dir)
elif file_path.endswith('.tar.gz'):
with tarfile.open(file_path, 'r:gz') as tar_ref:
file_count = len(tar_ref.getmembers())
print(f" [{lib_name}] 包含 {file_count} 个文件")
tar_ref.extractall(dest_dir)
elif file_path.endswith('.tar.xz'):
with tarfile.open(file_path, 'r:xz') as tar_ref:
file_count = len(tar_ref.getmembers())
print(f" [{lib_name}] 包含 {file_count} 个文件")
tar_ref.extractall(dest_dir)
else:
raise ValueError(f"[{lib_name}] 不支持的压缩格式: {file_path}")
elapsed_time = time.time() - start_time
print(f" [{lib_name}] 解压完成!")
print(f" [{lib_name}] 解压到: {dest_dir}")
print(f" [{lib_name}] 耗时: {int(elapsed_time // 60)}{int(elapsed_time % 60)}")
logging.info(bi(f"[{lib_name}] 解压完成: {dest_dir}", f"[{lib_name}] Extraction completed: {dest_dir}"))
except Exception as e:
error_msg = f"[{lib_name}] 解压失败: {e}"
print(f" [{lib_name}] ❌ {error_msg}")
print(f" [{lib_name}] 错误类型: {type(e).__name__}")
logging.error(error_msg)
wait_for_input("按 Enter 键继续...")
raise
def install_ffmpeg(target_arch):
ffmpeg_dir = os.path.join(LIB_DIR, 'ffmpeg')
bin_dir = os.path.join(ffmpeg_dir, 'bin')
print(f"\n📦 安装 FFmpeg ({target_arch})")
print("-" * 50)
if os.path.exists(bin_dir):
import shutil
print(f" [FFmpeg] 检测到已安装,正在清理旧版本...")
logging.info(bi("[FFmpeg] 检测到已安装,清理旧版本", "[FFmpeg] Already installed, cleaning up old version"))
shutil.rmtree(ffmpeg_dir)
if sys.platform.startswith('linux'):
print(f" [FFmpeg] Linux系统尝试使用包管理器安装...")
if install_with_package_manager('ffmpeg'):
print(f" [FFmpeg] ✅ 通过包管理器安装成功")
return
print(f" [FFmpeg] 包管理器安装失败,使用下载方式")
url = DEFAULT_DOWNLOAD_URLS['ffmpeg'].get(target_arch)
if not url:
raise ValueError(f"[FFmpeg] 不支持当前架构: {target_arch}")
print(f" [FFmpeg] 下载地址: {url}")
cache_file = os.path.join(DOWNLOAD_CACHE, f"ffmpeg-{target_arch}.zip")
download_file(url, cache_file, "FFmpeg")
extract_archive(cache_file, ffmpeg_dir, "FFmpeg")
extracted_contents = os.listdir(ffmpeg_dir)
if len(extracted_contents) == 1 and os.path.isdir(os.path.join(ffmpeg_dir, extracted_contents[0])):
import shutil
inner_dir = os.path.join(ffmpeg_dir, extracted_contents[0])
print(f" [FFmpeg] 整理文件结构...")
for item in os.listdir(inner_dir):
src = os.path.join(inner_dir, item)
dst = os.path.join(ffmpeg_dir, item)
if os.path.exists(dst):
if os.path.isdir(dst):
shutil.rmtree(dst)
else:
os.remove(dst)
shutil.move(src, ffmpeg_dir)
shutil.rmtree(inner_dir)
ffmpeg_bin = 'ffmpeg.exe' if sys.platform.startswith('win') else 'ffmpeg'
final_path = os.path.join(bin_dir, ffmpeg_bin)
if not os.path.exists(final_path):
raise RuntimeError(f"[FFmpeg] 安装失败,未找到可执行文件: {final_path}")
os.chmod(final_path, 0o755)
final_size = sum(os.path.getsize(os.path.join(bin_dir, f)) for f in os.listdir(bin_dir) if os.path.isfile(os.path.join(bin_dir, f)))
print(f" [FFmpeg] ✅ 安装成功!")
print(f" [FFmpeg] 可执行文件: {final_path}")
print(f" [FFmpeg] 安装大小: {format_size(final_size)}")
logging.info(bi(f"[FFmpeg] 安装成功: {final_path}", f"[FFmpeg] Installation successful: {final_path}"))
print("-" * 50)
def install_sanjuuni(target_arch):
sanjuuni_dir = os.path.join(LIB_DIR, 'sanjuuni')
print(f"\n📦 安装 Sanjuuni ({target_arch})")
print("-" * 50)
url = DEFAULT_DOWNLOAD_URLS['sanjuuni'].get(target_arch)
if not url:
print(f" [Sanjuuni] ⚠️ 当前架构不支持自动下载")
print(f" [Sanjuuni] 请手动安装 sanjuuni:")
print()
print(f" [Sanjuuni] Arch Linux (AUR):")
print(f" yay -S sanjuuni")
print()
print(f" [Sanjuuni] Nix/NixOS:")
print(f" nix-channel --update")
print(f" nix-shell -p sanjuuni")
print()
print(f" [Sanjuuni] 其他系统请从源码编译:")
print(f" git clone https://github.com/MCJack123/sanjuuni")
print(f" cd sanjuuni")
print(f" mkdir build && cd build")
print(f" cmake ..")
print(f" make")
print()
print(f" [Sanjuuni] 安装完成后,请确保 sanjuuni 可执行文件在系统 PATH 中")
logging.warning(bi(f"[Sanjuuni] 当前架构 {target_arch} 需要手动安装", f"[Sanjuuni] Current architecture {target_arch} requires manual installation"))
wait_for_input("按 Enter 键继续...")
return
if os.path.exists(sanjuuni_dir):
import shutil
print(f" [Sanjuuni] 检测到已安装,正在清理旧版本...")
logging.info(bi("[Sanjuuni] 检测到已安装,清理旧版本", "[Sanjuuni] Already installed, cleaning up old version"))
shutil.rmtree(sanjuuni_dir)
print(f" [Sanjuuni] 下载地址: {url}")
cache_file = os.path.join(DOWNLOAD_CACHE, f"sanjuuni-{target_arch}.zip")
download_file(url, cache_file, "Sanjuuni")
extract_archive(cache_file, sanjuuni_dir, "Sanjuuni")
sanjuuni_bin = 'sanjuuni.exe' if sys.platform.startswith('win') else 'sanjuuni'
final_path = os.path.join(sanjuuni_dir, sanjuuni_bin)
if not os.path.exists(final_path):
extracted_contents = os.listdir(sanjuuni_dir)
print(f" [Sanjuuni] 解压内容: {extracted_contents}")
for item in extracted_contents:
item_path = os.path.join(sanjuuni_dir, item)
if os.path.isdir(item_path):
inner_bin = os.path.join(item_path, sanjuuni_bin)
if os.path.exists(inner_bin):
import shutil
for file in os.listdir(item_path):
src = os.path.join(item_path, file)
dst = os.path.join(sanjuuni_dir, file)
if os.path.exists(dst):
if os.path.isdir(dst):
shutil.rmtree(dst)
else:
os.remove(dst)
shutil.move(src, sanjuuni_dir)
shutil.rmtree(item_path)
final_path = os.path.join(sanjuuni_dir, sanjuuni_bin)
break
if not os.path.exists(final_path):
raise RuntimeError(f"[Sanjuuni] 安装失败,未找到可执行文件: {final_path}")
os.chmod(final_path, 0o755)
final_size = sum(os.path.getsize(os.path.join(sanjuuni_dir, f)) for f in os.listdir(sanjuuni_dir) if os.path.isfile(os.path.join(sanjuuni_dir, f)))
print(f" [Sanjuuni] ✅ 安装成功!")
print(f" [Sanjuuni] 可执行文件: {final_path}")
print(f" [Sanjuuni] 安装大小: {format_size(final_size)}")
logging.info(bi(f"[Sanjuuni] 安装成功: {final_path}", f"[Sanjuuni] Installation successful: {final_path}"))
print("-" * 50)
def check_and_install_libraries(force=False):
print("\n🔍 运行库检查")
print("=" * 50)
target_arch = get_system_architecture()
if not target_arch:
error_msg = f"无法识别当前系统架构: {platform.system()} {platform.machine()}"
print(f"{error_msg}")
logging.error(error_msg)
wait_for_input("按 Enter 键继续...")
return False
print(f"系统架构: {platform.system()} {platform.machine()}")
print(f"目标架构: {target_arch}")
logging.info(bi(f"检测到系统架构: {target_arch}", f"Detected system architecture: {target_arch}"))
os.makedirs(DOWNLOAD_CACHE, exist_ok=True)
print(f"缓存目录: {DOWNLOAD_CACHE}")
needs_install = False
if not is_library_installed('ffmpeg', target_arch) or force:
print(f"FFmpeg: ❌ 未安装")
install_ffmpeg(target_arch)
needs_install = True
else:
print(f"FFmpeg: ✅ 已安装")
if not is_library_installed('sanjuuni', target_arch) or force:
print(f"Sanjuuni: ❌ 未安装")
install_sanjuuni(target_arch)
needs_install = True
else:
print(f"Sanjuuni: ✅ 已安装")
if needs_install:
print("\n✅ 所有运行库已安装完成")
logging.info(bi("所有运行库已安装完成", "All runtime libraries installed successfully"))
cleanup_download_cache()
cleanup_video_cache()
elif not force:
print("\n✅ 所有运行库已安装完成")
logging.info(bi("所有运行库已安装完成", "All runtime libraries installed successfully"))
cleanup_download_cache()
cleanup_video_cache()
print("=" * 50)
return True

93
main.py
View File

@@ -1,6 +1,9 @@
import logging
import json
import os
def bi(zh, en):
return f"{zh} | {en}"
import hashlib
from flask import Flask, request, jsonify
import threading
@@ -36,7 +39,7 @@ try:
from video_frame_utils import process_video_frame_extraction
from file_cleanup import start_cleanup_thread
except ImportError as e:
logging.error(f"导入模块时出错: {e}")
logging.error(bi(f"导入模块时出错: {e}", f"Error importing module: {e}"))
# 定义空函数作为备用
def process_ffmpeg(*args, **kwargs):
return {'error': 'FFmpeg模块未正确导入'}
@@ -134,10 +137,10 @@ def validate_request(data, api_name):
for key in enter_parameter_table[api_name]:
is_optional = key in optional_parameters.get(api_name, set())
if key not in data and key != "subtitle" and not is_optional:
logging.warning(f"请求中没有提供{key}参数")
logging.warning(bi(f"请求中没有提供{key}参数", f"Request missing {key} parameter"))
return jsonify({'error': f"未提供{key}参数"}), 400
if key in data and isinstance(data[key], enter_parameter_table[api_name][key]) == False:
logging.warning(f"请求中{key}参数类型错误,应为{enter_parameter_table[api_name][key]}")
logging.warning(bi(f"请求中{key}参数类型错误,应为{enter_parameter_table[api_name][key]}", f"Request {key} parameter type error, expected {enter_parameter_table[api_name][key]}"))
return jsonify({'error': f"{key}参数类型错误,您输入为{type(data[key])},应为{enter_parameter_table[api_name][key]}" }), 400
return None, None
@@ -172,7 +175,7 @@ def run_async_task(task_id, process_func, data):
save_persistent_state()
except Exception as e:
logging.error(f"任务 {task_id} 执行异常: {e}")
logging.error(bi(f"任务 {task_id} 执行异常: {e}", f"Task {task_id} execution exception: {e}"))
with task_lock:
if task_id in task_registry:
task_registry[task_id]['status'] = 'error'
@@ -186,7 +189,7 @@ def run_async_task(task_id, process_func, data):
@app.route('/api/ffmpeg/async', methods=['POST'])
def ffmpeg_async_api():
"""创建异步FFmpeg任务"""
logging.info("收到异步FFmpeg API请求")
logging.info(bi("收到异步FFmpeg API请求", "Received async FFmpeg API request"))
data = request.get_json()
# 检测参数类型
@@ -222,7 +225,7 @@ def ffmpeg_async_api():
status_url = f"{scheme}://{host}/api/task/{task_id}"
logging.info(f"创建异步FFmpeg任务: {task_id}")
logging.info(bi(f"创建异步FFmpeg任务: {task_id}", f"Created async FFmpeg task: {task_id}"))
return jsonify({
'status': 'success',
'task_id': task_id,
@@ -233,7 +236,7 @@ def ffmpeg_async_api():
@app.route('/api/sanjuuni/async', methods=['POST'])
def sanjuuni_async_api():
"""创建异步Sanjuuni任务"""
logging.info("收到异步Sanjuuni API请求")
logging.info(bi("收到异步Sanjuuni API请求", "Received async Sanjuuni API request"))
data = request.get_json()
# 检测参数类型
@@ -269,7 +272,7 @@ def sanjuuni_async_api():
status_url = f"{scheme}://{host}/api/task/{task_id}"
logging.info(f"创建异步Sanjuuni任务: {task_id}")
logging.info(bi(f"创建异步Sanjuuni任务: {task_id}", f"Created async Sanjuuni task: {task_id}"))
return jsonify({
'status': 'success',
'task_id': task_id,
@@ -280,7 +283,7 @@ def sanjuuni_async_api():
@app.route('/api/video_frame/async', methods=['POST'])
def video_frame_async_api():
"""创建异步视频帧提取任务"""
logging.info("收到异步视频帧提取API请求")
logging.info(bi("收到异步视频帧提取API请求", "Received async video frame extraction API request"))
data = request.get_json()
# 检测参数类型
@@ -320,7 +323,7 @@ def video_frame_async_api():
renew_video_frame_task_expiry(task_info, now)
task_info['create_time'] = now
save_persistent_state()
logging.info(f"命中视频帧缓存,复用任务: {cached_task_id}, signature: {request_signature}")
logging.info(bi(f"命中视频帧缓存,复用任务: {cached_task_id}, signature: {request_signature}", f"Video frame cache hit, reusing task: {cached_task_id}, signature: {request_signature}"))
return jsonify({
'status': 'success',
'task_id': cached_task_id,
@@ -394,7 +397,7 @@ def video_frame_async_api():
status_url = f"{scheme}://{host}/api/task/{task_id}"
logging.info(f"创建异步视频帧提取任务: {task_id}")
logging.info(bi(f"创建异步视频帧提取任务: {task_id}", f"Created async video frame extraction task: {task_id}"))
return jsonify({
'status': 'success',
'task_id': task_id,
@@ -472,7 +475,7 @@ def get_task_status(task_id):
# 原有的同步接口保持不变
@app.route('/api/ffmpeg', methods=['POST'])
def ffmpeg_api():
logging.info("收到FFmpeg API请求")
logging.info(bi("收到FFmpeg API请求", "Received FFmpeg API request"))
data = request.get_json()
# 检测参数类型
@@ -494,10 +497,10 @@ def ffmpeg_api():
# 等待处理结果
result = result_queue.get()
if 'error' in result:
logging.error(f"处理过程中出错: {result['error']}")
logging.error(bi(f"处理过程中出错: {result['error']}", f"Error during processing: {result['error']}"))
return jsonify({'status': 'error', 'error': result['error']}), 500
else:
logging.info(f"处理成功返回下载URL: {result['download_url']}")
logging.info(bi(f"处理成功返回下载URL: {result['download_url']}", f"Processing successful, returning download URL: {result['download_url']}"))
return jsonify({
'status': 'success',
'download_url': result['download_url'],
@@ -506,7 +509,7 @@ def ffmpeg_api():
@app.route('/api/sanjuuni', methods=['POST'])
def sanjuuni_api():
logging.info("收到Sanjuuni API请求")
logging.info(bi("收到Sanjuuni API请求", "Received Sanjuuni API request"))
data = request.get_json()
# 检测参数类型
@@ -528,10 +531,10 @@ def sanjuuni_api():
# 等待处理结果
result = result_queue.get()
if 'error' in result:
logging.error(f"处理过程中出错: {result['error']}")
logging.error(bi(f"处理过程中出错: {result['error']}", f"Error during processing: {result['error']}"))
return jsonify({'status': 'error', 'error': result['error']}), 500
else:
logging.info(f"处理成功返回下载URL: {result['download_url']}")
logging.info(bi(f"处理成功返回下载URL: {result['download_url']}", f"Processing successful, returning download URL: {result['download_url']}"))
return jsonify({
'status': 'success',
'download_url': result['download_url'],
@@ -540,10 +543,10 @@ def sanjuuni_api():
@app.route('/download/<file_id>/<filename>', methods=['GET'])
def download_file_endpoint(file_id, filename):
logging.info(f"收到文件下载请求 - 文件ID: {file_id}, 文件名: {filename}")
logging.info(bi(f"收到文件下载请求 - 文件ID: {file_id}, 文件名: {filename}", f"Received file download request - File ID: {file_id}, filename: {filename}"))
with file_lock:
if file_id not in file_registry:
logging.warning(f"文件ID: {file_id} 不存在")
logging.warning(bi(f"文件ID: {file_id} 不存在", f"File ID: {file_id} does not exist"))
return jsonify({'error': '文件不存在'}), 404
file_info = file_registry[file_id]
file_info['last_access'] = time.time()
@@ -555,7 +558,7 @@ def download_file_endpoint(file_id, filename):
file_data = f.read()
return file_data, 200, {'Content-Disposition': f'attachment; filename={filename}'}
except Exception as e:
logging.error(f"下载文件时出错: {e}")
logging.error(bi(f"下载文件时出错: {e}", f"Error downloading file: {e}"))
return jsonify({'status': 'error', 'error': str(e)}), 500
@app.route('/frames/<job_id>/<filename>', methods=['GET'])
@@ -579,7 +582,7 @@ def serve_video_frames(job_id, filename):
try:
return send_from_directory(dir_path, safe_file)
except Exception as e:
logging.error(f"提供文件时出错: {e}")
logging.error(bi(f"提供文件时出错: {e}", f"Error serving file: {e}"))
return jsonify({"error": "文件访问失败"}), 500
@@ -594,9 +597,39 @@ def create_framepack():
touched_job_ids = set()
file_paths = []
# 新模式GET /api/framepack?<task_id>-<start>-<end>
# 获取查询字符串可能是范围表达式或第一个URL
range_expr = request.query_string.decode('utf-8').strip() if request.query_string else ''
if range_expr:
# 获取POST请求体
data = request.get_json(silent=True) or {}
# 检查是否是客户端格式查询参数是第一个URLbody包含完整URL列表
# 格式: POST /api/framepack?/frames/task_id/frame_000001.png Body: {"urls": [...]}
if range_expr.startswith('/frames/') and 'urls' in data:
# 这是客户端正常格式使用body中的urls列表
urls = data.get('urls', [])
if not isinstance(urls, list) or len(urls) == 0:
return jsonify({'error': 'Missing or empty urls list'}), 400
for url in urls:
if not url.startswith('/frames/'):
return jsonify({'error': f'Invalid URL prefix: {url}'}), 400
parts = url[len('/frames/'):].split('/', 1)
if len(parts) != 2:
return jsonify({'error': f'Malformed URL: {url}'}), 400
job_id, filename = parts
touched_job_ids.add(os.path.basename(job_id))
safe_job = os.path.basename(job_id)
safe_file = os.path.basename(filename)
if not safe_file.endswith('.png'):
return jsonify({'error': f'Only .png allowed: {filename}'}), 400
full_path = os.path.join(FRAMES_ROOT, safe_job, safe_file)
if not os.path.isfile(full_path):
return jsonify({'error': f'File not found: {full_path}'}), 404
file_paths.append(full_path)
# 新模式GET /api/framepack?<task_id>-<start>-<end>
elif range_expr:
try:
parts = range_expr.split('-')
if len(parts) != 3:
@@ -724,6 +757,16 @@ if __name__ == '__main__':
parser.add_argument('--port', type=int, default=5000, help='服务监听的端口号(默认: 5000')
args = parser.parse_args()
print("🔍 正在检查运行库...")
try:
from lib_downloader import check_and_install_libraries
check_and_install_libraries()
print("✅ 运行库检查完成")
except Exception as e:
logging.error(bi(f"运行库安装失败: {e}", f"Runtime library installation failed: {e}"))
print(f"❌ 运行库安装失败: {e}")
sys.exit(1)
load_persistent_state()
interrupted_count = mark_interrupted_tasks_on_startup()
if interrupted_count > 0:
@@ -732,7 +775,7 @@ if __name__ == '__main__':
from logo import print_logo
print_logo()
except Exception as e:
logging.warning(f"LOGO显示失败: {e}")
logging.info(f"启动应用程序,端口: {args.port}...")
logging.warning(bi(f"LOGO显示失败: {e}", f"LOGO display failed: {e}"))
logging.info(bi(f"启动应用程序,端口: {args.port}...", f"Starting application, port: {args.port}..."))
start_cleanup_thread()
app.run(host='0.0.0.0', port=args.port, threaded=True)

View File

@@ -62,9 +62,27 @@ def download_file(url, temp_dir, task_id=None, task_registry=None, task_lock=Non
logging.error(error_msg)
raise
def find_sanjuuni_binary():
if sys.platform.startswith('win'):
exe_name = 'sanjuuni.exe'
else:
exe_name = 'sanjuuni'
local_path = os.path.join('lib', 'sanjuuni', exe_name)
if os.path.exists(local_path):
return local_path
for path in os.environ.get('PATH', '').split(os.pathsep):
exe_path = os.path.join(path, exe_name)
if os.path.exists(exe_path):
return exe_path
return 'sanjuuni'
def execute_sanjuuni(input_path, output_path, sanjuuni_args, task_id=None, task_registry=None, task_lock=None):
try:
cmd = ['lib/sanjuuni/sanjuuni', '-i', input_path] + sanjuuni_args + ['-o', output_path]
sanjuuni_bin = find_sanjuuni_binary()
cmd = [sanjuuni_bin, '-i', input_path] + sanjuuni_args + ['-o', output_path]
if task_id and task_registry and task_lock:
add_task_log(task_id, bi(f"执行Sanjuuni命令: {' '.join(cmd)}", f"Execute Sanjuuni command: {' '.join(cmd)}"), task_registry, task_lock)

View File

@@ -7,19 +7,34 @@
import os
import sys
import logging
from main import app
from shared_utils import load_persistent_state, mark_interrupted_tasks_on_startup, save_persistent_state
from lib_downloader import check_and_install_libraries
from logo import print_logo
def bi(zh, en):
return f"{zh} | {en}"
from config import (
load_config,
get_server_port,
get_server_debug,
get_server_threaded,
get_cleanup_interval,
get_cleanup_retention,
get_streaming_enabled,
get_logging_level,
get_logging_file_path,
get_hwaccel_enabled,
get_hwaccel_method
)
os.makedirs('data', exist_ok=True)
# 配置日志
logging_level = getattr(logging, get_logging_level())
logging.basicConfig(
level=logging.INFO,
level=logging_level,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[
logging.StreamHandler(sys.stdout),
logging.FileHandler(os.path.join('data', 'server.log'))
logging.FileHandler(get_logging_file_path())
]
)
@@ -27,7 +42,19 @@ def main():
try:
print_logo()
except Exception as e:
logging.warning(f"LOGO显示失败: {e}")
logging.warning(bi(f"LOGO显示失败: {e}", f"LOGO display failed: {e}"))
print("🔍 正在检查运行库...")
try:
check_and_install_libraries()
print("✅ 运行库检查完成")
except Exception as e:
logging.error(bi(f"运行库安装失败: {e}", f"Runtime library installation failed: {e}"))
print(f"❌ 运行库安装失败: {e}")
sys.exit(1)
port = get_server_port()
streaming_mode = "流式" if get_streaming_enabled() else "非流式"
print("🚀 GMapiServer 启动中...")
print("=" * 50)
@@ -37,11 +64,25 @@ def main():
print(" • 视频帧提取 (异步,支持B站BV号)")
print()
print("🔧 配置信息:")
print(f" 端口: 5000")
print(f" 端口: {port}")
print(f" 转码模式: {streaming_mode}")
hwaccel_enabled = get_hwaccel_enabled()
hwaccel_method = get_hwaccel_method()
if hwaccel_enabled:
import ffmpeg_utils
ffmpeg_utils.init_hwaccel()
actual_method = ffmpeg_utils._hwaccel_method if ffmpeg_utils._hwaccel_method else "检测失败(使用软件解码)"
print(f" 硬件加速: 已启用 ({actual_method})")
else:
print(f" 硬件加速: 已禁用")
print(f" FFmpeg缓存目录: data/ffmpeg/")
print(f" Sanjuuni缓存目录: data/sanjuuni/")
print(f" 视频帧目录: data/videoframe/")
print(f" 清理间隔: 1小时")
print(f" 清理间隔: {get_cleanup_interval()}小时")
print(f" 文件保留时间: {get_cleanup_retention()}小时")
print()
print("🔗 API 端点:")
print(" 同步接口:")
@@ -64,20 +105,22 @@ def main():
print(" GET /api/tasks - 任务列表(调试)")
print("=" * 50)
print()
print("🌐 服务器地址: http://0.0.0.0:5000")
print("🔄 自动清理机制已启用 - 临时文件将在2小时后自动删除")
print(f"🌐 服务器地址: http://0.0.0.0:{port}")
print(f"🔄 自动清理机制已启用 - 临时文件将在{get_cleanup_retention()}小时后自动删除")
print()
from main import app
from shared_utils import load_persistent_state, mark_interrupted_tasks_on_startup, save_persistent_state
load_persistent_state()
interrupted_count = mark_interrupted_tasks_on_startup()
if interrupted_count > 0:
save_persistent_state()
# 启动Flask应用
from file_cleanup import start_cleanup_thread
start_cleanup_thread()
app.run(host='0.0.0.0', port=5000, debug=False, threaded=True)
app.run(host='0.0.0.0', port=port, debug=get_server_debug(), threaded=get_server_threaded())
if __name__ == '__main__':
main()

View File

@@ -11,6 +11,87 @@ import glob
import requests
import logging
from shared_utils import task_registry, file_registry, file_lock, task_lock, add_task_log, save_persistent_state
from config import get_video_download_limit_resolution, get_video_download_max_resolution, get_video_download_preferred_codec
def find_yt_dlp_executable():
result = subprocess.run(['which', 'yt-dlp'], capture_output=True, text=True)
if result.returncode == 0:
return result.stdout.strip()
return None
def detect_video_codec(file_path):
try:
result = subprocess.run(
['ffprobe', '-v', 'quiet', '-print_format', 'json', '-show_streams', file_path],
capture_output=True,
text=True,
timeout=30
)
if result.returncode == 0:
info = json.loads(result.stdout)
for stream in info.get('streams', []):
if stream.get('codec_type') == 'video':
return stream.get('codec_name')
except Exception as e:
logging.warning(bi(f"检测视频编码失败: {e}", f"Failed to detect video codec: {e}"))
return None
def ensure_yt_dlp():
try:
import yt_dlp
return True, None
except ImportError:
pass
print("🔧 yt-dlp 未安装,正在尝试自动安装...")
logging.info(bi("yt-dlp 未安装,正在尝试自动安装", "yt-dlp not installed, attempting automatic installation"))
venv_python = os.path.join(os.path.dirname(sys.executable), '..', '..', 'bin', 'python3')
if os.path.exists(venv_python):
venv_python = os.path.abspath(venv_python)
install_methods = [
([sys.executable, '-m', 'pip', 'install', 'yt-dlp', '--quiet'], "直接安装"),
([sys.executable, '-m', 'pip', 'install', 'yt-dlp', '--quiet', '--break-system-packages'], "使用 --break-system-packages"),
]
if os.path.exists(venv_python) and venv_python != sys.executable:
install_methods.insert(0, ([venv_python, '-m', 'pip', 'install', 'yt-dlp', '--quiet'], "虚拟环境安装"))
if sys.platform.startswith('linux'):
try:
result = subprocess.run(['which', 'apt-get'], capture_output=True)
if result.returncode == 0:
install_methods.append((['sudo', 'apt-get', 'install', '-y', 'yt-dlp'], "使用 apt-get"))
except Exception:
pass
for cmd, method in install_methods:
try:
print(f" 尝试 {method}: {' '.join(cmd)}")
subprocess.check_call(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print("✅ yt-dlp 安装成功")
logging.info(bi(f"yt-dlp 安装成功({method}", f"yt-dlp installed successfully ({method})"))
return True, None
except subprocess.CalledProcessError as e:
print(f"{method} 失败")
logging.warning(bi(f"yt-dlp {method} 安装失败: {e}", f"yt-dlp {method} installation failed: {e}"))
except Exception as e:
print(f"{method} 异常: {e}")
logging.warning(bi(f"yt-dlp {method} 安装异常: {e}", f"yt-dlp {method} installation exception: {e}"))
yt_dlp_path = find_yt_dlp_executable()
if yt_dlp_path:
print(f"⚠️ 检测到系统中有 yt-dlp 命令: {yt_dlp_path}")
logging.info(bi(f"检测到系统中有 yt-dlp 命令: {yt_dlp_path}", f"Found yt-dlp executable in system: {yt_dlp_path}"))
return True, yt_dlp_path
print("\n❌ yt-dlp 安装失败,请手动安装:")
print(" 方法1 (推荐): sudo apt-get install yt-dlp")
print(" 方法2: pip install yt-dlp --break-system-packages")
print(" 方法3: 创建虚拟环境后安装")
logging.error(bi("yt-dlp 所有安装方法均失败,请手动安装", "All yt-dlp installation methods failed, please install manually"))
return False, None
def bi(zh, en):
return f"{zh} | {en}"
@@ -34,12 +115,30 @@ def renew_video_frame_task_expiry(task_info, now):
# 确保frames目录存在
os.makedirs(FRAMES_ROOT, exist_ok=True)
if sys.platform.startswith('win'):
FFMPEG_PATH = os.path.join(BASE_DIR, 'lib', 'ffmpeg', 'bin', 'ffmpeg.exe')
FFPROBE_PATH = os.path.join(BASE_DIR, 'lib', 'ffmpeg', 'bin', 'ffprobe.exe')
else:
FFMPEG_PATH = os.path.join(BASE_DIR, 'lib', 'ffmpeg', 'bin', 'ffmpeg')
FFPROBE_PATH = os.path.join(BASE_DIR, 'lib', 'ffmpeg', 'bin', 'ffprobe')
def find_ffmpeg_tool(tool_name):
"""查找FFmpeg工具路径优先系统路径"""
if sys.platform.startswith('win'):
exe_name = f"{tool_name}.exe"
local_path = os.path.join(BASE_DIR, 'lib', 'ffmpeg', 'bin', exe_name)
else:
exe_name = tool_name
local_path = os.path.join(BASE_DIR, 'lib', 'ffmpeg', 'bin', tool_name)
if os.path.exists(local_path):
return local_path
if sys.platform.startswith('linux'):
try:
result = subprocess.run(['which', exe_name], capture_output=True, text=True)
if result.returncode == 0:
return result.stdout.strip()
except Exception:
pass
return local_path
FFMPEG_PATH = find_ffmpeg_tool('ffmpeg')
FFPROBE_PATH = find_ffmpeg_tool('ffprobe')
def parse_ffmpeg_frame_progress(line, task_id, task_registry, task_lock):
"""解析FFmpeg进度输出并更新任务状态"""
@@ -165,16 +264,64 @@ def process_video_frame_extraction(data, file_registry, file_lock, task_id=None,
if task_id in task_registry:
task_registry[task_id]['progress'] = 20 # 开始下载进度20%
yt_dlp_cmd = [
sys.executable, '-m', 'yt_dlp',
video_url,
'-o', temp_base,
'--no-warnings',
'--progress', # 启用进度显示
'--newline', # 确保换行符正常
'--console-title', # 确保进度信息正确输出
'--no-colors', # 禁用颜色,避免控制字符干扰
]
yt_dlp_ok, yt_dlp_path = ensure_yt_dlp()
if not yt_dlp_ok:
raise Exception(bi("yt-dlp 安装失败,请手动安装: pip install yt-dlp", "yt-dlp installation failed, please install manually: pip install yt-dlp"))
limit_resolution = get_video_download_limit_resolution()
max_resolution = get_video_download_max_resolution()
preferred_codec = get_video_download_preferred_codec()
format_spec = ''
filters = []
if preferred_codec != 'auto':
codec_map = {
'h264': 'avc1',
'h265': 'hvc1',
'av1': 'av01',
'vp9': 'vp09'
}
codec_id = codec_map.get(preferred_codec)
if codec_id:
filters.append(f'vcodec^={codec_id}')
logging.info(bi(f"使用首选编码: {preferred_codec} (vcodec^={codec_id})", f"Using preferred codec: {preferred_codec} (vcodec^={codec_id})"))
if limit_resolution and max_resolution != 'best':
filters.append(f'height<={max_resolution[:-1]}')
logging.info(bi(f"限制下载分辨率为: {max_resolution}", f"Limit download resolution to: {max_resolution}"))
if filters:
format_spec = f'bv*[{"][".join(filters)}]+ba*'
else:
format_spec = 'bestvideo+bestaudio'
if yt_dlp_path:
yt_dlp_cmd = [
yt_dlp_path,
video_url,
'-o', temp_base,
'-f', format_spec,
'--no-playlist',
'--no-warnings',
'--progress',
'--newline',
'--console-title',
'--no-colors',
]
else:
yt_dlp_cmd = [
sys.executable, '-m', 'yt_dlp',
video_url,
'-o', temp_base,
'-f', format_spec,
'--no-playlist',
'--no-warnings',
'--progress',
'--newline',
'--console-title',
'--no-colors',
]
# 使用 Popen 实时捕获 stdout 和 stderr
proc = subprocess.Popen(
@@ -306,9 +453,16 @@ def process_video_frame_extraction(data, file_registry, file_lock, task_id=None,
if task_id in task_registry:
task_registry[task_id]['progress'] = 40
hwaccel_args_audio = []
codec = detect_video_codec(temp_audio)
if codec == 'av1':
logging.info(bi("检测到AV1编码视频音频提取时禁用硬件加速", "Detected AV1 encoded video, disabling hardware acceleration for audio extraction"))
hwaccel_args_audio = ['-hwaccel', 'none']
# 公共参数
dfpwm_args = [
FFMPEG_PATH, '-y',
FFMPEG_PATH, '-y'
] + hwaccel_args_audio + [
'-i', temp_audio,
'-vn',
'-ar', '48000',
@@ -381,8 +535,21 @@ def process_video_frame_extraction(data, file_registry, file_lock, task_id=None,
frame_pattern = os.path.join(job_dir, "frame_%06d.png")
try:
from ffmpeg_utils import get_hwaccel_args
hwaccel_args = get_hwaccel_args()
codec = detect_video_codec(temp_video)
if codec == 'av1':
logging.info(bi("检测到AV1编码视频禁用硬件加速", "Detected AV1 encoded video, disabling hardware acceleration"))
hwaccel_args = []
except Exception as e:
logging.warning(bi(f"获取硬件加速参数失败: {e}", f"Failed to get hardware acceleration args: {e}"))
hwaccel_args = []
ffmpeg_cmd = [
FFMPEG_PATH, '-y',
FFMPEG_PATH, '-y'
] + hwaccel_args + [
'-i', temp_video,
'-vf', vf,
'-pix_fmt', pix_fmt,