refactor(backend): 将Celery替换为Arq进行协程任务处理

本次提交将后端的任务队列系统从Celery迁移到了Arq,以支持基于协程的任务处理。主要改动包括:
- 更新文档和配置文件,反映架构变化。
- 修改健康检查和服务初始化逻辑,以适应Arq的使用。
- 移除与Celery相关的代码,并添加Arq任务定义和调度器。
- 更新Dockerfile和相关脚本,确保Arq worker能够正确运行。
- 调整API和业务服务中的任务处理逻辑,移除对Celery的依赖。

这些改动旨在提高系统的异步处理能力和整体性能。
This commit is contained in:
danial
2025-09-18 16:02:05 +08:00
parent ccb2969663
commit 8ad2a5366a
40 changed files with 1137 additions and 960 deletions

View File

@@ -16,6 +16,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
### Backend (FastAPI)
- `cd backend && uv sync` - Install dependencies with UV
- `cd backend && uv run python app/main.py` - Run development server
- `cd backend && uv run python scripts/start_arq_worker.py` - Start ARQ worker
- `cd backend && uv run pytest` - Run tests
- `cd backend && uv run black .` - Format code with Black
- `cd backend && uv run isort .` - Sort imports with isort
@@ -47,11 +48,11 @@ This is a full-stack application with:
### Backend Architecture
- **Framework**: FastAPI with async/await
- **Architecture**: Distributed microservices with Celery task queues
- **Architecture**: Distributed microservices with ARQ task queues
- **Browser Automation**: Playwright for web scraping
- **Database**: PostgreSQL with asyncpg
- **Cache**: Redis for distributed locking and state management
- **Task Queue**: Celery with Redis broker
- **Task Queue**: ARQ with Redis broker
- **Monitoring**: OpenTelemetry integration
- **Package Manager**: UV (configured in pyproject.toml)
@@ -77,7 +78,7 @@ This is a full-stack application with:
- `app/models/` - SQLAlchemy data models
- `app/schemas/` - Pydantic schemas
- `app/services/` - Business logic services
- `app/tasks/` - Celery task definitions
- `app/tasks/` - ARQ task definitions
- `app/repositories/` - Data access layer
- `deploy/` - Docker and deployment configurations
- `docs/` - Architecture and project documentation
@@ -96,7 +97,7 @@ This is a full-stack application with:
#### Backend Features
- Distributed web scraping with Playwright
- Redis-based distributed locking
- Celery task queue system
- ARQ task queue system
- Automatic task recovery and failure handling
- Kubernetes deployment support
- Graceful shutdown handling
@@ -120,7 +121,7 @@ This is a full-stack application with:
#### Backend State
- **Database**: PostgreSQL with SQLAlchemy ORM
- **Cache**: Redis for distributed state and locking
- **Task State**: Celery backend for task progress tracking
- **Task State**: ARQ backend for task progress tracking
- **Session State**: Redis-based session management
### Deployment Configuration
@@ -160,7 +161,7 @@ This is a full-stack application with:
#### Key Architectural Patterns
- **Microservices**: Backend uses service-oriented architecture
- **Distributed Systems**: Redis-based coordination and locking
- **Event-Driven**: Celery task queues for async processing
- **Event-Driven**: ARQ task queues for async processing
- **Reactive UI**: Frontend uses real-time updates with polling
- **Type Safety**: Full TypeScript coverage on both ends