kami_spider_monorepo/.qoder/quests/project-architecture-setup.md

Project Architecture Setup Design

1. Overview

1.1 Project Vision

Build a stateless, production-ready FastAPI-based web service platform that hosts multiple independent web applications through a unified entry point. The architecture emphasizes observability, scalability, and modern Python ecosystem practices.

1.2 Core Objectives

• Stateless Service: Application state externalized to Redis and MySQL
• Multi-Application Support: Single web service hosting independent applications (App A, App B, etc.)
• Modern Python Stack: Python 3.13, UV package manager, FastAPI, SQLModel, Pydantic 2.x
• Full Observability: OpenTelemetry integration with trace context propagation
• Production Deployment: Docker, Docker Compose, Docker Swarm, and Kubernetes support
• API Quality: Pydantic-validated I/O, auto-generated documentation, RESTful response format

1.3 Technology Stack

Component	Technology	Version
Language	Python	3.13
Package Manager	UV	Latest
Web Framework	FastAPI	Latest
ORM	SQLModel	Latest
Validation	Pydantic	2.x Latest
ASGI Server	Uvicorn	Latest
Process Manager	Gunicorn	Latest
Database	MySQL	8.x (future: PostgreSQL)
Cache/Session	Redis	Latest
Observability	OpenTelemetry	Latest
Telemetry Protocol	gRPC	-

2. Architecture Design

2.1 System Architecture

graph TB
    subgraph "Load Balancer / Ingress"
        LB[Ingress Controller]
    end
    
    subgraph "Application Layer - Stateless"
        LB --> APP1[FastAPI Instance 1]
        LB --> APP2[FastAPI Instance 2]
        LB --> APP3[FastAPI Instance N]
    end
    
    subgraph "Application Routing"
        APP1 --> ROUTER[Unified Router]
        ROUTER --> APPA[Web App A]
        ROUTER --> APPB[Web App B]
        ROUTER --> APPC[Web App N]
    end
    
    subgraph "Shared Stateful Services - Single Instance"
        REDIS[(Single Redis Instance)]
        MYSQL[(Single MySQL Instance)]
    end
    
    subgraph "Data Layer Access"
        APPA --> REDIS
        APPB --> REDIS
        APPC --> REDIS
        APPA --> MYSQL
        APPB --> MYSQL
        APPC --> MYSQL
    end
    
    subgraph "Observability"
        APP1 -.Traces/Logs.-> OTEL[OpenTelemetry Collector]
        APP2 -.Traces/Logs.-> OTEL
        APP3 -.Traces/Logs.-> OTEL
        OTEL -.gRPC.-> BACKEND[38.38.251.113:31547]
    end
    
    style REDIS fill:#ff9999
    style MYSQL fill:#ff9999

2.2 Layered Architecture

graph LR
    subgraph "Presentation Layer"
        A[FastAPI Routes]
        B[Request Validation]
        C[Response Formatting]
    end
    
    subgraph "Business Layer"
        D[Service Classes]
        E[Business Logic]
    end
    
    subgraph "Data Layer"
        F[SQLModel Models]
        G[Repository Pattern]
        H[Cache Layer]
    end
    
    subgraph "Cross-Cutting"
        I[Middleware: TraceID Injection]
        J[Exception Handlers]
        K[Logging with Trace Context]
    end
    
    A --> D
    B --> A
    A --> C
    D --> F
    F --> G
    G --> H
    I -.-> A
    I -.-> D
    K -.-> A
    K -.-> D

2.3 Multi-Application Architecture

graph TD
    subgraph "Main Application"
        MAIN[main.py - FastAPI App]
        MAIN --> MIDDLEWARE[Middleware Stack]
        MIDDLEWARE --> CORE_ROUTER[Core Router]
    end
    
    subgraph "Application Modules"
        CORE_ROUTER --> APP_A_ROUTER["/app-a/*"]
        CORE_ROUTER --> APP_B_ROUTER["/app-b/*"]
        CORE_ROUTER --> APP_N_ROUTER["/app-n/*"]
        
        APP_A_ROUTER --> APP_A_SERVICE[App A Services]
        APP_B_ROUTER --> APP_B_SERVICE[App B Services]
        APP_N_ROUTER --> APP_N_SERVICE[App N Services]
    end
    
    subgraph "Shared Infrastructure - Single Instance"
        APP_A_SERVICE --> SHARED_DB[Shared Database Pool]
        APP_B_SERVICE --> SHARED_DB
        APP_N_SERVICE --> SHARED_DB
        
        APP_A_SERVICE --> SHARED_REDIS[Shared Redis Pool]
        APP_B_SERVICE --> SHARED_REDIS
        APP_N_SERVICE --> SHARED_REDIS
        
        SHARED_DB --> MYSQL[(Single MySQL Instance)]
        SHARED_REDIS --> REDIS[(Single Redis Instance)]
    end

3. Project Structure

3.1 Directory Organization

kami_spider/
├── apps/                          # Independent web applications
│   ├── __init__.py
│   ├── app_a/                     # Application A module
│   │   ├── __init__.py
│   │   ├── router.py              # FastAPI routes for App A
│   │   ├── services.py            # Business logic
│   │   ├── models.py              # SQLModel definitions
│   │   ├── schemas.py             # Pydantic request/response schemas
│   │   └── dependencies.py        # App-specific dependencies
│   ├── app_b/                     # Application B module
│   │   └── ...                    # Same structure as App A
│   └── shared/                    # Shared utilities across apps
│       ├── base_models.py         # Base SQLModel classes
│       └── common_schemas.py      # Common Pydantic schemas
├── core/                          # Core infrastructure
│   ├── __init__.py
│   ├── config.py                  # Pydantic Settings configuration
│   ├── database.py                # SQLModel database engine
│   ├── redis.py                   # Redis client setup
│   ├── responses.py               # RESTful response generics
│   ├── exceptions.py              # Custom exception classes
│   └── dependencies.py            # Global dependencies
├── middleware/                    # Middleware components
│   ├── __init__.py
│   ├── trace_context.py           # TraceID injection
│   ├── error_handler.py           # Global error handling
│   └── correlation.py             # Request correlation
├── observability/                 # OpenTelemetry integration
│   ├── __init__.py
│   ├── tracing.py                 # Tracer configuration
│   ├── logging.py                 # Structured logging with trace
│   └── metrics.py                 # Metrics instrumentation
├── deployments/                   # Deployment configurations
│   ├── docker/
│   │   ├── Dockerfile
│   │   └── docker-compose.yml
│   ├── swarm/
│   │   └── docker-compose.swarm.yml
│   └── k8s/
│       ├── deployment.yaml
│       ├── service.yaml
│       ├── configmap.yaml
│       └── ingress.yaml
├── tests/                         # Test suites
│   ├── unit/
│   ├── integration/
│   └── conftest.py
├── main.py                        # Application entry point
├── pyproject.toml                 # UV project configuration
└── README.md

3.2 Module Responsibilities

Module	Responsibility
apps/	Contains all independent web applications with isolated business logic
core/	Shared infrastructure: config, database, cache, response formatting
middleware/	Request/response interceptors for cross-cutting concerns
observability/	OpenTelemetry setup, logging, tracing, metrics
deployments/	Container and orchestration manifests
main.py	FastAPI application bootstrap and router registration

4. Core Components Design

4.1 Configuration Management

4.1.1 Settings Structure

Use Pydantic Settings to manage all configuration with environment variable support and validation.

Configuration Categories:

Category	Purpose	Example Fields
Application	Runtime settings	APP_NAME, DEBUG, ENVIRONMENT
Database	MySQL connection	DB_HOST, DB_PORT, DB_NAME, DB_USER, DB_PASSWORD
Redis	Cache configuration	REDIS_HOST, REDIS_PORT, REDIS_DB, REDIS_PASSWORD
OpenTelemetry	Observability	OTEL_EXPORTER_ENDPOINT, OTEL_SERVICE_NAME
Server	Gunicorn/Uvicorn	WORKERS, HOST, PORT, LOG_LEVEL

4.1.2 Settings Validation

• All settings validated at startup using Pydantic validators
• Support for .env files in development
• Environment-specific overrides (dev, staging, production)
• Sensitive values (passwords, tokens) loaded from environment only

4.2 Database Layer

4.2.1 SQLModel Integration

Connection Management:

• Async SQLAlchemy engine with connection pooling
• Session factory using dependency injection
• Automatic session cleanup via context managers

Migration Strategy:

• Alembic for schema migrations
• Version-controlled migration files
• Support for MySQL → PostgreSQL future migration

4.2.2 Repository Pattern

Base Repository:

• Generic CRUD operations
• Pagination support
• Filter and query building
• Transaction management
• Shared across all applications

Application-Specific Repositories:

• Each app defines repositories extending base
• Encapsulates data access logic
• Returns domain models, not ORM entities
• Accesses same database instance as other applications
• May use discriminator columns to logically filter data by application

Cross-Application Data Access:

• Applications can access each other's tables if needed
• Shared models can be defined in apps/shared/base_models.py
• Business logic enforces data boundaries, not database constraints

4.3 Redis Integration

4.3.1 Connection Strategy

• Single Redis async client using redis-py
• Shared connection pool across all applications
• Health check integration
• All applications connect to the same Redis instance

4.3.2 Usage Patterns

Pattern	Use Case	Key Strategy
Session Store	User session data with TTL	Optional app prefix: session:{app}:{id}
Cache	Query result caching with invalidation	Optional app prefix: cache:{app}:{key}
Rate Limiting	API rate limit tracking	Shared or per-app: ratelimit:{endpoint}:{user}
Distributed Lock	Coordination across instances	Shared locks: lock:{resource}
Queue	Background task queueing (optional)	Shared queue: queue:{task_type}

Key Namespace Convention (Optional):

• Use prefixes for organizational clarity: {app_name}:{resource_type}:{identifier}
• Not enforced at infrastructure level
• Applications can read/write any keys
• Helps with debugging and monitoring

4.4 RESTful Response Format

4.4.1 Unified Response Structure

All API responses follow a consistent structure using generic types. Both success and error responses share the same top-level structure to ensure consistency.

Unified Response Schema:

Field	Type	Description
code	integer	Business status code (0 = success, >0 = various business errors)
message	string	Human-readable message describing the result
data	Generic[T] | null	Response payload for success, null for errors
trace_id	string	Request trace ID for debugging and log correlation
timestamp	datetime	Response timestamp (ISO 8601 format)

Response Examples:

Success Response (code = 0):

{
  "code": 0,
  "message": "Success",
  "data": {
    "user_id": 12345,
    "username": "john_doe"
  },
  "trace_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "timestamp": "2024-01-15T10:30:00Z"
}

Error Response (code > 0):

{
  "code": 1001,
  "message": "Login failed: Invalid credentials",
  "data": null,
  "trace_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "timestamp": "2024-01-15T10:30:00Z"
}

4.4.2 Business Code Convention

Code Range Assignment:

Code Range	Category	Description
0	Success	Operation completed successfully
1000-1999	Authentication & Authorization	Login failures, permission errors
2000-2999	Business Logic Errors	Order failures, insufficient balance, etc.
3000-3999	Validation Errors	Invalid input, missing required fields
4000-4999	Resource Errors	Not found, already exists, conflicts
5000-5999	System Errors	Database errors, external service failures
9000-9999	Unknown Errors	Unexpected errors

Example Business Codes:

Code	Message	Use Case
0	Success	Any successful operation
1001	Login failed	Invalid username or password
1002	Token expired	JWT token has expired
1003	Insufficient permissions	User lacks required role
2001	Order creation failed	Inventory insufficient
2002	Payment failed	Payment gateway error
2003	Insufficient balance	User account balance too low
3001	Invalid input	Field validation failure
3002	Missing required field	Required parameter not provided
4001	Resource not found	Requested entity doesn't exist
4002	Resource already exists	Duplicate creation attempt
4003	Resource conflict	State conflict or version mismatch
5001	Database error	Database connection or query failure
5002	External service error	Third-party API unavailable
9000	Unknown error	Unhandled exception

Application-Specific Code Ranges (Optional):

Each application can define sub-ranges within the main categories:

• App A: 10000-19999
• App B: 20000-29999
• App N: 30000-39999

4.4.3 HTTP Status Code Strategy

HTTP Status Codes vs Business Codes:

• HTTP Status Codes: Indicate transport/protocol-level status
• Business Codes: Indicate application-level result within the response body

HTTP Status Code Usage:

HTTP Status	Usage	Business Code Range
200 OK	Request processed successfully (business success or failure)	0, 1000-9999
201 Created	Resource successfully created	0
204 No Content	Successful DELETE with no response body	N/A (no body)
400 Bad Request	Malformed request, cannot be processed	3000-3999
401 Unauthorized	Missing or invalid authentication	1001-1099
403 Forbidden	Authenticated but insufficient permissions	1003
404 Not Found	Endpoint does not exist	N/A
422 Unprocessable Entity	Pydantic validation failures	3001-3099
500 Internal Server Error	Unhandled server errors	9000-9999
503 Service Unavailable	Database/Redis/external service down	5000-5999

Response Strategy:

• Business-level errors (login failed, order failed, insufficient balance): Return HTTP 200 with appropriate business code
• Protocol-level errors (invalid JSON, missing auth header): Return appropriate HTTP 4xx/5xx status
• Most application responses use HTTP 200 with business code indicating actual result
• Client checks code field to determine success (code = 0) or specific failure reason (code > 0)

4.4.4 Generic Response Types

TypeVar Definitions:

• T for data payload type

Generic Response Class:

• ApiResponse[T] for all responses (success and error)
  • code: Business status code (integer)
  • message: Human-readable message (string)
  • data: Typed payload or null (Generic[T] | None)
  • trace_id: Trace identifier (string)
  • timestamp: Response timestamp (datetime)

Specialized Response Types:

• SuccessResponse[T]: ApiResponse with code=0 and typed data
• ErrorResponse: ApiResponse with code>0 and data=null
• PaginatedResponse[T]: ApiResponse where data contains:
  • items: List[T]
  • total: Total record count
  • page: Current page number
  • page_size: Records per page
  • total_pages: Total page count

Response Builder Pattern:

Utility functions for creating responses:

• success(data: T, message: str = "Success") → Returns ApiResponse with code=0
• error(code: int, message: str) → Returns ApiResponse with specified error code
• paginated(items: List[T], total: int, page: int, page_size: int) → Returns paginated response

4.5 Pydantic Validation

4.5.1 Input Validation

• All route parameters validated via Pydantic models
• Query parameters use Pydantic schemas
• Request body validated with strict mode
• Custom validators for business rules

4.5.2 Output Validation

• Response models enforce output contracts
• Auto-filtering of sensitive fields
• Serialization customization (alias, exclude)

4.5.3 API Documentation

• FastAPI auto-generates OpenAPI spec from Pydantic models
• Swagger UI and ReDoc available at /docs and /redoc
• Schema examples in Pydantic models populate API docs
• Response model declarations generate accurate documentation

5. OpenTelemetry Integration

5.1 Instrumentation Architecture

graph LR
    subgraph "Request Flow"
        A[Incoming Request] --> B[ASGI Middleware]
        B --> C[TraceID Extraction/Generation]
        C --> D[Span Creation]
        D --> E[Business Logic]
        E --> F[Database Query]
        E --> G[Redis Operation]
        F --> H[Response]
        G --> H
    end
    
    subgraph "Trace Context"
        C --> I[Context Propagation]
        I --> J[Logging Context]
        I --> K[DB Span]
        I --> L[Redis Span]
    end
    
    subgraph "Export"
        D --> M[Trace Exporter]
        J --> N[Log Exporter]
        M --> O[gRPC: 38.38.251.113:31547]
        N --> O
    end

5.2 TraceID Management

5.2.1 Trace Context Injection

Middleware Responsibilities:

• Extract traceparent from incoming request headers (W3C Trace Context)
• Generate new trace ID if not present
• Inject trace ID into request context
• Add trace ID to response headers
• Propagate context to all sub-spans

5.2.2 Context Propagation

Component	Propagation Method
HTTP Requests	W3C Trace Context headers
Database Queries	SQLAlchemy instrumentation auto-propagates
Redis Operations	Manual span creation in Redis client
Logging	ContextVar injection into log records
Background Tasks	Explicit context passing

5.3 Logging with Trace Context

5.3.1 Structured Logging

Log Format:

• JSON-structured logs for production
• Human-readable for development
• Automatic trace_id injection into every log record
• Log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL

Standard Log Fields:

Field	Description
timestamp	ISO 8601 timestamp
level	Log level
logger	Logger name (module)
message	Log message
trace_id	Current trace ID
span_id	Current span ID
service.name	Service name from config
extras	Additional context (user_id, request_id, etc.)

5.3.2 Log Correlation

• Every log statement automatically includes trace_id
• Logs associated with spans exported to OpenTelemetry collector
• Correlation between traces and logs in observability backend

5.4 Telemetry Export

5.4.1 Exporter Configuration

Protocol: gRPC
Endpoint: 38.38.251.113:31547
Exported Data:

• Traces (spans with timing, attributes, events)
• Logs (structured with trace context)
• Metrics (optional: request rate, latency, error rate)

5.4.2 Instrumentation Targets

Target	Library	Auto/Manual
FastAPI	opentelemetry-instrumentation-fastapi	Auto
SQLAlchemy	opentelemetry-instrumentation-sqlalchemy	Auto
Redis	opentelemetry-instrumentation-redis	Auto
HTTP Client	opentelemetry-instrumentation-httpx	Auto
Custom Logic	opentelemetry-api (manual spans)	Manual

5.5 Performance Considerations

• Sampling strategy for high-traffic endpoints (e.g., 10% sampling)
• Batch span export to reduce network overhead
• Async export to avoid blocking request processing
• Circuit breaker for collector unavailability

6. Multi-Application Routing

6.1 Application Registration

Router Mounting Strategy:

• Each application provides a FastAPI router
• Main application mounts routers with path prefixes
• Example: App A mounted at /app-a, App B at /app-b

Router Structure:

graph TD
    MAIN[Main FastAPI App] --> HEALTH[/health]
    MAIN --> DOCS[/docs, /redoc]
    MAIN --> APP_A[/app-a/*]
    MAIN --> APP_B[/app-b/*]
    
    APP_A --> A_USERS[/app-a/users]
    APP_A --> A_ORDERS[/app-a/orders]
    
    APP_B --> B_PRODUCTS[/app-b/products]
    APP_B --> B_INVENTORY[/app-b/inventory]

6.2 Application Isolation

Namespace Isolation:

Aspect	Isolation Strategy
Routes	Path prefix separation
Database Tables	Shared tables, logical separation via application identifier column
Redis Keys	Shared Redis, optional key prefixing for clarity
Logging	Logger name includes app identifier
Metrics	Service labels distinguish apps
Dependencies	Shared dependency injection with optional app-specific overrides

Shared Resource Strategy:

• All applications share the same database instance and connection pool
• All applications share the same Redis instance and connection pool
• Data isolation achieved through application logic, not physical separation
• Tables may include app_id or similar discriminator columns where needed
• Redis keys can use prefixes for organizational clarity but not enforced isolation

6.3 Shared Infrastructure

Shared Components:

• Database connection pool

• Redis connection pool Shared Components:

• Single Database Instance: All applications connect to the same MySQL database

  • Shared connection pool across all applications
  • Shared transaction management
  • Tables accessible by all applications
  • Optional: Use discriminator columns for logical separation

• Single Redis Instance: All applications use the same Redis cache

  • Shared connection pool
  • Shared key space (namespace prefixing optional)
  • Shared session storage
  • Shared rate limiting counters

• Common Dependencies:

  • OpenTelemetry tracer (single service name or app-tagged spans)
  • Middleware stack (applied to all applications)
  • Exception handlers (global error handling)
  • Response formatting utilities (consistent API responses)
  • Configuration management (single settings instance)
  • Logging configuration (shared formatters and handlers)

Benefits of Shared Infrastructure:

• Simplified operations and maintenance
• Reduced resource consumption
• Easier data sharing between applications
• Centralized monitoring and observability
• Single source of truth for configuration

6.4 Application Lifecycle

Application Bootstrap Flow:

sequenceDiagram
    participant Main
    participant Config
    participant DB
    participant Redis
    participant OTEL
    participant AppA
    participant AppB
    
    Main->>Config: Load Settings
    Main->>OTEL: Initialize Tracing
    Main->>DB: Create Engine & Pool
    Main->>Redis: Create Client & Pool
    Main->>AppA: Register Router
    Main->>AppB: Register Router
    Main->>Main: Apply Middleware
    Main->>Main: Register Exception Handlers
    Main->>Main: Start Server

7. Middleware Stack

7.1 Middleware Execution Order

graph TD
    REQ[Incoming Request] --> M1[CORS Middleware]
    M1 --> M2[TraceID Injection]
    M2 --> M3[Request Logging]
    M3 --> M4[Error Handler]
    M4 --> M5[Authentication Optional]
    M5 --> ROUTE[Route Handler]
    ROUTE --> RES[Response]
    RES --> L1[Response Logging]
    L1 --> L2[Trace Header Addition]
    L2 --> OUT[Outgoing Response]

7.2 Middleware Components

Middleware	Purpose	Configuration
CORS	Cross-origin resource sharing	Allow origins from settings
TraceID Injection	Extract/generate trace context	W3C Trace Context format
Request Logging	Log incoming requests with trace	Log level: INFO
Error Handler	Catch and format exceptions	Return standardized error response
Authentication	Validate JWT/API keys (if needed)	Optional, app-specific
Rate Limiting	Prevent abuse	Redis-backed sliding window
Request ID	Generate correlation ID	UUID v4

7.3 Exception Handling Strategy

Exception Hierarchy:

graph TD
    BASE[BaseException] --> APP[ApplicationException]
    APP --> VALIDATION[ValidationException - 400]
    APP --> NOTFOUND[NotFoundException - 404]
    APP --> CONFLICT[ConflictException - 409]
    APP --> AUTH[AuthenticationException - 401]
    APP --> PERM[PermissionException - 403]
    APP --> BUSINESS[BusinessLogicException - 400]
    
    BASE --> SYSTEM[SystemException]
    SYSTEM --> DB_ERROR[DatabaseException - 503]
    SYSTEM --> CACHE_ERROR[CacheException - 503]
    SYSTEM --> EXTERNAL[ExternalServiceException - 502]

Exception Handler Behavior:

• Catch all exceptions at top level
• Log exception with trace context
• Map exception type to HTTP status code
• Return standardized ErrorResponse
• Include trace_id for debugging
• Sanitize sensitive information in error messages

8. Deployment Strategy

8.1 Container Design

8.1.1 Dockerfile Strategy

Base Image: python:3.13-slim

Build Stages:

1. Builder Stage: Install UV, copy dependencies, install packages
2. Runtime Stage: Copy virtual environment, copy application code
3. Optimization: Multi-stage build to reduce image size

Runtime Configuration:

• Use UV in production for dependency resolution
• Gunicorn as process manager
• Uvicorn as ASGI worker
• Non-root user for security
• Health check endpoint

Environment Variables:

• All configuration via environment variables
• Secrets via Docker secrets or K8s secrets
• Default to production settings

8.1.2 Gunicorn + Uvicorn Configuration

Parameter	Value	Rationale
Worker Class	uvicorn.workers.UvicornWorker	ASGI support
Workers	(CPU cores * 2) + 1	Optimal concurrency
Worker Connections	1000	Max concurrent connections per worker
Timeout	30s	Prevent hanging requests
Graceful Timeout	30s	Allow ongoing requests to complete
Keep-Alive	5s	Connection reuse
Access Log	stdout	Container log collection
Error Log	stderr	Container log collection

8.2 Docker Compose Deployment

8.2.1 Service Topology

graph TB
    subgraph "Docker Compose Stack"
        APP[kami_spider app]
        MYSQL[(MySQL)]
        REDIS[(Redis)]
        
        APP --> MYSQL
        APP --> REDIS
    end
    
    OTEL_EXT[External OTEL Collector<br/>38.38.251.113:31547]
    APP -.gRPC.-> OTEL_EXT

8.2.2 Service Definitions

Service	Image	Ports	Volumes	Dependencies
app	kami_spider:latest	8000:8000	None (stateless)	mysql, redis
mysql	mysql:8.x	3306:3306	mysql_data	None
redis	redis:latest	6379:6379	redis_data	None

Health Checks:

• App: GET /health every 30s
• MySQL: mysqladmin ping every 10s
• Redis: redis-cli ping every 10s

Restart Policy: unless-stopped

8.3 Docker Swarm Deployment

8.3.1 Swarm Enhancements

Differences from Compose:

• Replicas for app service (e.g., 3 replicas)
• Placement constraints (e.g., avoid same host)
• Update strategy: rolling updates
• Resource limits and reservations

Stack Configuration:

Service	Replicas	Update Strategy	Resources
app	3	Parallelism: 1, Delay: 10s	CPU: 0.5-2, Memory: 512M-2G
mysql	1	N/A (stateful)	CPU: 1-4, Memory: 2G-8G
redis	1	N/A (stateful)	CPU: 0.5-2, Memory: 512M-4G

Network:

• Overlay network for service discovery
• Ingress routing mesh for load balancing

8.4 Kubernetes Deployment

8.4.1 Resource Topology

graph TB
    subgraph "Kubernetes Cluster"
        ING[Ingress] --> SVC[Service: kami-spider]
        SVC --> POD1[Pod 1]
        SVC --> POD2[Pod 2]
        SVC --> POD3[Pod 3]
        
        POD1 --> MYSQL_SVC[MySQL Service]
        POD2 --> MYSQL_SVC
        POD3 --> MYSQL_SVC
        
        POD1 --> REDIS_SVC[Redis Service]
        POD2 --> REDIS_SVC
        POD3 --> REDIS_SVC
        
        MYSQL_SVC --> MYSQL_POD[MySQL StatefulSet]
        REDIS_SVC --> REDIS_POD[Redis StatefulSet]
    end

8.4.2 Kubernetes Resources

Deployment (kami-spider-app):

Aspect	Configuration
Replicas	3 (HPA: 3-10)
Update Strategy	RollingUpdate, maxSurge: 1, maxUnavailable: 0
Resources	Requests: 500m CPU, 512Mi RAM; Limits: 2 CPU, 2Gi RAM
Health Probes	Liveness: /health, Readiness: /health
Environment	ConfigMap + Secret references

StatefulSet (MySQL):

Aspect	Configuration
Replicas	1 (or 3 for HA)
Storage	PersistentVolumeClaim, 20Gi
Service	ClusterIP, headless for StatefulSet

StatefulSet (Redis):

Aspect	Configuration
Replicas	1 (or 3 for cluster mode)
Storage	PersistentVolumeClaim, 5Gi
Service	ClusterIP

ConfigMap:

• Non-sensitive configuration (DB host, Redis host, OTEL endpoint)

Secret:

• Sensitive data (DB password, Redis password)

Service:

• ClusterIP for app (internal)
• LoadBalancer or NodePort (if needed)
• Ingress for HTTP routing

Ingress:

• Path-based routing
• TLS termination (optional)
• Annotations for ingress controller

HorizontalPodAutoscaler:

• Target CPU: 70%
• Target Memory: 80%
• Min replicas: 3, Max: 10

8.5 Database Migration in Deployment

Strategy:

• Run Alembic migrations as init container in K8s
• Run migrations as part of entrypoint script in Docker Compose/Swarm
• Ensure migrations complete before app starts

Rollback Plan:

• Keep migration reversibility (downgrade scripts)
• Database backups before deployment
• Blue-green deployment for zero downtime

9. Additional Recommended Features

9.1 Authentication & Authorization

Recommendation: Implement JWT-based authentication

Rationale:

• Stateless authentication aligns with stateless service goal
• Supports multi-application architecture with shared auth
• Can be made optional per application

Components:

• JWT token generation and validation
• Password hashing (bcrypt/argon2)
• Role-based access control (RBAC)
• OAuth2 integration (optional)

9.2 API Versioning

Recommendation: URL path-based versioning

Strategy:

• Prefix routes with version: /v1/app-a/users
• Each app can have independent versioning
• Deprecation policy for old versions

Benefits:

• Clear version identification
• Backward compatibility support
• Gradual migration path

9.3 Rate Limiting

Recommendation: Redis-based rate limiting

Implementation:

• Sliding window algorithm
• Per-endpoint or per-user limits
• Configurable limits via settings
• Return 429 Too Many Requests with Retry-After header

Use Cases:

• Protect against abuse
• Fair resource allocation
• Cost control for external APIs

9.4 Caching Strategy

Recommendation: Multi-layer caching

Layers:

Layer	Technology	Use Case	TTL
Application	In-memory (lru_cache)	Configuration, static data	App lifetime
Distributed	Redis	Query results, computed data	5m - 1h
HTTP	Response headers	Client-side caching	Varies

Cache Invalidation:

• Time-based expiration
• Event-based invalidation (on write operations)
• Cache-aside pattern

9.5 Background Task Processing

Recommendation: Celery or ARQ for async tasks

Use Cases:

• Email sending
• Report generation
• Data synchronization
• Cleanup jobs

Integration:

• Redis as message broker
• Task result backend in Redis
• Separate worker processes
• Task monitoring and retry logic

9.6 Database Connection Pooling

Recommendation: SQLAlchemy pool configuration

Configuration:

Parameter	Value	Rationale
Pool Size	10 per worker	Balance connection usage
Max Overflow	20	Handle traffic spikes
Pool Recycle	3600s	Prevent stale connections
Pool Pre-ping	True	Validate connections before use
Echo	False (production)	Reduce log noise

9.7 Request/Response Compression

Recommendation: GZip middleware

Configuration:

• Enable for responses > 1KB
• Compression level: 6 (balance speed/size)
• Exclude already-compressed content types

9.8 Security Headers

Recommendation: Add security headers middleware

Headers:

• X-Content-Type-Options: nosniff
• X-Frame-Options: DENY
• X-XSS-Protection: 1; mode=block
• Strict-Transport-Security (if HTTPS)
• Content-Security-Policy (if serving UI)

9.9 Health Check Endpoint

Recommendation: Comprehensive health endpoint

Endpoint: GET /health

Checks:

Component	Check	Healthy Criteria
API	Self	Always true if responding
Database	Connection test	Query executes successfully
Redis	Ping	Returns PONG
OTEL	Export queue	Queue not full/blocked

Response Format:

• 200 OK if all healthy
• 503 Service Unavailable if any unhealthy
• JSON body with component status details

9.10 Metrics Collection

Recommendation: Prometheus metrics

Metrics:

• Request count by endpoint, method, status
• Request duration histogram
• Active request gauge
• Database connection pool stats
• Redis connection pool stats
• Custom business metrics

Export:

• /metrics endpoint for Prometheus scraping
• OpenTelemetry metrics export (optional)

9.11 Environment-Based Configuration

Recommendation: Configuration profiles

Environments:

• Development: Debug on, verbose logging, local DB
• Staging: Production-like, test data
• Production: Optimized, minimal logging, secure

Override Hierarchy:

1. Default values in code
2. .env file
3. Environment variables
4. K8s ConfigMap/Secrets (in K8s)

9.12 API Documentation Enhancements

Recommendation: Rich OpenAPI documentation

Enhancements:

• Add description, summary, tags to all endpoints
• Provide request/response examples in schemas
• Document error responses
• Add security scheme definitions
• Custom OpenAPI metadata (contact, license)

9.13 Database Read Replicas

Recommendation: Support read/write splitting (future)

Strategy:

• Primary for writes
• Replicas for reads
• Routing logic in repository layer
• Configuration for replica endpoints

Benefits:

• Horizontal scaling for read-heavy workloads
• High availability

9.14 Graceful Shutdown

Recommendation: Handle SIGTERM properly

Implementation:

• Register shutdown event handlers
• Stop accepting new requests
• Wait for ongoing requests to complete (up to timeout)
• Close database connections
• Close Redis connections
• Flush telemetry data

9.15 Feature Flags

Recommendation: Feature toggle system

Use Cases:

• Gradual feature rollout
• A/B testing
• Emergency kill switches
• Environment-specific features

Storage: Redis or configuration service

10. Testing Strategy

10.1 Unit Testing

Scope:

• Business logic in services
• Utility functions
• Validation logic
• Repository methods (with mocked DB)

Tools:

• pytest
• pytest-asyncio for async tests
• pytest-cov for coverage reports

Target Coverage: > 80%

10.2 Integration Testing

Scope:

• API endpoints (FastAPI TestClient)
• Database operations (test database)
• Redis operations (test Redis instance)
• Middleware behavior

Strategy:

• Use TestClient with dependency overrides
• Spin up test containers (testcontainers-python)
• Test complete request/response cycle

10.3 Test Data Management

Approach:

• Fixtures for common test data
• Factory pattern for model creation
• Database rollback after each test
• Redis flush after each test

10.4 Mocking Strategy

Mock Targets:

• External API calls
• OTEL exporter (avoid sending test traces)
• Time-dependent functions
• File I/O

Tools:

• pytest-mock
• unittest.mock
• responses library for HTTP mocking