# DreamChat Implementation Plan

## Overview

This document outlines the phased implementation approach for DreamChat.

---

## Phase 0: Project Setup & Infrastructure

**Timeline**: Week 1

### Goals
- Set up pnpm monorepo structure
- Configure development environment with DevContainer
- Establish shared packages for WebSocket types
- Configure CI/CD pipeline

### Tasks

#### Monorepo Setup
- [ ] Initialize pnpm workspace
- [ ] Create root package.json and pnpm-workspace.yaml
- [ ] Set up shared package (`packages/shared`)
- [ ] Configure ESLint and Prettier at root
- [ ] Set up TypeScript project references

#### Backend Setup
- [ ] Initialize NestJS app in `apps/backend`
- [ ] Configure Prisma ORM with PostgreSQL
- [ ] Install and configure pgvector extension
- [ ] Define Prisma schema using multi-file structure (`prisma/models/`)
- [ ] Configure Jest for unit testing
- [ ] Set up Swagger/OpenAPI
- [ ] Implement basic logging (Winston/Pino)

#### Frontend Setup
- [ ] Initialize Vite + React + TypeScript in `apps/frontend`
- [ ] Configure Tailwind CSS (or similar)
- [ ] Configure React Query / Zustand
- [ ] Set up React Router
- [ ] Configure testing (Vitest + React Testing Library)
- [ ] Link shared package (`pnpm add @dreamchat/shared@workspace:*`)

#### Infrastructure
- [ ] Create DevContainer configuration (supports pnpm monorepo)
- [ ] Create Docker Compose for development
- [ ] Set up GitHub Actions for CI (pnpm caching)
- [ ] Configure pre-commit hooks

#### Shared Types Package
- [ ] Create `packages/shared` structure
- [ ] Define WebSocket event types and payloads
- [ ] Define shared API DTOs
- [ ] Set up build configuration
- [ ] Link to apps (`pnpm install` with workspace protocol)

### Deliverables
- [ ] Running development environment via DevContainer
- [ ] Database connection established
- [ ] Basic "Hello World" API endpoint
- [ ] Frontend dev server running

---

## Phase 1: MVP - Single Character Chat

**Timeline**: Weeks 2-5

### Goals
- User authentication (Keycloak + password)
- Character CRUD with complex attributes
- Real-time chat via WebSocket
- Vector-based memory system

### 1.1 Authentication Module (Week 2)

#### Backend Tasks
- [ ] User Prisma model and repository
- [ ] Local authentication strategy (Passport)
- [ ] JWT token generation and validation
- [ ] Keycloak strategy implementation
- [ ] Auth guards and decorators
- [ ] Password hashing (bcrypt)

#### Frontend Tasks
- [ ] Login page with dual auth options
- [ ] Keycloak integration
- [ ] JWT storage and refresh logic
- [ ] Protected route wrapper

#### Testing
- [ ] Unit tests: AuthService (100% coverage)
- [ ] Unit tests: JwtStrategy, LocalStrategy
- [ ] E2E tests: Login flow

### 1.2 User Management (Week 2)

#### Backend Tasks
- [ ] User CRUD endpoints
- [ ] User profile endpoints
- [ ] Email/username validation

#### Frontend Tasks
- [ ] Registration page
- [ ] Profile management page

#### Testing
- [ ] Unit tests: UserService
- [ ] Unit tests: UserController

### 1.3 Character Module (Week 3)

#### Backend Tasks
- [ ] Character Prisma model with JSON attributes
- [ ] Character CRUD endpoints
- [ ] Character validation DTOs
- [ ] Attribute schema validation (optional)

#### Frontend Tasks
- [ ] Character list page
- [ ] Character creation form
- [ ] Character detail/editor page
- [ ] Complex attributes editor (JSON editor)
- [ ] Avatar upload

#### Testing
- [ ] Unit tests: CharacterService
- [ ] Unit tests: CharacterController
- [ ] Component tests: Character forms

### 1.4 LLM Integration (Week 3)

#### Backend Tasks
- [ ] LLM Provider interface
- [ ] OpenRouter provider implementation
- [ ] LLM configuration service
- [ ] Token counting utilities
- [ ] Error handling and retry logic

#### Testing
- [ ] Unit tests: OpenRouterProvider (mocked)
- [ ] Unit tests: LLMService

### 1.5 Vector Memory System (Week 4)

#### Backend Tasks
- [ ] Embedding service interface
- [ ] Local embedding provider (@xenova/transformers)
  - [ ] Dynamic model loading
  - [ ] Model caching strategy
  - [ ] Dimension configuration
- [ ] HuggingFace API provider (optional/fallback)
- [ ] Vector store service (pgvector)
- [ ] LangChain integration
- [ ] Document chunking utilities
- [ ] Similarity search implementation
- [ ] Memory manager service

#### Environment Configuration
```bash
# Embedding Configuration
EMBEDDING_PROVIDER=local  # or 'huggingface-api'
EMBEDDING_MODEL=Xenova/all-MiniLM-L6-v2
EMBEDDING_DIMENSION=384
EMBEDDING_DEVICE=cpu  # or 'gpu' if available

# HuggingFace API (if using API instead of local)
HUGGINGFACE_API_KEY=hf_...
```

#### Testing
- [ ] Unit tests: EmbeddingService
- [ ] Unit tests: LocalEmbeddingProvider
- [ ] Unit tests: VectorStoreService
- [ ] Unit tests: MemoryManager

### 1.6 Chat Module (Week 4-5)

#### Backend Tasks
- [ ] Conversation & Message Prisma models
- [ ] ChatGateway (WebSocket)
- [ ] ChatService (business logic)
- [ ] Message streaming implementation
- [ ] Context building with memory retrieval

#### Frontend Tasks
- [ ] Chat page layout
- [ ] Message list component
- [ ] Message bubble component
- [ ] Chat input with send button
- [ ] WebSocket client integration
- [ ] Streaming message display
- [ ] Conversation sidebar

#### Testing
- [ ] Unit tests: ChatService
- [ ] Unit tests: MessageService
- [ ] Unit tests: ChatGateway
- [ ] Component tests: Chat components
- [ ] E2E tests: Full chat flow

### 1.7 Import Module - MVP (Week 5)

#### Backend Tasks
- [ ] Import adapter interface
- [ ] Text file adapter (txt, md)
- [ ] File upload endpoint
- [ ] Data preprocessor (cleaning)
- [ ] Basic text chunking
- [ ] CharacterKnowledge Prisma model
- [ ] CharacterKnowledgeService
  - [ ] Import file/URL as knowledge
  - [ ] Chunk and embed content
  - [ ] Store in VectorMemory with type='character'

#### Frontend Tasks
- [ ] Import page
- [ ] File upload component (drag & drop)
- [ ] Import progress/status

#### Testing
- [ ] Unit tests: File adapters
- [ ] Unit tests: DataPreprocessor
- [ ] Unit tests: ImportController

### MVP Deliverables
- [ ] Users can register/login (Keycloak or password)
- [ ] Users can create/edit characters with complex attributes
- [ ] Users can start conversations with characters
- [ ] Real-time chat with streaming responses
- [ ] Vector-based memory for context
- [ ] Text file import for character training data
- [ ] Unit tests for all backend services
- [ ] Docker Compose for deployment

---

## Phase 2: Story Generation

**Timeline**: Weeks 6-8

### Goals
- Branching narrative generation
- Tree view visualization
- URL-based web scraping for imports

### 2.1 Story Module Backend (Week 6)

#### Backend Tasks
- [ ] StoryBranch Prisma model (tree structure with self-relation)
- [ ] Story generation service
- [ ] Branching narrative endpoints
- [ ] Open-ended generation mode
- [ ] Integration with chat history

#### Testing
- [ ] Unit tests: StoryService
- [ ] Unit tests: StoryController

### 2.2 Story Module Frontend (Week 7)

#### Frontend Tasks
- [ ] Tree view component for branches
- [ ] Story generation page
- [ ] Branch creation/editing UI
- [ ] Direction input for branching
- [ ] Story navigation controls

#### Testing
- [ ] Component tests: Tree view
- [ ] Component tests: Story page

### 2.3 Import Module - Web Scraping (Week 8)

#### Backend Tasks
- [ ] Web scraper interface
- [ ] Puppeteer/Playwright integration
- [ ] Predefined scraper: AO3
- [ ] Predefined scraper: FanFiction.net
- [ ] URL validation and scraper selection
- [ ] URL import endpoint (stores as CharacterKnowledge)
- [ ] PDF and Markdown support for character knowledge

#### Frontend Tasks
- [ ] URL import component
- [ ] Scraper status feedback
- [ ] Content preview before import

#### Testing
- [ ] Unit tests: Web scraper adapters
- [ ] Integration tests: Scraping (mocked browser)

### Phase 2 Deliverables
- [ ] Branching story generation from conversations
- [ ] Tree view for navigating story branches
- [ ] User can input direction per branch
- [ ] URL import with predefined scrapers
- [ ] PDF and Markdown file support

---

## Phase 3: Multi-Character Group Chat

**Timeline**: Weeks 9-10

### Goals
- Multiple characters in one conversation
- Group chat UI
- Character-to-character interactions

### 3.1 Multi-Character Backend (Week 9)

#### Backend Tasks
- [ ] ConversationParticipant Prisma model
- [ ] Multi-character chat service
- [ ] Auto-response orchestration
- [ ] Character addressing detection (@Character)
- [ ] Group context management

#### Testing
- [ ] Unit tests: MultiCharacterService
- [ ] Unit tests: Participant management

### 3.2 Multi-Character Frontend (Week 10)

#### Frontend Tasks
- [ ] Group chat layout
- [ ] Participant list sidebar
- [ ] Character mention support
- [ ] Message attribution (which character)
- [ ] Add/remove participants UI

#### Testing
- [ ] Component tests: Group chat
- [ ] E2E tests: Multi-character flow

### Phase 3 Deliverables
- [ ] Group chat with multiple characters
- [ ] Characters can address each other
- [ ] Auto-response per character
- [ ] Participant management UI

---

## Phase 4: Polish & Production

**Timeline**: Week 11-12

### Goals
- Performance optimization
- Security hardening
- Documentation
- Production deployment

### 4.1 Performance & Optimization
- [ ] Database query optimization
- [ ] WebSocket connection pooling
- [ ] Frontend code splitting
- [ ] Image optimization
- [ ] Caching layer (Redis optional)

### 4.2 Security
- [ ] Security audit
- [ ] Input sanitization review
- [ ] Rate limiting implementation
- [ ] CORS configuration
- [ ] Helmet.js integration

### 4.3 Documentation
- [ ] API documentation complete
- [ ] User guide
- [ ] Deployment guide
- [ ] Development guide

### 4.4 Production Setup
- [ ] Production Docker Compose
- [ ] Environment configuration
- [ ] SSL/TLS setup
- [ ] Backup scripts
- [ ] Monitoring setup (Prometheus/Grafana optional)

### Phase 4 Deliverables
- [ ] Production-ready deployment
- [ ] Complete documentation
- [ ] Security audit passed
- [ ] Performance benchmarks met

---

## Development Guidelines

### Code Quality Standards
- **Backend Unit Test Coverage**: Minimum 80%
- **Frontend Component Tests**: Critical paths covered
- **Code Reviews**: Required for all PRs
- **Linting**: Zero warnings policy
- **TypeScript**: Strict mode enabled

### Git Workflow
```
main (production)
  ↓
develop (integration)
  ↓
feature/XXX (feature branches)
  ↓
PR → Code Review → Merge
```

### Commit Message Convention
```
feat: add character creation form
fix: resolve WebSocket reconnection issue
docs: update API specification
test: add unit tests for ChatService
refactor: simplify memory retrieval logic
```

### Definition of Done
- [ ] Feature implemented
- [ ] Unit tests written (backend)
- [ ] Component tests written (frontend critical paths)
- [ ] Code reviewed and approved
- [ ] Documentation updated
- [ ] No linting errors
- [ ] CI pipeline passing

---

## Risk Assessment

| Risk | Impact | Mitigation |
|------|--------|------------|
| OpenRouter API changes | High | Adapter pattern allows easy provider switch |
| Web scraping site changes | Medium | Scraper versioning, graceful degradation |
| Token limits for memory | Medium | Summarization, smart chunking |
| WebSocket scaling | Medium | Horizontal scaling with Redis adapter (future) |
| pgvector performance | Low | HNSW indexing, query optimization |

---

## Post-MVP Roadmap

### Future Enhancements
- [ ] Character templates/marketplace
- [ ] Voice input/output (TTS/STT)
- [ ] Export conversations (PDF, EPUB)
- [ ] Mobile app (React Native)
- [ ] Plugin system for custom scrapers
- [ ] AI-powered character attribute suggestions
- [ ] Collaborative story writing
- [ ] Public character sharing