454 lines
12 KiB
Markdown
454 lines
12 KiB
Markdown
# DreamChat Implementation Plan
|
|
|
|
## Overview
|
|
|
|
This document outlines the phased implementation approach for DreamChat.
|
|
|
|
---
|
|
|
|
## Phase 0: Project Setup & Infrastructure
|
|
|
|
**Timeline**: Week 1
|
|
|
|
### Goals
|
|
- Set up pnpm monorepo structure
|
|
- Configure development environment with DevContainer
|
|
- Establish shared packages for WebSocket types
|
|
- Configure CI/CD pipeline
|
|
|
|
### Tasks
|
|
|
|
#### Monorepo Setup
|
|
- [ ] Initialize pnpm workspace
|
|
- [ ] Create root package.json and pnpm-workspace.yaml
|
|
- [ ] Set up shared package (`packages/shared`)
|
|
- [ ] Configure ESLint and Prettier at root
|
|
- [ ] Set up TypeScript project references
|
|
|
|
#### Backend Setup
|
|
- [ ] Initialize NestJS app in `apps/backend`
|
|
- [ ] Configure Prisma ORM with PostgreSQL
|
|
- [ ] Install and configure pgvector extension
|
|
- [ ] Define Prisma schema using multi-file structure (`prisma/models/`)
|
|
- [ ] Configure Jest for unit testing
|
|
- [ ] Set up Swagger/OpenAPI
|
|
- [ ] Implement basic logging (Winston/Pino)
|
|
|
|
#### Frontend Setup
|
|
- [ ] Initialize Vite + React + TypeScript in `apps/frontend`
|
|
- [ ] Configure Tailwind CSS (or similar)
|
|
- [ ] Configure React Query / Zustand
|
|
- [ ] Set up React Router
|
|
- [ ] Configure testing (Vitest + React Testing Library)
|
|
- [ ] Link shared package (`pnpm add @dreamchat/shared@workspace:*`)
|
|
|
|
#### Infrastructure
|
|
- [ ] Create DevContainer configuration (supports pnpm monorepo)
|
|
- [ ] Create Docker Compose for development
|
|
- [ ] Set up GitHub Actions for CI (pnpm caching)
|
|
- [ ] Configure pre-commit hooks
|
|
|
|
#### Shared Types Package
|
|
- [ ] Create `packages/shared` structure
|
|
- [ ] Define WebSocket event types and payloads
|
|
- [ ] Define shared API DTOs
|
|
- [ ] Set up build configuration
|
|
- [ ] Link to apps (`pnpm install` with workspace protocol)
|
|
|
|
### Deliverables
|
|
- [ ] Running development environment via DevContainer
|
|
- [ ] Database connection established
|
|
- [ ] Basic "Hello World" API endpoint
|
|
- [ ] Frontend dev server running
|
|
|
|
---
|
|
|
|
## Phase 1: MVP - Single Character Chat
|
|
|
|
**Timeline**: Weeks 2-5
|
|
|
|
### Goals
|
|
- User authentication (Keycloak + password)
|
|
- Character CRUD with complex attributes
|
|
- Real-time chat via WebSocket
|
|
- Vector-based memory system
|
|
|
|
### 1.1 Authentication Module (Week 2)
|
|
|
|
#### Backend Tasks
|
|
- [ ] User Prisma model and repository
|
|
- [ ] Local authentication strategy (Passport)
|
|
- [ ] JWT token generation and validation
|
|
- [ ] Keycloak strategy implementation
|
|
- [ ] Auth guards and decorators
|
|
- [ ] Password hashing (bcrypt)
|
|
|
|
#### Frontend Tasks
|
|
- [ ] Login page with dual auth options
|
|
- [ ] Keycloak integration
|
|
- [ ] JWT storage and refresh logic
|
|
- [ ] Protected route wrapper
|
|
|
|
#### Testing
|
|
- [ ] Unit tests: AuthService (100% coverage)
|
|
- [ ] Unit tests: JwtStrategy, LocalStrategy
|
|
- [ ] E2E tests: Login flow
|
|
|
|
### 1.2 User Management (Week 2)
|
|
|
|
#### Backend Tasks
|
|
- [ ] User CRUD endpoints
|
|
- [ ] User profile endpoints
|
|
- [ ] Email/username validation
|
|
|
|
#### Frontend Tasks
|
|
- [ ] Registration page
|
|
- [ ] Profile management page
|
|
|
|
#### Testing
|
|
- [ ] Unit tests: UserService
|
|
- [ ] Unit tests: UserController
|
|
|
|
### 1.3 Character Module (Week 3)
|
|
|
|
#### Backend Tasks
|
|
- [ ] Character Prisma model with JSON attributes
|
|
- [ ] Character CRUD endpoints
|
|
- [ ] Character validation DTOs
|
|
- [ ] Attribute schema validation (optional)
|
|
|
|
#### Frontend Tasks
|
|
- [ ] Character list page
|
|
- [ ] Character creation form
|
|
- [ ] Character detail/editor page
|
|
- [ ] Complex attributes editor (JSON editor)
|
|
- [ ] Avatar upload
|
|
|
|
#### Testing
|
|
- [ ] Unit tests: CharacterService
|
|
- [ ] Unit tests: CharacterController
|
|
- [ ] Component tests: Character forms
|
|
|
|
### 1.4 LLM Integration (Week 3)
|
|
|
|
#### Backend Tasks
|
|
- [ ] LLM Provider interface
|
|
- [ ] OpenRouter provider implementation
|
|
- [ ] LLM configuration service
|
|
- [ ] Token counting utilities
|
|
- [ ] Error handling and retry logic
|
|
|
|
#### Testing
|
|
- [ ] Unit tests: OpenRouterProvider (mocked)
|
|
- [ ] Unit tests: LLMService
|
|
|
|
### 1.5 Vector Memory System (Week 4)
|
|
|
|
#### Backend Tasks
|
|
- [ ] Embedding service interface
|
|
- [ ] Local embedding provider (@xenova/transformers)
|
|
- [ ] Dynamic model loading
|
|
- [ ] Model caching strategy
|
|
- [ ] Dimension configuration
|
|
- [ ] HuggingFace API provider (optional/fallback)
|
|
- [ ] Vector store service (pgvector)
|
|
- [ ] LangChain integration
|
|
- [ ] Document chunking utilities
|
|
- [ ] Similarity search implementation
|
|
- [ ] Memory manager service
|
|
|
|
#### Environment Configuration
|
|
```bash
|
|
# Embedding Configuration
|
|
EMBEDDING_PROVIDER=local # or 'huggingface-api'
|
|
EMBEDDING_MODEL=Xenova/all-MiniLM-L6-v2
|
|
EMBEDDING_DIMENSION=384
|
|
EMBEDDING_DEVICE=cpu # or 'gpu' if available
|
|
|
|
# HuggingFace API (if using API instead of local)
|
|
HUGGINGFACE_API_KEY=hf_...
|
|
```
|
|
|
|
#### Testing
|
|
- [ ] Unit tests: EmbeddingService
|
|
- [ ] Unit tests: LocalEmbeddingProvider
|
|
- [ ] Unit tests: VectorStoreService
|
|
- [ ] Unit tests: MemoryManager
|
|
|
|
### 1.6 Chat Module (Week 4-5)
|
|
|
|
#### Backend Tasks
|
|
- [ ] Conversation & Message Prisma models
|
|
- [ ] ChatGateway (WebSocket)
|
|
- [ ] ChatService (business logic)
|
|
- [ ] Message streaming implementation
|
|
- [ ] Context building with memory retrieval
|
|
|
|
#### Frontend Tasks
|
|
- [ ] Chat page layout
|
|
- [ ] Message list component
|
|
- [ ] Message bubble component
|
|
- [ ] Chat input with send button
|
|
- [ ] WebSocket client integration
|
|
- [ ] Streaming message display
|
|
- [ ] Conversation sidebar
|
|
|
|
#### Testing
|
|
- [ ] Unit tests: ChatService
|
|
- [ ] Unit tests: MessageService
|
|
- [ ] Unit tests: ChatGateway
|
|
- [ ] Component tests: Chat components
|
|
- [ ] E2E tests: Full chat flow
|
|
|
|
### 1.7 Import Module - MVP (Week 5)
|
|
|
|
#### Backend Tasks
|
|
- [ ] Import adapter interface
|
|
- [ ] Text file adapter (txt, md)
|
|
- [ ] File upload endpoint
|
|
- [ ] Data preprocessor (cleaning)
|
|
- [ ] Basic text chunking
|
|
- [ ] CharacterKnowledge Prisma model
|
|
- [ ] CharacterKnowledgeService
|
|
- [ ] Import file/URL as knowledge
|
|
- [ ] Chunk and embed content
|
|
- [ ] Store in VectorMemory with type='character'
|
|
|
|
#### Frontend Tasks
|
|
- [ ] Import page
|
|
- [ ] File upload component (drag & drop)
|
|
- [ ] Import progress/status
|
|
|
|
#### Testing
|
|
- [ ] Unit tests: File adapters
|
|
- [ ] Unit tests: DataPreprocessor
|
|
- [ ] Unit tests: ImportController
|
|
|
|
### MVP Deliverables
|
|
- [ ] Users can register/login (Keycloak or password)
|
|
- [ ] Users can create/edit characters with complex attributes
|
|
- [ ] Users can start conversations with characters
|
|
- [ ] Real-time chat with streaming responses
|
|
- [ ] Vector-based memory for context
|
|
- [ ] Text file import for character training data
|
|
- [ ] Unit tests for all backend services
|
|
- [ ] Docker Compose for deployment
|
|
|
|
---
|
|
|
|
## Phase 2: Story Generation
|
|
|
|
**Timeline**: Weeks 6-8
|
|
|
|
### Goals
|
|
- Branching narrative generation
|
|
- Tree view visualization
|
|
- URL-based web scraping for imports
|
|
|
|
### 2.1 Story Module Backend (Week 6)
|
|
|
|
#### Backend Tasks
|
|
- [ ] StoryBranch Prisma model (tree structure with self-relation)
|
|
- [ ] Story generation service
|
|
- [ ] Branching narrative endpoints
|
|
- [ ] Open-ended generation mode
|
|
- [ ] Integration with chat history
|
|
|
|
#### Testing
|
|
- [ ] Unit tests: StoryService
|
|
- [ ] Unit tests: StoryController
|
|
|
|
### 2.2 Story Module Frontend (Week 7)
|
|
|
|
#### Frontend Tasks
|
|
- [ ] Tree view component for branches
|
|
- [ ] Story generation page
|
|
- [ ] Branch creation/editing UI
|
|
- [ ] Direction input for branching
|
|
- [ ] Story navigation controls
|
|
|
|
#### Testing
|
|
- [ ] Component tests: Tree view
|
|
- [ ] Component tests: Story page
|
|
|
|
### 2.3 Import Module - Web Scraping (Week 8)
|
|
|
|
#### Backend Tasks
|
|
- [ ] Web scraper interface
|
|
- [ ] Puppeteer/Playwright integration
|
|
- [ ] Predefined scraper: AO3
|
|
- [ ] Predefined scraper: FanFiction.net
|
|
- [ ] URL validation and scraper selection
|
|
- [ ] URL import endpoint (stores as CharacterKnowledge)
|
|
- [ ] PDF and Markdown support for character knowledge
|
|
|
|
#### Frontend Tasks
|
|
- [ ] URL import component
|
|
- [ ] Scraper status feedback
|
|
- [ ] Content preview before import
|
|
|
|
#### Testing
|
|
- [ ] Unit tests: Web scraper adapters
|
|
- [ ] Integration tests: Scraping (mocked browser)
|
|
|
|
### Phase 2 Deliverables
|
|
- [ ] Branching story generation from conversations
|
|
- [ ] Tree view for navigating story branches
|
|
- [ ] User can input direction per branch
|
|
- [ ] URL import with predefined scrapers
|
|
- [ ] PDF and Markdown file support
|
|
|
|
---
|
|
|
|
## Phase 3: Multi-Character Group Chat
|
|
|
|
**Timeline**: Weeks 9-10
|
|
|
|
### Goals
|
|
- Multiple characters in one conversation
|
|
- Group chat UI
|
|
- Character-to-character interactions
|
|
|
|
### 3.1 Multi-Character Backend (Week 9)
|
|
|
|
#### Backend Tasks
|
|
- [ ] ConversationParticipant Prisma model
|
|
- [ ] Multi-character chat service
|
|
- [ ] Auto-response orchestration
|
|
- [ ] Character addressing detection (@Character)
|
|
- [ ] Group context management
|
|
|
|
#### Testing
|
|
- [ ] Unit tests: MultiCharacterService
|
|
- [ ] Unit tests: Participant management
|
|
|
|
### 3.2 Multi-Character Frontend (Week 10)
|
|
|
|
#### Frontend Tasks
|
|
- [ ] Group chat layout
|
|
- [ ] Participant list sidebar
|
|
- [ ] Character mention support
|
|
- [ ] Message attribution (which character)
|
|
- [ ] Add/remove participants UI
|
|
|
|
#### Testing
|
|
- [ ] Component tests: Group chat
|
|
- [ ] E2E tests: Multi-character flow
|
|
|
|
### Phase 3 Deliverables
|
|
- [ ] Group chat with multiple characters
|
|
- [ ] Characters can address each other
|
|
- [ ] Auto-response per character
|
|
- [ ] Participant management UI
|
|
|
|
---
|
|
|
|
## Phase 4: Polish & Production
|
|
|
|
**Timeline**: Week 11-12
|
|
|
|
### Goals
|
|
- Performance optimization
|
|
- Security hardening
|
|
- Documentation
|
|
- Production deployment
|
|
|
|
### 4.1 Performance & Optimization
|
|
- [ ] Database query optimization
|
|
- [ ] WebSocket connection pooling
|
|
- [ ] Frontend code splitting
|
|
- [ ] Image optimization
|
|
- [ ] Caching layer (Redis optional)
|
|
|
|
### 4.2 Security
|
|
- [ ] Security audit
|
|
- [ ] Input sanitization review
|
|
- [ ] Rate limiting implementation
|
|
- [ ] CORS configuration
|
|
- [ ] Helmet.js integration
|
|
|
|
### 4.3 Documentation
|
|
- [ ] API documentation complete
|
|
- [ ] User guide
|
|
- [ ] Deployment guide
|
|
- [ ] Development guide
|
|
|
|
### 4.4 Production Setup
|
|
- [ ] Production Docker Compose
|
|
- [ ] Environment configuration
|
|
- [ ] SSL/TLS setup
|
|
- [ ] Backup scripts
|
|
- [ ] Monitoring setup (Prometheus/Grafana optional)
|
|
|
|
### Phase 4 Deliverables
|
|
- [ ] Production-ready deployment
|
|
- [ ] Complete documentation
|
|
- [ ] Security audit passed
|
|
- [ ] Performance benchmarks met
|
|
|
|
---
|
|
|
|
## Development Guidelines
|
|
|
|
### Code Quality Standards
|
|
- **Backend Unit Test Coverage**: Minimum 80%
|
|
- **Frontend Component Tests**: Critical paths covered
|
|
- **Code Reviews**: Required for all PRs
|
|
- **Linting**: Zero warnings policy
|
|
- **TypeScript**: Strict mode enabled
|
|
|
|
### Git Workflow
|
|
```
|
|
main (production)
|
|
↓
|
|
develop (integration)
|
|
↓
|
|
feature/XXX (feature branches)
|
|
↓
|
|
PR → Code Review → Merge
|
|
```
|
|
|
|
### Commit Message Convention
|
|
```
|
|
feat: add character creation form
|
|
fix: resolve WebSocket reconnection issue
|
|
docs: update API specification
|
|
test: add unit tests for ChatService
|
|
refactor: simplify memory retrieval logic
|
|
```
|
|
|
|
### Definition of Done
|
|
- [ ] Feature implemented
|
|
- [ ] Unit tests written (backend)
|
|
- [ ] Component tests written (frontend critical paths)
|
|
- [ ] Code reviewed and approved
|
|
- [ ] Documentation updated
|
|
- [ ] No linting errors
|
|
- [ ] CI pipeline passing
|
|
|
|
---
|
|
|
|
## Risk Assessment
|
|
|
|
| Risk | Impact | Mitigation |
|
|
|------|--------|------------|
|
|
| OpenRouter API changes | High | Adapter pattern allows easy provider switch |
|
|
| Web scraping site changes | Medium | Scraper versioning, graceful degradation |
|
|
| Token limits for memory | Medium | Summarization, smart chunking |
|
|
| WebSocket scaling | Medium | Horizontal scaling with Redis adapter (future) |
|
|
| pgvector performance | Low | HNSW indexing, query optimization |
|
|
|
|
---
|
|
|
|
## Post-MVP Roadmap
|
|
|
|
### Future Enhancements
|
|
- [ ] Character templates/marketplace
|
|
- [ ] Voice input/output (TTS/STT)
|
|
- [ ] Export conversations (PDF, EPUB)
|
|
- [ ] Mobile app (React Native)
|
|
- [ ] Plugin system for custom scrapers
|
|
- [ ] AI-powered character attribute suggestions
|
|
- [ ] Collaborative story writing
|
|
- [ ] Public character sharing
|