GW_MC/DreamChat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
932f384f0d77962deecb4920510749e74d8c4d62
DreamChat/doc/implementation-plan.md

12 KiB
Raw Blame History

DreamChat Implementation Plan

Overview

This document outlines the phased implementation approach for DreamChat.

Phase 0: Project Setup & Infrastructure

Timeline: Week 1

Goals

  • Set up pnpm monorepo structure
  • Configure development environment with DevContainer
  • Establish shared packages for WebSocket types
  • Configure CI/CD pipeline

Tasks

Monorepo Setup

  • Initialize pnpm workspace
  • Create root package.json and pnpm-workspace.yaml
  • Set up shared package (packages/shared)
  • Configure ESLint and Prettier at root
  • Set up TypeScript project references

Backend Setup

  • Initialize NestJS app in apps/backend
  • Configure Prisma ORM with PostgreSQL
  • Install and configure pgvector extension
  • Define Prisma schema using multi-file structure (prisma/models/)
  • Configure Jest for unit testing
  • Set up Swagger/OpenAPI
  • Implement basic logging (Winston/Pino)

Frontend Setup

  • Initialize Vite + React + TypeScript in apps/frontend
  • Configure Tailwind CSS (or similar)
  • Configure React Query / Zustand
  • Set up React Router
  • Configure testing (Vitest + React Testing Library)
  • Link shared package (pnpm add @dreamchat/shared@workspace:*)

Infrastructure

  • Create DevContainer configuration (supports pnpm monorepo)
  • Create Docker Compose for development
  • Set up GitHub Actions for CI (pnpm caching)
  • Configure pre-commit hooks

Shared Types Package

  • Create packages/shared structure
  • Define WebSocket event types and payloads
  • Define shared API DTOs
  • Set up build configuration
  • Link to apps (pnpm install with workspace protocol)

Deliverables

  • Running development environment via DevContainer
  • Database connection established
  • Basic "Hello World" API endpoint
  • Frontend dev server running

Phase 1: MVP - Single Character Chat

Timeline: Weeks 2-5

Goals

  • User authentication (Keycloak + password)
  • Character CRUD with complex attributes
  • Real-time chat via WebSocket
  • Vector-based memory system

1.1 Authentication Module (Week 2)

Backend Tasks

  • User Prisma model and repository
  • Local authentication strategy (Passport)
  • JWT token generation and validation
  • Keycloak strategy implementation
  • Auth guards and decorators
  • Password hashing (bcrypt)

Frontend Tasks

  • Login page with dual auth options
  • Keycloak integration
  • JWT storage and refresh logic
  • Protected route wrapper

Testing

  • Unit tests: AuthService (100% coverage)
  • Unit tests: JwtStrategy, LocalStrategy
  • E2E tests: Login flow

1.2 User Management (Week 2)

Backend Tasks

  • User CRUD endpoints
  • User profile endpoints
  • Email/username validation

Frontend Tasks

  • Registration page
  • Profile management page

Testing

  • Unit tests: UserService
  • Unit tests: UserController

1.3 Character Module (Week 3)

Backend Tasks

  • Character Prisma model with JSON attributes
  • Character CRUD endpoints
  • Character validation DTOs
  • Attribute schema validation (optional)

Frontend Tasks

  • Character list page
  • Character creation form
  • Character detail/editor page
  • Complex attributes editor (JSON editor)
  • Avatar upload

Testing

  • Unit tests: CharacterService
  • Unit tests: CharacterController
  • Component tests: Character forms

1.4 LLM Integration (Week 3)

Backend Tasks

  • LLM Provider interface
  • OpenRouter provider implementation
  • LLM configuration service
  • Token counting utilities
  • Error handling and retry logic

Testing

  • Unit tests: OpenRouterProvider (mocked)
  • Unit tests: LLMService

1.5 Vector Memory System (Week 4)

Backend Tasks

  • Embedding service interface
  • Local embedding provider (@xenova/transformers)
    • Dynamic model loading
    • Model caching strategy
    • Dimension configuration
  • HuggingFace API provider (optional/fallback)
  • Vector store service (pgvector)
  • LangChain integration
  • Document chunking utilities
  • Similarity search implementation
  • Memory manager service

Environment Configuration

# Embedding Configuration
EMBEDDING_PROVIDER=local  # or 'huggingface-api'
EMBEDDING_MODEL=Xenova/all-MiniLM-L6-v2
EMBEDDING_DIMENSION=384
EMBEDDING_DEVICE=cpu  # or 'gpu' if available

# HuggingFace API (if using API instead of local)
HUGGINGFACE_API_KEY=hf_...

Testing

  • Unit tests: EmbeddingService
  • Unit tests: LocalEmbeddingProvider
  • Unit tests: VectorStoreService
  • Unit tests: MemoryManager

1.6 Chat Module (Week 4-5)

Backend Tasks

  • Conversation & Message Prisma models
  • ChatGateway (WebSocket)
  • ChatService (business logic)
  • Message streaming implementation
  • Context building with memory retrieval

Frontend Tasks

  • Chat page layout
  • Message list component
  • Message bubble component
  • Chat input with send button
  • WebSocket client integration
  • Streaming message display
  • Conversation sidebar

Testing

  • Unit tests: ChatService
  • Unit tests: MessageService
  • Unit tests: ChatGateway
  • Component tests: Chat components
  • E2E tests: Full chat flow

1.7 Import Module - MVP (Week 5)

Backend Tasks

  • Import adapter interface
  • Text file adapter (txt, md)
  • File upload endpoint
  • Data preprocessor (cleaning)
  • Basic text chunking
  • CharacterKnowledge Prisma model
  • CharacterKnowledgeService
    • Import file/URL as knowledge
    • Chunk and embed content
    • Store in VectorMemory with type='character'

Frontend Tasks

  • Import page
  • File upload component (drag & drop)
  • Import progress/status

Testing

  • Unit tests: File adapters
  • Unit tests: DataPreprocessor
  • Unit tests: ImportController

MVP Deliverables

  • Users can register/login (Keycloak or password)
  • Users can create/edit characters with complex attributes
  • Users can start conversations with characters
  • Real-time chat with streaming responses
  • Vector-based memory for context
  • Text file import for character training data
  • Unit tests for all backend services
  • Docker Compose for deployment

Phase 2: Story Generation

Timeline: Weeks 6-8

Goals

  • Branching narrative generation
  • Tree view visualization
  • URL-based web scraping for imports

2.1 Story Module Backend (Week 6)

Backend Tasks

  • StoryBranch Prisma model (tree structure with self-relation)
  • Story generation service
  • Branching narrative endpoints
  • Open-ended generation mode
  • Integration with chat history

Testing

  • Unit tests: StoryService
  • Unit tests: StoryController

2.2 Story Module Frontend (Week 7)

Frontend Tasks

  • Tree view component for branches
  • Story generation page
  • Branch creation/editing UI
  • Direction input for branching
  • Story navigation controls

Testing

  • Component tests: Tree view
  • Component tests: Story page

2.3 Import Module - Web Scraping (Week 8)

Backend Tasks

  • Web scraper interface
  • Puppeteer/Playwright integration
  • Predefined scraper: AO3
  • Predefined scraper: FanFiction.net
  • URL validation and scraper selection
  • URL import endpoint (stores as CharacterKnowledge)
  • PDF and Markdown support for character knowledge

Frontend Tasks

  • URL import component
  • Scraper status feedback
  • Content preview before import

Testing

  • Unit tests: Web scraper adapters
  • Integration tests: Scraping (mocked browser)

Phase 2 Deliverables

  • Branching story generation from conversations
  • Tree view for navigating story branches
  • User can input direction per branch
  • URL import with predefined scrapers
  • PDF and Markdown file support

Phase 3: Multi-Character Group Chat

Timeline: Weeks 9-10

Goals

  • Multiple characters in one conversation
  • Group chat UI
  • Character-to-character interactions

3.1 Multi-Character Backend (Week 9)

Backend Tasks

  • ConversationParticipant Prisma model
  • Multi-character chat service
  • Auto-response orchestration
  • Character addressing detection (@Character)
  • Group context management

Testing

  • Unit tests: MultiCharacterService
  • Unit tests: Participant management

3.2 Multi-Character Frontend (Week 10)

Frontend Tasks

  • Group chat layout
  • Participant list sidebar
  • Character mention support
  • Message attribution (which character)
  • Add/remove participants UI

Testing

  • Component tests: Group chat
  • E2E tests: Multi-character flow

Phase 3 Deliverables

  • Group chat with multiple characters
  • Characters can address each other
  • Auto-response per character
  • Participant management UI

Phase 4: Polish & Production

Timeline: Week 11-12

Goals

  • Performance optimization
  • Security hardening
  • Documentation
  • Production deployment

4.1 Performance & Optimization

  • Database query optimization
  • WebSocket connection pooling
  • Frontend code splitting
  • Image optimization
  • Caching layer (Redis optional)

4.2 Security

  • Security audit
  • Input sanitization review
  • Rate limiting implementation
  • CORS configuration
  • Helmet.js integration

4.3 Documentation

  • API documentation complete
  • User guide
  • Deployment guide
  • Development guide

4.4 Production Setup

  • Production Docker Compose
  • Environment configuration
  • SSL/TLS setup
  • Backup scripts
  • Monitoring setup (Prometheus/Grafana optional)

Phase 4 Deliverables

  • Production-ready deployment
  • Complete documentation
  • Security audit passed
  • Performance benchmarks met

Development Guidelines

Code Quality Standards

  • Backend Unit Test Coverage: Minimum 80%
  • Frontend Component Tests: Critical paths covered
  • Code Reviews: Required for all PRs
  • Linting: Zero warnings policy
  • TypeScript: Strict mode enabled

Git Workflow

main (production)
  ↓
develop (integration)
  ↓
feature/XXX (feature branches)
  ↓
PR → Code Review → Merge

Commit Message Convention

feat: add character creation form
fix: resolve WebSocket reconnection issue
docs: update API specification
test: add unit tests for ChatService
refactor: simplify memory retrieval logic

Definition of Done

  • Feature implemented
  • Unit tests written (backend)
  • Component tests written (frontend critical paths)
  • Code reviewed and approved
  • Documentation updated
  • No linting errors
  • CI pipeline passing

Risk Assessment

Risk Impact Mitigation
OpenRouter API changes High Adapter pattern allows easy provider switch
Web scraping site changes Medium Scraper versioning, graceful degradation
Token limits for memory Medium Summarization, smart chunking
WebSocket scaling Medium Horizontal scaling with Redis adapter (future)
pgvector performance Low HNSW indexing, query optimization

Post-MVP Roadmap

Future Enhancements

  • Character templates/marketplace
  • Voice input/output (TTS/STT)
  • Export conversations (PDF, EPUB)
  • Mobile app (React Native)
  • Plugin system for custom scrapers
  • AI-powered character attribute suggestions
  • Collaborative story writing
  • Public character sharing
Reference in New Issue View Git Blame Copy Permalink