GW_MC/DreamChat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add implementation plan and monorepo guide for DreamChat project

Browse Source
This commit is contained in:
GW_MC
2026-02-23 19:14:45 +08:00
parent fbd5c84eca
commit ab02758382
8 changed files with 4286 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
doc/README.md Normal file
View File

@@ -0,0 +1,44 @@
# DreamChat Project Documentation
This directory contains comprehensive planning and implementation documentation for the DreamChat project.
## Documentation Structure
| Document | Description |
|----------|-------------|
| [architecture.md](./architecture.md) | System architecture, tech stack, component diagrams |
| [monorepo-guide.md](./monorepo-guide.md) | pnpm workspace setup, shared packages |
| [database-schema.md](./database-schema.md) | PostgreSQL schema, vector store design, entities |
| [api-spec.md](./api-spec.md) | REST API & WebSocket specifications, OpenAPI |
| [implementation-plan.md](./implementation-plan.md) | Phased roadmap, milestones, deliverables |
| [frontend-guide.md](./frontend-guide.md) | Frontend architecture, component hierarchy |
| [deployment.md](./deployment.md) | Docker Compose, devcontainer, deployment guide |
## Quick Reference
### Tech Stack
- **Backend**: NestJS (TypeScript)
- **Frontend**: React + Vite (TypeScript)
- **Package Manager**: pnpm (monorepo workspaces)
- **Database**: PostgreSQL with pgvector
- **ORM**: Prisma
- **Vector Store**: PostgreSQL (pgvector) + LangChain + Local HuggingFace Embeddings
- **LLM**: OpenRouter (flexible adapter pattern)
- **Real-time**: WebSocket (shared types package)
- **Auth**: Keycloak + Password-based
- **Web Scraping**: Puppeteer/Playwright (headless browser)
### Implementation Phases
1. **MVP**: Single character chat with memory
2. **Phase 2**: Story generation with branching tree view
3. **Phase 3**: Multi-character group chat
### Key Design Patterns
- **Adapter Pattern**: LLM providers, data import sources
- **Repository Pattern**: Database access
- **Strategy Pattern**: Predefined web scrapers
- **Observer Pattern**: WebSocket events
---
*Last updated: 2026-02-23*

572
doc/api-spec.md Normal file
View File

@@ -0,0 +1,572 @@
# DreamChat API Specification
## Overview
This document defines the REST API and WebSocket specifications for DreamChat.
- **REST API**: For CRUD operations, file uploads, and synchronous requests
- **WebSocket**: For real-time chat streaming
- **OpenAPI**: Auto-generated from NestJS decorators for frontend client generation
## Base URL
```
Development: http://localhost:3000/api
Production: https://api.dreamchat.example/api
```
## Authentication
### JWT Token Flow
```http
POST /api/auth/login
Content-Type: application/json
{
"username": "user@example.com",
"password": "securepassword"
}
Response:
{
"accessToken": "eyJhbGciOiJIUzI1NiIs...",
"refreshToken": "eyJhbGciOiJIUzI1NiIs...",
"expiresIn": 3600
}
```
### Keycloak Integration
```http
GET /api/auth/keycloak
Redirect to Keycloak login
Callback:
GET /api/auth/keycloak/callback?code=...
Response: { accessToken, refreshToken, expiresIn }
```
### WebSocket Auth
WebSocket connections authenticate via query parameter:
```
ws://localhost:3000/chat?token=eyJhbGciOiJIUzI1NiIs...
```
## REST API Endpoints
### Authentication Module
| Method | Endpoint | Description | Auth |
|--------|----------|-------------|------|
| POST | `/auth/login` | Local login | Public |
| POST | `/auth/refresh` | Refresh token | Public |
| POST | `/auth/logout` | Logout | Bearer |
| GET | `/auth/me` | Get current user | Bearer |
| GET | `/auth/keycloak` | Keycloak login URL | Public |
| GET | `/auth/keycloak/callback` | Keycloak callback | Public |
### Users Module
| Method | Endpoint | Description | Auth |
|--------|----------|-------------|------|
| POST | `/users` | Register new user | Public |
| GET | `/users/:id` | Get user profile | Bearer |
| PATCH | `/users/:id` | Update user | Bearer (own only) |
| DELETE | `/users/:id` | Delete user | Bearer (own/admin) |
### Characters Module
```typescript
// DTOs
class CreateCharacterDto {
name: string;
personalityPrompt: string;
backstory?: string;
attributes?: Record<string, any>;
avatarUrl?: string;
config?: Record<string, any>;
}
class CharacterResponseDto {
id: string;
name: string;
avatarUrl: string;
personalityPrompt: string;
backstory: string;
attributes: Record<string, any>;
createdAt: Date;
}
```
| Method | Endpoint | Description | Auth |
|--------|----------|-------------|------|
| GET | `/characters` | List user's characters | Bearer |
| POST | `/characters` | Create character | Bearer |
| GET | `/characters/:id` | Get character details | Bearer |
| PATCH | `/characters/:id` | Update character | Bearer (owner) |
| DELETE | `/characters/:id` | Delete character | Bearer (owner) |
**Example Requests:**
```http
POST /api/characters
Authorization: Bearer {token}
Content-Type: application/json
{
"name": "Alice",
"personalityPrompt": "You are Alice, a curious and adventurous explorer...",
"backstory": "Alice grew up in a small village...",
"attributes": {
"traits": ["curious", "brave", "witty"],
"age": 25,
"species": "human",
"skills": ["navigation", "survival"]
}
}
Response: 201 Created
{
"id": "uuid",
"name": "Alice",
"personalityPrompt": "...",
"attributes": { ... },
"createdAt": "2026-02-23T10:00:00Z"
}
```
### Conversations Module
```typescript
class CreateConversationDto {
characterId: string;
title?: string;
}
class ConversationResponseDto {
id: string;
title: string;
character: CharacterSummaryDto;
messageCount: number;
createdAt: Date;
}
```
| Method | Endpoint | Description | Auth |
|--------|----------|-------------|------|
| GET | `/conversations` | List conversations | Bearer |
| POST | `/conversations` | Create conversation | Bearer |
| GET | `/conversations/:id` | Get conversation | Bearer |
| PATCH | `/conversations/:id` | Update conversation | Bearer |
| DELETE | `/conversations/:id` | Delete conversation | Bearer |
### Messages Module
```typescript
class CreateMessageDto {
content: string;
}
class MessageResponseDto {
id: string;
role: 'user' | 'assistant';
content: string;
tokensUsed?: number;
model?: string;
createdAt: Date;
}
```
| Method | Endpoint | Description | Auth |
|--------|----------|-------------|------|
| GET | `/conversations/:id/messages` | Get messages (paginated) | Bearer |
| POST | `/conversations/:id/messages` | Send message (non-streaming) | Bearer |
| DELETE | `/messages/:id` | Delete message | Bearer |
### Import Module
```typescript
class FileImportDto {
// Multipart form data
file: File;
}
class UrlImportDto {
url: string;
}
class ImportResponseDto {
id: string;
status: 'pending' | 'processing' | 'completed' | 'failed';
sourceName: string;
fileSize?: number;
content?: string;
errorMessage?: string;
createdAt: Date;
}
```
| Method | Endpoint | Description | Auth |
|--------|----------|-------------|------|
| POST | `/import/file` | Import file (txt, pdf, md) | Bearer |
| POST | `/import/url` | Import from URL | Bearer |
| GET | `/import/:id` | Get import status | Bearer |
| GET | `/import` | List imports | Bearer |
| DELETE | `/import/:id` | Delete import | Bearer |
**File Upload Request:**
```http
POST /api/import/file
Authorization: Bearer {token}
Content-Type: multipart/form-data
------Boundary
Content-Disposition: form-data; name="file"; filename="story.txt"
Content-Type: text/plain
(file content here)
------Boundary--
Response: 202 Accepted
{
"id": "import-uuid",
"status": "processing",
"sourceName": "story.txt",
"createdAt": "2026-02-23T10:00:00Z"
}
```
**URL Import Request:**
```http
POST /api/import/url
Authorization: Bearer {token}
Content-Type: application/json
{
"url": "https://archiveofourown.org/works/12345678"
}
Response: 202 Accepted
{
"id": "import-uuid",
"status": "processing",
"sourceName": "https://archiveofourown.org/works/12345678"
}
// If no scraper available:
Response: 400 Bad Request
{
"error": "UNSUPPORTED_URL",
"message": "No scraper available for this URL"
}
```
### Story Module (Phase 2)
```typescript
class CreateStoryBranchDto {
parentId?: string; // null for root
userDirection: string;
}
class StoryBranchResponseDto {
id: string;
conversationId: string;
parentId?: string;
title: string;
content: string;
userDirection: string;
depth: number;
children: StoryBranchResponseDto[];
createdAt: Date;
}
```
| Method | Endpoint | Description | Auth |
|--------|----------|-------------|------|
| GET | `/conversations/:id/story` | Get story tree | Bearer |
| POST | `/conversations/:id/story/branches` | Create new branch | Bearer |
| GET | `/story-branches/:id` | Get branch details | Bearer |
| PATCH | `/story-branches/:id` | Update branch | Bearer |
| DELETE | `/story-branches/:id` | Delete branch | Bearer |
**Get Story Tree:**
```http
GET /api/conversations/conv-id/story
Authorization: Bearer {token}
Response:
{
"root": {
"id": "root-uuid",
"title": "The Beginning",
"content": "Once upon a time...",
"depth": 0,
"children": [
{
"id": "branch-1",
"title": "The Left Path",
"content": "You chose the left path...",
"userDirection": "Go left into the dark forest",
"depth": 1,
"children": []
},
{
"id": "branch-2",
"title": "The Right Path",
"content": "You chose the right path...",
"userDirection": "Go right towards the castle",
"depth": 1,
"children": []
}
]
}
}
```
### Multi-Character Module (Phase 3)
```typescript
class AddParticipantDto {
characterId: string;
autoRespond: boolean;
}
class ParticipantResponseDto {
id: string;
character: CharacterSummaryDto;
isActive: boolean;
autoRespond: boolean;
}
```
| Method | Endpoint | Description | Auth |
|--------|----------|-------------|------|
| GET | `/conversations/:id/participants` | List participants | Bearer |
| POST | `/conversations/:id/participants` | Add participant | Bearer |
| DELETE | `/conversations/:id/participants/:charId` | Remove participant | Bearer |
## WebSocket Specification
### Connection
```javascript
const ws = new WebSocket('ws://localhost:3000/chat?token=JWT_TOKEN');
ws.onopen = () => {
console.log('Connected');
};
ws.onmessage = (event) => {
const message = JSON.parse(event.data);
handleMessage(message);
};
```
### Message Protocol
All WebSocket messages use JSON format:
```typescript
interface WebSocketMessage {
type: string;
payload: any;
timestamp: string;
requestId?: string; // For correlating responses
}
```
### Client → Server Events
#### 1. Join Conversation
```typescript
// Join a conversation room
{
"type": "JOIN_CONVERSATION",
"payload": {
"conversationId": "conv-uuid"
},
"requestId": "req-123"
}
// Response
{
"type": "CONVERSATION_JOINED",
"payload": {
"conversationId": "conv-uuid",
"history": [ /* last N messages */ ]
},
"requestId": "req-123"
}
```
#### 2. Send Message (Streaming)
```typescript
{
"type": "SEND_MESSAGE",
"payload": {
"conversationId": "conv-uuid",
"content": "Hello, how are you?",
"streaming": true // Enable streaming response
},
"requestId": "msg-456"
}
```
#### 3. Stop Generation
```typescript
{
"type": "STOP_GENERATION",
"payload": {
"conversationId": "conv-uuid"
}
}
```
#### 4. Leave Conversation
```typescript
{
"type": "LEAVE_CONVERSATION",
"payload": {
"conversationId": "conv-uuid"
}
}
```
### Server → Client Events
#### 1. Message Acknowledged
```typescript
{
"type": "MESSAGE_ACK",
"payload": {
"messageId": "msg-uuid",
"status": "received"
},
"requestId": "msg-456"
}
```
#### 2. Stream Chunk
```typescript
{
"type": "STREAM_CHUNK",
"payload": {
"conversationId": "conv-uuid",
"chunk": " part of response",
"isComplete": false
},
"requestId": "msg-456"
}
```
#### 3. Stream Complete
```typescript
{
"type": "STREAM_COMPLETE",
"payload": {
"conversationId": "conv-uuid",
"messageId": "assistant-msg-uuid",
"content": "Full response text",
"tokensUsed": 150,
"model": "openai/gpt-4o"
},
"requestId": "msg-456"
}
```
#### 4. Error
```typescript
{
"type": "ERROR",
"payload": {
"code": "LLM_ERROR",
"message": "Failed to generate response",
"details": { ... }
},
"requestId": "msg-456"
}
```
### Error Codes
| Code | Description | HTTP Status |
|------|-------------|-------------|
| `UNAUTHORIZED` | Invalid or expired token | 401 |
| `FORBIDDEN` | Insufficient permissions | 403 |
| `NOT_FOUND` | Resource not found | 404 |
| `VALIDATION_ERROR` | Invalid input data | 400 |
| `LLM_ERROR` | LLM provider error | 502 |
| `RATE_LIMITED` | Too many requests | 429 |
| `FILE_TOO_LARGE` | File exceeds 50MB | 413 |
| `UNSUPPORTED_URL` | No scraper for URL | 400 |
## OpenAPI Configuration
### NestJS Setup
```typescript
// main.ts
const config = new DocumentBuilder()
.setTitle('DreamChat API')
.setDescription('Character simulation and chat API')
.setVersion('1.0')
.addBearerAuth()
.addTag('auth', 'Authentication endpoints')
.addTag('users', 'User management')
.addTag('characters', 'Character CRUD')
.addTag('conversations', 'Chat sessions')
.addTag('messages', 'Chat messages')
.addTag('import', 'Data import')
.build();
const document = SwaggerModule.createDocument(app, config);
SwaggerModule.setup('api/docs', app, document);
// Also serve raw JSON for client generation
writeFileSync('./openapi-spec.json', JSON.stringify(document, null, 2));
```
### Frontend Client Generation
```bash
# Generate TypeScript client
npx openapi-generator-cli generate \
-i ./openapi-spec.json \
-g typescript-fetch \
-o ./apps/frontend/src/api/generated \
--additional-properties=supportsES6=true,npmName=dreamchat-api
# Or use Orval for React Query hooks
npx orval --config ./orval.config.js
```
## Rate Limiting
| Endpoint | Limit |
|----------|-------|
| Auth endpoints | 10 req/min |
| General API | 100 req/min |
| File upload | 5 req/min |
| WebSocket messages | 60 msg/min |
| LLM streaming | 20 requests/min |
```typescript
// Rate limit headers
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1645603200
```

490
doc/architecture.md Normal file
View File

@@ -0,0 +1,490 @@
# DreamChat System Architecture
## Overview
DreamChat is a character simulation platform built with a modular, extensible architecture. The system follows clean architecture principles with clear separation of concerns.
## High-Level Architecture
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Client Layer │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ React │ │ Vite │ │ WebSocket │ │ OpenAPI Generator │ │
│ │ (UI) │ │ (Build) │ │ Client │ │ (API Client) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ API Gateway Layer │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ NestJS Backend │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Auth │ │ Guards │ │ Validation │ │ WebSocket │ │ │
│ │ │ Module │ │ (JWT/Keycloak)│ │ Pipes │ │ Gateway │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
│ Domain Modules │ │ Service Layer │ │ Infrastructure │
│ ┌────────────────┐ │ │ ┌────────────────┐ │ │ ┌────────────────┐ │
│ │ Character │ │ │ │ LLM Service │ │ │ │ LangChain │ │
│ │ Module │ │ │ │ (OpenRouter) │ │ │ │ Integration │ │
│ └────────────────┘ │ │ └────────────────┘ │ │ └────────────────┘ │
│ ┌────────────────┐ │ │ ┌────────────────┐ │ │ ┌────────────────┐ │
│ │ Chat Module │ │ │ │ Memory Service │ │ │ │ Vector Store │ │
│ │ (MVP Focus) │ │ │ │ (Vector DB) │ │ │ │ (pgvector) │ │
│ └────────────────┘ │ │ └────────────────┘ │ │ └────────────────┘ │
│ ┌────────────────┐ │ │ ┌────────────────┐ │ │ ┌────────────────┐ │
│ │ Story Module │ │ │ │ Import Service │ │ │ │ Puppeteer │ │
│ │ (Phase 2) │ │ │ │ (Adapter) │ │ │ │ (Scraper) │ │
│ └────────────────┘ │ │ └────────────────┘ │ │ └────────────────┘ │
│ ┌────────────────┐ │ │ ┌────────────────┐ │ │ ┌────────────────┐ │
│ │ Multi-Char │ │ │ │ File Processor │ │ │ │ PDF Parser │ │
│ │ (Phase 3) │ │ │ │ Service │ │ │ │ MD Parser │ │
│ └────────────────┘ │ │ └────────────────┘ │ │ └────────────────┘ │
└──────────────────────┘ └──────────────────────┘ └──────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ Data Layer │
│ ┌─────────────┐ ┌─────────────────────┐ ┌─────────────────────────────┐ │
│ │ PostgreSQL │ │ pgvector Extension │ │ Keycloak (External) │ │
│ │ (Primary) │ │ (Vector Store) │ │ (Auth Provider) │ │
│ └─────────────┘ └─────────────────────┘ └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
## Module Dependencies
```
┌─────────────────────────────────────────────────────────────────┐
│ Application Module │
├─────────────────────────────────────────────────────────────────┤
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │
│ │ Auth │ │ User │ │ Config │ │
│ │ Module │──│ Module │ │ (Global) │ │
│ └─────────────┘ └─────────────┘ └─────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│
┌─────────────────────┼─────────────────────┐
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────────┐
│ Character │ │ Chat │ │ Import/Export │
│ Module │ │ Module │ │ Module │
│ │ │ (MVP Focus) │ │ │
│ • Attributes │ │ │ │ • File Adapter │
│ • Personality│ │ • Messages │ │ • Web Adapter │
│ • Backstory │ │ • WebSocket │ │ • Preprocessing │
└───────────────┘ └───────────────┘ └───────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ Story Module │ │ Multi-Char Module│
│ (Phase 2) │ │ (Phase 3) │
│ │ │ │
│ • Branching Tree │ │ • Group Chat │
│ • Open-ended Gen │ │ • Char-to-Char │
│ • Tree View API │ │ • Address Direct │
└─────────────────────┘ └───────────────────┘
```
## Component Details
### Backend (NestJS)
#### 1. Auth Module
```typescript
// Dual authentication strategy
- KeycloakStrategy (OAuth2/OIDC)
- LocalStrategy (Password-based)
- JWT Guard for stateless auth
- Roles: USER, ADMIN
// Prisma User model
- id, email, username
- passwordHash, keycloakSub
- role, isActive
```
#### 2. Character Module
```typescript
- CharacterController (REST)
- CharacterService (Business Logic)
- CharacterRepository (Prisma)
- DTOs: CreateCharacterDto, UpdateCharacterDto, CharacterResponseDto
Entities:
- Character
- id, name, avatar
- personalityPrompt: string
- attributes: JSON (complex attribute system)
- backstory: string
- createdBy: User
- createdAt, updatedAt
```
#### 3. Chat Module (MVP)
```typescript
- ChatGateway (WebSocket)
- ChatService
- MessageService
- ConversationRepository (Prisma)
Prisma Models:
- Conversation
- id, title
- characterId (relation)
- userId (relation)
- messages: Message[]
- messageCount, totalTokens
- createdAt, updatedAt
- Message
- id, role (MessageRole enum: user | assistant | system)
- content: String
- tokensUsed: Int?
- model: String?
- metadata: Json?
- conversationId (relation)
- createdAt: DateTime
WebSocket Events:
- client:send_message → server:receive_message → llm:generate → server:stream_response → client:receive_chunk
```
#### 4. Memory Service (LangChain + pgvector + Local Embeddings)
```typescript
- EmbeddingService (Adapter Pattern)
- generateEmbeddings(texts: string[]): Promise<number[][]>
- getDimension(): number
Implementations:
- LocalEmbeddingProvider: Loads HuggingFace model via @xenova/transformers
- HuggingFaceAPIProvider: Uses HuggingFace Inference API
- VectorStoreService (uses Prisma with pgvector extension)
- addDocument(conversationId, content, metadata)
- similaritySearch(conversationId, query, k=5)
- Uses raw Prisma queries with pgvector operators
- MemoryManager
- buildContext(conversationId, currentMessage): string
- summarizeOldMessages(conversationId): Promise<void>
- retrieveRelevantMemories(conversationId, query): Document[]
Prisma Model:
- VectorMemory
- id
- conversationId (relation)
- content: String
- embedding: Unsupported("vector") // pgvector type
- metadata: Json?
- createdAt: DateTime
```
#### 5. LLM Service (Adapter Pattern)
```typescript
interface LLMProvider {
generate(messages: Message[]): Promise<string>;
stream(messages: Message[]): AsyncIterable<string>;
getTokenCount(text: string): number;
}
class OpenRouterProvider implements LLMProvider { ... }
class OpenAIProvider implements LLMProvider { ... }
class OllamaProvider implements LLMProvider { ... }
// Configuration via environment
LLM_PROVIDER=openrouter
LLM_MODEL=openai/gpt-4o
LLM_API_KEY=...
```
#### 6. Import Module (Adapter Pattern)
```typescript
interface ImportAdapter {
canHandle(source: ImportSource): boolean;
import(source: ImportSource): Promise<Document[]>;
}
// File Adapters
class TextFileAdapter implements ImportAdapter { ... }
class PdfFileAdapter implements ImportAdapter { ... }
class MarkdownFileAdapter implements ImportAdapter { ... }
// Web Adapters (Predefined scrapers)
abstract class WebScraperAdapter implements ImportAdapter {
protected abstract canHandleUrl(url: string): boolean;
protected abstract extractContent(page: Page): Promise<string>;
}
class AO3Scraper extends WebScraperAdapter { ... }
class FanfictionNetScraper extends WebScraperAdapter { ... }
// Each scraper validates URL pattern before processing
// Uses Puppeteer for headless browser
// Data Preprocessing Pipeline
class DataPreprocessor {
clean(text: string): string;
chunk(text: string, maxChunkSize: number): string[];
extractEntities(text: string): Entity[];
}
```
### Frontend (React + Vite)
#### Component Hierarchy
```
App
├── AuthProvider (Keycloak + Local)
├── Router
│ ├── /login
│ │ └── LoginPage
│ │ ├── KeycloakLoginButton
│ │ └── PasswordLoginForm
│ ├── /characters
│ │ └── CharacterListPage
│ │ ├── CharacterCard[]
│ │ └── CreateCharacterButton
│ ├── /characters/:id
│ │ └── CharacterDetailPage
│ │ ├── CharacterAttributesEditor
│ │ ├── PersonalityPromptEditor
│ │ └── ChatHistory (if any)
│ ├── /chat/:conversationId (MVP Focus)
│ │ └── ChatPage
│ │ ├── ChatHeader (character info)
│ │ ├── MessageList
│ │ │ └── MessageBubble[]
│ │ └── ChatInput
│ │ └── MessageComposer
│ ├── /stories (Phase 2)
│ │ └── StoryListPage
│ │ └── StoryTreeView
│ └── /import
│ └── ImportPage
│ ├── FileUpload (Drag & Drop)
│ │ └── FileDropzone
│ ├── UrlInput
│ │ └── ScraperSelector
│ └── ProcessingProgress
└── Layout
├── Sidebar
└── Header
```
#### State Management
```typescript
// Using Zustand or React Query
- authStore: AuthState
- characterStore: Character[]
- chatStore:
- currentConversation: Conversation
- messages: Message[]
- isStreaming: boolean
- wsConnection: WebSocket
- importStore: ImportJob[]
```
#### API Client Generation
```bash
# Generated from OpenAPI spec
npx openapi-generator-cli generate \
-i http://localhost:3000/api-json \
-g typescript-fetch \
-o src/api/generated
```
## Data Flow
### Chat Flow (MVP)
```
┌─────────┐ ┌──────────┐ ┌────────────┐ ┌─────────────┐
│ User │────▶│ Frontend │────▶│ WebSocket │────▶│ Chat │
│ │ │ │ │ Gateway │ │ Gateway │
└─────────┘ └──────────┘ └────────────┘ └──────┬──────┘
│
┌────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ ChatService │
│ 1. Save user message to DB │
│ 2. Call MemoryManager.buildContext() │
│ 3. Retrieve relevant memories (vector search) │
│ 4. Build system prompt + context + user message │
│ 5. Call LLMService.generateStream() │
└────────────────────────┬───────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ LLMService │
│ 1. Select provider (OpenRouter) │
│ 2. Format messages for provider │
│ 3. Stream response chunks │
│ 4. Return async iterator │
└────────────────────────┬───────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ Stream Response │
│ 1. Send chunks via WebSocket │
│ 2. Accumulate full response │
│ 3. Save assistant message to DB │
│ 4. Store in vector memory │
└────────────────────────────────────────────────────────┘
```
### File Import Flow
```
┌──────────┐ ┌─────────────┐ ┌──────────────┐ ┌─────────────┐
│ User │────▶│ Frontend │────▶│ POST /api │────▶│ Import │
│ Upload │ │ FileSelect │ │ /import/file│ │ Controller│
└──────────┘ └─────────────┘ └──────────────┘ └──────┬──────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ ImportService │
│ 1. Validate file (type, size < 50MB) │
│ 2. Select adapter based on mime-type │
│ 3. Parse file to raw text │
│ 4. Run DataPreprocessor.clean() │
│ 5. Chunk into segments │
│ 6. Store in import_documents table │
└─────────────────────────────────────────────────────────────────┘
```
### Web Scraping Flow
```
┌──────────┐ ┌─────────────┐ ┌──────────────┐ ┌─────────────┐
│ User │────▶│ Frontend │────▶│ POST /api │────▶│ Import │
│ Enter URL│ │ URL Input │ │ /import/url │ │ Controller│
└──────────┘ └─────────────┘ └──────────────┘ └──────┬──────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ WebImportService │
│ 1. Validate URL format │
│ 2. Find matching scraper (or reject) │
│ 3. Launch Puppeteer, navigate to URL │
│ 4. Extract content using scraper selectors │
│ 5. Run DataPreprocessor.clean() │
│ 6. Chunk and store │
└─────────────────────────────────────────────────────────────────┘
```
## Directory Structure (pnpm Monorepo)
```
dreamchat/
├── apps/
│ ├── backend/
│ │ ├── src/
│ │ │ ├── app.module.ts
│ │ │ ├── main.ts
│ │ │ ├── config/
│ │ │ ├── common/
│ │ │ ├── modules/
│ │ │ │ ├── auth/
│ │ │ │ ├── users/
│ │ │ │ ├── characters/
│ │ │ │ ├── chat/
│ │ │ │ ├── import/
│ │ │ │ ├── story/
│ │ │ │ └── multi-character/
│ │ │ └── shared/
│ │ │ ├── services/
│ │ │ └── prisma/
│ │ ├── test/
│ │ ├── Dockerfile
│ │ └── package.json
│ │
│ └── frontend/
│ ├── src/
│ │ ├── main.tsx
│ │ ├── App.tsx
│ │ ├── api/
│ │ ├── components/
│ │ ├── pages/
│ │ ├── stores/
│ │ ├── hooks/
│ │ └── utils/
│ ├── public/
│ ├── Dockerfile
│ └── package.json
│
├── packages/
│ ├── shared/ # Shared types & WebSocket definitions
│ │ ├── src/
│ │ │ ├── websocket/
│ │ │ │ ├── events.ts # WebSocket event types
│ │ │ │ ├── messages.ts
│ │ │ │ └── index.ts
│ │ │ ├── api/
│ │ │ │ ├── dto.ts # Shared DTOs
│ │ │ │ └── index.ts
│ │ │ └── index.ts
│ │ ├── package.json
│ │ └── tsconfig.json
│ │
│ └── config/ # Shared configurations
│ ├── eslint/
│ └── typescript/
│
├── prisma/ # Database schema (shared)
│ ├── schema.prisma
│ ├── migrations/
│ └── seed.ts
│
├── docker-compose.yml
├── pnpm-workspace.yaml
├── package.json # Root package.json
├── .npmrc
├── .devcontainer/
└── doc/
```
### Package Management
```yaml
# pnpm-workspace.yaml
packages:
- 'apps/*'
- 'packages/*'
```
```bash
# Install all dependencies
pnpm install
# Add dependency to specific app
pnpm --filter @dreamchat/backend add @nestjs/jwt
# Add shared package to apps
pnpm --filter @dreamchat/backend add @dreamchat/shared@workspace:*
```
## Security Considerations
1. **Authentication**: JWT tokens with refresh strategy
2. **Authorization**: Role-based access control (RBAC)
3. **File Upload**:
- Size limit: 50MB
- Mime-type validation
- Storage outside web root
4. **Web Scraping**:
- URL whitelist (predefined scrapers only)
- Rate limiting per domain
- Content sanitization
5. **WebSocket**:
- Auth token validation on connection
- Message rate limiting per user
6. **Database**:
- Prepared statements (Prisma)
- Connection pooling

709
doc/database-schema.md Normal file
View File

@@ -0,0 +1,709 @@
# DreamChat Database Schema
## Overview
PostgreSQL with pgvector extension for vector storage. All data is stored locally (offline-first).
## Extensions Required
```sql
-- Enable required extensions
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pgvector";
```
## Entity Relationship Diagram
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ users │ │ characters │ │ conversations │
├─────────────────┤ ├─────────────────┤ ├─────────────────┤
│ id (PK) │◄──────│ user_id (FK) │ │ id (PK) │
│ email │ │ id (PK) │◄──────│ character_id(FK)│
│ username │ │ name │ │ user_id (FK) │
│ password_hash │ │ avatar_url │ │ title │
│ keycloak_sub │ │ personality │ │ created_at │
│ role │ │ backstory │ │ updated_at │
│ created_at │ │ attributes │ └────────┬────────┘
│ updated_at │ │ created_at │ │
└─────────────────┘ │ updated_at │ │
└─────────────────┘ │
┌─────────────────┐ │
│import_documents │ │
├─────────────────┤ │
│ id (PK) │ │
│ user_id (FK) │ │
│ source_type │ ┌─────────────────┐ │
│ source_name │ │ messages │◄───────────────┘
│ content │ ├─────────────────┤
│ metadata │ │ id (PK) │
│ vector_id │ │ conversation_id │
│ created_at │ │ role │
└─────────────────┘ │ content │
│ tokens_used │
│ model │
┌─────────────────┐ │ metadata │
│vector_memories │ │ created_at │
├─────────────────┤ └─────────────────┘
│ id (PK) │
│ conversation_id │ ┌─────────────────┐
│ content │ │ story_branches │ (Phase 2)
│ embedding │ ├─────────────────┤
│ metadata │ │ id (PK) │
│ created_at │ │ conversation_id │
└─────────────────┘ │ parent_id (FK) │
│ content │
│ direction │
│ metadata │
│ created_at │
└─────────────────┘
```
## Table Definitions
### 1. users
Stores user account information. Supports both Keycloak (OIDC) and local password authentication.
```sql
CREATE TABLE users (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
email VARCHAR(255) UNIQUE NOT NULL,
username VARCHAR(100) UNIQUE NOT NULL,
password_hash VARCHAR(255), -- NULL if using Keycloak
keycloak_sub VARCHAR(255) UNIQUE, -- NULL if using password auth
role VARCHAR(20) DEFAULT 'USER' CHECK (role IN ('USER', 'ADMIN')),
is_active BOOLEAN DEFAULT true,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
-- At least one auth method must be set
CONSTRAINT auth_method_check CHECK (
(password_hash IS NOT NULL) OR (keycloak_sub IS NOT NULL)
)
);
CREATE INDEX idx_users_email ON users(email);
CREATE INDEX idx_users_keycloak_sub ON users(keycloak_sub);
```
### 2. characters
Character definitions with complex attribute system (JSONB for flexibility).
```sql
CREATE TABLE characters (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
name VARCHAR(255) NOT NULL,
avatar_url TEXT,
-- Core personality prompt sent to LLM
personality_prompt TEXT NOT NULL,
-- Backstory context for the character
backstory TEXT,
-- Complex attribute system (structured JSON)
-- Example: {"traits": ["brave", "witty"], "age": 25, "species": "human"}
attributes JSONB DEFAULT '{}',
-- Character configuration
config JSONB DEFAULT '{
"model": "openai/gpt-4o",
"temperature": 0.7,
"max_tokens": 2048,
"memory_enabled": true
}',
is_public BOOLEAN DEFAULT false,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_characters_user_id ON characters(user_id);
CREATE INDEX idx_characters_name ON characters(name);
CREATE INDEX idx_characters_attributes ON characters USING GIN(attributes);
```
### 3. conversations
Chat sessions between user and character.
```sql
CREATE TABLE conversations (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
character_id UUID NOT NULL REFERENCES characters(id) ON DELETE CASCADE,
title VARCHAR(255), -- Auto-generated or user-defined
-- Context window management
message_count INTEGER DEFAULT 0,
total_tokens INTEGER DEFAULT 0,
-- Conversation settings
settings JSONB DEFAULT '{}',
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_conversations_user_id ON conversations(user_id);
CREATE INDEX idx_conversations_character_id ON conversations(character_id);
CREATE INDEX idx_conversations_created_at ON conversations(created_at);
```
### 4. messages
Individual chat messages.
```sql
CREATE TYPE message_role AS ENUM ('user', 'assistant', 'system');
CREATE TABLE messages (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
conversation_id UUID NOT NULL REFERENCES conversations(id) ON DELETE CASCADE,
role message_role NOT NULL,
content TEXT NOT NULL,
-- LLM metadata
tokens_used INTEGER,
model VARCHAR(100),
-- Additional metadata (temperature, latency, etc.)
metadata JSONB DEFAULT '{}',
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_messages_conversation_id ON messages(conversation_id);
CREATE INDEX idx_messages_created_at ON messages(created_at);
CREATE INDEX idx_messages_conversation_created ON messages(conversation_id, created_at);
```
### 5. vector_memories
Vector embeddings for conversation memory using pgvector. Stores chunked content for semantic search.
```sql
-- Create vector extension first
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE vector_memories (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
conversation_id UUID NOT NULL REFERENCES conversations(id) ON DELETE CASCADE,
-- The text content
content TEXT NOT NULL,
-- Vector embedding (configurable dimension based on model)
-- Common sizes: 384 (all-MiniLM-L6-v2), 768 (all-mpnet-base-v2), 1024 (BGE)
-- Must match the EMBEDDING_DIMENSION env var
embedding VECTOR({{EMBEDDING_DIMENSION}}),
-- Metadata for filtering
metadata JSONB DEFAULT '{
"chunk_index": 0,
"source": "conversation",
"timestamp": null
}',
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
-- HNSW index for efficient similarity search
-- Note: Index is created after table creation based on actual dimension
-- CREATE INDEX idx_vector_memories_embedding ON vector_memories
-- USING hnsw (embedding vector_cosine_ops);
CREATE INDEX idx_vector_memories_conversation ON vector_memories(conversation_id);
```
### 6. import_documents
Raw imported documents from files or web scraping.
```sql
CREATE TYPE import_source_type AS ENUM ('file', 'url');
CREATE TYPE import_status AS ENUM ('pending', 'processing', 'completed', 'failed');
CREATE TABLE import_documents (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
source_type import_source_type NOT NULL,
source_name VARCHAR(255) NOT NULL, -- filename or URL
-- Mime type for files
mime_type VARCHAR(100),
-- File size in bytes
file_size BIGINT,
-- Raw content (preprocessed)
content TEXT,
-- Processing status
status import_status DEFAULT 'pending',
error_message TEXT,
-- Metadata (source info, extraction method, etc.)
metadata JSONB DEFAULT '{}',
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_import_documents_user_id ON import_documents(user_id);
CREATE INDEX idx_import_documents_status ON import_documents(status);
```
### 7. story_branches (Phase 2)
Tree structure for branching narratives.
```sql
CREATE TABLE story_branches (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
conversation_id UUID NOT NULL REFERENCES conversations(id) ON DELETE CASCADE,
-- Self-referential for tree structure
parent_id UUID REFERENCES story_branches(id) ON DELETE CASCADE,
-- Branch content
title VARCHAR(255),
content TEXT NOT NULL, -- The generated story content
-- User direction that led to this branch
user_direction TEXT,
-- Branch metadata
generation_params JSONB DEFAULT '{}',
-- Tree position
depth INTEGER DEFAULT 0,
branch_order INTEGER DEFAULT 0,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_story_branches_conversation ON story_branches(conversation_id);
CREATE INDEX idx_story_branches_parent ON story_branches(parent_id);
```
### 8. conversation_participants (Phase 3 - Multi-Character)
Supports multiple characters in a single conversation.
```sql
CREATE TABLE conversation_participants (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
conversation_id UUID NOT NULL REFERENCES conversations(id) ON DELETE CASCADE,
character_id UUID NOT NULL REFERENCES characters(id) ON DELETE CASCADE,
-- Participant settings
is_active BOOLEAN DEFAULT true,
auto_respond BOOLEAN DEFAULT true, -- Auto-generate responses
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
UNIQUE(conversation_id, character_id)
);
CREATE INDEX idx_participants_conversation ON conversation_participants(conversation_id);
```
## Prisma Schema (Reference)
### Full Schema Definition
```prisma
// schema.prisma
generator client {
provider = "prisma-client-js"
}
datasource db {
provider = "postgresql"
url = env("DATABASE_URL")
}
// Enums
enum UserRole {
USER
ADMIN
}
enum MessageRole {
user
assistant
system
}
enum ImportSourceType {
file
url
}
enum ImportStatus {
pending
processing
completed
failed
}
// Models
model User {
id String @id @default(uuid())
email String @unique
username String @unique
passwordHash String?
keycloakSub String? @unique
role UserRole @default(USER)
isActive Boolean @default(true)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
characters Character[]
conversations Conversation[]
importDocs ImportDocument[]
@@index([email])
@@index([keycloakSub])
}
model Character {
id String @id @default(uuid())
name String
avatarUrl String?
personalityPrompt String
backstory String?
attributes Json @default("{}")
config Json @default("{}")
isPublic Boolean @default(false)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
userId String
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
conversations Conversation[]
@@index([userId])
@@index([name])
}
model Conversation {
id String @id @default(uuid())
title String?
messageCount Int @default(0)
totalTokens Int @default(0)
settings Json @default("{}")
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
userId String
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
characterId String
character Character @relation(fields: [characterId], references: [id], onDelete: Cascade)
messages Message[]
vectorMemories VectorMemory[]
storyBranches StoryBranch[]
participants ConversationParticipant[]
@@index([userId])
@@index([characterId])
@@index([createdAt])
}
model Message {
id String @id @default(uuid())
role MessageRole
content String
tokensUsed Int?
model String?
metadata Json?
createdAt DateTime @default(now())
conversationId String
conversation Conversation @relation(fields: [conversationId], references: [id], onDelete: Cascade)
@@index([conversationId])
@@index([createdAt])
@@index([conversationId, createdAt])
}
model VectorMemory {
id String @id @default(uuid())
content String
embedding Unsupported("vector")? // pgvector extension
metadata Json?
createdAt DateTime @default(now())
conversationId String
conversation Conversation @relation(fields: [conversationId], references: [id], onDelete: Cascade)
@@index([conversationId])
}
model ImportDocument {
id String @id @default(uuid())
sourceType ImportSourceType
sourceName String
mimeType String?
fileSize BigInt?
content String?
status ImportStatus @default(pending)
errorMessage String?
metadata Json?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
userId String
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
@@index([userId])
@@index([status])
}
model StoryBranch {
id String @id @default(uuid())
title String?
content String
userDirection String
generationParams Json?
depth Int @default(0)
branchOrder Int @default(0)
createdAt DateTime @default(now())
conversationId String
conversation Conversation @relation(fields: [conversationId], references: [id], onDelete: Cascade)
parentId String?
parent StoryBranch? @relation("BranchTree", fields: [parentId], references: [id], onDelete: Cascade)
children StoryBranch[] @relation("BranchTree")
@@index([conversationId])
@@index([parentId])
}
model ConversationParticipant {
id String @id @default(uuid())
isActive Boolean @default(true)
autoRespond Boolean @default(true)
createdAt DateTime @default(now())
conversationId String
conversation Conversation @relation(fields: [conversationId], references: [id], onDelete: Cascade)
characterId String
@@unique([conversationId, characterId])
@@index([conversationId])
}
```
### Prisma Client Usage Examples
```typescript
// src/shared/prisma/prisma.service.ts
import { Injectable, OnModuleInit, OnModuleDestroy } from '@nestjs/common';
import { PrismaClient } from '@prisma/client';
@Injectable()
export class PrismaService extends PrismaClient implements OnModuleInit, OnModuleDestroy {
async onModuleInit() {
await this.$connect();
}
async onModuleDestroy() {
await this.$disconnect();
}
}
```
```typescript
// Repository pattern with Prisma
@Injectable()
export class CharacterRepository {
constructor(private prisma: PrismaService) {}
async findByUser(userId: string) {
return this.prisma.character.findMany({
where: { userId },
orderBy: { updatedAt: 'desc' },
});
}
async create(data: CreateCharacterDto, userId: string) {
return this.prisma.character.create({
data: { ...data, userId },
});
}
}
```
### Vector Memory Query with Prisma
```typescript
// Similarity search using pgvector with Prisma
async similaritySearch(
conversationId: string,
queryEmbedding: number[],
k: number = 5
) {
// Using raw query for pgvector-specific operations
const results = await this.prisma.$queryRaw`
SELECT
id,
content,
metadata,
embedding <=> ${queryEmbedding}::vector as distance
FROM "VectorMemory"
WHERE "conversationId" = ${conversationId}
ORDER BY embedding <=> ${queryEmbedding}::vector
LIMIT ${k}
`;
return results;
}
// Alternative: using cosine similarity
async similaritySearchCosine(
conversationId: string,
queryEmbedding: number[],
k: number = 5
) {
const results = await this.prisma.$queryRaw`
SELECT
id,
content,
metadata,
1 - (embedding <=> ${queryEmbedding}::vector) as similarity
FROM "VectorMemory"
WHERE "conversationId" = ${conversationId}
ORDER BY similarity DESC
LIMIT ${k}
`;
return results;
}
```
### Embedding Configuration
```typescript
// Configuration for embedding providers
interface EmbeddingConfig {
provider: 'local' | 'huggingface-api';
model: string;
dimension: number;
// For local provider
localModelPath?: string;
// For HuggingFace API
apiKey?: string;
apiEndpoint?: string;
}
// Example configurations:
// Local: { provider: 'local', model: 'Xenova/all-MiniLM-L6-v2', dimension: 384 }
// HF API: { provider: 'huggingface-api', model: 'sentence-transformers/all-mpnet-base-v2', dimension: 768 }
```
## Prisma Migration Strategy
### Initial Migration
```bash
# 1. Initialize Prisma
npx prisma init
# 2. Define schema in prisma/schema.prisma
# 3. Create first migration
npx prisma migrate dev --name init
# 4. Generate Prisma Client
npx prisma generate
```
### Migration Workflow
```bash
# After schema changes
npx prisma migrate dev --name descriptive_name
# Production deployment
npx prisma migrate deploy
# Generate client (in CI/CD)
npx prisma generate
```
### Migration File Example
```sql
-- migrations/20240223120000_init/migration.sql
-- Enable extensions
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "vector";
-- CreateEnum
CREATE TYPE "UserRole" AS ENUM ('USER', 'ADMIN');
-- CreateEnum
CREATE TYPE "MessageRole" AS ENUM ('user', 'assistant', 'system');
-- CreateTable
CREATE TABLE "User" (
"id" UUID NOT NULL DEFAULT uuid_generate_v4(),
"email" TEXT NOT NULL,
"username" TEXT NOT NULL,
-- ... etc
);
-- CreateIndex
CREATE UNIQUE INDEX "User_email_key" ON "User"("email");
-- Vector index (HNSW) - created manually after migration
CREATE INDEX idx_vector_memories_embedding ON "VectorMemory"
USING hnsw (embedding vector_cosine_ops);
```
### Seeding
```typescript
// prisma/seed.ts
import { PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();
async function main() {
// Seed default admin or test data
await prisma.user.create({
data: {
email: 'admin@dreamchat.local',
username: 'admin',
role: 'ADMIN',
},
});
}
main()
.catch(console.error)
.finally(() => prisma.$disconnect());
```
```bash
# Run seed
npx prisma db seed
```
## Backup Strategy
```bash
# pg_dump with custom format
docker exec dreamchat-postgres pg_dump -U postgres -Fc dreamchat > backup.dump
# Restore
pg_restore -U postgres -d dreamchat backup.dump
```

725
doc/deployment.md Normal file
View File

@@ -0,0 +1,725 @@
# DreamChat Deployment Guide
## Overview
This document covers the deployment configuration including Docker Compose setup, DevContainer configuration, and production deployment procedures.
## DevContainer Setup
### .devcontainer/devcontainer.json
```json
{
"name": "DreamChat Development",
"dockerComposeFile": "docker-compose.yml",
"service": "app",
"workspaceFolder": "/workspace",
"features": {
"ghcr.io/devcontainers/features/node:1": {
"version": "20"
},
"ghcr.io/devcontainers/features/docker-in-docker:2": {}
},
"customizations": {
"vscode": {
"extensions": [
"dbaeumer.vscode-eslint",
"esbenp.prettier-vscode",
"bradlc.vscode-tailwindcss",
"ms-vscode.vscode-typescript-next",
"nestjs.vscode-nestjs",
"prisma.prisma"
],
"settings": {
"editor.formatOnSave": true,
"editor.defaultFormatter": "esbenp.prettier-vscode",
"typescript.preferences.importModuleSpecifier": "relative"
}
}
},
"forwardPorts": [3000, 5173, 5432, 8080],
"portsAttributes": {
"3000": {
"label": "Backend API",
"onAutoForward": "notify"
},
"5173": {
"label": "Frontend Dev Server",
"onAutoForward": "notify"
},
"5432": {
"label": "PostgreSQL",
"onAutoForward": "silent"
},
"8080": {
"label": "Keycloak",
"onAutoForward": "notify"
}
},
"postCreateCommand": "bash .devcontainer/post-create.sh",
"remoteUser": "node",
"mounts": [
"source=${localWorkspaceFolderBasename}-node_modules,target=${containerWorkspaceFolder}/node_modules,type=volume",
"source=${localWorkspaceFolderBasename}-pnpm-store,target=/home/node/.local/share/pnpm/store,type=volume"
]
}
```
### .devcontainer/docker-compose.yml
```yaml
version: '3.8'
services:
app:
build:
context: ..
dockerfile: .devcontainer/Dockerfile
volumes:
- ..:/workspace:cached
- /var/run/docker.sock:/var/run/docker.sock
command: sleep infinity
environment:
- DATABASE_URL=postgresql://postgres:postgres@db:5432/dreamchat
- REDIS_URL=redis://redis:6379
- KEYCLOAK_URL=http://keycloak:8080
depends_on:
- db
- redis
- keycloak
networks:
- dreamchat-network
db:
image: ankane/pgvector:latest
restart: unless-stopped
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: dreamchat
volumes:
- postgres-data:/var/lib/postgresql/data
ports:
- "5432:5432"
networks:
- dreamchat-network
redis:
image: redis:7-alpine
restart: unless-stopped
ports:
- "6379:6379"
networks:
- dreamchat-network
keycloak:
image: quay.io/keycloak/keycloak:23.0
restart: unless-stopped
command: start-dev
environment:
KEYCLOAK_ADMIN: admin
KEYCLOAK_ADMIN_PASSWORD: admin
KC_DB: postgres
KC_DB_URL: jdbc:postgresql://db:5432/keycloak
KC_DB_USERNAME: postgres
KC_DB_PASSWORD: postgres
ports:
- "8080:8080"
depends_on:
- db
networks:
- dreamchat-network
volumes:
postgres-data:
networks:
dreamchat-network:
driver: bridge
```
### .devcontainer/Dockerfile
```dockerfile
FROM mcr.microsoft.com/devcontainers/typescript-node:20
# Install additional tools
RUN apt-get update && apt-get install -y \
postgresql-client \
redis-tools \
&& rm -rf /var/lib/apt/lists/*
# Install pnpm
RUN npm install -g pnpm@8
# Set working directory
WORKDIR /workspace
# Install global packages
RUN npm install -g @nestjs/cli@latest
# Create non-root user
USER node
```
### .devcontainer/post-create.sh
```bash
#!/bin/bash
set -e
echo "🚀 Setting up DreamChat monorepo development environment..."
# Install pnpm globally
npm install -g pnpm@8
# Install all dependencies (uses pnpm workspaces)
echo "📦 Installing dependencies..."
cd /workspace
pnpm install
# Build shared packages first
echo "📦 Building shared packages..."
pnpm --filter @dreamchat/shared build
# Generate Prisma client
pnpm db:generate
# Copy environment files if they don't exist
if [ ! -f /workspace/apps/backend/.env ]; then
echo "⚙️ Creating backend .env file..."
cat > /workspace/apps/backend/.env << EOF
NODE_ENV=development
PORT=3000
DATABASE_URL=postgresql://postgres:postgres@db:5432/dreamchat
JWT_SECRET=dev-jwt-secret-change-in-production
JWT_EXPIRES_IN=1h
JWT_REFRESH_EXPIRES_IN=7d
LLM_PROVIDER=openrouter
LLM_API_KEY=your-openrouter-api-key
LLM_MODEL=openai/gpt-4o
KEYCLOAK_URL=http://localhost:8080
KEYCLOAK_REALM=dreamchat
KEYCLOAK_CLIENT_ID=dreamchat-backend
EOF
fi
if [ ! -f /workspace/apps/frontend/.env ]; then
echo "⚙️ Creating frontend .env file..."
cat > /workspace/apps/frontend/.env << EOF
VITE_API_URL=http://localhost:3000/api
VITE_WS_URL=ws://localhost:3000
VITE_KEYCLOAK_URL=http://localhost:8080
VITE_KEYCLOAK_REALM=dreamchat
VITE_KEYCLOAK_CLIENT_ID=dreamchat-frontend
EOF
fi
echo "✅ Development environment setup complete!"
echo ""
echo "Next steps:"
echo "1. Start all apps: pnpm dev"
echo "2. Or start individually:"
echo " - Backend: pnpm --filter @dreamchat/backend dev"
echo " - Frontend: pnpm --filter @dreamchat/frontend dev"
echo "3. Access Keycloak admin at http://localhost:8080 (admin/admin)"
```
## Docker Compose Production
### docker-compose.yml (Root)
```yaml
version: '3.8'
services:
# Backend API
backend:
build:
context: .
dockerfile: apps/backend/Dockerfile
restart: unless-stopped
environment:
- NODE_ENV=production
- PORT=3000
- DATABASE_URL=postgresql://postgres:${POSTGRES_PASSWORD}@db:5432/dreamchat
- JWT_SECRET=${JWT_SECRET}
- JWT_EXPIRES_IN=${JWT_EXPIRES_IN:-1h}
- JWT_REFRESH_EXPIRES_IN=${JWT_REFRESH_EXPIRES_IN:-7d}
- LLM_PROVIDER=${LLM_PROVIDER}
- LLM_API_KEY=${LLM_API_KEY}
- LLM_MODEL=${LLM_MODEL}
- EMBEDDING_PROVIDER=${EMBEDDING_PROVIDER:-local}
- EMBEDDING_MODEL=${EMBEDDING_MODEL:-Xenova/all-MiniLM-L6-v2}
- EMBEDDING_DIMENSION=${EMBEDDING_DIMENSION:-384}
- EMBEDDING_DEVICE=${EMBEDDING_DEVICE:-cpu}
- HUGGINGFACE_API_KEY=${HUGGINGFACE_API_KEY}
- KEYCLOAK_URL=${KEYCLOAK_URL}
- KEYCLOAK_REALM=${KEYCLOAK_REALM}
- KEYCLOAK_CLIENT_ID=${KEYCLOAK_CLIENT_ID}
- KEYCLOAK_CLIENT_SECRET=${KEYCLOAK_CLIENT_SECRET}
ports:
- "3000:3000"
depends_on:
db:
condition: service_healthy
volumes:
- backend-logs:/app/logs
- model-cache:/app/models
networks:
- dreamchat-network
healthcheck:
test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3000/health"]
interval: 30s
timeout: 10s
retries: 3
# Frontend
frontend:
build:
context: .
dockerfile: apps/frontend/Dockerfile
restart: unless-stopped
ports:
- "80:80"
- "443:443"
environment:
- VITE_API_URL=/api
- VITE_WS_URL=/ws
depends_on:
- backend
networks:
- dreamchat-network
# Database
db:
image: ankane/pgvector:latest
restart: unless-stopped
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: dreamchat
volumes:
- postgres-data:/var/lib/postgresql/data
- ./init-scripts:/docker-entrypoint-initdb.d
networks:
- dreamchat-network
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 10s
timeout: 5s
retries: 5
# Redis (optional, for session storage and caching)
redis:
image: redis:7-alpine
restart: unless-stopped
volumes:
- redis-data:/data
networks:
- dreamchat-network
# Nginx Reverse Proxy (optional)
nginx:
image: nginx:alpine
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./nginx/ssl:/etc/nginx/ssl:ro
- model-cache:/model-cache:ro
depends_on:
- backend
- frontend
networks:
- dreamchat-network
volumes:
postgres-data:
redis-data:
backend-logs:
model-cache: # Persist downloaded embedding models
networks:
dreamchat-network:
driver: bridge
```
### Embedding Model Cache (Optional)
To avoid re-downloading models on every restart, models are cached in a Docker volume:
```yaml
# Add to docker-compose volumes
volumes:
model-cache:
```
Models are downloaded on first use and cached at `/app/models` in the backend container.
Common models and their sizes:
- `Xenova/all-MiniLM-L6-v2`: ~80MB (384 dimensions)
- `Xenova/all-mpnet-base-v2`: ~420MB (768 dimensions)
- `BAAI/bge-small-en`: ~130MB (384 dimensions)
### .env.example
```bash
# Database
POSTGRES_PASSWORD=your_secure_password_here
# JWT
JWT_SECRET=your_jwt_secret_key_min_32_chars
JWT_EXPIRES_IN=1h
JWT_REFRESH_EXPIRES_IN=7d
# LLM Configuration
LLM_PROVIDER=openrouter
LLM_API_KEY=sk-or-v1-...
LLM_MODEL=openai/gpt-4o
# Embedding Configuration (Local HuggingFace by default)
EMBEDDING_PROVIDER=local
EMBEDDING_MODEL=Xenova/all-MiniLM-L6-v2
EMBEDDING_DIMENSION=384
EMBEDDING_DEVICE=cpu
# HuggingFace API (optional - if not using local embeddings)
# HUGGINGFACE_API_KEY=hf_...
# Keycloak (optional - for external auth)
KEYCLOAK_URL=http://keycloak:8080
KEYCLOAK_REALM=dreamchat
KEYCLOAK_CLIENT_ID=dreamchat-backend
KEYCLOAK_CLIENT_SECRET=your_keycloak_secret
```
## Backend Dockerfile
```dockerfile
# apps/backend/Dockerfile
FROM node:20-alpine AS base
RUN npm install -g pnpm@8
FROM base AS dependencies
WORKDIR /app
# Copy workspace configuration
COPY pnpm-workspace.yaml package.json ./
COPY apps/backend/package.json ./apps/backend/
COPY packages/shared/package.json ./packages/shared/
# Install dependencies
RUN pnpm install --frozen-lockfile
FROM base AS build
WORKDIR /app
COPY --from=dependencies /app/node_modules ./node_modules
COPY --from=dependencies /app/apps/backend/node_modules ./apps/backend/node_modules
COPY --from=dependencies /app/packages/shared/node_modules ./packages/shared/node_modules
# Copy source code
COPY packages/shared ./packages/shared
COPY apps/backend ./apps/backend
# Build shared packages first
RUN pnpm --filter @dreamchat/shared build
# Build backend
RUN pnpm --filter @dreamchat/backend build
FROM base AS production
WORKDIR /app
# Copy only production dependencies
COPY --from=dependencies /app/node_modules ./node_modules
COPY --from=dependencies /app/apps/backend/node_modules ./apps/backend/node_modules
COPY --from=build /app/apps/backend/dist ./dist
COPY --from=build /app/packages/shared/dist ./node_modules/@dreamchat/shared/dist
# Create logs directory
RUN mkdir -p /app/logs
# Non-root user
RUN addgroup -g 1001 -S nodejs
RUN adduser -S nodejs -u 1001
USER nodejs
EXPOSE 3000
CMD ["node", "dist/main.js"]
```
## Frontend Dockerfile
```dockerfile
# apps/frontend/Dockerfile
FROM node:20-alpine AS base
RUN npm install -g pnpm@8
FROM base AS dependencies
WORKDIR /app
# Copy workspace configuration
COPY pnpm-workspace.yaml package.json ./
COPY apps/frontend/package.json ./apps/frontend/
COPY packages/shared/package.json ./packages/shared/
# Install dependencies
RUN pnpm install --frozen-lockfile
FROM base AS build
WORKDIR /app
COPY --from=dependencies /app/node_modules ./node_modules
COPY --from=dependencies /app/apps/frontend/node_modules ./apps/frontend/node_modules
COPY --from=dependencies /app/packages/shared/node_modules ./packages/shared/node_modules
# Copy source code
COPY packages/shared ./packages/shared
COPY apps/frontend ./apps/frontend
# Build shared packages first
RUN pnpm --filter @dreamchat/shared build
# Build frontend
RUN pnpm --filter @dreamchat/frontend build
# Production with Nginx
FROM nginx:alpine
# Copy built assets
COPY --from=build /app/apps/frontend/dist /usr/share/nginx/html
# Copy nginx config
COPY apps/frontend/nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
```
### frontend/nginx.conf
```nginx
server {
listen 80;
server_name localhost;
root /usr/share/nginx/html;
index index.html;
# Gzip compression
gzip on;
gzip_vary on;
gzip_min_length 1024;
gzip_types text/plain text/css text/xml text/javascript application/json application/javascript application/xml+rss application/rss+xml font/truetype font/opentype application/vnd.ms-fontobject image/svg+xml;
# API proxy
location /api {
proxy_pass http://backend:3000/api;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
}
# WebSocket proxy
location /ws {
proxy_pass http://backend:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Static files
location / {
try_files $uri $uri/ /index.html;
add_header Cache-Control "public, max-age=31536000, immutable";
}
# Health check
location /health {
access_log off;
return 200 "healthy\n";
add_header Content-Type text/plain;
}
}
```
## Keycloak Configuration
### Initial Setup
1. Access Keycloak admin console: `http://localhost:8080/admin`
2. Login with admin credentials
3. Create new realm: `dreamchat`
4. Create client: `dreamchat-backend`
- Client authentication: ON
- Authorization: ON
- Valid redirect URIs: `http://localhost:3000/*`
- Web origins: `http://localhost:3000`
5. Create client: `dreamchat-frontend`
- Client authentication: OFF
- Valid redirect URIs: `http://localhost:5173/*`
- Web origins: `http://localhost:5173`
### realm-export.json (Optional)
```json
{
"realm": "dreamchat",
"enabled": true,
"clients": [
{
"clientId": "dreamchat-backend",
"enabled": true,
"clientAuthenticatorType": "client-secret",
"secret": "**********",
"redirectUris": ["http://localhost:3000/*"],
"webOrigins": ["http://localhost:3000"],
"protocol": "openid-connect"
},
{
"clientId": "dreamchat-frontend",
"enabled": true,
"publicClient": true,
"redirectUris": ["http://localhost:5173/*"],
"webOrigins": ["http://localhost:5173"],
"protocol": "openid-connect"
}
]
}
```
## Deployment Procedures
### Local Development
```bash
# Using DevContainer
1. Open project in VS Code
2. Click "Reopen in Container"
3. Wait for setup to complete
4. Start services:
- Terminal 1: pnpm --filter @dreamchat/backend dev
- Terminal 2: pnpm --filter @dreamchat/frontend dev
- Or run all: pnpm dev
```
### Production Deployment
```bash
# 1. Clone repository
git clone https://github.com/yourusername/dreamchat.git
cd dreamchat
# 2. Create environment file
cp .env.example .env
# Edit .env with production values
# 3. Build and start
docker-compose up -d --build
# 4. Run database migrations
docker-compose exec backend pnpm db:migrate
# 5. Check health
curl http://localhost:3000/health
# 6. View logs
docker-compose logs -f backend
```
### Backup and Restore
```bash
# Backup database
docker-compose exec db pg_dump -U postgres -Fc dreamchat > backup_$(date +%Y%m%d).dump
# Restore database
docker-compose exec -T db pg_restore -U postgres -d dreamchat < backup_20240223.dump
# Backup volumes
docker run --rm -v dreamchat_postgres-data:/data -v $(pwd):/backup alpine tar czf /backup/postgres-backup.tar.gz -C /data .
```
### Updates
```bash
# Pull latest changes
git pull origin main
# Rebuild and restart
docker-compose down
docker-compose up -d --build
# Run migrations
docker-compose exec backend npx prisma migrate deploy
```
## Monitoring and Logging
### Health Checks
```bash
# Backend health
curl http://localhost:3000/health
# Database health
docker-compose exec db pg_isready -U postgres
# Full stack
docker-compose ps
```
### Log Management
```bash
# View all logs
docker-compose logs
# View specific service
docker-compose logs -f backend
# View last 100 lines
docker-compose logs --tail=100 backend
```
## Security Considerations
1. **Change default passwords** in production
2. **Use HTTPS** with valid SSL certificates
3. **Enable firewall** rules for required ports only
4. **Regular updates** of base images and dependencies
5. **Secrets management** - use Docker secrets or external vault
6. **Network isolation** - separate networks for different services
## Troubleshooting
### Common Issues
1. **Database connection failed**
- Check DATABASE_URL environment variable
- Ensure db service is healthy: `docker-compose ps`
2. **WebSocket not connecting**
- Verify VITE_WS_URL matches your domain
- Check Nginx proxy configuration
3. **Keycloak authentication issues**
- Verify client configuration in Keycloak admin
- Check redirect URIs match exactly
4. **File upload fails**
- Check file size limits in Nginx config
- Verify disk space in backend container

635
doc/frontend-guide.md Normal file
View File

@@ -0,0 +1,635 @@
# DreamChat Frontend Guide
## Overview
This document outlines the frontend architecture, component structure, and development guidelines for the DreamChat React application.
## Tech Stack
| Category | Technology | Purpose |
|----------|------------|---------|
| Framework | React 18+ | UI library |
| Build Tool | Vite | Fast development and building |
| Language | TypeScript | Type safety |
| Styling | Tailwind CSS | Utility-first CSS |
| State Management | Zustand | Global state |
| Server State | TanStack Query | API data caching |
| Routing | React Router v6 | Navigation |
| WebSocket | Socket.io-client | Real-time communication |
| Shared Types | `@dreamchat/shared` | Monorepo shared package |
| API Client | OpenAPI Generator | Auto-generated API client |
| Forms | React Hook Form + Zod | Form handling and validation |
| UI Components | Radix UI + shadcn/ui | Accessible primitives |
| Icons | Lucide React | Icon library |
| Testing | Vitest + React Testing Library | Unit/component tests |
## Project Structure
```
apps/frontend/
├── src/
│ ├── api/
│ │ ├── generated/ # Auto-generated from OpenAPI
│ │ │ ├── models/ # DTO types
│ │ │ └── apis/ # API client classes
│ │ └── client.ts # Axios/fetch configuration
│ │
│ ├── websocket/
│ │ ├── socket.ts # Socket.io client setup
│ │ └── hooks.ts # WebSocket hooks (uses @dreamchat/shared)
│ │
│ ├── components/
│ │ ├── ui/ # Reusable UI components
│ │ │ ├── button.tsx
│ │ │ ├── input.tsx
│ │ │ ├── card.tsx
│ │ │ ├── dialog.tsx
│ │ │ └── ...
│ │ ├── layout/
│ │ │ ├── sidebar.tsx
│ │ │ ├── header.tsx
│ │ │ └── main-layout.tsx
│ │ ├── character/
│ │ │ ├── character-card.tsx
│ │ │ ├── character-form.tsx
│ │ │ ├── attribute-editor.tsx
│ │ │ └── personality-editor.tsx
│ │ ├── chat/
│ │ │ ├── chat-container.tsx
│ │ │ ├── message-list.tsx
│ │ │ ├── message-bubble.tsx
│ │ │ ├── chat-input.tsx
│ │ │ └── typing-indicator.tsx
│ │ └── import/
│ │ ├── file-dropzone.tsx
│ │ ├── url-input.tsx
│ │ └── import-progress.tsx
│ │
│ ├── pages/
│ │ ├── login-page.tsx
│ │ ├── register-page.tsx
│ │ ├── character-list-page.tsx
│ │ ├── character-detail-page.tsx
│ │ ├── chat-page.tsx
│ │ ├── story-page.tsx
│ │ └── import-page.tsx
│ │
│ ├── hooks/
│ │ ├── use-auth.ts
│ │ ├── use-chat.ts
│ │ ├── use-characters.ts
│ │ └── use-import.ts
│ │
│ ├── stores/
│ │ ├── auth-store.ts
│ │ ├── chat-store.ts
│ │ └── ui-store.ts
│ │
│ ├── lib/
│ │ ├── utils.ts # Utility functions
│ │ └── constants.ts # App constants
│ │
│ ├── types/
│ │ └── index.ts # App-specific types
│ │
│ ├── styles/
│ │ └── globals.css
│ │
│ ├── main.tsx
│ └── App.tsx
│
├── public/
│ └── assets/
│
├── index.html
├── vite.config.ts
├── tailwind.config.ts
├── tsconfig.json
└── package.json
```
## Component Architecture
### UI Components (shadcn/ui pattern)
Base components built on Radix UI primitives:
```typescript
// components/ui/button.tsx
import * as React from "react";
import { cva, type VariantProps } from "class-variance-authority";
import { cn } from "@/lib/utils";
const buttonVariants = cva(
"inline-flex items-center justify-center rounded-md text-sm font-medium transition-colors",
{
variants: {
variant: {
default: "bg-primary text-primary-foreground hover:bg-primary/90",
destructive: "bg-destructive text-destructive-foreground",
outline: "border border-input bg-background hover:bg-accent",
secondary: "bg-secondary text-secondary-foreground",
ghost: "hover:bg-accent hover:text-accent-foreground",
link: "underline-offset-4 hover:underline text-primary",
},
size: {
default: "h-10 px-4 py-2",
sm: "h-9 rounded-md px-3",
lg: "h-11 rounded-md px-8",
icon: "h-10 w-10",
},
},
defaultVariants: {
variant: "default",
size: "default",
},
}
);
export interface ButtonProps
extends React.ButtonHTMLAttributes<HTMLButtonElement>,
VariantProps<typeof buttonVariants> {}
const Button = React.forwardRef<HTMLButtonElement, ButtonProps>(
({ className, variant, size, ...props }, ref) => {
return (
<button
className={cn(buttonVariants({ variant, size, className }))}
ref={ref}
{...props}
/>
);
}
);
Button.displayName = "Button";
export { Button, buttonVariants };
```
### Feature Components
Character form example:
```typescript
// components/character/character-form.tsx
import { useForm } from "react-hook-form";
import { zodResolver } from "@hookform/resolvers/zod";
import * as z from "zod";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { Textarea } from "@/components/ui/textarea";
import { AttributeEditor } from "./attribute-editor";
const characterSchema = z.object({
name: z.string().min(1, "Name is required"),
personalityPrompt: z.string().min(10, "Personality prompt too short"),
backstory: z.string().optional(),
attributes: z.record(z.any()).default({}),
});
type CharacterFormData = z.infer<typeof characterSchema>;
interface CharacterFormProps {
initialData?: Partial<CharacterFormData>;
onSubmit: (data: CharacterFormData) => Promise<void>;
}
export function CharacterForm({ initialData, onSubmit }: CharacterFormProps) {
const form = useForm<CharacterFormData>({
resolver: zodResolver(characterSchema),
defaultValues: initialData || {
name: "",
personalityPrompt: "",
backstory: "",
attributes: {},
},
});
return (
<form onSubmit={form.handleSubmit(onSubmit)} className="space-y-6">
<div>
<label>Name</label>
<Input {...form.register("name")} />
{form.formState.errors.name && (
<span className="text-red-500">{form.formState.errors.name.message}</span>
)}
</div>
<div>
<label>Personality Prompt</label>
<Textarea {...form.register("personalityPrompt")} rows={5} />
</div>
<div>
<label>Backstory</label>
<Textarea {...form.register("backstory")} rows={5} />
</div>
<div>
<label>Attributes</label>
<AttributeEditor
value={form.watch("attributes")}
onChange={(attrs) => form.setValue("attributes", attrs)}
/>
</div>
<Button type="submit" disabled={form.formState.isSubmitting}>
{initialData ? "Update" : "Create"} Character
</Button>
</form>
);
}
```
## State Management
### Zustand Store Example
```typescript
// stores/auth-store.ts
import { create } from "zustand";
import { persist } from "zustand/middleware";
interface User {
id: string;
email: string;
username: string;
}
interface AuthState {
user: User | null;
accessToken: string | null;
refreshToken: string | null;
isAuthenticated: boolean;
setAuth: (user: User, accessToken: string, refreshToken: string) => void;
clearAuth: () => void;
updateAccessToken: (token: string) => void;
}
export const useAuthStore = create<AuthState>()(
persist(
(set) => ({
user: null,
accessToken: null,
refreshToken: null,
isAuthenticated: false,
setAuth: (user, accessToken, refreshToken) =>
set({ user, accessToken, refreshToken, isAuthenticated: true }),
clearAuth: () =>
set({
user: null,
accessToken: null,
refreshToken: null,
isAuthenticated: false,
}),
updateAccessToken: (token) => set({ accessToken: token }),
}),
{
name: "auth-storage",
partialize: (state) => ({
accessToken: state.accessToken,
refreshToken: state.refreshToken,
user: state.user,
}),
}
)
);
```
### TanStack Query for Server State
```typescript
// hooks/use-characters.ts
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
import { CharactersApi } from "@/api/generated";
const charactersApi = new CharactersApi();
export function useCharacters() {
return useQuery({
queryKey: ["characters"],
queryFn: () => charactersApi.charactersControllerFindAll(),
});
}
export function useCreateCharacter() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: (data: CreateCharacterDto) =>
charactersApi.charactersControllerCreate(data),
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["characters"] });
},
});
}
```
## WebSocket Integration
Uses shared types from `@dreamchat/shared` for type-safe communication.
```typescript
// src/websocket/socket.ts
import { io, Socket } from "socket.io-client";
import { useAuthStore } from "@/stores/auth-store";
let socket: Socket | null = null;
export function getSocket(): Socket {
if (!socket) {
const token = useAuthStore.getState().accessToken;
socket = io(import.meta.env.VITE_WS_URL, {
auth: { token },
transports: ["websocket"],
});
socket.on("connect", () => {
console.log("WebSocket connected");
});
socket.on("disconnect", () => {
console.log("WebSocket disconnected");
});
socket.on("ERROR", (error) => {
console.error("WebSocket error:", error);
});
}
return socket;
}
export function disconnectSocket() {
if (socket) {
socket.disconnect();
socket = null;
}
}
```
```typescript
// src/websocket/hooks.ts
import { useEffect, useCallback } from "react";
import { getSocket } from "./socket";
import {
WebSocketEventType,
WebSocketMessage,
JoinConversationPayload,
SendMessagePayload,
StreamChunkPayload,
StreamCompletePayload,
ErrorPayload,
} from "@dreamchat/shared";
export function useChatSocket(conversationId: string) {
const socket = getSocket();
useEffect(() => {
const payload: JoinConversationPayload = { conversationId };
socket.emit(WebSocketEventType.JOIN_CONVERSATION, payload);
return () => {
socket.emit(WebSocketEventType.LEAVE_CONVERSATION, { conversationId });
};
}, [conversationId, socket]);
const sendMessage = useCallback(
(content: string, streaming = true) => {
const payload: SendMessagePayload = {
conversationId,
content,
streaming,
};
socket.emit(WebSocketEventType.SEND_MESSAGE, payload);
},
[conversationId, socket]
);
const stopGeneration = useCallback(() => {
socket.emit(WebSocketEventType.STOP_GENERATION, { conversationId });
}, [conversationId, socket]);
return { sendMessage, stopGeneration };
}
export function useChatEvents(
onStreamChunk: (chunk: string) => void,
onStreamComplete: (message: StreamCompletePayload) => void,
onError: (error: ErrorPayload) => void
) {
const socket = getSocket();
useEffect(() => {
const handleChunk = (msg: WebSocketMessage<StreamChunkPayload>) => {
onStreamChunk(msg.payload.chunk);
};
const handleComplete = (msg: WebSocketMessage<StreamCompletePayload>) => {
onStreamComplete(msg.payload);
};
const handleError = (msg: WebSocketMessage<ErrorPayload>) => {
onError(msg.payload);
};
socket.on(WebSocketEventType.STREAM_CHUNK, handleChunk);
socket.on(WebSocketEventType.STREAM_COMPLETE, handleComplete);
socket.on(WebSocketEventType.ERROR, handleError);
return () => {
socket.off(WebSocketEventType.STREAM_CHUNK, handleChunk);
socket.off(WebSocketEventType.STREAM_COMPLETE, handleComplete);
socket.off(WebSocketEventType.ERROR, handleError);
};
}, [socket, onStreamChunk, onStreamComplete, onError]);
}
```
## Routing
```typescript
// App.tsx
import { BrowserRouter, Routes, Route, Navigate } from "react-router-dom";
import { useAuthStore } from "@/stores/auth-store";
import { MainLayout } from "@/components/layout/main-layout";
import { LoginPage } from "@/pages/login-page";
import { CharacterListPage } from "@/pages/character-list-page";
import { ChatPage } from "@/pages/chat-page";
function ProtectedRoute({ children }: { children: React.ReactNode }) {
const isAuthenticated = useAuthStore((state) => state.isAuthenticated);
return isAuthenticated ? <>{children}</> : <Navigate to="/login" />;
}
function App() {
return (
<BrowserRouter>
<Routes>
<Route path="/login" element={<LoginPage />} />
<Route
path="/"
element={
<ProtectedRoute>
<MainLayout />
</ProtectedRoute>
}
>
<Route index element={<Navigate to="/characters" />} />
<Route path="characters" element={<CharacterListPage />} />
<Route path="characters/:id" element={<CharacterDetailPage />} />
<Route path="chat/:conversationId" element={<ChatPage />} />
<Route path="stories" element={<StoryPage />} />
<Route path="import" element={<ImportPage />} />
</Route>
</Routes>
</BrowserRouter>
);
}
```
## Styling with Tailwind
```typescript
// tailwind.config.ts
import type { Config } from "tailwindcss";
const config: Config = {
darkMode: ["class"],
content: ["./src/**/*.{js,ts,jsx,tsx}"],
theme: {
extend: {
colors: {
border: "hsl(var(--border))",
input: "hsl(var(--input))",
ring: "hsl(var(--ring))",
background: "hsl(var(--background))",
foreground: "hsl(var(--foreground))",
primary: {
DEFAULT: "hsl(var(--primary))",
foreground: "hsl(var(--primary-foreground))",
},
secondary: {
DEFAULT: "hsl(var(--secondary))",
foreground: "hsl(var(--secondary-foreground))",
},
destructive: {
DEFAULT: "hsl(var(--destructive))",
foreground: "hsl(var(--destructive-foreground))",
},
muted: {
DEFAULT: "hsl(var(--muted))",
foreground: "hsl(var(--muted-foreground))",
},
accent: {
DEFAULT: "hsl(var(--accent))",
foreground: "hsl(var(--accent-foreground))",
},
},
},
},
plugins: [require("tailwindcss-animate")],
};
export default config;
```
## Testing Strategy
### Component Tests
```typescript
// components/character/__tests__/character-card.test.tsx
import { render, screen } from "@testing-library/react";
import { CharacterCard } from "../character-card";
const mockCharacter = {
id: "1",
name: "Alice",
avatarUrl: "https://example.com/avatar.png",
personalityPrompt: "A curious explorer",
};
describe("CharacterCard", () => {
it("renders character name", () => {
render(<CharacterCard character={mockCharacter} />);
expect(screen.getByText("Alice")).toBeInTheDocument();
});
it("renders avatar when provided", () => {
render(<CharacterCard character={mockCharacter} />);
expect(screen.getByAltText("Alice")).toHaveAttribute(
"src",
"https://example.com/avatar.png"
);
});
});
```
### Hook Tests
```typescript
// hooks/__tests__/use-auth.test.ts
import { renderHook, act } from "@testing-library/react";
import { useAuthStore } from "@/stores/auth-store";
describe("useAuthStore", () => {
it("sets authentication state", () => {
const { result } = renderHook(() => useAuthStore());
act(() => {
result.current.setAuth(
{ id: "1", email: "test@test.com", username: "test" },
"access-token",
"refresh-token"
);
});
expect(result.current.isAuthenticated).toBe(true);
expect(result.current.user?.username).toBe("test");
});
});
```
## Environment Variables
```typescript
// apps/frontend/.env
VITE_API_URL=http://localhost:3000/api
VITE_WS_URL=ws://localhost:3000
VITE_KEYCLOAK_URL=http://localhost:8080
VITE_KEYCLOAK_REALM=dreamchat
VITE_KEYCLOAK_CLIENT_ID=dreamchat-frontend
```
## Build Configuration
```typescript
// vite.config.ts
import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";
import path from "path";
export default defineConfig({
plugins: [react()],
resolve: {
alias: {
"@": path.resolve(__dirname, "./src"),
},
},
server: {
port: 5173,
proxy: {
"/api": {
target: "http://localhost:3000",
changeOrigin: true,
},
},
},
build: {
outDir: "dist",
sourcemap: true,
},
});
```

447
doc/implementation-plan.md Normal file
View File

@@ -0,0 +1,447 @@
# DreamChat Implementation Plan
## Overview
This document outlines the phased implementation approach for DreamChat.
---
## Phase 0: Project Setup & Infrastructure
**Timeline**: Week 1
### Goals
- Set up pnpm monorepo structure
- Configure development environment with DevContainer
- Establish shared packages for WebSocket types
- Configure CI/CD pipeline
### Tasks
#### Monorepo Setup
- [ ] Initialize pnpm workspace
- [ ] Create root package.json and pnpm-workspace.yaml
- [ ] Set up shared package (`packages/shared`)
- [ ] Configure ESLint and Prettier at root
- [ ] Set up TypeScript project references
#### Backend Setup
- [ ] Initialize NestJS app in `apps/backend`
- [ ] Configure Prisma ORM with PostgreSQL
- [ ] Install and configure pgvector extension
- [ ] Define Prisma schema for all entities
- [ ] Configure Jest for unit testing
- [ ] Set up Swagger/OpenAPI
- [ ] Implement basic logging (Winston/Pino)
#### Frontend Setup
- [ ] Initialize Vite + React + TypeScript in `apps/frontend`
- [ ] Configure Tailwind CSS (or similar)
- [ ] Configure React Query / Zustand
- [ ] Set up React Router
- [ ] Configure testing (Vitest + React Testing Library)
- [ ] Link shared package (`pnpm add @dreamchat/shared@workspace:*`)
#### Infrastructure
- [ ] Create DevContainer configuration (supports pnpm monorepo)
- [ ] Create Docker Compose for development
- [ ] Set up GitHub Actions for CI (pnpm caching)
- [ ] Configure pre-commit hooks
#### Shared Types Package
- [ ] Create `packages/shared` structure
- [ ] Define WebSocket event types and payloads
- [ ] Define shared API DTOs
- [ ] Set up build configuration
- [ ] Link to apps (`pnpm install` with workspace protocol)
### Deliverables
- [ ] Running development environment via DevContainer
- [ ] Database connection established
- [ ] Basic "Hello World" API endpoint
- [ ] Frontend dev server running
---
## Phase 1: MVP - Single Character Chat
**Timeline**: Weeks 2-5
### Goals
- User authentication (Keycloak + password)
- Character CRUD with complex attributes
- Real-time chat via WebSocket
- Vector-based memory system
### 1.1 Authentication Module (Week 2)
#### Backend Tasks
- [ ] User Prisma model and repository
- [ ] Local authentication strategy (Passport)
- [ ] JWT token generation and validation
- [ ] Keycloak strategy implementation
- [ ] Auth guards and decorators
- [ ] Password hashing (bcrypt)
#### Frontend Tasks
- [ ] Login page with dual auth options
- [ ] Keycloak integration
- [ ] JWT storage and refresh logic
- [ ] Protected route wrapper
#### Testing
- [ ] Unit tests: AuthService (100% coverage)
- [ ] Unit tests: JwtStrategy, LocalStrategy
- [ ] E2E tests: Login flow
### 1.2 User Management (Week 2)
#### Backend Tasks
- [ ] User CRUD endpoints
- [ ] User profile endpoints
- [ ] Email/username validation
#### Frontend Tasks
- [ ] Registration page
- [ ] Profile management page
#### Testing
- [ ] Unit tests: UserService
- [ ] Unit tests: UserController
### 1.3 Character Module (Week 3)
#### Backend Tasks
- [ ] Character Prisma model with JSON attributes
- [ ] Character CRUD endpoints
- [ ] Character validation DTOs
- [ ] Attribute schema validation (optional)
#### Frontend Tasks
- [ ] Character list page
- [ ] Character creation form
- [ ] Character detail/editor page
- [ ] Complex attributes editor (JSON editor)
- [ ] Avatar upload
#### Testing
- [ ] Unit tests: CharacterService
- [ ] Unit tests: CharacterController
- [ ] Component tests: Character forms
### 1.4 LLM Integration (Week 3)
#### Backend Tasks
- [ ] LLM Provider interface
- [ ] OpenRouter provider implementation
- [ ] LLM configuration service
- [ ] Token counting utilities
- [ ] Error handling and retry logic
#### Testing
- [ ] Unit tests: OpenRouterProvider (mocked)
- [ ] Unit tests: LLMService
### 1.5 Vector Memory System (Week 4)
#### Backend Tasks
- [ ] Embedding service interface
- [ ] Local embedding provider (@xenova/transformers)
- [ ] Dynamic model loading
- [ ] Model caching strategy
- [ ] Dimension configuration
- [ ] HuggingFace API provider (optional/fallback)
- [ ] Vector store service (pgvector)
- [ ] LangChain integration
- [ ] Document chunking utilities
- [ ] Similarity search implementation
- [ ] Memory manager service
#### Environment Configuration
```bash
# Embedding Configuration
EMBEDDING_PROVIDER=local # or 'huggingface-api'
EMBEDDING_MODEL=Xenova/all-MiniLM-L6-v2
EMBEDDING_DIMENSION=384
EMBEDDING_DEVICE=cpu # or 'gpu' if available
# HuggingFace API (if using API instead of local)
HUGGINGFACE_API_KEY=hf_...
```
#### Testing
- [ ] Unit tests: EmbeddingService
- [ ] Unit tests: LocalEmbeddingProvider
- [ ] Unit tests: VectorStoreService
- [ ] Unit tests: MemoryManager
### 1.6 Chat Module (Week 4-5)
#### Backend Tasks
- [ ] Conversation & Message Prisma models
- [ ] ChatGateway (WebSocket)
- [ ] ChatService (business logic)
- [ ] Message streaming implementation
- [ ] Context building with memory retrieval
#### Frontend Tasks
- [ ] Chat page layout
- [ ] Message list component
- [ ] Message bubble component
- [ ] Chat input with send button
- [ ] WebSocket client integration
- [ ] Streaming message display
- [ ] Conversation sidebar
#### Testing
- [ ] Unit tests: ChatService
- [ ] Unit tests: MessageService
- [ ] Unit tests: ChatGateway
- [ ] Component tests: Chat components
- [ ] E2E tests: Full chat flow
### 1.7 Import Module - MVP (Week 5)
#### Backend Tasks
- [ ] Import adapter interface
- [ ] Text file adapter
- [ ] File upload endpoint
- [ ] Data preprocessor (cleaning)
- [ ] Basic text chunking
#### Frontend Tasks
- [ ] Import page
- [ ] File upload component (drag & drop)
- [ ] Import progress/status
#### Testing
- [ ] Unit tests: File adapters
- [ ] Unit tests: DataPreprocessor
- [ ] Unit tests: ImportController
### MVP Deliverables
- [ ] Users can register/login (Keycloak or password)
- [ ] Users can create/edit characters with complex attributes
- [ ] Users can start conversations with characters
- [ ] Real-time chat with streaming responses
- [ ] Vector-based memory for context
- [ ] Text file import for character training data
- [ ] Unit tests for all backend services
- [ ] Docker Compose for deployment
---
## Phase 2: Story Generation
**Timeline**: Weeks 6-8
### Goals
- Branching narrative generation
- Tree view visualization
- URL-based web scraping for imports
### 2.1 Story Module Backend (Week 6)
#### Backend Tasks
- [ ] StoryBranch Prisma model (tree structure with self-relation)
- [ ] Story generation service
- [ ] Branching narrative endpoints
- [ ] Open-ended generation mode
- [ ] Integration with chat history
#### Testing
- [ ] Unit tests: StoryService
- [ ] Unit tests: StoryController
### 2.2 Story Module Frontend (Week 7)
#### Frontend Tasks
- [ ] Tree view component for branches
- [ ] Story generation page
- [ ] Branch creation/editing UI
- [ ] Direction input for branching
- [ ] Story navigation controls
#### Testing
- [ ] Component tests: Tree view
- [ ] Component tests: Story page
### 2.3 Import Module - Web Scraping (Week 8)
#### Backend Tasks
- [ ] Web scraper interface
- [ ] Puppeteer/Playwright integration
- [ ] Predefined scraper: AO3
- [ ] Predefined scraper: FanFiction.net
- [ ] URL validation and scraper selection
- [ ] URL import endpoint
#### Frontend Tasks
- [ ] URL import component
- [ ] Scraper status feedback
- [ ] Content preview before import
#### Testing
- [ ] Unit tests: Web scraper adapters
- [ ] Integration tests: Scraping (mocked browser)
### Phase 2 Deliverables
- [ ] Branching story generation from conversations
- [ ] Tree view for navigating story branches
- [ ] User can input direction per branch
- [ ] URL import with predefined scrapers
- [ ] PDF and Markdown file support
---
## Phase 3: Multi-Character Group Chat
**Timeline**: Weeks 9-10
### Goals
- Multiple characters in one conversation
- Group chat UI
- Character-to-character interactions
### 3.1 Multi-Character Backend (Week 9)
#### Backend Tasks
- [ ] ConversationParticipant Prisma model
- [ ] Multi-character chat service
- [ ] Auto-response orchestration
- [ ] Character addressing detection (@Character)
- [ ] Group context management
#### Testing
- [ ] Unit tests: MultiCharacterService
- [ ] Unit tests: Participant management
### 3.2 Multi-Character Frontend (Week 10)
#### Frontend Tasks
- [ ] Group chat layout
- [ ] Participant list sidebar
- [ ] Character mention support
- [ ] Message attribution (which character)
- [ ] Add/remove participants UI
#### Testing
- [ ] Component tests: Group chat
- [ ] E2E tests: Multi-character flow
### Phase 3 Deliverables
- [ ] Group chat with multiple characters
- [ ] Characters can address each other
- [ ] Auto-response per character
- [ ] Participant management UI
---
## Phase 4: Polish & Production
**Timeline**: Week 11-12
### Goals
- Performance optimization
- Security hardening
- Documentation
- Production deployment
### 4.1 Performance & Optimization
- [ ] Database query optimization
- [ ] WebSocket connection pooling
- [ ] Frontend code splitting
- [ ] Image optimization
- [ ] Caching layer (Redis optional)
### 4.2 Security
- [ ] Security audit
- [ ] Input sanitization review
- [ ] Rate limiting implementation
- [ ] CORS configuration
- [ ] Helmet.js integration
### 4.3 Documentation
- [ ] API documentation complete
- [ ] User guide
- [ ] Deployment guide
- [ ] Development guide
### 4.4 Production Setup
- [ ] Production Docker Compose
- [ ] Environment configuration
- [ ] SSL/TLS setup
- [ ] Backup scripts
- [ ] Monitoring setup (Prometheus/Grafana optional)
### Phase 4 Deliverables
- [ ] Production-ready deployment
- [ ] Complete documentation
- [ ] Security audit passed
- [ ] Performance benchmarks met
---
## Development Guidelines
### Code Quality Standards
- **Backend Unit Test Coverage**: Minimum 80%
- **Frontend Component Tests**: Critical paths covered
- **Code Reviews**: Required for all PRs
- **Linting**: Zero warnings policy
- **TypeScript**: Strict mode enabled
### Git Workflow
```
main (production)
↓
develop (integration)
↓
feature/XXX (feature branches)
↓
PR → Code Review → Merge
```
### Commit Message Convention
```
feat: add character creation form
fix: resolve WebSocket reconnection issue
docs: update API specification
test: add unit tests for ChatService
refactor: simplify memory retrieval logic
```
### Definition of Done
- [ ] Feature implemented
- [ ] Unit tests written (backend)
- [ ] Component tests written (frontend critical paths)
- [ ] Code reviewed and approved
- [ ] Documentation updated
- [ ] No linting errors
- [ ] CI pipeline passing
---
## Risk Assessment
| Risk | Impact | Mitigation |
|------|--------|------------|
| OpenRouter API changes | High | Adapter pattern allows easy provider switch |
| Web scraping site changes | Medium | Scraper versioning, graceful degradation |
| Token limits for memory | Medium | Summarization, smart chunking |
| WebSocket scaling | Medium | Horizontal scaling with Redis adapter (future) |
| pgvector performance | Low | HNSW indexing, query optimization |
---
## Post-MVP Roadmap
### Future Enhancements
- [ ] Character templates/marketplace
- [ ] Voice input/output (TTS/STT)
- [ ] Export conversations (PDF, EPUB)
- [ ] Mobile app (React Native)
- [ ] Plugin system for custom scrapers
- [ ] AI-powered character attribute suggestions
- [ ] Collaborative story writing
- [ ] Public character sharing

664
doc/monorepo-guide.md Normal file
View File

@@ -0,0 +1,664 @@
# DreamChat Monorepo Guide
## Overview
DreamChat uses a pnpm workspace monorepo structure to share code between frontend and backend, particularly for WebSocket types and API DTOs.
## Monorepo Structure
```
dreamchat/
├── apps/
│ ├── backend/ # NestJS application
│ └── frontend/ # React + Vite application
├── packages/
│ ├── shared/ # Shared types & interfaces
│ └── config/ # Shared configurations (ESLint, TS)
├── prisma/ # Database schema (shared)
├── pnpm-workspace.yaml
└── package.json # Root package.json
```
## Root Configuration
### package.json
```json
{
"name": "dreamchat",
"version": "1.0.0",
"private": true,
"packageManager": "pnpm@8.15.0",
"scripts": {
"build": "pnpm -r build",
"dev": "pnpm -r --parallel dev",
"test": "pnpm -r test",
"lint": "pnpm -r lint",
"db:migrate": "pnpm --filter backend db:migrate",
"db:generate": "prisma generate",
"db:studio": "prisma studio",
"clean": "pnpm -r clean && rm -rf node_modules"
},
"devDependencies": {
"@types/node": "^20.0.0",
"typescript": "^5.3.0"
}
}
```
### pnpm-workspace.yaml
```yaml
packages:
- 'apps/*'
- 'packages/*'
```
### .npmrc
```
shamefully-hoist=true
auto-install-peers=true
strict-peer-dependencies=false
```
## Package Structure
### 1. Shared Package (`packages/shared`)
Contains all shared types, interfaces, and DTOs used by both frontend and backend.
#### packages/shared/package.json
```json
{
"name": "@dreamchat/shared",
"version": "1.0.0",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
"exports": {
".": {
"import": "./dist/index.js",
"require": "./dist/index.js",
"types": "./dist/index.d.ts"
},
"./websocket": {
"import": "./dist/websocket/index.js",
"types": "./dist/websocket/index.d.ts"
},
"./api": {
"import": "./dist/api/index.js",
"types": "./dist/api/index.d.ts"
}
},
"scripts": {
"build": "tsc",
"dev": "tsc --watch",
"clean": "rm -rf dist"
},
"devDependencies": {
"typescript": "^5.3.0"
}
}
```
#### packages/shared/tsconfig.json
```json
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"declaration": true,
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}
```
#### packages/shared/src/websocket/events.ts
```typescript
/**
* WebSocket Event Types
* Shared between frontend and backend for type-safe communication
*/
export enum WebSocketEventType {
// Client -> Server
JOIN_CONVERSATION = 'JOIN_CONVERSATION',
LEAVE_CONVERSATION = 'LEAVE_CONVERSATION',
SEND_MESSAGE = 'SEND_MESSAGE',
STOP_GENERATION = 'STOP_GENERATION',
// Server -> Client
CONVERSATION_JOINED = 'CONVERSATION_JOINED',
MESSAGE_ACK = 'MESSAGE_ACK',
STREAM_CHUNK = 'STREAM_CHUNK',
STREAM_COMPLETE = 'STREAM_COMPLETE',
ERROR = 'ERROR',
}
// Base message interface
export interface WebSocketMessage<T = unknown> {
type: WebSocketEventType;
payload: T;
timestamp: string;
requestId?: string;
}
// Client -> Server payloads
export interface JoinConversationPayload {
conversationId: string;
}
export interface LeaveConversationPayload {
conversationId: string;
}
export interface SendMessagePayload {
conversationId: string;
content: string;
streaming?: boolean;
}
export interface StopGenerationPayload {
conversationId: string;
}
// Server -> Client payloads
export interface ConversationJoinedPayload {
conversationId: string;
history: MessageHistoryItem[];
}
export interface MessageHistoryItem {
id: string;
role: 'user' | 'assistant' | 'system';
content: string;
createdAt: string;
}
export interface MessageAckPayload {
messageId: string;
status: 'received' | 'processing' | 'error';
}
export interface StreamChunkPayload {
conversationId: string;
chunk: string;
isComplete: boolean;
}
export interface StreamCompletePayload {
conversationId: string;
messageId: string;
content: string;
tokensUsed?: number;
model?: string;
}
export interface ErrorPayload {
code: string;
message: string;
details?: Record<string, unknown>;
}
```
#### packages/shared/src/websocket/index.ts
```typescript
export * from './events.js';
```
#### packages/shared/src/api/dto.ts
```typescript
/**
* Shared API DTOs
* Used by backend for validation and frontend for type safety
*/
// Character DTOs
export interface CreateCharacterDto {
name: string;
personalityPrompt: string;
backstory?: string;
attributes?: Record<string, unknown>;
avatarUrl?: string;
config?: Record<string, unknown>;
}
export interface UpdateCharacterDto extends Partial<CreateCharacterDto> {}
export interface CharacterResponseDto {
id: string;
name: string;
avatarUrl?: string;
personalityPrompt: string;
backstory?: string;
attributes: Record<string, unknown>;
config: Record<string, unknown>;
isPublic: boolean;
createdAt: string;
updatedAt: string;
}
// Conversation DTOs
export interface CreateConversationDto {
characterId: string;
title?: string;
}
export interface ConversationResponseDto {
id: string;
title?: string;
characterId: string;
messageCount: number;
totalTokens: number;
createdAt: string;
updatedAt: string;
}
// Message DTOs
export interface CreateMessageDto {
content: string;
}
export interface MessageResponseDto {
id: string;
role: 'user' | 'assistant' | 'system';
content: string;
tokensUsed?: number;
model?: string;
metadata?: Record<string, unknown>;
createdAt: string;
}
// Auth DTOs
export interface LoginDto {
username: string;
password: string;
}
export interface RegisterDto {
email: string;
username: string;
password: string;
}
export interface AuthResponseDto {
accessToken: string;
refreshToken: string;
expiresIn: number;
user: UserResponseDto;
}
export interface UserResponseDto {
id: string;
email: string;
username: string;
role: 'USER' | 'ADMIN';
}
```
#### packages/shared/src/api/index.ts
```typescript
export * from './dto.js';
```
#### packages/shared/src/index.ts
```typescript
export * from './websocket/index.js';
export * from './api/index.js';
```
### 2. Backend (`apps/backend`)
#### apps/backend/package.json
```json
{
"name": "@dreamchat/backend",
"version": "1.0.0",
"scripts": {
"build": "nest build",
"dev": "nest start --watch",
"start": "node dist/main",
"test": "jest",
"test:watch": "jest --watch",
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\"",
"db:migrate": "prisma migrate deploy",
"db:generate": "prisma generate",
"clean": "rm -rf dist"
},
"dependencies": {
"@dreamchat/shared": "workspace:*",
"@nestjs/common": "^10.0.0",
"@nestjs/core": "^10.0.0",
"@nestjs/platform-socket.io": "^10.0.0",
"@nestjs/websockets": "^10.0.0",
"@prisma/client": "^5.10.0",
"@xenova/transformers": "^2.15.0",
"socket.io": "^4.7.0",
"reflect-metadata": "^0.2.0",
"rxjs": "^7.8.0"
},
"devDependencies": {
"@nestjs/cli": "^10.0.0",
"@nestjs/testing": "^10.0.0",
"@types/node": "^20.0.0",
"jest": "^29.0.0",
"prisma": "^5.10.0",
"typescript": "^5.3.0"
}
}
```
#### Using Shared Package in Backend
```typescript
// apps/backend/src/chat/chat.gateway.ts
import {
WebSocketEventType,
WebSocketMessage,
JoinConversationPayload,
SendMessagePayload,
StreamChunkPayload,
ConversationJoinedPayload,
} from '@dreamchat/shared';
@WebSocketGateway({ namespace: 'chat' })
export class ChatGateway implements OnGatewayConnection {
@SubscribeMessage(WebSocketEventType.JOIN_CONVERSATION)
async handleJoin(
@MessageBody() payload: JoinConversationPayload,
@ConnectedSocket() client: Socket,
) {
// Implementation
const response: WebSocketMessage<ConversationJoinedPayload> = {
type: WebSocketEventType.CONVERSATION_JOINED,
payload: { conversationId: payload.conversationId, history: [] },
timestamp: new Date().toISOString(),
};
client.emit(response.type, response);
}
@SubscribeMessage(WebSocketEventType.SEND_MESSAGE)
async handleMessage(
@MessageBody() payload: SendMessagePayload,
@ConnectedSocket() client: Socket,
) {
// Stream chunks with type safety
const chunk: WebSocketMessage<StreamChunkPayload> = {
type: WebSocketEventType.STREAM_CHUNK,
payload: {
conversationId: payload.conversationId,
chunk: ' partial text',
isComplete: false,
},
timestamp: new Date().toISOString(),
};
client.emit(chunk.type, chunk);
}
}
```
### 3. Frontend (`apps/frontend`)
#### apps/frontend/package.json
```json
{
"name": "@dreamchat/frontend",
"version": "1.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "tsc && vite build",
"preview": "vite preview",
"test": "vitest",
"lint": "eslint . --ext ts,tsx",
"clean": "rm -rf dist"
},
"dependencies": {
"@dreamchat/shared": "workspace:*",
"@tanstack/react-query": "^5.0.0",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"react-router-dom": "^6.20.0",
"socket.io-client": "^4.7.0",
"zustand": "^4.5.0"
},
"devDependencies": {
"@types/react": "^18.2.0",
"@types/react-dom": "^18.2.0",
"@vitejs/plugin-react": "^4.2.0",
"typescript": "^5.3.0",
"vite": "^5.0.0",
"vitest": "^1.0.0"
}
}
```
#### Using Shared Package in Frontend
```typescript
// apps/frontend/src/hooks/use-chat.ts
import { useCallback } from 'react';
import { io, Socket } from 'socket.io-client';
import {
WebSocketEventType,
WebSocketMessage,
JoinConversationPayload,
SendMessagePayload,
StreamChunkPayload,
StreamCompletePayload,
} from '@dreamchat/shared';
export function useChat(conversationId: string) {
const [socket, setSocket] = useState<Socket | null>(null);
const [messages, setMessages] = useState<string[]>([]);
useEffect(() => {
const s = io('ws://localhost:3000/chat', {
auth: { token: getAuthToken() },
});
s.on(WebSocketEventType.STREAM_CHUNK, (msg: WebSocketMessage<StreamChunkPayload>) => {
setMessages((prev) => [...prev, msg.payload.chunk]);
});
s.on(WebSocketEventType.STREAM_COMPLETE, (msg: WebSocketMessage<StreamCompletePayload>) => {
// Handle completion
});
setSocket(s);
// Join conversation
const joinPayload: JoinConversationPayload = { conversationId };
s.emit(WebSocketEventType.JOIN_CONVERSATION, joinPayload);
return () => {
s.disconnect();
};
}, [conversationId]);
const sendMessage = useCallback((content: string) => {
if (!socket) return;
const payload: SendMessagePayload = {
conversationId,
content,
streaming: true,
};
socket.emit(WebSocketEventType.SEND_MESSAGE, payload);
}, [socket, conversationId]);
return { messages, sendMessage };
}
```
## Development Workflow
### Initial Setup
```bash
# 1. Install pnpm (if not already installed)
npm install -g pnpm
# 2. Clone and setup
git clone <repo-url>
cd dreamchat
# 3. Install all dependencies
pnpm install
# 4. Build shared packages first
pnpm --filter @dreamchat/shared build
# 5. Generate Prisma client
pnpm db:generate
# 6. Start development (runs all apps in parallel)
pnpm dev
```
### Adding Dependencies
```bash
# Add dependency to root
pnpm add -D typescript
# Add dependency to specific app
pnpm --filter @dreamchat/backend add @nestjs/jwt
# Add dependency to shared package
pnpm --filter @dreamchat/shared add zod
# Add shared package to backend/frontend
pnpm --filter @dreamchat/backend add @dreamchat/shared@workspace:*
```
### Building
```bash
# Build all packages
pnpm build
# Build specific package
pnpm --filter @dreamchat/shared build
# Watch mode for shared package (during development)
pnpm --filter @dreamchat/shared dev
```
### Testing
```bash
# Test all
pnpm test
# Test specific package
pnpm --filter @dreamchat/backend test
# Test with coverage
pnpm --filter @dreamchat/backend test --coverage
```
## Docker Integration
### Development Docker Compose
```yaml
# docker-compose.dev.yml
version: '3.8'
services:
backend:
build:
context: .
dockerfile: apps/backend/Dockerfile
target: development
volumes:
- .:/workspace
- /workspace/node_modules
- /workspace/apps/backend/node_modules
- /workspace/packages/shared/node_modules
environment:
- NODE_ENV=development
command: pnpm --filter @dreamchat/backend dev
frontend:
build:
context: .
dockerfile: apps/frontend/Dockerfile
target: development
volumes:
- .:/workspace
- /workspace/node_modules
- /workspace/apps/frontend/node_modules
- /workspace/packages/shared/node_modules
command: pnpm --filter @dreamchat/frontend dev
```
### Production Dockerfile (Backend)
```dockerfile
# apps/backend/Dockerfile
FROM node:20-alpine AS base
RUN npm install -g pnpm
FROM base AS dependencies
WORKDIR /app
COPY pnpm-workspace.yaml package.json ./
COPY apps/backend/package.json ./apps/backend/
COPY packages/shared/package.json ./packages/shared/
RUN pnpm install --frozen-lockfile
FROM base AS build
WORKDIR /app
COPY --from=dependencies /app/node_modules ./node_modules
COPY --from=dependencies /app/apps/backend/node_modules ./apps/backend/node_modules
COPY --from=dependencies /app/packages/shared/node_modules ./packages/shared/node_modules
COPY . .
RUN pnpm --filter @dreamchat/shared build
RUN pnpm --filter @dreamchat/backend build
FROM base AS production
WORKDIR /app
COPY --from=build /app/apps/backend/dist ./dist
COPY --from=build /app/apps/backend/node_modules ./node_modules
COPY --from=build /app/node_modules ./node_modules
EXPOSE 3000
CMD ["node", "dist/main.js"]
```
## Benefits of This Structure
1. **Type Safety**: Shared WebSocket events and DTOs ensure frontend and backend stay in sync
2. **Single Source of Truth**: Changes to types in `packages/shared` propagate to both apps
3. **Efficient Development**: pnpm workspaces deduplicate dependencies, saving disk space
4. **Simplified CI/CD**: Single repository with unified versioning
5. **Code Reuse**: Utilities and constants can be shared
## Migration from Non-Monorepo
If migrating an existing project:
1. Create new monorepo structure
2. Move backend to `apps/backend`
3. Move frontend to `apps/frontend`
4. Extract shared types to `packages/shared`
5. Update all imports to use `@dreamchat/shared`
6. Update Docker files to use pnpm