diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index b178b306bf..b1b052f90c 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -9,6 +9,8 @@ /src/main/data/ @0xfullex /src/renderer/src/data/ @0xfullex /v2-refactor-temp/ @0xfullex +/docs/en/references/data/ @0xfullex +/docs/zh/references/data/ @0xfullex /packages/ui/ @MyPrototypeWhat diff --git a/docs/en/references/data/README.md b/docs/en/references/data/README.md new file mode 100644 index 0000000000..cd247160c5 --- /dev/null +++ b/docs/en/references/data/README.md @@ -0,0 +1,193 @@ +# Data System Reference + +This is the main entry point for Cherry Studio's data management documentation. The application uses three distinct data systems based on data characteristics. + +## Quick Navigation + +### System Overview (Architecture) +- [Cache Overview](./cache-overview.md) - Three-tier caching architecture +- [Preference Overview](./preference-overview.md) - User settings management +- [DataApi Overview](./data-api-overview.md) - Business data API architecture + +### Usage Guides (Code Examples) +- [Cache Usage](./cache-usage.md) - useCache hooks, CacheService examples +- [Preference Usage](./preference-usage.md) - usePreference hook, PreferenceService examples +- [DataApi in Renderer](./data-api-in-renderer.md) - useQuery/useMutation, DataApiService +- [DataApi in Main](./data-api-in-main.md) - Handlers, Services, Repositories patterns + +### Reference Guides (Coding Standards) +- [API Design Guidelines](./api-design-guidelines.md) - RESTful design rules +- [Database Patterns](./database-patterns.md) - DB naming, schema patterns +- [API Types](./api-types.md) - API type system, schemas, error handling +- [V2 Migration Guide](./v2-migration-guide.md) - Migration system + +--- + +## Choosing the Right System + +### Quick Decision Table + +| Service | Data Characteristics | Lifecycle | Data Loss Impact | Examples | +|---------|---------------------|-----------|------------------|----------| +| **CacheService** | Regenerable, temporary | ≤ App process or survives restart | None to minimal | API responses, computed results, UI state | +| **PreferenceService** | User settings, key-value | Permanent until changed | Low (can rebuild) | Theme, language, font size, shortcuts | +| **DataApiService** | Business data, structured | Permanent | **Severe** (irreplaceable) | Topics, messages, files, knowledge base | + +### Decision Flowchart + +Ask these questions in order: + +1. **Can this data be regenerated or lost without affecting the user?** + - Yes → **CacheService** + - No → Continue to #2 + +2. **Is this a user-configurable setting that affects app behavior?** + - Yes → Does it have a fixed key and stable value structure? + - Yes → **PreferenceService** + - No (structure changes often) → **DataApiService** + - No → Continue to #3 + +3. **Is this business data created/accumulated through user activity?** + - Yes → **DataApiService** + - No → Reconsider #1 (most data falls into one of these categories) + +--- + +## System Characteristics + +### CacheService - Runtime & Cache Data + +Use CacheService when: +- Data can be **regenerated or lost without user impact** +- No backup or cross-device synchronization needed +- Lifecycle is tied to component, window, or app session + +**Two sub-categories**: +1. **Performance cache**: Computed results, API responses, expensive calculations +2. **UI state cache**: Temporary settings, scroll positions, panel states + +**Three tiers based on persistence needs**: +- `useCache` (memory): Lost on app restart, component-level sharing +- `useSharedCache` (shared): Cross-window sharing, lost on restart +- `usePersistCache` (persist): Survives app restarts via localStorage + +```typescript +// Good: Temporary computed results +const [searchResults, setSearchResults] = useCache('search.results', []) + +// Good: UI state that can be lost +const [sidebarCollapsed, setSidebarCollapsed] = useSharedCache('ui.sidebar.collapsed', false) + +// Good: Recent items (nice to have, not critical) +const [recentSearches, setRecentSearches] = usePersistCache('search.recent', []) +``` + +### PreferenceService - User Preferences + +Use PreferenceService when: +- Data is a **user-modifiable setting that affects app behavior** +- Structure is key-value with **predefined keys** (users modify values, not keys) +- **Value structure is stable** (won't change frequently) +- Data loss has **low impact** (user can reconfigure) + +**Key characteristics**: +- Auto-syncs across all windows +- Each preference item should be **atomic** (one setting = one key) +- Values are typically: boolean, string, number, or simple array/object + +```typescript +// Good: App behavior settings +const [theme, setTheme] = usePreference('app.theme.mode') +const [language, setLanguage] = usePreference('app.language') +const [fontSize, setFontSize] = usePreference('chat.message.font_size') + +// Good: Feature toggles +const [showTimestamp, setShowTimestamp] = usePreference('chat.display.show_timestamp') +``` + +### DataApiService - User Data + +Use DataApiService when: +- Data is **business data accumulated through user activity** +- Data is **structured with dedicated schemas/tables** +- Users can **create, delete, modify records** (no fixed limit) +- Data loss would be **severe and irreplaceable** +- Data volume can be **large** (potentially GBs) + +**Key characteristics**: +- No automatic window sync (fetch on demand for fresh data) +- May contain sensitive data (encryption consideration) +- Requires proper CRUD operations and transactions + +```typescript +// Good: User-generated business data +const { data: topics } = useQuery('/topics') +const { trigger: createTopic } = useMutation('/topics', 'POST') + +// Good: Conversation history (irreplaceable) +const { data: messages } = useQuery('/messages', { query: { topicId } }) + +// Good: User files and knowledge base +const { data: files } = useQuery('/files') +``` + +--- + +## Common Anti-patterns + +| Wrong Choice | Why It's Wrong | Correct Choice | +|--------------|----------------|----------------| +| Storing AI provider configs in Cache | User loses configured providers on restart | **PreferenceService** | +| Storing conversation history in Preferences | Unbounded growth, complex structure | **DataApiService** | +| Storing topic list in Preferences | User-created records, can grow large | **DataApiService** | +| Storing theme/language in DataApi | Overkill for simple key-value settings | **PreferenceService** | +| Storing API responses in DataApi | Regenerable data, doesn't need persistence | **CacheService** | +| Storing window positions in Preferences | Can be lost without impact | **CacheService** (persist tier) | + +## Edge Cases + +- **Recently used items** (e.g., recent files, recent searches): Use `usePersistCache` - nice to have but not critical if lost +- **Draft content** (e.g., unsaved message): Use `useSharedCache` for cross-window, consider auto-save to DataApi for recovery +- **Computed statistics**: Use `useCache` with TTL - regenerate when expired +- **User-created templates/presets**: Use **DataApiService** - user-generated content that can grow + +--- + +## Architecture Overview + +``` +┌─────────────────┐ +│ React Components│ +└─────────┬───────┘ + │ +┌─────────▼───────┐ +│ React Hooks │ ← useDataApi, usePreference, useCache +└─────────┬───────┘ + │ +┌─────────▼───────┐ +│ Services │ ← DataApiService, PreferenceService, CacheService +└─────────┬───────┘ + │ +┌─────────▼───────┐ +│ IPC Layer │ ← Main Process Communication +└─────────────────┘ +``` + +## Related Source Code + +### Type Definitions +- `packages/shared/data/api/` - API type system +- `packages/shared/data/cache/` - Cache type definitions +- `packages/shared/data/preference/` - Preference type definitions + +### Main Process Implementation +- `src/main/data/api/` - API server and handlers +- `src/main/data/CacheService.ts` - Cache service +- `src/main/data/PreferenceService.ts` - Preference service +- `src/main/data/db/` - Database schemas + +### Renderer Process Implementation +- `src/renderer/src/data/DataApiService.ts` - API client +- `src/renderer/src/data/CacheService.ts` - Cache service +- `src/renderer/src/data/PreferenceService.ts` - Preference service +- `src/renderer/src/data/hooks/` - React hooks diff --git a/packages/shared/data/api/api-design-guidelines.md b/docs/en/references/data/api-design-guidelines.md similarity index 100% rename from packages/shared/data/api/api-design-guidelines.md rename to docs/en/references/data/api-design-guidelines.md diff --git a/docs/en/references/data/api-types.md b/docs/en/references/data/api-types.md new file mode 100644 index 0000000000..452b3a22c0 --- /dev/null +++ b/docs/en/references/data/api-types.md @@ -0,0 +1,238 @@ +# Data API Type System + +This directory contains the type definitions and utilities for Cherry Studio's Data API system, which provides type-safe IPC communication between renderer and main processes. + +## Directory Structure + +``` +packages/shared/data/api/ +├── index.ts # Barrel export for infrastructure types +├── apiTypes.ts # Core request/response types and API utilities +├── apiPaths.ts # Path template literal type utilities +├── apiErrors.ts # Error handling: ErrorCode, DataApiError class, factory +└── schemas/ + ├── index.ts # Schema composition (merges all domain schemas) + └── test.ts # Test API schema and DTOs +``` + +## File Responsibilities + +| File | Purpose | +|------|---------| +| `apiTypes.ts` | Core types (`DataRequest`, `DataResponse`, `ApiClient`) and schema utilities | +| `apiPaths.ts` | Template literal types for path resolution (`/items/:id` → `/items/${string}`) | +| `apiErrors.ts` | `ErrorCode` enum, `DataApiError` class, `DataApiErrorFactory`, retryability config | +| `index.ts` | Unified export of infrastructure types (not domain DTOs) | +| `schemas/index.ts` | Composes all domain schemas into `ApiSchemas` using intersection types | +| `schemas/*.ts` | Domain-specific API definitions and DTOs | + +## Import Conventions + +### Infrastructure Types (via barrel export) + +Use the barrel export for common API infrastructure: + +```typescript +import type { + DataRequest, + DataResponse, + ApiClient, + PaginatedResponse +} from '@shared/data/api' + +import { + ErrorCode, + DataApiError, + DataApiErrorFactory, + isDataApiError, + toDataApiError +} from '@shared/data/api' +``` + +### Domain DTOs (directly from schema files) + +Import domain-specific types directly from their schema files: + +```typescript +// Topic domain +import type { Topic, CreateTopicDto, UpdateTopicDto } from '@shared/data/api/schemas/topic' + +// Message domain +import type { Message, CreateMessageDto } from '@shared/data/api/schemas/message' + +// Test domain (development) +import type { TestItem, CreateTestItemDto } from '@shared/data/api/schemas/test' +``` + +## Adding a New Domain Schema + +1. Create the schema file (e.g., `schemas/topic.ts`): + +```typescript +import type { PaginatedResponse } from '../apiTypes' + +// Domain models +export interface Topic { + id: string + name: string + createdAt: string +} + +export interface CreateTopicDto { + name: string +} + +// API Schema - validation happens via AssertValidSchemas in index.ts +export interface TopicSchemas { + '/topics': { + GET: { + response: PaginatedResponse // response is required + } + POST: { + body: CreateTopicDto + response: Topic + } + } + '/topics/:id': { + GET: { + params: { id: string } + response: Topic + } + } +} +``` + +**Validation**: Schemas are validated at composition level via `AssertValidSchemas` in `schemas/index.ts`: +- Ensures only valid HTTP methods (GET, POST, PUT, DELETE, PATCH) +- Requires `response` field for each endpoint +- Invalid schemas cause TypeScript errors at the composition point + +> **Design Guidelines**: Before creating new schemas, review the [API Design Guidelines](./api-design-guidelines.md) for path naming, HTTP methods, and error handling conventions. + +2. Register in `schemas/index.ts`: + +```typescript +import type { TopicSchemas } from './topic' + +// AssertValidSchemas provides fallback validation even if ValidateSchema is forgotten +export type ApiSchemas = AssertValidSchemas +``` + +3. Implement handlers in `src/main/data/api/handlers/` + +## Type Safety Features + +### Path Resolution + +The system uses template literal types to map concrete paths to schema paths: + +```typescript +// Concrete path '/topics/abc123' maps to schema path '/topics/:id' +api.get('/topics/abc123') // TypeScript knows this returns Topic +``` + +### Exhaustive Handler Checking + +`ApiImplementation` type ensures all schema endpoints have handlers: + +```typescript +// TypeScript will error if any endpoint is missing +const handlers: ApiImplementation = { + '/topics': { + GET: async () => { /* ... */ }, + POST: async ({ body }) => { /* ... */ } + } + // Missing '/topics/:id' would cause compile error +} +``` + +### Type-Safe Client + +`ApiClient` provides fully typed methods: + +```typescript +const topic = await api.get('/topics/123') // Returns Topic +const topics = await api.get('/topics', { query: { page: 1 } }) // Returns PaginatedResponse +await api.post('/topics', { body: { name: 'New' } }) // Body is typed as CreateTopicDto +``` + +## Error Handling + +The error system provides type-safe error handling with automatic retryability detection: + +```typescript +import { + DataApiError, + DataApiErrorFactory, + ErrorCode, + isDataApiError, + toDataApiError +} from '@shared/data/api' + +// Create errors using the factory (recommended) +throw DataApiErrorFactory.notFound('Topic', id) +throw DataApiErrorFactory.validation({ name: ['Name is required'] }) +throw DataApiErrorFactory.timeout('fetch topics', 3000) +throw DataApiErrorFactory.database(originalError, 'insert topic') + +// Or create directly with the class +throw new DataApiError( + ErrorCode.NOT_FOUND, + 'Topic not found', + 404, + { resource: 'Topic', id: 'abc123' } +) + +// Check if error is retryable (for automatic retry logic) +if (error instanceof DataApiError && error.isRetryable) { + await retry(operation) +} + +// Check error type +if (error instanceof DataApiError) { + if (error.isClientError) { + // 4xx - issue with the request + } else if (error.isServerError) { + // 5xx - server-side issue + } +} + +// Convert any error to DataApiError +const apiError = toDataApiError(unknownError, 'context') + +// Serialize for IPC (Main → Renderer) +const serialized = apiError.toJSON() + +// Reconstruct from IPC response (Renderer) +const reconstructed = DataApiError.fromJSON(response.error) +``` + +### Retryable Error Codes + +The following errors are automatically considered retryable: +- `SERVICE_UNAVAILABLE` (503) +- `TIMEOUT` (504) +- `RATE_LIMIT_EXCEEDED` (429) +- `DATABASE_ERROR` (500) +- `INTERNAL_SERVER_ERROR` (500) +- `RESOURCE_LOCKED` (423) + +## Architecture Overview + +``` +Renderer Main +──────────────────────────────────────────────────── +DataApiService ──IPC──► IpcAdapter ──► ApiServer + │ │ + │ ▼ + ApiClient MiddlewareEngine + (typed) │ + ▼ + Handlers + (typed) +``` + +- **Renderer**: Uses `DataApiService` with type-safe `ApiClient` interface +- **IPC**: Requests serialized via `IpcAdapter` +- **Main**: `ApiServer` routes to handlers through `MiddlewareEngine` +- **Type Safety**: End-to-end types from client call to handler implementation diff --git a/docs/en/references/data/cache-overview.md b/docs/en/references/data/cache-overview.md new file mode 100644 index 0000000000..6e28e061fb --- /dev/null +++ b/docs/en/references/data/cache-overview.md @@ -0,0 +1,125 @@ +# Cache System Overview + +The Cache system provides a three-tier caching architecture for temporary and regenerable data across the Cherry Studio application. + +## Purpose + +CacheService handles data that: +- Can be **regenerated or lost without user impact** +- Requires no backup or cross-device synchronization +- Has lifecycle tied to component, window, or app session + +## Three-Tier Architecture + +| Tier | Scope | Persistence | Use Case | +|------|-------|-------------|----------| +| **Memory Cache** | Component-level | Lost on app restart | API responses, computed results | +| **Shared Cache** | Cross-window | Lost on app restart | Window state, cross-window coordination | +| **Persist Cache** | Cross-window + localStorage | Survives app restarts | Recent items, non-critical preferences | + +### Memory Cache +- Fastest access, in-process memory +- Isolated per renderer process +- Best for: expensive computations, API response caching + +### Shared Cache +- Synchronized across all windows via IPC +- Main process acts as the source of truth +- Best for: window layouts, shared UI state + +### Persist Cache +- Backed by localStorage in renderer +- Main process maintains authoritative copy +- Best for: recent files, search history, non-critical state + +## Key Features + +### TTL (Time To Live) Support +```typescript +// Cache with 30-second expiration +cacheService.set('temp.calculation', result, 30000) +``` + +### Hook Reference Tracking +- Prevents deletion of cache entries while React hooks are subscribed +- Automatic cleanup when components unmount + +### Cross-Window Synchronization +- Shared and Persist caches sync across all windows +- Uses IPC broadcast for real-time updates +- Main process resolves conflicts + +### Type Safety +- Schema-based keys for compile-time checking +- Casual methods for dynamic keys with manual typing + +## Data Categories + +### Performance Cache (Memory tier) +- Computed results from expensive operations +- API response caching +- Parsed/transformed data + +### UI State Cache (Shared tier) +- Sidebar collapsed state +- Panel dimensions +- Scroll positions + +### Non-Critical Persistence (Persist tier) +- Recently used items +- Search history +- User-customized but regenerable data + +## Architecture Diagram + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Renderer Process │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ useCache │ │useSharedCache│ │usePersistCache│ │ +│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ +│ │ │ │ │ +│ └────────────────┼────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────┐ │ +│ │ CacheService │ │ +│ │ (Renderer) │ │ +│ └──────────┬──────────┘ │ +└─────────────────────────┼────────────────────────────────────┘ + │ IPC (shared/persist only) +┌─────────────────────────┼────────────────────────────────────┐ +│ Main Process ▼ │ +│ ┌─────────────────────┐ │ +│ │ CacheService │ │ +│ │ (Main) │ │ +│ └─────────────────────┘ │ +│ - Source of truth for shared/persist │ +│ - Broadcasts updates to all windows │ +└──────────────────────────────────────────────────────────────┘ +``` + +## Main vs Renderer Responsibilities + +### Main Process CacheService +- Manages shared and persist cache storage +- Handles IPC requests from renderers +- Broadcasts updates to all windows +- Manages TTL expiration for shared caches + +### Renderer Process CacheService +- Manages local memory cache +- Proxies shared/persist operations to Main +- Handles hook subscriptions and updates +- Local TTL management for memory cache + +## Usage Summary + +For detailed code examples and API usage, see [Cache Usage Guide](./cache-usage.md). + +| Method | Tier | Type Safety | +|--------|------|-------------| +| `useCache` / `get` / `set` | Memory | Schema-based keys | +| `getCasual` / `setCasual` | Memory | Dynamic keys (manual typing) | +| `useSharedCache` / `getShared` / `setShared` | Shared | Schema-based keys | +| `getSharedCasual` / `setSharedCasual` | Shared | Dynamic keys (manual typing) | +| `usePersistCache` / `getPersist` / `setPersist` | Persist | Schema-based keys only | diff --git a/docs/en/references/data/cache-usage.md b/docs/en/references/data/cache-usage.md new file mode 100644 index 0000000000..2dfd0f8453 --- /dev/null +++ b/docs/en/references/data/cache-usage.md @@ -0,0 +1,246 @@ +# Cache Usage Guide + +This guide covers how to use the Cache system in React components and services. + +## React Hooks + +### useCache (Memory Cache) + +Memory cache is lost on app restart. Best for temporary computed results. + +```typescript +import { useCache } from '@data/hooks/useCache' + +// Basic usage with default value +const [counter, setCounter] = useCache('ui.counter', 0) + +// Update the value +setCounter(counter + 1) + +// With TTL (30 seconds) +const [searchResults, setSearchResults] = useCache('search.results', [], { ttl: 30000 }) +``` + +### useSharedCache (Cross-Window Cache) + +Shared cache syncs across all windows, lost on app restart. + +```typescript +import { useSharedCache } from '@data/hooks/useCache' + +// Cross-window state +const [layout, setLayout] = useSharedCache('window.layout', defaultLayout) + +// Sidebar state shared between windows +const [sidebarCollapsed, setSidebarCollapsed] = useSharedCache('ui.sidebar.collapsed', false) +``` + +### usePersistCache (Persistent Cache) + +Persist cache survives app restarts via localStorage. + +```typescript +import { usePersistCache } from '@data/hooks/useCache' + +// Recent files list (survives restart) +const [recentFiles, setRecentFiles] = usePersistCache('app.recent_files', []) + +// Search history +const [searchHistory, setSearchHistory] = usePersistCache('search.history', []) +``` + +## CacheService Direct Usage + +For non-React code or more control, use CacheService directly. + +### Memory Cache + +```typescript +import { cacheService } from '@data/CacheService' + +// Type-safe (schema key) +cacheService.set('temp.calculation', result) +const result = cacheService.get('temp.calculation') + +// With TTL (30 seconds) +cacheService.set('temp.calculation', result, 30000) + +// Casual (dynamic key, manual type) +cacheService.setCasual(`topic:${id}`, topicData) +const topic = cacheService.getCasual(`topic:${id}`) + +// Check existence +if (cacheService.has('temp.calculation')) { + // ... +} + +// Delete +cacheService.delete('temp.calculation') +cacheService.deleteCasual(`topic:${id}`) +``` + +### Shared Cache + +```typescript +// Type-safe (schema key) +cacheService.setShared('window.layout', layoutConfig) +const layout = cacheService.getShared('window.layout') + +// Casual (dynamic key) +cacheService.setSharedCasual(`window:${windowId}`, state) +const state = cacheService.getSharedCasual(`window:${windowId}`) + +// Delete +cacheService.deleteShared('window.layout') +cacheService.deleteSharedCasual(`window:${windowId}`) +``` + +### Persist Cache + +```typescript +// Schema keys only (no Casual methods for persist) +cacheService.setPersist('app.recent_files', recentFiles) +const files = cacheService.getPersist('app.recent_files') + +// Delete +cacheService.deletePersist('app.recent_files') +``` + +## Type-Safe vs Casual Methods + +### Type-Safe Methods +- Use predefined keys from cache schema +- Full auto-completion and type inference +- Compile-time key validation + +```typescript +// Key 'ui.counter' must exist in schema +const [counter, setCounter] = useCache('ui.counter', 0) +``` + +### Casual Methods +- Use dynamically constructed keys +- Require manual type specification via generics +- No compile-time key validation + +```typescript +// Dynamic key, must specify type +const topic = cacheService.getCasual(`topic:${id}`) +``` + +### When to Use Which + +| Scenario | Method | Example | +|----------|--------|---------| +| Fixed cache keys | Type-safe | `useCache('ui.counter')` | +| Entity caching by ID | Casual | `getCasual(\`topic:${id}\`)` | +| Session-based keys | Casual | `setCasual(\`session:${sessionId}\`)` | +| UI state | Type-safe | `useSharedCache('window.layout')` | + +## Common Patterns + +### Caching Expensive Computations + +```typescript +function useExpensiveData(input: string) { + const [cached, setCached] = useCache(`computed:${input}`, null) + + useEffect(() => { + if (cached === null) { + const result = expensiveComputation(input) + setCached(result) + } + }, [input, cached, setCached]) + + return cached +} +``` + +### Cross-Window Coordination + +```typescript +// Window A: Update shared state +const [activeFile, setActiveFile] = useSharedCache('editor.activeFile', null) +setActiveFile(selectedFile) + +// Window B: Reacts to change automatically +const [activeFile] = useSharedCache('editor.activeFile', null) +// activeFile updates when Window A changes it +``` + +### Recent Items with Limit + +```typescript +const [recentItems, setRecentItems] = usePersistCache('app.recentItems', []) + +const addRecentItem = (item: Item) => { + setRecentItems(prev => { + const filtered = prev.filter(i => i.id !== item.id) + return [item, ...filtered].slice(0, 10) // Keep last 10 + }) +} +``` + +### Cache with Expiration Check + +```typescript +interface CachedData { + data: T + timestamp: number +} + +function useCachedWithExpiry(key: string, fetcher: () => Promise, maxAge: number) { + const [cached, setCached] = useCache | null>(key, null) + const [data, setData] = useState(cached?.data ?? null) + + useEffect(() => { + const isExpired = !cached || Date.now() - cached.timestamp > maxAge + + if (isExpired) { + fetcher().then(result => { + setCached({ data: result, timestamp: Date.now() }) + setData(result) + }) + } + }, [key, maxAge]) + + return data +} +``` + +## Adding New Cache Keys + +### 1. Add to Cache Schema + +```typescript +// packages/shared/data/cache/cacheSchemas.ts +export interface CacheSchema { + // Existing keys... + 'myFeature.data': MyDataType +} +``` + +### 2. Define Value Type (if complex) + +```typescript +// packages/shared/data/cache/cacheValueTypes.ts +export interface MyDataType { + items: string[] + lastUpdated: number +} +``` + +### 3. Use in Code + +```typescript +// Now type-safe +const [data, setData] = useCache('myFeature.data', defaultValue) +``` + +## Best Practices + +1. **Choose the right tier**: Memory for temp, Shared for cross-window, Persist for survival +2. **Use TTL for stale data**: Prevent serving outdated cached values +3. **Prefer type-safe keys**: Add to schema when possible +4. **Clean up dynamic keys**: Remove casual cache entries when no longer needed +5. **Consider data size**: Persist cache uses localStorage (limited to ~5MB) diff --git a/docs/en/references/data/data-api-in-main.md b/docs/en/references/data/data-api-in-main.md new file mode 100644 index 0000000000..90ca93ad0d --- /dev/null +++ b/docs/en/references/data/data-api-in-main.md @@ -0,0 +1,360 @@ +# DataApi in Main Process + +This guide covers how to implement API handlers, services, and repositories in the Main process. + +## Architecture Layers + +``` +Handlers → Services → Repositories → Database +``` + +- **Handlers**: Thin layer, extract params, call service, transform response +- **Services**: Business logic, validation, transaction coordination +- **Repositories**: Data access (for complex domains) +- **Database**: Drizzle ORM + SQLite + +## Implementing Handlers + +### Location +`src/main/data/api/handlers/` + +### Handler Responsibilities +- Extract parameters from request +- Delegate to business service +- Transform response for IPC +- **NO business logic here** + +### Example Handler + +```typescript +// handlers/topic.ts +import type { ApiImplementation } from '@shared/data/api' +import { TopicService } from '@data/services/TopicService' + +export const topicHandlers: Partial = { + '/topics': { + GET: async ({ query }) => { + const { page = 1, limit = 20 } = query ?? {} + return await TopicService.getInstance().list({ page, limit }) + }, + POST: async ({ body }) => { + return await TopicService.getInstance().create(body) + } + }, + '/topics/:id': { + GET: async ({ params }) => { + return await TopicService.getInstance().getById(params.id) + }, + PUT: async ({ params, body }) => { + return await TopicService.getInstance().replace(params.id, body) + }, + PATCH: async ({ params, body }) => { + return await TopicService.getInstance().update(params.id, body) + }, + DELETE: async ({ params }) => { + await TopicService.getInstance().delete(params.id) + } + } +} +``` + +### Register Handlers + +```typescript +// handlers/index.ts +import { topicHandlers } from './topic' +import { messageHandlers } from './message' + +export const allHandlers: ApiImplementation = { + ...topicHandlers, + ...messageHandlers +} +``` + +## Implementing Services + +### Location +`src/main/data/services/` + +### Service Responsibilities +- Business validation +- Transaction coordination +- Domain workflows +- Call repositories or direct Drizzle + +### Example Service + +```typescript +// services/TopicService.ts +import { DbService } from '@data/db/DbService' +import { TopicRepository } from '@data/repositories/TopicRepository' +import { DataApiErrorFactory } from '@shared/data/api' + +export class TopicService { + private static instance: TopicService + private topicRepo: TopicRepository + + private constructor() { + this.topicRepo = new TopicRepository() + } + + static getInstance(): TopicService { + if (!this.instance) { + this.instance = new TopicService() + } + return this.instance + } + + async list(options: { page: number; limit: number }) { + return await this.topicRepo.findAll(options) + } + + async getById(id: string) { + const topic = await this.topicRepo.findById(id) + if (!topic) { + throw DataApiErrorFactory.notFound('Topic', id) + } + return topic + } + + async create(data: CreateTopicDto) { + // Business validation + this.validateTopicData(data) + + return await this.topicRepo.create(data) + } + + async update(id: string, data: Partial) { + const existing = await this.getById(id) // Throws if not found + + return await this.topicRepo.update(id, data) + } + + async delete(id: string) { + await this.getById(id) // Throws if not found + await this.topicRepo.delete(id) + } + + private validateTopicData(data: CreateTopicDto) { + if (!data.name?.trim()) { + throw DataApiErrorFactory.validation({ name: ['Name is required'] }) + } + } +} +``` + +### Service with Transaction + +```typescript +async createTopicWithMessage(data: CreateTopicWithMessageDto) { + return await DbService.transaction(async (tx) => { + // Create topic + const topic = await this.topicRepo.create(data.topic, tx) + + // Create initial message + const message = await this.messageRepo.create({ + ...data.message, + topicId: topic.id + }, tx) + + return { topic, message } + }) +} +``` + +## Implementing Repositories + +### When to Use Repository Pattern + +Use repositories for **complex domains**: +- ✅ Complex queries (joins, subqueries, aggregations) +- ✅ GB-scale data requiring pagination +- ✅ Complex transactions involving multiple tables +- ✅ Reusable data access patterns +- ✅ High testing requirements + +### When to Use Direct Drizzle + +Use direct Drizzle for **simple domains**: +- ✅ Simple CRUD operations +- ✅ Small datasets (< 100MB) +- ✅ Domain-specific queries with no reuse +- ✅ Fast development is priority + +### Example Repository + +```typescript +// repositories/TopicRepository.ts +import { eq, desc, sql } from 'drizzle-orm' +import { DbService } from '@data/db/DbService' +import { topicTable } from '@data/db/schemas/topic' + +export class TopicRepository { + async findAll(options: { page: number; limit: number }) { + const { page, limit } = options + const offset = (page - 1) * limit + + const [items, countResult] = await Promise.all([ + DbService.db + .select() + .from(topicTable) + .orderBy(desc(topicTable.updatedAt)) + .limit(limit) + .offset(offset), + DbService.db + .select({ count: sql`count(*)` }) + .from(topicTable) + ]) + + return { + items, + total: countResult[0].count, + page, + limit + } + } + + async findById(id: string, tx?: Transaction) { + const db = tx || DbService.db + const [topic] = await db + .select() + .from(topicTable) + .where(eq(topicTable.id, id)) + .limit(1) + return topic ?? null + } + + async create(data: CreateTopicDto, tx?: Transaction) { + const db = tx || DbService.db + const [topic] = await db + .insert(topicTable) + .values(data) + .returning() + return topic + } + + async update(id: string, data: Partial, tx?: Transaction) { + const db = tx || DbService.db + const [topic] = await db + .update(topicTable) + .set(data) + .where(eq(topicTable.id, id)) + .returning() + return topic + } + + async delete(id: string, tx?: Transaction) { + const db = tx || DbService.db + await db + .delete(topicTable) + .where(eq(topicTable.id, id)) + } +} +``` + +### Example: Direct Drizzle in Service + +For simple domains, skip the repository: + +```typescript +// services/TagService.ts +import { eq } from 'drizzle-orm' +import { DbService } from '@data/db/DbService' +import { tagTable } from '@data/db/schemas/tag' + +export class TagService { + async getAll() { + return await DbService.db.select().from(tagTable) + } + + async create(name: string) { + const [tag] = await DbService.db + .insert(tagTable) + .values({ name }) + .returning() + return tag + } + + async delete(id: string) { + await DbService.db + .delete(tagTable) + .where(eq(tagTable.id, id)) + } +} +``` + +## Error Handling + +### Using DataApiErrorFactory + +```typescript +import { DataApiErrorFactory } from '@shared/data/api' + +// Not found +throw DataApiErrorFactory.notFound('Topic', id) + +// Validation error +throw DataApiErrorFactory.validation({ + name: ['Name is required', 'Name must be at least 3 characters'], + email: ['Invalid email format'] +}) + +// Database error +try { + await db.insert(table).values(data) +} catch (error) { + throw DataApiErrorFactory.database(error, 'insert topic') +} + +// Invalid operation +throw DataApiErrorFactory.invalidOperation( + 'delete root message', + 'cascade=true required' +) + +// Conflict +throw DataApiErrorFactory.conflict('Topic name already exists') + +// Timeout +throw DataApiErrorFactory.timeout('fetch topics', 3000) +``` + +## Adding New Endpoints + +### Step-by-Step + +1. **Define schema** in `packages/shared/data/api/schemas/` + +```typescript +// schemas/topic.ts +export interface TopicSchemas { + '/topics': { + GET: { response: PaginatedResponse } + POST: { body: CreateTopicDto; response: Topic } + } +} +``` + +2. **Register schema** in `schemas/index.ts` + +```typescript +export type ApiSchemas = AssertValidSchemas +``` + +3. **Create service** in `services/` + +4. **Create repository** (if complex) in `repositories/` + +5. **Implement handler** in `handlers/` + +6. **Register handler** in `handlers/index.ts` + +## Best Practices + +1. **Keep handlers thin**: Only extract params and call services +2. **Put logic in services**: All business rules belong in services +3. **Use repositories selectively**: Simple CRUD doesn't need a repository +4. **Always use `.returning()`**: Get inserted/updated data without re-querying +5. **Support transactions**: Accept optional `tx` parameter in repositories +6. **Validate in services**: Business validation belongs in the service layer +7. **Use error factory**: Consistent error creation with `DataApiErrorFactory` diff --git a/docs/en/references/data/data-api-in-renderer.md b/docs/en/references/data/data-api-in-renderer.md new file mode 100644 index 0000000000..abe79ca8c1 --- /dev/null +++ b/docs/en/references/data/data-api-in-renderer.md @@ -0,0 +1,298 @@ +# DataApi in Renderer + +This guide covers how to use the DataApi system in React components and the renderer process. + +## React Hooks + +### useQuery (GET Requests) + +Fetch data with automatic caching and revalidation via SWR. + +```typescript +import { useQuery } from '@data/hooks/useDataApi' + +// Basic usage +const { data, loading, error } = useQuery('/topics') + +// With query parameters +const { data: messages } = useQuery('/messages', { + query: { topicId: 'abc123', page: 1, limit: 20 } +}) + +// With path parameters (inferred from path) +const { data: topic } = useQuery('/topics/abc123') + +// Conditional fetching +const { data } = useQuery(topicId ? `/topics/${topicId}` : null) + +// With refresh callback +const { data, mutate } = useQuery('/topics') +// Refresh data +await mutate() +``` + +### useMutation (POST/PUT/PATCH/DELETE) + +Perform data modifications with loading states. + +```typescript +import { useMutation } from '@data/hooks/useDataApi' + +// Create (POST) +const { trigger: createTopic, isMutating } = useMutation('/topics', 'POST') +const newTopic = await createTopic({ body: { name: 'New Topic' } }) + +// Update (PUT - full replacement) +const { trigger: replaceTopic } = useMutation('/topics/abc123', 'PUT') +await replaceTopic({ body: { name: 'Updated Name', description: '...' } }) + +// Partial Update (PATCH) +const { trigger: updateTopic } = useMutation('/topics/abc123', 'PATCH') +await updateTopic({ body: { name: 'New Name' } }) + +// Delete +const { trigger: deleteTopic } = useMutation('/topics/abc123', 'DELETE') +await deleteTopic() +``` + +## DataApiService Direct Usage + +For non-React code or more control. + +```typescript +import { dataApiService } from '@data/DataApiService' + +// GET request +const topics = await dataApiService.get('/topics') +const topic = await dataApiService.get('/topics/abc123') +const messages = await dataApiService.get('/topics/abc123/messages', { + query: { page: 1, limit: 20 } +}) + +// POST request +const newTopic = await dataApiService.post('/topics', { + body: { name: 'New Topic' } +}) + +// PUT request (full replacement) +const updatedTopic = await dataApiService.put('/topics/abc123', { + body: { name: 'Updated', description: 'Full update' } +}) + +// PATCH request (partial update) +const patchedTopic = await dataApiService.patch('/topics/abc123', { + body: { name: 'Just update name' } +}) + +// DELETE request +await dataApiService.delete('/topics/abc123') +``` + +## Error Handling + +### With Hooks + +```typescript +function TopicList() { + const { data, loading, error } = useQuery('/topics') + + if (loading) return + if (error) { + if (error.code === ErrorCode.NOT_FOUND) { + return + } + return + } + + return +} +``` + +### With Try-Catch + +```typescript +import { DataApiError, ErrorCode } from '@shared/data/api' + +try { + await dataApiService.post('/topics', { body: data }) +} catch (error) { + if (error instanceof DataApiError) { + switch (error.code) { + case ErrorCode.VALIDATION_ERROR: + // Handle validation errors + const fieldErrors = error.details?.fieldErrors + break + case ErrorCode.NOT_FOUND: + // Handle not found + break + case ErrorCode.CONFLICT: + // Handle conflict + break + default: + // Handle other errors + } + } +} +``` + +### Retryable Errors + +```typescript +if (error instanceof DataApiError && error.isRetryable) { + // Safe to retry: SERVICE_UNAVAILABLE, TIMEOUT, etc. + await retry(operation) +} +``` + +## Common Patterns + +### List with Pagination + +```typescript +function TopicListWithPagination() { + const [page, setPage] = useState(1) + const { data, loading } = useQuery('/topics', { + query: { page, limit: 20 } + }) + + return ( + <> + + + + ) +} +``` + +### Create Form + +```typescript +function CreateTopicForm() { + const { trigger: createTopic, isMutating } = useMutation('/topics', 'POST') + const { mutate } = useQuery('/topics') // For revalidation + + const handleSubmit = async (data: CreateTopicDto) => { + try { + await createTopic({ body: data }) + await mutate() // Refresh list + toast.success('Topic created') + } catch (error) { + toast.error('Failed to create topic') + } + } + + return ( +
+ {/* form fields */} + +
+ ) +} +``` + +### Optimistic Updates + +```typescript +function TopicItem({ topic }: { topic: Topic }) { + const { trigger: updateTopic } = useMutation(`/topics/${topic.id}`, 'PATCH') + const { mutate } = useQuery('/topics') + + const handleToggleStar = async () => { + // Optimistically update the cache + await mutate( + current => ({ + ...current, + items: current.items.map(t => + t.id === topic.id ? { ...t, starred: !t.starred } : t + ) + }), + { revalidate: false } + ) + + try { + await updateTopic({ body: { starred: !topic.starred } }) + } catch (error) { + // Revert on failure + await mutate() + toast.error('Failed to update') + } + } + + return ( +
+ {topic.name} + +
+ ) +} +``` + +### Dependent Queries + +```typescript +function MessageList({ topicId }: { topicId: string }) { + // First query: get topic + const { data: topic } = useQuery(`/topics/${topicId}`) + + // Second query: depends on first (only runs when topic exists) + const { data: messages } = useQuery( + topic ? `/topics/${topicId}/messages` : null + ) + + if (!topic) return + + return ( +
+

{topic.name}

+ +
+ ) +} +``` + +### Polling for Updates + +```typescript +function LiveTopicList() { + const { data } = useQuery('/topics', { + refreshInterval: 5000 // Poll every 5 seconds + }) + + return +} +``` + +## Type Safety + +The API is fully typed based on schema definitions: + +```typescript +// Types are inferred from schema +const { data } = useQuery('/topics') +// data is typed as PaginatedResponse + +const { trigger } = useMutation('/topics', 'POST') +// trigger expects { body: CreateTopicDto } +// returns Topic + +// Path parameters are type-checked +const { data: topic } = useQuery('/topics/abc123') +// TypeScript knows this returns Topic +``` + +## Best Practices + +1. **Use hooks for components**: `useQuery` and `useMutation` handle loading/error states +2. **Handle loading states**: Always show feedback while data is loading +3. **Handle errors gracefully**: Provide meaningful error messages to users +4. **Revalidate after mutations**: Keep the UI in sync with the database +5. **Use conditional fetching**: Pass `null` to skip queries when dependencies aren't ready +6. **Batch related operations**: Consider using transactions for multiple updates diff --git a/docs/en/references/data/data-api-overview.md b/docs/en/references/data/data-api-overview.md new file mode 100644 index 0000000000..883bd57f49 --- /dev/null +++ b/docs/en/references/data/data-api-overview.md @@ -0,0 +1,158 @@ +# DataApi System Overview + +The DataApi system provides type-safe IPC communication for business data operations between the Renderer and Main processes. + +## Purpose + +DataApiService handles data that: +- Is **business data accumulated through user activity** +- Has **dedicated database schemas/tables** +- Users can **create, delete, modify records** without fixed limits +- Would be **severe and irreplaceable** if lost +- Can grow to **large volumes** (potentially GBs) + +## Key Characteristics + +### Type-Safe Communication +- End-to-end TypeScript types from client call to handler +- Path parameter inference from route definitions +- Compile-time validation of request/response shapes + +### RESTful-Style API +- Familiar HTTP semantics (GET, POST, PUT, PATCH, DELETE) +- Resource-based URL patterns (`/topics/:id/messages`) +- Standard status codes and error responses + +### On-Demand Data Access +- No automatic caching (fetch fresh data when needed) +- Explicit cache control via query options +- Supports large datasets with pagination + +## Architecture Diagram + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Renderer Process │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ React Components │ │ +│ │ - useQuery('/topics') │ │ +│ │ - useMutation('/topics', 'POST') │ │ +│ └──────────────────────────┬──────────────────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ DataApiService (Renderer) │ │ +│ │ - Type-safe ApiClient interface │ │ +│ │ - Request serialization │ │ +│ │ - Automatic retry with exponential backoff │ │ +│ │ - Error handling and transformation │ │ +│ └──────────────────────────┬──────────────────────────────┘ │ +└──────────────────────────────┼───────────────────────────────────┘ + │ IPC +┌──────────────────────────────┼───────────────────────────────────┐ +│ Main Process ▼ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ IpcAdapter │ │ +│ │ - Receives IPC requests │ │ +│ │ - Routes to ApiServer │ │ +│ └──────────────────────────┬──────────────────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ ApiServer │ │ +│ │ - Request routing by path and method │ │ +│ │ - Middleware pipeline processing │ │ +│ └──────────────────────────┬──────────────────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Handlers (api/handlers/) │ │ +│ │ - Thin layer: extract params, call service, transform │ │ +│ │ - NO business logic here │ │ +│ └──────────────────────────┬──────────────────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Services (services/) │ │ +│ │ - Business logic and validation │ │ +│ │ - Transaction coordination │ │ +│ │ - Domain workflows │ │ +│ └──────────────────────────┬──────────────────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────┴─────────────────────┐ │ +│ ▼ ▼ │ +│ ┌───────────────┐ ┌─────────────────────┐ │ +│ │ Repositories │ │ Direct Drizzle │ │ +│ │ (Complex) │ │ (Simple domains) │ │ +│ │ - Query logic │ │ - Inline queries │ │ +│ └───────┬───────┘ └──────────┬──────────┘ │ +│ │ │ │ +│ └────────────────────┬───────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ SQLite Database (via Drizzle ORM) │ │ +│ │ - topic, message, file tables │ │ +│ │ - Full-text search indexes │ │ +│ └─────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────┘ +``` + +## Four-Layer Architecture + +### 1. API Layer (Handlers) +- **Location**: `src/main/data/api/handlers/` +- **Responsibility**: HTTP-like interface layer +- **Does**: Extract parameters, call services, transform responses +- **Does NOT**: Contain business logic + +### 2. Business Logic Layer (Services) +- **Location**: `src/main/data/services/` +- **Responsibility**: Domain logic and workflows +- **Does**: Validation, transaction coordination, orchestration +- **Uses**: Repositories or direct Drizzle queries + +### 3. Data Access Layer (Repositories) +- **Location**: `src/main/data/repositories/` +- **Responsibility**: Complex data operations +- **When to use**: Complex queries, large datasets, reusable patterns +- **Alternative**: Direct Drizzle for simple CRUD + +### 4. Database Layer +- **Location**: `src/main/data/db/` +- **Technology**: SQLite + Drizzle ORM +- **Schemas**: `db/schemas/` directory + +## Data Access Pattern Decision + +### Use Repository Pattern When: +- ✅ Complex queries (joins, subqueries, aggregations) +- ✅ GB-scale data requiring optimization and pagination +- ✅ Complex transactions involving multiple tables +- ✅ Reusable data access patterns across services +- ✅ High testing requirements (mock data access) + +### Use Direct Drizzle When: +- ✅ Simple CRUD operations +- ✅ Small datasets (< 100MB) +- ✅ Domain-specific queries with no reuse potential +- ✅ Fast development is priority + +## Key Features + +### Automatic Retry +- Exponential backoff for transient failures +- Configurable retry count and delays +- Skips retry for client errors (4xx) + +### Error Handling +- Typed error codes (`ErrorCode` enum) +- `DataApiError` class with retryability detection +- Factory methods for consistent error creation + +### Request Timeout +- Configurable per-request timeouts +- Automatic cancellation of stale requests + +## Usage Summary + +For detailed code examples, see: +- [DataApi in Renderer](./data-api-in-renderer.md) - Client-side usage +- [DataApi in Main](./data-api-in-main.md) - Server-side implementation +- [API Design Guidelines](./api-design-guidelines.md) - RESTful conventions +- [API Types](./api-types.md) - Type system details diff --git a/docs/en/references/data/database-patterns.md b/docs/en/references/data/database-patterns.md new file mode 100644 index 0000000000..10d3a44593 --- /dev/null +++ b/docs/en/references/data/database-patterns.md @@ -0,0 +1,199 @@ +# Database Schema Guidelines + +## Naming Conventions + +- **Table names**: Use **singular** form with snake_case (e.g., `topic`, `message`, `app_state`) +- **Export names**: Use `xxxTable` pattern (e.g., `topicTable`, `messageTable`) +- **Column names**: Drizzle auto-infers from property names, no need to specify explicitly + +## Column Helpers + +All helpers are exported from `./schemas/columnHelpers.ts`. + +### Primary Keys + +| Helper | UUID Version | Use Case | +|--------|--------------|----------| +| `uuidPrimaryKey()` | v4 (random) | General purpose tables | +| `uuidPrimaryKeyOrdered()` | v7 (time-ordered) | Large tables with time-based queries | + +**Usage:** + +```typescript +import { uuidPrimaryKey, uuidPrimaryKeyOrdered } from './columnHelpers' + +// General purpose table +export const topicTable = sqliteTable('topic', { + id: uuidPrimaryKey(), + name: text(), + ... +}) + +// Large table with time-ordered data +export const messageTable = sqliteTable('message', { + id: uuidPrimaryKeyOrdered(), + content: text(), + ... +}) +``` + +**Behavior:** + +- ID is auto-generated if not provided during insert +- Can be manually specified for migration scenarios +- Use `.returning()` to get the generated ID after insert + +### Timestamps + +| Helper | Fields | Use Case | +|--------|--------|----------| +| `createUpdateTimestamps` | `createdAt`, `updatedAt` | Tables without soft delete | +| `createUpdateDeleteTimestamps` | `createdAt`, `updatedAt`, `deletedAt` | Tables with soft delete | + +**Usage:** + +```typescript +import { createUpdateTimestamps, createUpdateDeleteTimestamps } from './columnHelpers' + +// Without soft delete +export const tagTable = sqliteTable('tag', { + id: uuidPrimaryKey(), + name: text(), + ...createUpdateTimestamps +}) + +// With soft delete +export const topicTable = sqliteTable('topic', { + id: uuidPrimaryKey(), + name: text(), + ...createUpdateDeleteTimestamps +}) +``` + +**Behavior:** + +- `createdAt`: Auto-set to `Date.now()` on insert +- `updatedAt`: Auto-set on insert, auto-updated on update +- `deletedAt`: `null` by default, set to timestamp for soft delete + +## JSON Fields + +For JSON column support, use `{ mode: 'json' }`: + +```typescript +data: text({ mode: 'json' }).$type() +``` + +Drizzle handles JSON serialization/deserialization automatically. + +## Foreign Keys + +### Basic Usage + +```typescript +// SET NULL: preserve record when referenced record is deleted +groupId: text().references(() => groupTable.id, { onDelete: 'set null' }) + +// CASCADE: delete record when referenced record is deleted +topicId: text().references(() => topicTable.id, { onDelete: 'cascade' }) +``` + +### Self-Referencing Foreign Keys + +For self-referencing foreign keys (e.g., tree structures with parentId), **always use the `foreignKey` operator** in the table's third parameter: + +```typescript +import { foreignKey, sqliteTable, text } from 'drizzle-orm/sqlite-core' + +export const messageTable = sqliteTable( + 'message', + { + id: uuidPrimaryKeyOrdered(), + parentId: text(), // Do NOT use .references() here + // ...other fields + }, + (t) => [ + // Use foreignKey operator for self-referencing + foreignKey({ columns: [t.parentId], foreignColumns: [t.id] }).onDelete('set null') + ] +) +``` + +**Why this approach:** +- Avoids TypeScript circular reference issues (no need for `AnySQLiteColumn` type annotation) +- More explicit and readable +- Allows chaining `.onDelete()` / `.onUpdate()` actions + +### Circular Foreign Key References + +**Avoid circular foreign key references between tables.** For example: + +```typescript +// ❌ BAD: Circular FK between tables +// tableA.currentItemId -> tableB.id +// tableB.ownerId -> tableA.id +``` + +If you encounter a scenario that seems to require circular references: + +1. **Identify which relationship is "weaker"** - typically the one that can be null or is less critical for data integrity +2. **Remove the FK constraint from the weaker side** - let the application layer handle validation and consistency (this is known as "soft references" pattern) +3. **Document the application-layer constraint** in code comments + +```typescript +// ✅ GOOD: Break the cycle by handling one side at application layer +export const topicTable = sqliteTable('topic', { + id: uuidPrimaryKey(), + // Application-managed reference (no FK constraint) + // Validated by TopicService.setCurrentMessage() + currentMessageId: text(), +}) + +export const messageTable = sqliteTable('message', { + id: uuidPrimaryKeyOrdered(), + // Database-enforced FK + topicId: text().references(() => topicTable.id, { onDelete: 'cascade' }), +}) +``` + +**Why soft references for SQLite:** +- SQLite does not support `DEFERRABLE` constraints (unlike PostgreSQL/Oracle) +- Application-layer validation provides equivalent data integrity +- Simplifies insert/update operations without transaction ordering concerns + +## Migrations + +Generate migrations after schema changes: + +```bash +yarn db:migrations:generate +``` + +## Field Generation Rules + +The schema uses Drizzle's auto-generation features. Follow these rules: + +### Auto-generated fields (NEVER set manually) + +- `id`: Uses `$defaultFn()` with UUID v4/v7, auto-generated on insert +- `createdAt`: Uses `$defaultFn()` with `Date.now()`, auto-generated on insert +- `updatedAt`: Uses `$defaultFn()` and `$onUpdateFn()`, auto-updated on every update + +### Using `.returning()` pattern + +Always use `.returning()` to get inserted/updated data instead of re-querying: + +```typescript +// Good: Use returning() +const [row] = await db.insert(table).values(data).returning() +return rowToEntity(row) + +// Avoid: Re-query after insert (unnecessary database round-trip) +await db.insert(table).values({ id, ...data }) +return this.getById(id) +``` + +### Soft delete support + +The schema supports soft delete via `deletedAt` field (see `createUpdateDeleteTimestamps`). +Business logic can choose to use soft delete or hard delete based on requirements. diff --git a/docs/en/references/data/preference-overview.md b/docs/en/references/data/preference-overview.md new file mode 100644 index 0000000000..755571c659 --- /dev/null +++ b/docs/en/references/data/preference-overview.md @@ -0,0 +1,144 @@ +# Preference System Overview + +The Preference system provides centralized management for user configuration and application settings with cross-window synchronization. + +## Purpose + +PreferenceService handles data that: +- Is a **user-modifiable setting that affects app behavior** +- Has a **fixed key structure** with stable value types +- Needs to **persist permanently** until explicitly changed +- Should **sync automatically** across all application windows + +## Key Characteristics + +### Fixed Key Structure +- Predefined keys in the schema (users modify values, not keys) +- Supports 158 configuration items +- Nested key paths supported (e.g., `app.theme.mode`) + +### Atomic Values +- Each preference item represents one logical setting +- Values are typically: boolean, string, number, or simple array/object +- Changes are independent (updating one doesn't affect others) + +### Cross-Window Synchronization +- Changes automatically broadcast to all windows +- Consistent state across main window, mini window, etc. +- Conflict resolution handled by Main process + +## Update Strategies + +### Optimistic Updates (Default) +```typescript +// UI updates immediately, then syncs to database +await preferenceService.set('app.theme.mode', 'dark') +``` +- Best for: frequent, non-critical settings +- Behavior: Local state updates first, then persists +- Rollback: Automatic revert if persistence fails + +### Pessimistic Updates +```typescript +// Waits for database confirmation before updating UI +await preferenceService.set('api.key', 'secret', { optimistic: false }) +``` +- Best for: critical settings (API keys, security options) +- Behavior: Persists first, then updates local state +- No rollback needed: UI only updates on success + +## Architecture Diagram + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Renderer Process │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ usePreference Hook │ │ +│ │ - Subscribe to preference changes │ │ +│ │ - Optimistic/pessimistic update support │ │ +│ └──────────────────────┬──────────────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ PreferenceService (Renderer) │ │ +│ │ - Local cache for fast reads │ │ +│ │ - IPC proxy to Main process │ │ +│ │ - Subscription management │ │ +│ └──────────────────────┬──────────────────────────┘ │ +└─────────────────────────┼────────────────────────────────────┘ + │ IPC +┌─────────────────────────┼────────────────────────────────────┐ +│ Main Process ▼ │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ PreferenceService (Main) │ │ +│ │ - Full memory cache of all preferences │ │ +│ │ - SQLite persistence via Drizzle ORM │ │ +│ │ - Cross-window broadcast │ │ +│ └──────────────────────┬──────────────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ SQLite Database (preference table) │ │ +│ │ - scope + key structure │ │ +│ │ - JSON value storage │ │ +│ └─────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────┘ +``` + +## Main vs Renderer Responsibilities + +### Main Process PreferenceService +- **Source of truth** for all preferences +- Full memory cache for fast access +- SQLite persistence via preference table +- Broadcasts changes to all renderer windows +- Handles batch operations and transactions + +### Renderer Process PreferenceService +- Local cache for read performance +- Proxies write operations to Main +- Manages React hook subscriptions +- Handles optimistic update rollbacks +- Listens for cross-window updates + +## Database Schema + +Preferences are stored in the `preference` table: + +```typescript +// Simplified schema +{ + scope: string // e.g., 'default', 'user' + key: string // e.g., 'app.theme.mode' + value: json // The preference value + createdAt: number + updatedAt: number +} +``` + +## Preference Categories + +### Application Settings +- Theme mode, language, font sizes +- Window behavior, startup options + +### Feature Toggles +- Show/hide UI elements +- Enable/disable features + +### User Customization +- Keyboard shortcuts +- Default values for operations + +### Provider Configuration +- AI provider settings +- API endpoints and tokens + +## Usage Summary + +For detailed code examples and API usage, see [Preference Usage Guide](./preference-usage.md). + +| Operation | Hook | Service Method | +|-----------|------|----------------| +| Read single | `usePreference(key)` | `preferenceService.get(key)` | +| Write single | `setPreference(value)` | `preferenceService.set(key, value)` | +| Read multiple | `usePreferences([...keys])` | `preferenceService.getMultiple([...keys])` | +| Write multiple | - | `preferenceService.setMultiple({...})` | diff --git a/docs/en/references/data/preference-usage.md b/docs/en/references/data/preference-usage.md new file mode 100644 index 0000000000..70a0586724 --- /dev/null +++ b/docs/en/references/data/preference-usage.md @@ -0,0 +1,260 @@ +# Preference Usage Guide + +This guide covers how to use the Preference system in React components and services. + +## React Hooks + +### usePreference (Single Preference) + +```typescript +import { usePreference } from '@data/hooks/usePreference' + +// Basic usage - optimistic updates (default) +const [theme, setTheme] = usePreference('app.theme.mode') + +// Update the value +await setTheme('dark') + +// With pessimistic updates (wait for confirmation) +const [apiKey, setApiKey] = usePreference('api.key', { optimistic: false }) +``` + +### usePreferences (Multiple Preferences) + +```typescript +import { usePreferences } from '@data/hooks/usePreference' + +// Read multiple preferences at once +const { theme, language, fontSize } = usePreferences([ + 'app.theme.mode', + 'app.language', + 'chat.message.font_size' +]) +``` + +## Update Strategies + +### Optimistic Updates (Default) + +UI updates immediately, then syncs to database. Automatic rollback on failure. + +```typescript +const [theme, setTheme] = usePreference('app.theme.mode') + +const handleThemeChange = async (newTheme: string) => { + try { + await setTheme(newTheme) // UI updates immediately + } catch (error) { + // UI automatically rolls back + console.error('Theme update failed:', error) + } +} +``` + +**Best for:** +- Frequent changes (theme, font size) +- Non-critical settings +- Better perceived performance + +### Pessimistic Updates + +Waits for database confirmation before updating UI. + +```typescript +const [apiKey, setApiKey] = usePreference('api.key', { optimistic: false }) + +const handleApiKeyChange = async (newKey: string) => { + try { + await setApiKey(newKey) // Waits for DB confirmation + toast.success('API key saved') + } catch (error) { + toast.error('Failed to save API key') + } +} +``` + +**Best for:** +- Security-sensitive settings (API keys, passwords) +- Settings that affect external services +- When confirmation feedback is important + +## PreferenceService Direct Usage + +For non-React code or batch operations. + +### Get Preferences + +```typescript +import { preferenceService } from '@data/PreferenceService' + +// Get single preference +const theme = await preferenceService.get('app.theme.mode') + +// Get multiple preferences +const settings = await preferenceService.getMultiple([ + 'app.theme.mode', + 'app.language' +]) +// Returns: { 'app.theme.mode': 'dark', 'app.language': 'en' } + +// Get with default value +const fontSize = await preferenceService.get('chat.message.font_size') ?? 14 +``` + +### Set Preferences + +```typescript +// Set single preference (optimistic by default) +await preferenceService.set('app.theme.mode', 'dark') + +// Set with pessimistic update +await preferenceService.set('api.key', 'secret', { optimistic: false }) + +// Set multiple preferences at once +await preferenceService.setMultiple({ + 'app.theme.mode': 'dark', + 'app.language': 'en', + 'chat.message.font_size': 16 +}) +``` + +### Subscribe to Changes + +```typescript +// Subscribe to preference changes (useful in services) +const unsubscribe = preferenceService.subscribe('app.theme.mode', (newValue) => { + console.log('Theme changed to:', newValue) +}) + +// Cleanup when done +unsubscribe() +``` + +## Common Patterns + +### Settings Form + +```typescript +function SettingsForm() { + const [theme, setTheme] = usePreference('app.theme.mode') + const [language, setLanguage] = usePreference('app.language') + const [fontSize, setFontSize] = usePreference('chat.message.font_size') + + return ( +
+ + + + + setFontSize(Number(e.target.value))} + min={12} + max={24} + /> +
+ ) +} +``` + +### Feature Toggle + +```typescript +function ChatMessage({ message }) { + const [showTimestamp] = usePreference('chat.display.show_timestamp') + + return ( +
+

{message.content}

+ {showTimestamp && {message.createdAt}} +
+ ) +} +``` + +### Conditional Rendering Based on Settings + +```typescript +function App() { + const [theme] = usePreference('app.theme.mode') + const [sidebarPosition] = usePreference('app.sidebar.position') + + return ( +
+ {sidebarPosition === 'left' && } + + {sidebarPosition === 'right' && } +
+ ) +} +``` + +### Batch Settings Update + +```typescript +async function resetToDefaults() { + await preferenceService.setMultiple({ + 'app.theme.mode': 'system', + 'app.language': 'en', + 'chat.message.font_size': 14, + 'chat.display.show_timestamp': true + }) +} +``` + +## Adding New Preference Keys + +### 1. Add to Preference Schema + +```typescript +// packages/shared/data/preference/preferenceSchemas.ts +export interface PreferenceSchema { + // Existing keys... + 'myFeature.enabled': boolean + 'myFeature.options': MyFeatureOptions +} +``` + +### 2. Set Default Value + +```typescript +// Same file or separate defaults file +export const preferenceDefaults: Partial = { + // Existing defaults... + 'myFeature.enabled': true, + 'myFeature.options': { mode: 'auto', limit: 100 } +} +``` + +### 3. Use in Code + +```typescript +// Now type-safe with auto-completion +const [enabled, setEnabled] = usePreference('myFeature.enabled') +``` + +## Best Practices + +1. **Choose update strategy wisely**: Optimistic for UX, pessimistic for critical settings +2. **Batch related updates**: Use `setMultiple` when changing multiple related settings +3. **Provide sensible defaults**: All preferences should have default values +4. **Keep values atomic**: One preference = one logical setting +5. **Use consistent naming**: Follow `domain.feature.setting` pattern + +## Preference vs Other Storage + +| Scenario | Use | +|----------|-----| +| User theme preference | `usePreference('app.theme.mode')` | +| Window position | `usePersistCache` (can be lost without impact) | +| API key | `usePreference` with pessimistic updates | +| Search history | `usePersistCache` (nice to have) | +| Conversation history | `DataApiService` (business data) | diff --git a/docs/en/references/data/v2-migration-guide.md b/docs/en/references/data/v2-migration-guide.md new file mode 100644 index 0000000000..86d597223e --- /dev/null +++ b/docs/en/references/data/v2-migration-guide.md @@ -0,0 +1,64 @@ +# Migration V2 (Main Process) + +Architecture for the new one-shot migration from the legacy Dexie + Redux Persist stores into the SQLite schema. This module owns orchestration, data access helpers, migrator plugins, and IPC entry points used by the renderer migration window. + +## Directory Layout + +``` +src/main/data/migration/v2/ +├── core/ # Engine + shared context +├── migrators/ # Domain-specific migrators and mappings +├── utils/ # Data source readers (Redux, Dexie, streaming JSON) +├── window/ # IPC handlers + migration window manager +└── index.ts # Public exports for main process +``` + +## Core Contracts + +- `core/MigrationEngine.ts` coordinates all migrators in order, surfaces progress to the UI, and marks status in `app_state.key = 'migration_v2_status'`. It will clear new-schema tables before running and abort on any validation failure. +- `core/MigrationContext.ts` builds the shared context passed to every migrator: + - `sources`: `ConfigManager` (ElectronStore), `ReduxStateReader` (parsed Redux Persist data), `DexieFileReader` (JSON exports) + - `db`: current SQLite connection + - `sharedData`: `Map` for passing cross-cutting info between migrators + - `logger`: `loggerService` scoped to migration +- `@shared/data/migration/v2/types` defines stages, results, and validation stats used across main and renderer. + +## Migrators + +- Base contract: extend `migrators/BaseMigrator.ts` and implement: + - `id`, `name`, `description`, `order` (lower runs first) + - `prepare(ctx)`: dry-run checks, counts, and staging data; return `PrepareResult` + - `execute(ctx)`: perform inserts/updates; manage your own transactions; report progress via `reportProgress` + - `validate(ctx)`: verify counts and integrity; return `ValidateResult` with stats (`sourceCount`, `targetCount`, `skippedCount`) and any `errors` +- Registration: list migrators (in order) in `migrators/index.ts` so the engine can sort and run them. +- Current migrators: + - `PreferencesMigrator` (implemented): maps ElectronStore + Redux settings to the `preference` table using `mappings/PreferencesMappings.ts`. + - `AssistantMigrator`, `KnowledgeMigrator`, `ChatMigrator` (placeholders): scaffolding and TODO notes for future tables. +- Conventions: + - All logging goes through `loggerService` with a migrator-specific context. + - Use `MigrationContext.sources` instead of accessing raw files/stores directly. + - Use `sharedData` to pass IDs or lookup tables between migrators (e.g., assistant -> chat references) instead of re-reading sources. + - Stream large Dexie exports (`JSONStreamReader`) and batch inserts to avoid memory spikes. + - Count validation is mandatory; engine will fail the run if `targetCount < sourceCount - skippedCount` or if `ValidateResult.errors` is non-empty. + - Keep migrations idempotent per run—engine clears target tables before it starts, but each migrator should tolerate retries within the same run. + +## Utilities + +- `utils/ReduxStateReader.ts`: safe accessor for categorized Redux Persist data with dot-path lookup. +- `utils/DexieFileReader.ts`: reads exported Dexie JSON tables; can stream large tables. +- `utils/JSONStreamReader.ts`: streaming reader with batching, counting, and sampling helpers for very large arrays. + +## Window & IPC Integration + +- `window/MigrationIpcHandler.ts` exposes IPC channels for the migration UI: + - Receives Redux data and Dexie export path, starts the engine, and streams progress back to renderer. + - Manages backup flow (dialogs via `BackupManager`) and retry/cancel/restart actions. +- `window/MigrationWindowManager.ts` creates the frameless migration window, handles lifecycle, and relaunch instructions after completion in production. + +## Implementation Checklist for New Migrators + +- [ ] Add mapping definitions (if needed) under `migrators/mappings/`. +- [ ] Implement `prepare/execute/validate` with explicit counts, batch inserts, and integrity checks. +- [ ] Wire progress updates through `reportProgress` so UI shows per-migrator progress. +- [ ] Register the migrator in `migrators/index.ts` with the correct `order`. +- [ ] Add any new target tables to `MigrationEngine.verifyAndClearNewTables` once those tables exist. diff --git a/packages/shared/data/README.md b/packages/shared/data/README.md index 9428522575..30d30ff54d 100644 --- a/packages/shared/data/README.md +++ b/packages/shared/data/README.md @@ -1,124 +1,50 @@ -# Cherry Studio Shared Data +# Shared Data Types -This directory contains shared type definitions and schemas for the Cherry Studio data management systems. These files provide type safety and consistency across the entire application. +This directory contains shared type definitions for Cherry Studio's data layer. + +## Documentation + +For comprehensive documentation, see: +- **Overview**: [docs/en/references/data/README.md](../../../docs/en/references/data/README.md) +- **Cache Types**: [cache-overview.md](../../../docs/en/references/data/cache-overview.md) +- **Preference Types**: [preference-overview.md](../../../docs/en/references/data/preference-overview.md) +- **API Types**: [api-types.md](../../../docs/en/references/data/api-types.md) ## Directory Structure ``` packages/shared/data/ -├── api/ # Data API type system (see api/README.md) -│ ├── index.ts # Barrel exports for infrastructure types -│ ├── apiTypes.ts # Core request/response types and utilities -│ ├── apiPaths.ts # Path template literal type utilities -│ ├── errorCodes.ts # Error handling utilities -│ ├── schemas/ # Domain-specific API schemas -│ │ ├── index.ts # Schema composition -│ │ └── test.ts # Test API schema and DTOs -│ └── README.md # Detailed API documentation +├── api/ # Data API type system +│ ├── index.ts # Barrel exports +│ ├── apiTypes.ts # Core request/response types +│ ├── apiPaths.ts # Path template utilities +│ ├── apiErrors.ts # Error handling +│ └── schemas/ # Domain-specific API schemas ├── cache/ # Cache system type definitions -│ ├── cacheTypes.ts # Core cache infrastructure types -│ ├── cacheSchemas.ts # Cache key schemas and type mappings -│ └── cacheValueTypes.ts # Cache value type definitions +│ ├── cacheTypes.ts # Core cache types +│ ├── cacheSchemas.ts # Cache key schemas +│ └── cacheValueTypes.ts # Cache value types ├── preference/ # Preference system type definitions -│ ├── preferenceTypes.ts # Core preference system types -│ └── preferenceSchemas.ts # Preference schemas and default values -├── types/ # Shared data types for Main/Renderer -└── README.md # This file +│ ├── preferenceTypes.ts # Core preference types +│ └── preferenceSchemas.ts # Preference schemas +└── types/ # Shared data types ``` -## System Overview +## Quick Reference -This directory provides type definitions for four main data management systems: +### Import Conventions -### Types System (`types/`) -- **Purpose**: Shared data types for cross-process (Main/Renderer) communication and database schemas -- **Features**: Database table field types, business entity definitions -- **Usage**: Used in Drizzle ORM schemas via `.$type()` and runtime type checking - -### API System (`api/`) -- **Purpose**: Type-safe IPC communication between Main and Renderer processes -- **Features**: RESTful patterns, modular schema design, error handling -- **Documentation**: See [`api/README.md`](./api/README.md) for detailed usage - -### Cache System (`cache/`) -- **Purpose**: Type definitions for three-layer caching architecture -- **Features**: Memory/shared/persist cache schemas, TTL support, hook integration -- **Usage**: Type-safe caching operations across the application - -### Preference System (`preference/`) -- **Purpose**: User configuration and settings management -- **Features**: 158 configuration items, default values, nested key support -- **Usage**: Type-safe preference access and synchronization - -## File Categories - -**Framework Infrastructure** - These are TypeScript type definitions that: -- ✅ Exist only at compile time -- ✅ Provide type safety and IntelliSense support -- ✅ Define contracts between application layers -- ✅ Enable static analysis and error detection - -## Usage Examples - -### API Types ```typescript -// Infrastructure types from barrel export +// API infrastructure types (from barrel) import type { DataRequest, DataResponse, ApiClient } from '@shared/data/api' -import { DataApiErrorFactory, ErrorCode } from '@shared/data/api' +import { ErrorCode, DataApiError, DataApiErrorFactory } from '@shared/data/api' -// Domain DTOs directly from schema files -import type { TestItem, CreateTestItemDto } from '@shared/data/api/schemas/test' -``` +// Domain DTOs (from schema files) +import type { Topic, CreateTopicDto } from '@shared/data/api/schemas/topic' -### Cache Types -```typescript -// Import cache types +// Cache types import type { UseCacheKey, UseSharedCacheKey } from '@shared/data/cache' + +// Preference types +import type { PreferenceKeyType } from '@shared/data/preference' ``` - -### Preference Types -```typescript -// Import preference types -import type { PreferenceKeyType, PreferenceDefaultScopeType } from '@shared/data/preference' -``` - -## Development Guidelines - -### Adding Shared Types -1. Create or update type file in `types/` directory -2. Use camelCase for field names -3. Reference types in Drizzle schemas using `.$type()` - -### Adding Cache Types -1. Add cache key to `cache/cacheSchemas.ts` -2. Define value type in `cache/cacheValueTypes.ts` -3. Update type mappings for type safety - -### Adding Preference Types -1. Add preference key to `preference/preferenceSchemas.ts` -2. Define default value and type -3. Preference system automatically picks up new keys - -### Adding API Types -1. Create schema file in `api/schemas/` (e.g., `topic.ts`) -2. Define domain models, DTOs, and API schema in the file -3. Register schema in `api/schemas/index.ts` using intersection type -4. See [`api/README.md`](./api/README.md) for detailed guide - -### Best Practices -- Use `import type` for type-only imports -- Infrastructure types from barrel, domain DTOs from schema files -- Follow existing naming conventions -- Document complex types with JSDoc - -## Related Implementation - -### Main Process -- `src/main/data/api/` - API server, handlers, and IPC adapter -- `src/main/data/cache/` - Cache service implementation -- `src/main/data/preference/` - Preference service implementation - -### Renderer Process -- `src/renderer/src/services/DataApiService.ts` - API client -- `src/renderer/src/services/CacheService.ts` - Cache service -- `src/renderer/src/services/PreferenceService.ts` - Preference service \ No newline at end of file diff --git a/packages/shared/data/api/README.md b/packages/shared/data/api/README.md index 452b3a22c0..eb06824d87 100644 --- a/packages/shared/data/api/README.md +++ b/packages/shared/data/api/README.md @@ -1,238 +1,42 @@ # Data API Type System -This directory contains the type definitions and utilities for Cherry Studio's Data API system, which provides type-safe IPC communication between renderer and main processes. +This directory contains type definitions for the DataApi system. + +## Documentation + +- **DataApi Overview**: [docs/en/references/data/data-api-overview.md](../../../../docs/en/references/data/data-api-overview.md) +- **API Types**: [api-types.md](../../../../docs/en/references/data/api-types.md) +- **API Design Guidelines**: [api-design-guidelines.md](../../../../docs/en/references/data/api-design-guidelines.md) ## Directory Structure ``` packages/shared/data/api/ -├── index.ts # Barrel export for infrastructure types -├── apiTypes.ts # Core request/response types and API utilities -├── apiPaths.ts # Path template literal type utilities -├── apiErrors.ts # Error handling: ErrorCode, DataApiError class, factory +├── index.ts # Barrel exports +├── apiTypes.ts # Core request/response types +├── apiPaths.ts # Path template utilities +├── apiErrors.ts # Error handling └── schemas/ - ├── index.ts # Schema composition (merges all domain schemas) - └── test.ts # Test API schema and DTOs + ├── index.ts # Schema composition + └── *.ts # Domain-specific schemas ``` -## File Responsibilities +## Quick Reference -| File | Purpose | -|------|---------| -| `apiTypes.ts` | Core types (`DataRequest`, `DataResponse`, `ApiClient`) and schema utilities | -| `apiPaths.ts` | Template literal types for path resolution (`/items/:id` → `/items/${string}`) | -| `apiErrors.ts` | `ErrorCode` enum, `DataApiError` class, `DataApiErrorFactory`, retryability config | -| `index.ts` | Unified export of infrastructure types (not domain DTOs) | -| `schemas/index.ts` | Composes all domain schemas into `ApiSchemas` using intersection types | -| `schemas/*.ts` | Domain-specific API definitions and DTOs | - -## Import Conventions - -### Infrastructure Types (via barrel export) - -Use the barrel export for common API infrastructure: +### Import Conventions ```typescript -import type { - DataRequest, - DataResponse, - ApiClient, - PaginatedResponse -} from '@shared/data/api' +// Infrastructure types (via barrel) +import type { DataRequest, DataResponse, ApiClient } from '@shared/data/api' +import { ErrorCode, DataApiError, DataApiErrorFactory } from '@shared/data/api' -import { - ErrorCode, - DataApiError, - DataApiErrorFactory, - isDataApiError, - toDataApiError -} from '@shared/data/api' -``` - -### Domain DTOs (directly from schema files) - -Import domain-specific types directly from their schema files: - -```typescript -// Topic domain -import type { Topic, CreateTopicDto, UpdateTopicDto } from '@shared/data/api/schemas/topic' - -// Message domain +// Domain DTOs (directly from schema files) +import type { Topic, CreateTopicDto } from '@shared/data/api/schemas/topic' import type { Message, CreateMessageDto } from '@shared/data/api/schemas/message' - -// Test domain (development) -import type { TestItem, CreateTestItemDto } from '@shared/data/api/schemas/test' ``` -## Adding a New Domain Schema - -1. Create the schema file (e.g., `schemas/topic.ts`): - -```typescript -import type { PaginatedResponse } from '../apiTypes' - -// Domain models -export interface Topic { - id: string - name: string - createdAt: string -} - -export interface CreateTopicDto { - name: string -} - -// API Schema - validation happens via AssertValidSchemas in index.ts -export interface TopicSchemas { - '/topics': { - GET: { - response: PaginatedResponse // response is required - } - POST: { - body: CreateTopicDto - response: Topic - } - } - '/topics/:id': { - GET: { - params: { id: string } - response: Topic - } - } -} -``` - -**Validation**: Schemas are validated at composition level via `AssertValidSchemas` in `schemas/index.ts`: -- Ensures only valid HTTP methods (GET, POST, PUT, DELETE, PATCH) -- Requires `response` field for each endpoint -- Invalid schemas cause TypeScript errors at the composition point - -> **Design Guidelines**: Before creating new schemas, review the [API Design Guidelines](./api-design-guidelines.md) for path naming, HTTP methods, and error handling conventions. - -2. Register in `schemas/index.ts`: - -```typescript -import type { TopicSchemas } from './topic' - -// AssertValidSchemas provides fallback validation even if ValidateSchema is forgotten -export type ApiSchemas = AssertValidSchemas -``` +### Adding New Schemas +1. Create schema file in `schemas/` (e.g., `topic.ts`) +2. Register in `schemas/index.ts` using intersection type 3. Implement handlers in `src/main/data/api/handlers/` - -## Type Safety Features - -### Path Resolution - -The system uses template literal types to map concrete paths to schema paths: - -```typescript -// Concrete path '/topics/abc123' maps to schema path '/topics/:id' -api.get('/topics/abc123') // TypeScript knows this returns Topic -``` - -### Exhaustive Handler Checking - -`ApiImplementation` type ensures all schema endpoints have handlers: - -```typescript -// TypeScript will error if any endpoint is missing -const handlers: ApiImplementation = { - '/topics': { - GET: async () => { /* ... */ }, - POST: async ({ body }) => { /* ... */ } - } - // Missing '/topics/:id' would cause compile error -} -``` - -### Type-Safe Client - -`ApiClient` provides fully typed methods: - -```typescript -const topic = await api.get('/topics/123') // Returns Topic -const topics = await api.get('/topics', { query: { page: 1 } }) // Returns PaginatedResponse -await api.post('/topics', { body: { name: 'New' } }) // Body is typed as CreateTopicDto -``` - -## Error Handling - -The error system provides type-safe error handling with automatic retryability detection: - -```typescript -import { - DataApiError, - DataApiErrorFactory, - ErrorCode, - isDataApiError, - toDataApiError -} from '@shared/data/api' - -// Create errors using the factory (recommended) -throw DataApiErrorFactory.notFound('Topic', id) -throw DataApiErrorFactory.validation({ name: ['Name is required'] }) -throw DataApiErrorFactory.timeout('fetch topics', 3000) -throw DataApiErrorFactory.database(originalError, 'insert topic') - -// Or create directly with the class -throw new DataApiError( - ErrorCode.NOT_FOUND, - 'Topic not found', - 404, - { resource: 'Topic', id: 'abc123' } -) - -// Check if error is retryable (for automatic retry logic) -if (error instanceof DataApiError && error.isRetryable) { - await retry(operation) -} - -// Check error type -if (error instanceof DataApiError) { - if (error.isClientError) { - // 4xx - issue with the request - } else if (error.isServerError) { - // 5xx - server-side issue - } -} - -// Convert any error to DataApiError -const apiError = toDataApiError(unknownError, 'context') - -// Serialize for IPC (Main → Renderer) -const serialized = apiError.toJSON() - -// Reconstruct from IPC response (Renderer) -const reconstructed = DataApiError.fromJSON(response.error) -``` - -### Retryable Error Codes - -The following errors are automatically considered retryable: -- `SERVICE_UNAVAILABLE` (503) -- `TIMEOUT` (504) -- `RATE_LIMIT_EXCEEDED` (429) -- `DATABASE_ERROR` (500) -- `INTERNAL_SERVER_ERROR` (500) -- `RESOURCE_LOCKED` (423) - -## Architecture Overview - -``` -Renderer Main -──────────────────────────────────────────────────── -DataApiService ──IPC──► IpcAdapter ──► ApiServer - │ │ - │ ▼ - ApiClient MiddlewareEngine - (typed) │ - ▼ - Handlers - (typed) -``` - -- **Renderer**: Uses `DataApiService` with type-safe `ApiClient` interface -- **IPC**: Requests serialized via `IpcAdapter` -- **Main**: `ApiServer` routes to handlers through `MiddlewareEngine` -- **Type Safety**: End-to-end types from client call to handler implementation diff --git a/src/main/data/README.md b/src/main/data/README.md index dde7f8188e..e596b87434 100644 --- a/src/main/data/README.md +++ b/src/main/data/README.md @@ -1,405 +1,44 @@ # Main Data Layer -This directory contains the main process data management system, providing unified data access for the entire application. +This directory contains the main process data management implementation. + +## Documentation + +- **Overview**: [docs/en/references/data/README.md](../../../docs/en/references/data/README.md) +- **DataApi in Main**: [data-api-in-main.md](../../../docs/en/references/data/data-api-in-main.md) +- **Database Patterns**: [database-patterns.md](../../../docs/en/references/data/database-patterns.md) ## Directory Structure ``` src/main/data/ -├── api/ # Data API framework (interface layer) -│ ├── core/ # Core API infrastructure -│ │ ├── ApiServer.ts # Request routing and handler execution -│ │ ├── MiddlewareEngine.ts # Request/response middleware -│ │ └── adapters/ # Communication adapters (IPC) -│ ├── handlers/ # API endpoint implementations -│ │ ├── index.ts # Handler aggregation and exports -│ │ └── test.ts # Test endpoint handlers -│ └── index.ts # API framework exports -│ +├── api/ # Data API framework +│ ├── core/ # ApiServer, MiddlewareEngine, adapters +│ └── handlers/ # API endpoint implementations ├── services/ # Business logic layer -│ ├── base/ # Service base classes and interfaces -│ │ └── IBaseService.ts # Service interface definitions -│ └── TestService.ts # Test service (placeholder for real services) -│ # Future business services: -│ # - TopicService.ts # Topic business logic -│ # - MessageService.ts # Message business logic -│ # - FileService.ts # File business logic -│ ├── repositories/ # Data access layer (selective usage) -│ # Repository pattern used selectively for complex domains -│ # Future repositories: -│ # - TopicRepository.ts # Complex: Topic data access -│ # - MessageRepository.ts # Complex: Message data access -│ -├── db/ # Database layer -│ ├── schemas/ # Drizzle table definitions -│ │ ├── preference.ts # Preference configuration table -│ │ ├── appState.ts # Application state table -│ │ ├── topic.ts # Topic/conversation table -│ │ ├── message.ts # Message table -│ │ ├── group.ts # Group table -│ │ ├── tag.ts # Tag table -│ │ ├── entityTag.ts # Entity-tag relationship table -│ │ ├── messageFts.ts # Message full-text search table -│ │ └── columnHelpers.ts # Reusable column definitions -│ ├── seeding/ # Database initialization -│ └── DbService.ts # Database connection and management -│ -├── migration/ # Data migration system -│ └── v2/ # v2 data refactoring migration tools -│ -├── CacheService.ts # Infrastructure: Cache management -├── DataApiService.ts # Infrastructure: API coordination -└── PreferenceService.ts # System service: User preferences +├── db/ # Database layer +│ ├── schemas/ # Drizzle table definitions +│ ├── seeding/ # Database initialization +│ └── DbService.ts # Database connection management +├── migration/ # Data migration system +├── CacheService.ts # Cache management +├── DataApiService.ts # API coordination +└── PreferenceService.ts # User preferences ``` -## Core Components - -### Naming Note - -Three components at the root of `data/` use the "Service" suffix but serve different purposes: - -#### CacheService (Infrastructure Component) -- **True Nature**: Cache Manager / Infrastructure Utility -- **Purpose**: Multi-tier caching system (memory/shared/persist) -- **Features**: TTL support, IPC synchronization, cross-window broadcasting -- **Characteristics**: Zero business logic, purely technical functionality -- **Note**: Named "Service" for management consistency, but is actually infrastructure - -#### DataApiService (Coordinator Component) -- **True Nature**: API Coordinator (Main) / API Client (Renderer) -- **Main Process Purpose**: Coordinates ApiServer and IpcAdapter initialization -- **Renderer Purpose**: HTTP-like client for IPC communication -- **Characteristics**: Zero business logic, purely coordination/communication plumbing -- **Note**: Named "Service" for management consistency, but is actually coordinator/client - -#### PreferenceService (System Service) -- **True Nature**: System-level Data Access Service -- **Purpose**: User configuration management with caching and multi-window sync -- **Features**: SQLite persistence, full memory cache, cross-window synchronization -- **Characteristics**: Minimal business logic (validation, defaults), primarily data access -- **Note**: Hybrid between data access and infrastructure, "Service" naming is acceptable - -**Key Takeaway**: Despite all being named "Service", these are infrastructure/coordination components, not business services. The "Service" suffix is kept for consistency with existing codebase conventions. - -## Architecture Layers - -### API Framework Layer (`api/`) - -The API framework provides the interface layer for data access: - -#### API Server (`api/core/ApiServer.ts`) -- Request routing and handler execution -- Middleware pipeline processing -- Type-safe endpoint definitions - -#### Handlers (`api/handlers/`) -- **Purpose**: Thin API endpoint implementations -- **Responsibilities**: - - HTTP-like parameter extraction from requests - - DTO/domain model conversion - - Delegating to business services - - Transforming responses for IPC -- **Anti-pattern**: Do NOT put business logic in handlers -- **Currently**: Contains test handlers (business handlers pending) -- **Type Safety**: Must implement all endpoints defined in `@shared/data/api/schemas/` - -### Business Logic Layer (`services/`) - -Business services implement domain logic and workflows: - -#### When to Create a Service -- Contains business rules and validation -- Orchestrates multiple repositories or data sources -- Implements complex workflows -- Manages transactions across multiple operations - -#### Service Pattern - -Just an example for understanding. - -```typescript -// services/TopicService.ts -export class TopicService { - constructor( - private topicRepo: TopicRepository, // Use repository for complex data access - private cacheService: CacheService // Use infrastructure utilities - ) {} - - async createTopicWithMessage(data: CreateTopicDto) { - // Business validation - this.validateTopicData(data) - - // Transaction coordination - return await DbService.transaction(async (tx) => { - const topic = await this.topicRepo.create(data.topic, tx) - const message = await this.messageRepo.create(data.message, tx) - return { topic, message } - }) - } -} -``` - -#### Current Services -- `TestService`: Placeholder service for testing API framework -- More business services will be added as needed (TopicService, MessageService, etc.) - -### Data Access Layer (`repositories/`) - -Repositories handle database operations with a **selective usage pattern**: - -#### When to Use Repository Pattern -Use repositories for **complex domains** that meet multiple criteria: -- ✅ Complex queries (joins, subqueries, aggregations) -- ✅ GB-scale data requiring optimization and pagination -- ✅ Complex transactions involving multiple tables -- ✅ Reusable data access patterns across services -- ✅ High testing requirements (mock data access in tests) - -#### When to Use Direct Drizzle in Services -Skip repository layer for **simple domains**: -- ✅ Simple CRUD operations -- ✅ Small datasets (< 100MB) -- ✅ Domain-specific queries with no reuse potential -- ✅ Fast development is priority - -#### Repository Pattern - -Just an example for understanding. - -```typescript -// repositories/TopicRepository.ts -export class TopicRepository { - async findById(id: string, tx?: Transaction): Promise { - const db = tx || DbService.db - return await db.select() - .from(topicTable) - .where(eq(topicTable.id, id)) - .limit(1) - } - - async findByIdWithMessages( - topicId: string, - pagination: PaginationOptions - ): Promise { - // Complex join query with pagination - // Handles GB-scale data efficiently - } -} -``` - -#### Direct Drizzle Pattern (Simple Services) -```typescript -// services/SimpleService.ts -export class SimpleService extends BaseService { - async getItem(id: string) { - // Direct Drizzle query for simple operations - return await this.database - .select() - .from(itemTable) - .where(eq(itemTable.id, id)) - } -} -``` - -#### Planned Repositories -- **TopicRepository**: Complex topic data access with message relationships -- **MessageRepository**: GB-scale message queries with pagination -- **FileRepository**: File reference counting and cleanup logic - -**Decision Principle**: Use the simplest approach that solves the problem. Add repository abstraction only when complexity demands it. - -## Database Layer - -### DbService -- SQLite database connection management -- Automatic migrations and seeding -- Drizzle ORM integration - -### Schemas (`db/schemas/`) -- Table definitions using Drizzle ORM -- Follow naming convention: `{entity}Table` exports -- Use `crudTimestamps` helper for timestamp fields -- See [db/README.md](./db/README.md#field-generation-rules) for detailed field generation rules and `.returning()` pattern - -### Current Tables -- `preference`: User configuration storage -- `appState`: Application state persistence -- `topic`: Conversation/topic storage -- `message`: Message storage with full-text search -- `group`: Group organization -- `tag`: Tag definitions -- `entityTag`: Entity-tag relationships -- `messageFts`: Message full-text search index - -## Usage Examples - -### Accessing Services -```typescript -// Get service instances -import { cacheService } from '@/data/CacheService' -import { preferenceService } from '@/data/PreferenceService' -import { dataApiService } from '@/data/DataApiService' - -// Services are singletons, initialized at app startup -``` +## Quick Reference ### Adding New API Endpoints -1. Create or update schema in `@shared/data/api/schemas/` (see `@shared/data/api/README.md`) -2. Register schema in `@shared/data/api/schemas/index.ts` -3. Implement handler in `api/handlers/` (thin layer, delegate to service) -4. Create business service in `services/` for domain logic -5. Create repository in `repositories/` if domain is complex (optional) -6. Add database schema in `db/schemas/` if required -### Adding Database Tables -1. Create schema in `db/schemas/{tableName}.ts` -2. Generate migration: `yarn run db:migrations:generate` -3. Add seeding data in `db/seeding/` if needed -4. Decide: Repository pattern or direct Drizzle? - - Complex domain → Create repository in `repositories/` - - Simple domain → Use direct Drizzle in service -5. Create business service in `services/` -6. Implement API handler in `api/handlers/` +1. Define schema in `@shared/data/api/schemas/` +2. Implement handler in `api/handlers/` +3. Create business service in `services/` +4. Create repository in `repositories/` (if complex domain) -### Creating a New Business Service +### Database Commands -**For complex domains (with repository)**: -```typescript -// 1. Create repository: repositories/ExampleRepository.ts -export class ExampleRepository { - async findById(id: string, tx?: Transaction) { /* ... */ } - async create(data: CreateDto, tx?: Transaction) { /* ... */ } -} - -// 2. Create service: services/ExampleService.ts -export class ExampleService { - constructor(private exampleRepo: ExampleRepository) {} - - async createExample(data: CreateDto) { - // Business validation - this.validate(data) - - // Use repository - return await this.exampleRepo.create(data) - } -} - -// 3. Create handler: api/handlers/example.ts -import { ExampleService } from '@data/services/ExampleService' - -export const exampleHandlers: Partial = { - '/examples': { - POST: async ({ body }) => { - return await ExampleService.getInstance().createExample(body) - } - } -} +```bash +# Generate migrations +yarn db:migrations:generate ``` - -**For simple domains (direct Drizzle)**: -```typescript -// 1. Create service: services/SimpleService.ts -export class SimpleService extends BaseService { - async getItem(id: string) { - // Direct database access - return await this.database - .select() - .from(itemTable) - .where(eq(itemTable.id, id)) - } -} - -// 2. Create handler: api/handlers/simple.ts -export const simpleHandlers: Partial = { - '/items/:id': { - GET: async ({ params }) => { - return await SimpleService.getInstance().getItem(params.id) - } - } -} -``` - -## Data Flow - -### Complete Request Flow - -``` -┌─────────────────────────────────────────────────────┐ -│ Renderer Process │ -│ React Component → useDataApi Hook │ -└────────────────┬────────────────────────────────────┘ - │ IPC Request -┌────────────────▼────────────────────────────────────┐ -│ Infrastructure Layer │ -│ DataApiService (coordinator) │ -│ ↓ │ -│ ApiServer (routing) → MiddlewareEngine │ -└────────────────┬────────────────────────────────────┘ - │ -┌────────────────▼────────────────────────────────────┐ -│ API Layer (api/handlers/) │ -│ Handler: Thin layer │ -│ - Extract parameters │ -│ - Call business service │ -│ - Transform response │ -└────────────────┬────────────────────────────────────┘ - │ -┌────────────────▼────────────────────────────────────┐ -│ Business Logic Layer (services/) │ -│ Service: Domain logic │ -│ - Business validation │ -│ - Transaction coordination │ -│ - Call repository or direct DB │ -└────────────────┬────────────────────────────────────┘ - │ - ┌──────────┴──────────┐ - │ │ -┌─────▼─────────┐ ┌──────▼──────────────────────────┐ -│ repositories/ │ │ Direct Drizzle │ -│ (Complex) │ │ (Simple domains) │ -│ - Repository │ │ - Service uses DbService.db │ -│ - Query logic │ │ - Inline queries │ -└─────┬─────────┘ └──────┬──────────────────────────┘ - │ │ - └──────────┬─────────┘ - │ -┌────────────────▼────────────────────────────────────┐ -│ Database Layer (db/) │ -│ DbService → SQLite (Drizzle ORM) │ -└─────────────────────────────────────────────────────┘ -``` - -### Architecture Principles - -1. **Separation of Concerns** - - Handlers: Request/response transformation only - - Services: Business logic and orchestration - - Repositories: Data access (when complexity demands it) - -2. **Dependency Flow** (top to bottom only) - - Handlers depend on Services - - Services depend on Repositories (or DbService directly) - - Repositories depend on DbService - - **Never**: Services depend on Handlers - - **Never**: Repositories contain business logic - -3. **Selective Repository Usage** - - Use Repository: Complex domains (Topic, Message, File) - - Direct Drizzle: Simple domains (Agent, Session, Translate) - - Decision based on: query complexity, data volume, testing needs - -## Development Guidelines - -- All services use singleton pattern -- Database operations must be type-safe (Drizzle) -- API endpoints require complete type definitions -- Services should handle errors gracefully -- Use existing logging system (`@logger`) - -## Integration Points - -- **IPC Communication**: All services expose IPC handlers for renderer communication -- **Type Safety**: Shared types in `@shared/data` ensure end-to-end type safety -- **Error Handling**: Standardized error codes and handling across all services -- **Logging**: Comprehensive logging for debugging and monitoring \ No newline at end of file diff --git a/src/main/data/db/README.md b/src/main/data/db/README.md index 10d3a44593..0e43e760eb 100644 --- a/src/main/data/db/README.md +++ b/src/main/data/db/README.md @@ -1,199 +1,46 @@ -# Database Schema Guidelines +# Database Layer -## Naming Conventions +This directory contains database schemas and configuration. -- **Table names**: Use **singular** form with snake_case (e.g., `topic`, `message`, `app_state`) -- **Export names**: Use `xxxTable` pattern (e.g., `topicTable`, `messageTable`) -- **Column names**: Drizzle auto-infers from property names, no need to specify explicitly +## Documentation -## Column Helpers +- **Database Patterns**: [docs/en/references/data/database-patterns.md](../../../../docs/en/references/data/database-patterns.md) -All helpers are exported from `./schemas/columnHelpers.ts`. +## Directory Structure -### Primary Keys - -| Helper | UUID Version | Use Case | -|--------|--------------|----------| -| `uuidPrimaryKey()` | v4 (random) | General purpose tables | -| `uuidPrimaryKeyOrdered()` | v7 (time-ordered) | Large tables with time-based queries | - -**Usage:** - -```typescript -import { uuidPrimaryKey, uuidPrimaryKeyOrdered } from './columnHelpers' - -// General purpose table -export const topicTable = sqliteTable('topic', { - id: uuidPrimaryKey(), - name: text(), - ... -}) - -// Large table with time-ordered data -export const messageTable = sqliteTable('message', { - id: uuidPrimaryKeyOrdered(), - content: text(), - ... -}) +``` +src/main/data/db/ +├── schemas/ # Drizzle table definitions +│ ├── columnHelpers.ts # Reusable column definitions +│ ├── topic.ts # Topic table +│ ├── message.ts # Message table +│ └── ... # Other tables +├── seeding/ # Database initialization +└── DbService.ts # Database connection management ``` -**Behavior:** +## Quick Reference -- ID is auto-generated if not provided during insert -- Can be manually specified for migration scenarios -- Use `.returning()` to get the generated ID after insert +### Naming Conventions -### Timestamps +- **Table names**: Singular snake_case (`topic`, `message`, `app_state`) +- **Export names**: `xxxTable` pattern (`topicTable`, `messageTable`) -| Helper | Fields | Use Case | -|--------|--------|----------| -| `createUpdateTimestamps` | `createdAt`, `updatedAt` | Tables without soft delete | -| `createUpdateDeleteTimestamps` | `createdAt`, `updatedAt`, `deletedAt` | Tables with soft delete | +### Common Commands -**Usage:** +```bash +# Generate migrations after schema changes +yarn db:migrations:generate +``` + +### Column Helpers ```typescript -import { createUpdateTimestamps, createUpdateDeleteTimestamps } from './columnHelpers' +import { uuidPrimaryKey, createUpdateTimestamps } from './columnHelpers' -// Without soft delete -export const tagTable = sqliteTable('tag', { +export const myTable = sqliteTable('my_table', { id: uuidPrimaryKey(), name: text(), ...createUpdateTimestamps }) - -// With soft delete -export const topicTable = sqliteTable('topic', { - id: uuidPrimaryKey(), - name: text(), - ...createUpdateDeleteTimestamps -}) ``` - -**Behavior:** - -- `createdAt`: Auto-set to `Date.now()` on insert -- `updatedAt`: Auto-set on insert, auto-updated on update -- `deletedAt`: `null` by default, set to timestamp for soft delete - -## JSON Fields - -For JSON column support, use `{ mode: 'json' }`: - -```typescript -data: text({ mode: 'json' }).$type() -``` - -Drizzle handles JSON serialization/deserialization automatically. - -## Foreign Keys - -### Basic Usage - -```typescript -// SET NULL: preserve record when referenced record is deleted -groupId: text().references(() => groupTable.id, { onDelete: 'set null' }) - -// CASCADE: delete record when referenced record is deleted -topicId: text().references(() => topicTable.id, { onDelete: 'cascade' }) -``` - -### Self-Referencing Foreign Keys - -For self-referencing foreign keys (e.g., tree structures with parentId), **always use the `foreignKey` operator** in the table's third parameter: - -```typescript -import { foreignKey, sqliteTable, text } from 'drizzle-orm/sqlite-core' - -export const messageTable = sqliteTable( - 'message', - { - id: uuidPrimaryKeyOrdered(), - parentId: text(), // Do NOT use .references() here - // ...other fields - }, - (t) => [ - // Use foreignKey operator for self-referencing - foreignKey({ columns: [t.parentId], foreignColumns: [t.id] }).onDelete('set null') - ] -) -``` - -**Why this approach:** -- Avoids TypeScript circular reference issues (no need for `AnySQLiteColumn` type annotation) -- More explicit and readable -- Allows chaining `.onDelete()` / `.onUpdate()` actions - -### Circular Foreign Key References - -**Avoid circular foreign key references between tables.** For example: - -```typescript -// ❌ BAD: Circular FK between tables -// tableA.currentItemId -> tableB.id -// tableB.ownerId -> tableA.id -``` - -If you encounter a scenario that seems to require circular references: - -1. **Identify which relationship is "weaker"** - typically the one that can be null or is less critical for data integrity -2. **Remove the FK constraint from the weaker side** - let the application layer handle validation and consistency (this is known as "soft references" pattern) -3. **Document the application-layer constraint** in code comments - -```typescript -// ✅ GOOD: Break the cycle by handling one side at application layer -export const topicTable = sqliteTable('topic', { - id: uuidPrimaryKey(), - // Application-managed reference (no FK constraint) - // Validated by TopicService.setCurrentMessage() - currentMessageId: text(), -}) - -export const messageTable = sqliteTable('message', { - id: uuidPrimaryKeyOrdered(), - // Database-enforced FK - topicId: text().references(() => topicTable.id, { onDelete: 'cascade' }), -}) -``` - -**Why soft references for SQLite:** -- SQLite does not support `DEFERRABLE` constraints (unlike PostgreSQL/Oracle) -- Application-layer validation provides equivalent data integrity -- Simplifies insert/update operations without transaction ordering concerns - -## Migrations - -Generate migrations after schema changes: - -```bash -yarn db:migrations:generate -``` - -## Field Generation Rules - -The schema uses Drizzle's auto-generation features. Follow these rules: - -### Auto-generated fields (NEVER set manually) - -- `id`: Uses `$defaultFn()` with UUID v4/v7, auto-generated on insert -- `createdAt`: Uses `$defaultFn()` with `Date.now()`, auto-generated on insert -- `updatedAt`: Uses `$defaultFn()` and `$onUpdateFn()`, auto-updated on every update - -### Using `.returning()` pattern - -Always use `.returning()` to get inserted/updated data instead of re-querying: - -```typescript -// Good: Use returning() -const [row] = await db.insert(table).values(data).returning() -return rowToEntity(row) - -// Avoid: Re-query after insert (unnecessary database round-trip) -await db.insert(table).values({ id, ...data }) -return this.getById(id) -``` - -### Soft delete support - -The schema supports soft delete via `deletedAt` field (see `createUpdateDeleteTimestamps`). -Business logic can choose to use soft delete or hard delete based on requirements. diff --git a/src/main/data/migration/v2/README.md b/src/main/data/migration/v2/README.md index 86d597223e..6e5e071f7d 100644 --- a/src/main/data/migration/v2/README.md +++ b/src/main/data/migration/v2/README.md @@ -1,64 +1,33 @@ -# Migration V2 (Main Process) +# Data Migration System -Architecture for the new one-shot migration from the legacy Dexie + Redux Persist stores into the SQLite schema. This module owns orchestration, data access helpers, migrator plugins, and IPC entry points used by the renderer migration window. +This directory contains the v2 data migration implementation. -## Directory Layout +## Documentation + +- **Migration Guide**: [docs/en/references/data/v2-migration-guide.md](../../../../../docs/en/references/data/v2-migration-guide.md) + +## Directory Structure ``` src/main/data/migration/v2/ -├── core/ # Engine + shared context -├── migrators/ # Domain-specific migrators and mappings -├── utils/ # Data source readers (Redux, Dexie, streaming JSON) -├── window/ # IPC handlers + migration window manager -└── index.ts # Public exports for main process +├── core/ # MigrationEngine, MigrationContext +├── migrators/ # Domain-specific migrators +│ └── mappings/ # Mapping definitions +├── utils/ # ReduxStateReader, DexieFileReader, JSONStreamReader +├── window/ # IPC handlers, window manager +└── index.ts # Public exports ``` -## Core Contracts +## Quick Reference -- `core/MigrationEngine.ts` coordinates all migrators in order, surfaces progress to the UI, and marks status in `app_state.key = 'migration_v2_status'`. It will clear new-schema tables before running and abort on any validation failure. -- `core/MigrationContext.ts` builds the shared context passed to every migrator: - - `sources`: `ConfigManager` (ElectronStore), `ReduxStateReader` (parsed Redux Persist data), `DexieFileReader` (JSON exports) - - `db`: current SQLite connection - - `sharedData`: `Map` for passing cross-cutting info between migrators - - `logger`: `loggerService` scoped to migration -- `@shared/data/migration/v2/types` defines stages, results, and validation stats used across main and renderer. +### Creating a New Migrator -## Migrators +1. Extend `BaseMigrator` in `migrators/` +2. Implement `prepare`, `execute`, `validate` methods +3. Register in `migrators/index.ts` -- Base contract: extend `migrators/BaseMigrator.ts` and implement: - - `id`, `name`, `description`, `order` (lower runs first) - - `prepare(ctx)`: dry-run checks, counts, and staging data; return `PrepareResult` - - `execute(ctx)`: perform inserts/updates; manage your own transactions; report progress via `reportProgress` - - `validate(ctx)`: verify counts and integrity; return `ValidateResult` with stats (`sourceCount`, `targetCount`, `skippedCount`) and any `errors` -- Registration: list migrators (in order) in `migrators/index.ts` so the engine can sort and run them. -- Current migrators: - - `PreferencesMigrator` (implemented): maps ElectronStore + Redux settings to the `preference` table using `mappings/PreferencesMappings.ts`. - - `AssistantMigrator`, `KnowledgeMigrator`, `ChatMigrator` (placeholders): scaffolding and TODO notes for future tables. -- Conventions: - - All logging goes through `loggerService` with a migrator-specific context. - - Use `MigrationContext.sources` instead of accessing raw files/stores directly. - - Use `sharedData` to pass IDs or lookup tables between migrators (e.g., assistant -> chat references) instead of re-reading sources. - - Stream large Dexie exports (`JSONStreamReader`) and batch inserts to avoid memory spikes. - - Count validation is mandatory; engine will fail the run if `targetCount < sourceCount - skippedCount` or if `ValidateResult.errors` is non-empty. - - Keep migrations idempotent per run—engine clears target tables before it starts, but each migrator should tolerate retries within the same run. +### Key Contracts -## Utilities - -- `utils/ReduxStateReader.ts`: safe accessor for categorized Redux Persist data with dot-path lookup. -- `utils/DexieFileReader.ts`: reads exported Dexie JSON tables; can stream large tables. -- `utils/JSONStreamReader.ts`: streaming reader with batching, counting, and sampling helpers for very large arrays. - -## Window & IPC Integration - -- `window/MigrationIpcHandler.ts` exposes IPC channels for the migration UI: - - Receives Redux data and Dexie export path, starts the engine, and streams progress back to renderer. - - Manages backup flow (dialogs via `BackupManager`) and retry/cancel/restart actions. -- `window/MigrationWindowManager.ts` creates the frameless migration window, handles lifecycle, and relaunch instructions after completion in production. - -## Implementation Checklist for New Migrators - -- [ ] Add mapping definitions (if needed) under `migrators/mappings/`. -- [ ] Implement `prepare/execute/validate` with explicit counts, batch inserts, and integrity checks. -- [ ] Wire progress updates through `reportProgress` so UI shows per-migrator progress. -- [ ] Register the migrator in `migrators/index.ts` with the correct `order`. -- [ ] Add any new target tables to `MigrationEngine.verifyAndClearNewTables` once those tables exist. +- `prepare(ctx)`: Dry-run checks, return counts +- `execute(ctx)`: Perform inserts, report progress +- `validate(ctx)`: Verify counts and integrity diff --git a/src/renderer/src/data/README.md b/src/renderer/src/data/README.md index bf4068de06..4cafe2284e 100644 --- a/src/renderer/src/data/README.md +++ b/src/renderer/src/data/README.md @@ -1,429 +1,40 @@ -# Data Layer - Renderer Process +# Renderer Data Layer -This directory contains the unified data access layer for Cherry Studio's renderer process, providing type-safe interfaces for data operations, preference management, and caching. +This directory contains the renderer process data services. -## Overview +## Documentation -The `src/renderer/src/data` directory implements the new data architecture as part of the ongoing database refactoring project. It provides three core services that handle all data operations in the renderer process: +- **Overview**: [docs/en/references/data/README.md](../../../../docs/en/references/data/README.md) +- **Cache**: [cache-overview.md](../../../../docs/en/references/data/cache-overview.md) | [cache-usage.md](../../../../docs/en/references/data/cache-usage.md) +- **Preference**: [preference-overview.md](../../../../docs/en/references/data/preference-overview.md) | [preference-usage.md](../../../../docs/en/references/data/preference-usage.md) +- **DataApi**: [data-api-in-renderer.md](../../../../docs/en/references/data/data-api-in-renderer.md) -- **DataApiService**: RESTful-style API for communication with the main process -- **PreferenceService**: Unified preference/configuration management with real-time sync -- **CacheService**: Three-tier caching system for optimal performance - -## Architecture +## Directory Structure ``` -┌─────────────────┐ -│ React Components│ -└─────────┬───────┘ - │ -┌─────────▼───────┐ -│ React Hooks │ ← useDataApi, usePreference, useCache -└─────────┬───────┘ - │ -┌─────────▼───────┐ -│ Services │ ← DataApiService, PreferenceService, CacheService -└─────────┬───────┘ - │ -┌─────────▼───────┐ -│ IPC Layer │ ← Main Process Communication -└─────────────────┘ +src/renderer/src/data/ +├── DataApiService.ts # User Data API service +├── PreferenceService.ts # Preferences management +├── CacheService.ts # Three-tier caching system +└── hooks/ + ├── useDataApi.ts # useQuery, useMutation + ├── usePreference.ts # usePreference, usePreferences + └── useCache.ts # useCache, useSharedCache, usePersistCache ``` ## Quick Start -### Data API Operations - ```typescript +// Data API import { useQuery, useMutation } from '@data/hooks/useDataApi' - -// Fetch data with auto-retry and caching -const { data, loading, error } = useQuery('/topics') - -// Create/update data with optimistic updates +const { data } = useQuery('/topics') const { trigger: createTopic } = useMutation('/topics', 'POST') -await createTopic({ title: 'New Topic', content: 'Hello World' }) -``` -### Preference Management - -```typescript +// Preferences import { usePreference } from '@data/hooks/usePreference' - -// Manage user preferences with real-time sync -const [theme, setTheme] = usePreference('app.theme.mode') -const [fontSize, setFontSize] = usePreference('chat.message.font_size') - -// Optimistic updates (default) -await setTheme('dark') // UI updates immediately, syncs to database -``` - -### Cache Management - -```typescript -import { useCache, useSharedCache, usePersistCache } from '@data/hooks/useCache' - -// Component-level cache (lost on app restart) -const [count, setCount] = useCache('ui.counter') - -// Cross-window cache (shared between all windows) -const [windowState, setWindowState] = useSharedCache('window.layout') - -// Persistent cache (survives app restarts) -const [recentFiles, setRecentFiles] = usePersistCache('app.recent_files') -``` - -## Core Services - -### DataApiService - -**Purpose**: Type-safe communication with the main process using RESTful-style APIs. - -**Key Features**: -- Type-safe request/response handling -- Automatic retry with exponential backoff -- Real-time subscriptions -- Request timeout handling - -**Basic Usage**: -```typescript -import { dataApiService } from '@data/DataApiService' - -// Simple GET request -const topics = await dataApiService.get('/topics') - -// POST with body -const newTopic = await dataApiService.post('/topics', { - body: { title: 'Hello', content: 'World' } -}) -``` - -### PreferenceService - -**Purpose**: Centralized preference/configuration management with cross-window synchronization. - -**Key Features**: -- Optimistic and pessimistic update strategies -- Real-time cross-window synchronization -- Local caching for performance -- Race condition handling -- Batch operations for multiple preferences - -**Basic Usage**: -```typescript -import { preferenceService } from '@data/PreferenceService' - -// Get single preference -const theme = await preferenceService.get('app.theme.mode') - -// Set with optimistic updates (default) -await preferenceService.set('app.theme.mode', 'dark') - -// Set with pessimistic updates -await preferenceService.set('api.key', 'secret', { optimistic: false }) - -// Batch operations -await preferenceService.setMultiple({ - 'app.theme.mode': 'dark', - 'chat.message.font_size': 14 -}) -``` - -### CacheService - -**Purpose**: Three-tier caching system for different data persistence needs. - -**Cache Tiers**: -1. **Memory Cache**: Component-level, lost on app restart -2. **Shared Cache**: Cross-window, lost on app restart -3. **Persist Cache**: Cross-window + localStorage, survives restarts - -**Key Features**: -- TTL (Time To Live) support -- Hook reference tracking (prevents deletion of active data) -- Cross-window synchronization -- Type-safe cache schemas -- Automatic default value handling - -**Basic Usage**: -```typescript -import { cacheService } from '@data/CacheService' - -// Memory cache - Type-safe (schema key, with auto-completion) -cacheService.set('temp.calculation', result, 30000) // 30s TTL -const result = cacheService.get('temp.calculation') - -// Memory cache - Casual (dynamic key, requires manual type) -cacheService.setCasual(`topic:${id}`, topicData) -const topic = cacheService.getCasual(`topic:${id}`) - -// Shared cache - Type-safe (schema key) -cacheService.setShared('window.layout', layoutConfig) -const layout = cacheService.getShared('window.layout') - -// Shared cache - Casual (dynamic key) -cacheService.setSharedCasual(`window:${windowId}`, state) -const state = cacheService.getSharedCasual(`window:${windowId}`) - -// Persist cache (survives restarts, schema keys only) -cacheService.setPersist('app.recent_files', recentFiles) -const files = cacheService.getPersist('app.recent_files') -``` - -**When to Use Type-safe vs Casual Methods**: -- **Type-safe** (`get`, `set`, `getShared`, `setShared`): Use when the key is predefined in the cache schema. Provides auto-completion and type inference. -- **Casual** (`getCasual`, `setCasual`, `getSharedCasual`, `setSharedCasual`): Use when the key is dynamically constructed (e.g., `topic:${id}`). Requires manual type specification via generics. -- **Persist Cache**: Only supports schema keys (no Casual methods) to ensure data integrity. - -## React Hooks - -### useDataApi - -Type-safe data fetching with SWR integration. - -```typescript -import { useQuery, useMutation } from '@data/hooks/useDataApi' - -// GET requests with auto-caching -const { data, loading, error, mutate } = useQuery('/topics', { - query: { page: 1, limit: 20 } -}) - -// Mutations with optimistic updates -const { trigger: updateTopic, isMutating } = useMutation('/topics/123', 'PUT') -await updateTopic({ title: 'Updated Title' }) -``` - -### usePreference - -Reactive preference management with automatic synchronization. - -```typescript -import { usePreference } from '@data/hooks/usePreference' - -// Basic usage with optimistic updates const [theme, setTheme] = usePreference('app.theme.mode') -// Pessimistic updates for critical settings -const [apiKey, setApiKey] = usePreference('api.key', { optimistic: false }) - -// Handle updates -const handleThemeChange = async (newTheme) => { - try { - await setTheme(newTheme) // Auto-rollback on failure - } catch (error) { - console.error('Theme update failed:', error) - } -} -``` - -### useCache Hooks - -Component-friendly cache management with automatic lifecycle handling. - -```typescript +// Cache import { useCache, useSharedCache, usePersistCache } from '@data/hooks/useCache' - -// Memory cache (useState-like, but shared between components) const [counter, setCounter] = useCache('ui.counter', 0) - -// Shared cache (cross-window) -const [layout, setLayout] = useSharedCache('window.layout') - -// Persistent cache (survives restarts) -const [recentFiles, setRecentFiles] = usePersistCache('app.recent_files') ``` - -## Best Practices - -### When to Use Which Service - -The three services map to distinct data categories based on the original architecture design. Use the following guide to choose the right service. - -#### Quick Decision Table - -| Service | Data Characteristics | Lifecycle | Data Loss Impact | Examples | -|---------|---------------------|-----------|------------------|----------| -| **CacheService** | Regenerable, temporary | ≤ App process or survives restart | None to minimal | API responses, computed results, UI state | -| **PreferenceService** | User settings, key-value | Permanent until changed | Low (can rebuild) | Theme, language, font size, shortcuts | -| **DataApiService** | Business data, structured | Permanent | **Severe** (irreplaceable) | Topics, messages, files, knowledge base | - -#### CacheService - Runtime & Cache Data - -Use CacheService when: -- Data can be **regenerated or lost without user impact** -- No backup or cross-device synchronization needed -- Lifecycle is tied to component, window, or app session - -**Two sub-categories**: -1. **Performance cache**: Computed results, API responses, expensive calculations -2. **UI state cache**: Temporary settings, scroll positions, panel states - -**Three tiers based on persistence needs**: -- `useCache` (memory): Lost on app restart, component-level sharing -- `useSharedCache` (shared): Cross-window sharing, lost on restart -- `usePersistCache` (persist): Survives app restarts via localStorage - -```typescript -// Good: Temporary computed results -const [searchResults, setSearchResults] = useCache('search.results', []) - -// Good: UI state that can be lost -const [sidebarCollapsed, setSidebarCollapsed] = useSharedCache('ui.sidebar.collapsed', false) - -// Good: Recent items (nice to have, not critical) -const [recentSearches, setRecentSearches] = usePersistCache('search.recent', []) -``` - -#### PreferenceService - User Preferences - -Use PreferenceService when: -- Data is a **user-modifiable setting that affects app behavior** -- Structure is key-value with **predefined keys** (users modify values, not keys) -- **Value structure is stable** (won't change frequently) -- Data loss has **low impact** (user can reconfigure) - -**Key characteristics**: -- Auto-syncs across all windows -- Each preference item should be **atomic** (one setting = one key) -- Values are typically: boolean, string, number, or simple array/object - -```typescript -// Good: App behavior settings -const [theme, setTheme] = usePreference('app.theme.mode') -const [language, setLanguage] = usePreference('app.language') -const [fontSize, setFontSize] = usePreference('chat.message.font_size') - -// Good: Feature toggles -const [showTimestamp, setShowTimestamp] = usePreference('chat.display.show_timestamp') -``` - -#### DataApiService - User Data - -Use DataApiService when: -- Data is **business data accumulated through user activity** -- Data is **structured with dedicated schemas/tables** -- Users can **create, delete, modify records** (no fixed limit) -- Data loss would be **severe and irreplaceable** -- Data volume can be **large** (potentially GBs) - -**Key characteristics**: -- No automatic window sync (fetch on demand for fresh data) -- May contain sensitive data (encryption consideration) -- Requires proper CRUD operations and transactions - -```typescript -// Good: User-generated business data -const { data: topics } = useQuery('/topics') -const { trigger: createTopic } = useMutation('/topics', 'POST') - -// Good: Conversation history (irreplaceable) -const { data: messages } = useQuery('/messages', { query: { topicId } }) - -// Good: User files and knowledge base -const { data: files } = useQuery('/files') -``` - -#### Decision Flowchart - -Ask these questions in order: - -1. **Can this data be regenerated or lost without affecting the user?** - - Yes → **CacheService** - - No → Continue to #2 - -2. **Is this a user-configurable setting that affects app behavior?** - - Yes → Does it have a fixed key and stable value structure? - - Yes → **PreferenceService** - - No (structure changes often) → **DataApiService** - - No → Continue to #3 - -3. **Is this business data created/accumulated through user activity?** - - Yes → **DataApiService** - - No → Reconsider #1 (most data falls into one of these categories) - -#### Common Anti-patterns - -| Wrong Choice | Why It's Wrong | Correct Choice | -|--------------|----------------|----------------| -| Storing AI provider configs in Cache | User loses configured providers on restart | **PreferenceService** | -| Storing conversation history in Preferences | Unbounded growth, complex structure | **DataApiService** | -| Storing topic list in Preferences | User-created records, can grow large | **DataApiService** | -| Storing theme/language in DataApi | Overkill for simple key-value settings | **PreferenceService** | -| Storing API responses in DataApi | Regenerable data, doesn't need persistence | **CacheService** | -| Storing window positions in Preferences | Can be lost without impact | **CacheService** (persist tier) | - -#### Edge Cases - -- **Recently used items** (e.g., recent files, recent searches): Use `usePersistCache` - nice to have but not critical if lost -- **Draft content** (e.g., unsaved message): Use `useSharedCache` for cross-window, consider auto-save to DataApi for recovery -- **Computed statistics**: Use `useCache` with TTL - regenerate when expired -- **User-created templates/presets**: Use **DataApiService** - user-generated content that can grow - -### Performance Guidelines - -1. **Prefer React Hooks**: Use `useQuery`, `usePreference`, `useCache` for component integration -2. **Batch Operations**: Use `setMultiple()` for updating multiple preferences -3. **Cache Strategically**: Use appropriate cache tiers based on data lifetime needs -4. **Optimize Re-renders**: SWR and useSyncExternalStore minimize unnecessary re-renders - -### Common Patterns - -```typescript -// Loading states with error handling -const { data, loading, error } = useQuery('/topics') -if (loading) return -if (error) return - -// Form handling with preferences -const [fontSize, setFontSize] = usePreference('chat.message.font_size') -const handleChange = (e) => setFontSize(Number(e.target.value)) - -// Temporary state with caching -const [searchQuery, setSearchQuery] = useCache('search.current_query', '') -const [searchResults, setSearchResults] = useCache('search.results', []) -``` - -## Type Safety - -All services provide full TypeScript support with auto-completion and type checking: - -- **API Types**: Defined in `@shared/data/api/` -- **Preference Types**: Defined in `@shared/data/preference/` -- **Cache Types**: Defined in `@shared/data/cache/` - -Type definitions are automatically inferred, providing: -- Request/response type safety -- Preference key validation -- Cache schema enforcement -- Auto-completion in IDEs - -## Migration from Legacy Systems - -This new data layer replaces multiple legacy systems: -- Redux-persist slices → PreferenceService -- localStorage direct access → CacheService -- Direct IPC calls → DataApiService -- Dexie database operations → DataApiService - -For migration guidelines, see the project's `.claude/` directory documentation. - -## File Structure - -``` -src/renderer/src/data/ -├── DataApiService.ts # User Data API querying service -├── PreferenceService.ts # Preferences management -├── CacheService.ts # Three-tier caching system -└── hooks/ - ├── useDataApi.ts # React hooks for user data operations - ├── usePreference.ts # React hooks for preferences - └── useCache.ts # React hooks for caching -``` - -## Related Documentation - -- **API Schemas**: `packages/shared/data/` - Type definitions and API contracts -- **Architecture Design**: `.claude/data-architecture.md` - Detailed system design -- **Migration Guide**: `.claude/migration-planning.md` - Legacy system migration -- **Project Overview**: `CLAUDE.local.md` - Complete refactoring context \ No newline at end of file