Commit Graph

8 Commits

Author SHA1 Message Date
LiuVaayne
72ae105166
[1.5.0-rc] Feat/memory (#7689)
* Merge memory into main

* Improvement/memory UI (#7655)

* feat: add auto-dimension detection to memory settings

- Add automatic embedding dimension detection for memory configuration
- Add toggle switch to enable/disable auto-detection (enabled by default)
- Detect dimensions by making test API call to embedding provider
- Show dimension input field only when auto-detection is disabled
- Add loading state and error handling during dimension detection
- Maintain consistency with knowledge base dimension handling

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

*  feat: implement unified embedding dimensions for memory service

- Add jaison dependency for robust JSON parsing
- Normalize all embeddings to 1536 dimensions for consistency
- Improve embedding dimension logging
- Update memory processor to use jaison for better error handling
- Handle various JSON response formats in fact extraction

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: refactor MemoriesPage layout with new styled components and improved user management features

---------

Co-authored-by: Claude <noreply@anthropic.com>

* Improvement/memory UI (#7656)

Co-authored-by: Claude <noreply@anthropic.com>

*  feat: add memory icon to sidebar for existing users

- Add migration version 118 to enable memory feature visibility
- Adds 'memory' icon to sidebar visible icons if not already present
- Updates store version to trigger migration for existing users

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): include last user message ID in processor config

*  feat(memory): enhance memory settings UI and add new translations

* Enhance memory management UI: Added settings, statistics, search, actions, and user management sections to the memory page. Updated translations for multiple languages to include new UI elements. Refactored component structure for improved layout and readability.

* feat: add i18n

* ui: Enhance memory modals and UI

* refactor(memory): replace direct message calls with window.message for error and success notifications

* fix: eslint error

* feat(memory): enhance memory restoration logic and queries

- Updated MemoryService to restore deleted memories instead of inserting new ones if a memory with the same hash exists.
- Added new SQL queries to check for deleted memories and restore them.
- Improved logging for memory restoration and embedding generation.
- Refactored related API service methods to handle updated memory processing logic.

* refactor: update memory configuration to use ApiClient structure

- Refactored memory-related services and components to utilize the new ApiClient structure for embedding and reranking models.
- Updated constructors and method signatures across multiple files to accept embedApiClient and rerankApiClient parameters.
- Enhanced memory settings UI to reflect changes in memory configuration management.
- Improved type definitions for KnowledgeBaseParams and MemoryConfig to align with the new structure.

* ui: improve user interface for adding new users in memory page

- Enhanced the button for adding new users by incorporating an icon and adjusting padding for better alignment.
- Updated the user selection options to ensure consistent alignment of avatars and user names.
- Refactored layout to improve overall user experience and visual consistency.

* refactor(memory): streamline MemoryProcessor usage in ApiService

- Removed the singleton instance of MemoryProcessor and instantiated it directly within the ApiService methods.
- Updated relevant methods to utilize the new instance for searching and processing memories, improving clarity and encapsulation of memory handling logic.

* chore: move knowledge dir

* fix: correct import paths in KnowledgeService.ts

* fix(Memory): memory deduplicate

* fix(Memory): memory llm provider

* fix: ci error

* fix(Memory): update fact extraction prompt to focus on personal information

* feat: Refactor memory fom sidebar to settings page

- Removed MemoryStick icon from Sidebar component.
- Updated navigation to point to the new memory settings page.
- Introduced MemoriesSettingsModal for managing memory configurations.
- Created MemorySettings component for comprehensive memory management.
- Added user management features including adding, editing, and deleting users.
- Implemented pagination and search functionality for memory items.
- Updated sidebar settings to remove memory icon and ensure proper migration.
- Adjusted Redux store settings to reflect changes in sidebar icons.

* feat: redesign memory settings page with improved UI and layout

* fix i18n

* fix: update citation titles to include memory hash and increment version number

* fix: remove unnecessary prop from KnowledgeCitation component

* feat: enhance fact extraction prompt with clearer guidelines and examples

* 🔧 feat: disable global memory by default and improve UI

- Set globalMemoryEnabled default to false for better user experience
- Remove manual localStorage handling to rely on redux-persist
- Add Beta badge to memory settings section
- Improve layout and styling of memory settings UI components

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Simplify external tool completion handling

* Fix whitespace in migrate config

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: suyao <sy20010504@gmail.com>
Co-authored-by: eeee0717 <chentao020717Work@outlook.com>
Co-authored-by: kangfenmao <kangfenmao@qq.com>
Co-authored-by: 自由的世界人 <3196812536@qq.com>
2025-07-15 10:24:41 +08:00
西街工坊
808a22d5c6
fix(Doc2xPreprocessProvider): replace filePath split with path.parse… (#8042) 2025-07-11 11:03:05 +08:00
Phantom
a7b78c547a
fix(encoding): encoding detection and decoding logic (#8024) 2025-07-10 22:13:40 +08:00
Phantom
c05c06b7a1
fix: VoyageEmbeddings (#8034)
* fix(embeddings): 修复VoyageAI嵌入格式和模型验证错误

修复OpenAIBaseClient中VoyageAI提供商的embedding格式设置问题
完善VoyageEmbeddings模型验证的错误提示信息

* refactor(embeddings): 移除VoyageEmbeddings的模型维度限制检查

简化VoyageEmbeddings的创建逻辑,不再对支持的模型维度进行校验

* fix(embeddings): 修复VoyageEmbeddings模型维度设置问题

修复VoyageEmbeddings中未正确校验模型是否支持设置outputDimension的问题
当provider为voyageai且模型不支持设置dimensions时,自动忽略传入的dimensions参数

* refactor(embeddings): 集中管理支持设置维度的模型列表

将各嵌入模型支持设置维度的模型列表集中到utils模块
不再让VoyageEmbeddings中getDimensions抛出错误,而是自动修复
2025-07-10 21:53:37 +08:00
Phantom
05b8afd681
feat: more encoding (#7898)
* feat(文件处理): 添加文件编码支持以正确处理不同编码的文本文件

添加文件编码检测和指定编码读取功能
- 在FileMetadata接口中添加encoding字段
- 添加iconv-lite和jschardet依赖用于编码处理和检测
- 文件上传时自动检测文本文件编码
- 文件读取时支持指定编码参数
- 更新所有API客户端以传递文件编码参数

* feat(文件处理): 添加文件编码检测和UTF-8读取功能

新增文件编码检测工具函数和UTF-8读取功能,统一处理不同编码的文件读取
移除重复的编码检测逻辑,优化代码结构

* refactor(FileStorage): 使用 readFileUTF8 替换 decodeBuffer 读取文件

移除冗余的 decodeBuffer 逻辑,直接使用封装好的 readFileUTF8 方法读取文件内容

* docs(utils): 为文件编码相关函数添加注释说明

添加对 detectEncoding、decodeBuffer 和 readFileUTF8 函数的详细注释,说明其功能和使用方法

* fix(utils): 为detectEncoding函数添加返回类型声明

* refactor(文件处理): 移除冗余的decodeBuffer函数并直接使用iconv.decode

简化文件读取逻辑,直接调用iconv.decode而不是通过中间函数decodeBuffer

* test(file): 添加文件编码检测的测试用例

* test(文件编码检测): 移除ISO-8859-1编码的测试匹配

* refactor(file): 移除文件编码相关逻辑,统一使用UTF-8读取文本文件

移除FileMetadata接口中的encoding字段及相关检测逻辑
将所有文件读取操作统一改为使用readTextFileUTF8方法

* fix(文件读取): 改进文本文件解码逻辑以处理编码识别错误

当自动识别的编码包含错误字符时,尝试其他可能的编码

* refactor(utils): 将 console 日志替换为 electron-log 记录器

* refactor(文件存储): 移除文件读取时的可选编码参数

简化文件读取逻辑,始终使用UTF-8编码读取文本文件

* fix(utils): 修复文件编码检测中的文件描述符泄漏

在detectEncoding函数中,文件描述符在使用后未关闭,可能导致资源泄漏

* refactor(文件处理): 将readTextFileUTF8重命名为readTextFileWithAutoEncoding并改进编码检测

修复文件编码检测中未正确关闭文件描述符的问题
改进文本文件读取功能以支持自动编码检测

* test(file): 重构编码检测测试用例并改进测试结构

- 将 describe 块重命名为更明确的 detectEncoding
- 提取公共的 mock 逻辑到 beforeEach
- 更新测试描述为英文并保持一致性
- 简化测试实现,移除重复代码

* test(file): 添加对readTextFileWithAutoEncoding的测试用例
2025-07-08 00:57:31 +08:00
Chen Tao
463ca6185b
chore: move ocr and preprocess into knowledge folder (#7896)
chore: move ocr and preprocess into knowledge file
2025-07-07 07:29:15 +08:00
Chen Tao
e35b4d9cd1
feat(knowledge): support doc2x, mistral, MacOS, MinerU... OCR (#3734)
Co-authored-by: suyao <sy20010504@gmail.com>
Co-authored-by: 亢奋猫 <kangfenmao@qq.com>
2025-07-03 16:23:02 +08:00
亢奋猫
9f29194180
refactor: Restructure the knowledge base directory (#7754)
重构知识库目录结构,代码逻辑完全不变

├── embeddings
│   ├── Embeddings.ts
│   ├── EmbeddingsFactory.ts
│   └── VoyageEmbeddings.ts
├── loader
│   ├── draftsExportLoader.ts
│   ├── epubLoader.ts
│   ├── index.ts
│   ├── noteLoader.ts
│   └── odLoader.ts
└── reranker
    ├── BaseReranker.ts
    ├── GeneralReranker.ts
    └── Reranker.ts

4 directories, 11 files
2025-07-02 15:23:02 +08:00