Commit Graph

39 Commits

Author SHA1 Message Date
千石
a6bd90a9b2
feat(driver/s3): Add OSS Archive Support (#9350)
* feat(s3): Add support for S3 object storage classes

Introduces a new 'storage_class' configuration option for S3 providers. Users can now specify the desired storage class (e.g., Standard, GLACIER, DEEP_ARCHIVE) for objects uploaded to S3-compatible services like AWS S3 and Tencent COS.

The input storage class string is normalized to match AWS SDK constants, supporting various common aliases. If an unknown storage class is provided, it will be used as a raw value with a warning. This enhancement provides greater control over storage costs and data access patterns.

* feat(storage): Support for displaying file storage classes

Adds storage class information to file metadata and API responses.

This change introduces the ability to store file storage classes in file metadata and display them in API responses. This allows users to view a file's storage tier (e.g., S3 Standard, Glacier), enhancing data management capabilities.

Implementation details include:
- Introducing the StorageClassProvider interface and the ObjWrapStorageClass structure to uniformly handle and communicate object storage class information.
- Updated file metadata structures (e.g., ArchiveObj, FileInfo, RespFile) to include a StorageClass field.
- Modified relevant API response functions (e.g., GetFileInfo, GetFileList) to populate and return storage classes.
- Integrated functionality for retrieving object storage classes from underlying storage systems (e.g., S3) and wrapping them in lists.

* feat(driver/s3): Added the "Other" interface and implemented it by the S3 driver.

A new `driver.Other` interface has been added and defined in the `other.go` file.
The S3 driver has been updated to implement this new interface, extending its functionality.

* feat(s3): Add S3 object archive and thaw task management

This commit introduces comprehensive support for S3 object archive and thaw operations, managed asynchronously through a new task system.

- **S3 Transition Task System**:
  - Adds a new `S3Transition` task configuration, including workers, max retries, and persistence options.
  - Initializes `S3TransitionTaskManager` to handle asynchronous S3 archive/thaw requests.
  - Registers dedicated API routes for monitoring S3 transition tasks.

- **Integrate S3 Archive/Thaw with Other API**:
  - Modifies the `Other` API handler to intercept `archive` and `thaw` methods for S3 storage drivers.
  - Dispatches these operations as `S3TransitionTask` instances to the task manager for background processing.
  - Returns a task ID to the client for tracking the status of the dispatched operation.

- **Refactor `other` package for improved API consistency**:
  - Exports previously internal structs such as `archiveRequest`, `thawRequest`, `objectDescriptor`, `archiveResponse`, `thawResponse`, and `restoreStatus` by making their names public.
  - Makes helper functions like `decodeOtherArgs`, `normalizeStorageClass`, and `normalizeRestoreTier` public.
  - Introduces new constants for various S3 `Other` API methods.
2025-10-16 17:22:54 +08:00
j2rong4cn
4f5cabc725
feat: add h2c for http server (#8294)
* feat: add h2c for http server

* chore(config): add EnableH2c option
2025-04-12 17:02:51 +08:00
KirCute_ECT
d5ec998699
feat(task): allow retry canceled (#7852) 2025-01-27 20:18:10 +08:00
j2rong4cn
2be0c3d1a0
feat(alias): add DownloadConcurrency and DownloadPartSize option (#7829)
* fix(net): goroutine logic bug (AlistGo/alist#7215)

* Fix goroutine logic bug

* Fix bug

---------

Co-authored-by: hpy hs <hshpy.pengyu@gmail.com>

* perf(net): sequential and dynamic concurrency

* fix(net): incorrect error return

* feat(alias):  add `DownloadConcurrency` and `DownloadPartSize` option

* feat(net): add `ConcurrencyLimit`

* pref(net): create `chunk` on demand

* refactor

* refactor

* fix(net): `r.Closers.Add` has no effect

* refactor

---------

Co-authored-by: hpy hs <hshpy.pengyu@gmail.com>
2025-01-27 20:08:39 +08:00
KirCute_ECT
bb40e2e2cd
feat(archive): archive manage (#7817)
* feat(archive): archive management

* fix(ftp-server): remove duplicate ReadAtSeeker realization

* fix(archive): bad seeking of SeekableStream

* fix(archive): split internal and driver extraction api

* feat(archive): patch

* fix(shutdown): clear decompress upload tasks

* chore

* feat(archive): support .iso format

* chore
2025-01-18 23:28:12 +08:00
KirCute_ECT
5994c17b4e
feat(patch): upgrade patch module (#7738)
* feat(patch): upgrade patch module

* chore(patch): add docs

* fix(patch): skip and rewrite invalid last launched version

* fix(patch): turn two functions into patches
2024-12-30 22:48:33 +08:00
Andy Hsu
cf58ab3a78 chore(config): disable FTP and SFTP by default 2024-12-12 21:04:14 +08:00
KirCute_ECT
33ba7f1521
feat: sftp server support (#7643)
* feat: sftp server support

* fix(sftp-server): try fix build failed

* fix: sftp download lack
2024-12-12 20:51:43 +08:00
KirCute_ECT
650b03aeb1
feat: ftp server support (#7634 close #1898)
* feat: ftp server support

* fix(ftp): incorrect mode for dirs in LIST returns
2024-12-10 20:17:46 +08:00
安稳
f58de9923a
refactor(aliyunopen,config): Modify default properties (#7476) 2024-11-08 22:07:35 +08:00
itsHenry
74f8295960
feat: persistant Task (#6925 close #5313) 2024-08-07 12:16:21 +08:00
Mix
cd5a8a011d
fix: typo about env of Meilisearch (#6316) 2024-04-08 18:35:23 +08:00
itsHenry
9c84b6596f
feat: stand-alone port s3 server (#6242)
* feat: single port s3 server

* fix: unable to PUT files if not in root dir
2024-03-24 15:16:00 +08:00
不插电
f1979a8bbc
feat(search): search with meilisearch (#6060)
* feat(search): search with meilisearch.

* feat(search): meilisearch supports auto update.

* chores: remove utils.Log.

* fix(search): the null pointer caused by deleting non-existing file/folder indexes.

---------

Co-authored-by: Andy Hsu <i@nn.ci>
2024-02-23 15:37:40 +08:00
Mars160
1f835502ba
feat: support customize dsn for mysql and pg (#6031)
* support for unixsocket to connect to mysql

* feat: customize dsn for mysql and pg

---------

Co-authored-by: Andy Hsu <i@nn.ci>
2024-02-23 15:28:48 +08:00
Andy Hsu
3f405de6a9 feat: customize allow origins, headers and methods 2023-11-24 19:18:34 +08:00
Andy Hsu
7583c4d734 feat: customize workers and retry of task (close #5493 fix #5274) 2023-11-21 15:51:57 +08:00
Andy Hsu
6fc6751463 feat: support using external dist files (close #5531) 2023-11-18 19:56:22 +08:00
Andy Hsu
95386d777b feat(aliyundrive_open): record token exchange 2023-08-08 20:38:13 +08:00
Andy Hsu
2b533e4b91 feat: allow customize perm of unix file (close #4709) 2023-07-08 20:17:05 +08:00
Andy Hsu
7d20a01dba feat!: support listen to the unix (close #4671)
Starting from this commit, the HTTP server related config all move to the scheme
2023-07-04 17:56:02 +08:00
BoYanZh
f646d2a699 feat!: listen to both http & https (#4536)
Co-authored-by: Andy Hsu <i@nn.ci>
2023-06-11 18:17:37 +08:00
Andy Hsu
f9e067abec feat: support delayed start (#4532) 2023-06-05 16:00:31 +08:00
Andy Hsu
c8f3e8ab4d feat!: skip tls insecure verify by default 2023-02-23 22:33:54 +08:00
BoYanZh
7bc46de8aa
feat: settings for tls insecure skip verify (close #3306 in #3307) 2023-02-08 22:01:26 +08:00
BoYanZh
7947ff1ae4
feat: limit max connection count (#2701) 2022-12-14 10:33:58 +08:00
Noah Hsu
ddcba93eea
feat: multiple search indexes (#2514)
* refactor: abstract search interface

* wip: ~

* fix cycle import

* objs update hook

* wip: ~

* Delete search/none

* auto update index while cache changed

* db searcher

TODO: bleve init issue

cannot open index, metadata missing

* fix size type

why float64??

* fix typo

* fix nil pointer using

* api adapt ui

* bleve: fix clear & change struct
2022-11-28 13:45:25 +08:00
BoYanZh
330a767fd7
feat: build index & search with bleve (close #1740 pr #2386)
* feat: build index & search with bleve (#1740)

* delete unused struct

Co-authored-by: Noah Hsu <i@nn.ci>
2022-11-24 11:46:47 +08:00
Noah Hsu
1c212f6c30 feat!: force to use the bin dir as the data dir (close #2108)
- move default log path to `data/log/log.log`
- replace `--conf` with `--data`
2022-11-01 19:16:23 +08:00
Noah Hsu
9d9c79179b feat: custom token expires in 2022-09-27 14:05:00 +08:00
Noah Hsu
7c32af4649 refactor!: move api_url and base_path to config file 2022-09-25 17:57:54 +08:00
Noah Hsu
03dbb3a403 chore: fix typo of env name 2022-09-25 17:41:04 +08:00
Noah Hsu
046bbb3a48 feat: use lumberjack for log rotate 2022-08-30 15:22:54 +08:00
Noah Hsu
d267c43556 feat: static file router 2022-08-28 23:13:03 +08:00
Noah Hsu
2e8322e99b feat: set cache_expiration for each storage (close #1455) 2022-08-07 13:33:53 +08:00
Noah Hsu
490df4f5fe fix: typo of environment variable (close #1280) 2022-06-27 14:01:15 +08:00
Noah Hsu
4cef3adc90 feat: meta manage api 2022-06-26 19:09:28 +08:00
Noah Hsu
c5295f4d72 feat: user jwt login 2022-06-25 21:34:44 +08:00
Noah Hsu
306b90399c chore: move conf package 2022-06-25 20:38:02 +08:00