As the number of Blocklets grows in our ecosystem, user experience and performance are always top priorities for our team. Due to some learnings from our Test Store, we've changed the default database of the Blocklet Store from NeDB to SQLite. With this change, we've improved memory consumption and data security. We invite you to try it out and to share your feedback.
Why SQLite#
Decreased RAM usage#
NeDB achieves querying capabilities by loading all data into memory, and as the volume of data increases, this method consumes a significant amount of runtime memory, which can lead to performance issues or even service crashes.
After migrating to SQLite, the database uses indexing, eliminating the need for extensive memory usage and significantly reducing the server's operational memory footprint. As shown below, the actual operational data indicate a marked decrease in memory usage post-migration leading to improved performance stability.
Risk of Data Loss#
Recently, an abnormal service crash has resulted in a Test Store data loss for a short period of time. During this time, users were unable to access the Test Store Blocklet. The reason for this service crash was NeDB, which is that Nedb's data persistence is achieved asynchronously, leading to data not being promptly written to the disk during a service crash, resulting in data loss.
After migrating to SQLite, the database employs a transactional mechanism and WAL (Write-Ahead Logging) mode, ensuring data consistency and security. Even in system anomalies or crashes, incomplete transactions do not affect the integrity of the database files, significantly reducing the risk of data loss.
Other Advantages#
- Enhanced Query Capabilities:
SQLite supports standard SQL queries, including complex multi-table joins, subqueries, and window functions, making it suitable for broader use cases and application features. - Better Scalability
SQLite handles small to medium-sized datasets effectively, maintaining efficient performance as data volumes grow. - Abundant Tools and Ecosystem Support
SQLite benefits from extensive tool support, such as DB Browser for SQLite, and an active community ecosystem, making debugging and management easier. It performs exceptionally well in batch inserts and high-concurrency scenarios within Node.js.
Note#
If you are using our Blocklet Store, we have set up an automatic migration script during the upgrade that will automatically move your database from NeDB to SQLite. Additionally, we will check for any data loss and attempt recovery. If recovery fails, you can check the failed recovery Blocklet via the /api/console/blocklets/meilisearch_sync_failed interface. After launching the Blocklet Store, you can also check whether any data has been lost. If you encounter any issues or need assistance, please feel free to contact us, and we will be glad to assist you.
Statement: The lost data has been recovered through technical means.
Community feedback:#
- blocklet store列表与详情下载量数据不一致 fatych CrypThor
- Test Store - 1/3 of my blocklet have been deleted bug CrypThor
- copy install url 的 demo 图片宽高比变形 Zhu Liang
- should render blocklet with correct host Shijun Wang
- limit changelog display to recent entries if too long Shijun Wang
Help Us Improve#
If you have any needs, suggestions, or problems using our new store experience, please go to the community for feedback.