v2.0.0-Release <- Read this for performance problems

[CommanderStorm]

This was originally a PR here, but has been transformed into an issue due to being pin-able

what has been done?

We are hard at work improving Uptime Kuma.
If you want to have a look at the full set of features for v2.0, have a look here.

Most notably this includes (a full migration guide will be made before the release, dont worry):

• better performance through external MariaDB, storing heartbeats in an aggregated form, full server-side pagination for important events, better async and database usage
  => even though benchmarking is still out, we are confident that this pushes the prominent "limits" (highly hardware-dependent, not a limit imposed by us) ~500 Monitor or ~1.5GB DB-size
  A large part of the focus in this release is on performance. We don't know if we are optimising for the right thing, see Can/should we add privacy-respecting usage metrics? #4456 for a discussion on this topic.

Tip

If you are affected by the performance problems of v1, you need to reduce the amount of data you store.
The core problem is that Uptime Kuma in v1 has to do a table scan of the entire heartbeat table for some operations.

The solution boils down to having a lower amount of data => making Uptime Kuma less worried about reading those gigabytes of data.:

• reduce the retention and execute a manual cleanup via the VACUUM button under /settings/monitor-history
• Increase the time between checks
• pause or reduce the amount of monitors
• delete the specific history of less essential monitors (to "lower" their retention below the configured maximum)

• tightening the security via offering rootless docker images images #2086 #4052
• deprecated backup feature has been dropped (see Drop backup #3892 (comment) for an explanation why)
• deprecated cacheable lookups (not the same as nscd) have been dropped CacheableLookup was never used for http monitors #3762
• A lot of bugfixes, some minor new features (see the milestone)

what is needed to get 2.0.0-beta.0 released

Note

We hope 2.0.0-beta.0 can be released in spring 2024, which doesn't include the e2e tests yet. During the beta test, we will try to work on implementing the e2e tests.

Tip

How can I get involved?

While most of the bugs noted below require reading some code
=> gaining some context about how the aggregated tables work, some work do not and are such better starting points:

• writing a synthetic benchmark which spawns some monitors and looks how the server handles it
• helping in writing e2e tests using playwright

• some features have not adopted to new aggregate tables yet
  • Migrate existing heartbeat data into aggregate tables
  • Clear stats (Update clear stats #4324)
  • History retention
  • Ping chart (Feat: Use UptimeCalculator for PingChart #4264)
  • Heartbeat table keeps 24-48 hours data only (excepted for important events, it will keep longer for the event list) (Can be deferred)
  • Test and fix the Badges feature (originally introduced in Add badges #1119, a big headache for us right now because the new table schemas break this feature a lot) (Fix uptime & ping badge duration #4850)
• Testing
  • Drop Jest e2e testing  #2174
  • Playwright + Native Node Test Runner #3893
  • Write more E2E Testing using Playwright
• Document how the aggregated tables work for our contributors in our contribution guide
• Migration guide from v1 to v2 (Wiki) Migration Guide from v1 to v2 uptime-kuma-wiki#53
  • New Docker tags
  • Notify about breaking changes and write a migration guide
    • removed variable badges
    • SMTP-templating Feature: SMTP-templating of customBody and customHeader via liquidjs #3414
      • dropped support for vArIaBlE Cased variables
      • deprecated UPPERCASE variables (this was added to make the v1⇒v2 migration more seamless, but will be removed in v3)
    • removed deprecations:
      • CacheableLookup was never used for http monitors #3762
      • Drop backup #3892
    • Update to respect docker compose v2 #4289
    • default retires is now 0 instead of 1
• Benchmark where the new performance limits are
• Write the changelog and Release 2.0.0-beta.0

[CommanderStorm]

@compgeniuses
Our installation guide only works for stable versions (it checks out a git tag to avoid users being on the wrong branch in npm run setup).
Pre-releases will be made once we don't have glaring bugs left, see my post above for which ones there are.

If you want to test-drive or contribute one of the areas named above, our contribution guide should have all the answers. If you are missing something for development, feel free to ping me.
We also have this way of testing PRs: https://github.com/louislam/uptime-kuma/wiki/Test-Pull-Requests

[CommanderStorm]

Closing as 2.0.0-beta.0 was released

[zenkraker]

@CommanderStorm I was looking at the beta, and it is not clear if the "external" MariaDB is included within the uptimeKuma image or (in a context of docker compose) to have uptimeKuma along with a mariadb as a separated containers

[otbutz]

Seems to embedded:

uptime-kuma/docker/debian-base.dockerfile

Line 69 in 4e144ec

apt --yes --no-install-recommends install chromium fonts-indic fonts-noto fonts-noto-cjk mariadb-server && \

I'd love if we could separate this.

[valdearg]

Setup option let's you choose either embedded or external depending on your preferences/requirements.

[zenkraker]

Personally, I'd prefer to either choose between using the embedded SQL-Lite (which we know the limitations in production, but good enough for a testing environment) OR go full power and leave the user running a separated MariaDB offering the following advantages (to my eyes of course):

1. UptimeKuma image will be smaller because you won't need to bloat it with mariaDB
2. Users can have control on the DB and backup is easier, because you just need to backup the volume of the MariaDB container and probably it is beneficial for uptimeKuma developers because, they won't need to think how to backup the DB or implement any feature for this and let the end-users deal with it

To summarize:
I'd add an env variable to uptimeKuma configuration, if it is filled it will use the mariaDB container + port, otherwise it will run with SQL-Lite by default
(Same concept could be applied to cloudflared image that at the moment is pulled inside the dockerfile as well)

But in general, the peculiarity of docker is the detachment of dependencies so having an image ALL-IN-ONE is in my opinion against this principle

[CommanderStorm]

If you don't want mariadb, please use the -slim variants like beta-slim instead.

But in general, the peculiarity of docker is the detachment of dependencies so having an image ALL-IN-ONE is in my opinion against this principle

Having all of the features in the default image and having slim variants is quite common.

[zenkraker]

@CommanderStorm thanks for the response.
So, if I understood correctly, the slim version uses sql lite or can I pass set a connection to an external mariaDB container?

[CommanderStorm]

The slim version can use sqlite or an external mariaDB container

[zenkraker]

@CommanderStorm
Thanks for the hint, the beta version works like a charm! <3

[msaleemCinnova]

@louislam , @CommanderStorm ,
I am currently running Uptime Kuma version 1.23.11 without Docker and would like to migrate to version 2. Could you kindly advise which branch I should switch to in order to access the beta/v2 release? Additionally, any guidance on the migration process would be greatly appreciated.

Thank you in advance for your support.

[CommanderStorm]

Please refer to https://github.com/louislam/uptime-kuma/wiki/Migration-From-v1-To-v2
If anything is missing, feedback is appreciated

[msaleemCinnova]

Dear Frank, I am running Uptime Kuma in a non-Docker environment because I have integrated a custom module into the existing setup and submitted a pull request for these changes.[#5382] To maintain my customizations, I update and manage the application without Docker. So i want to migrate from v1 to v2 without using Docker. Here are the steps I follow for setup and execution & updates: # Update from git git fetch --all git checkout 1.23.15 --force # Install dependencies and prebuilt npm install --production npm run download-dist npm run setup # Option 1. Try it node server/server.js # Start Server pm2 start server/server.js --name uptime-kuma Looking for the migration guide without using Docker.

…

________________________________ From: Frank Elsinga ***@***.***> Sent: Monday, December 2, 2024 1:31 PM To: louislam/uptime-kuma ***@***.***> Cc: Maryam Saleem ***@***.***>; Comment ***@***.***> Subject: Re: [louislam/uptime-kuma] `v2.0.0`-Release <- Read this for performance problems (Issue #4500) You don't often get email from ***@***.*** Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Please refer to https://github.com/louislam/uptime-kuma/wiki/Migration-From-v1-To-v2 If anything is missing, feedback is appreciated — Reply to this email directly, view it on GitHub<#4500 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHCBSCYNP3IPPZ2FH756ZEL2DQLGDAVCNFSM6AAAAABDNIX4NKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJQHA3TSMJWGA>. You are receiving this because you commented.Message ID: ***@***.***>

[msaleemCinnova]

@CommanderStorm ,
Dear Frank,

I am running Uptime Kuma in a non-Docker environment because I have integrated a custom module into the existing setup and submitted a pull request for these changes:

• Add SIP Request Ping and Registration Options Feature #5382

To maintain my customizations, I update and manage the application without Docker. So i want to migrate from v1 to v2 without using Docker.

Here are the steps I follow for setup and execution & updates:

# Update from git
git fetch --all
git checkout 1.23.15 --force

# Install dependencies and prebuilt
npm install --production
npm run download-dist

npm run setup

# Option 1. Try it
node server/server.js
# Start Server
pm2 start server/server.js --name uptime-kuma

Looking for the migration guide without using Docker.

[Wraaath]

You could build your own Docker image with the patch maybe? Unless you really don't wanna use Docker

[CommanderStorm]

It sounds like you are looking for a development environment.
We have documented this here: https://github.com/louislam/uptime-kuma/blob/master/CONTRIBUTING.md

What you are trying to do is otherwise not well supported as it is not the main path.
You can extend it, but make sure you actually have the state as you expect it. Both npm run setup and the git part may reset your changes.
In any case, there is no official support from us for this kind of deployment as I don't have capacity for this.

You can test pull requests via this path: https://github.com/louislam/uptime-kuma/wiki/Test-Pull-Requests

[msaleemCinnova]

@CommanderStorm ,
Thank you for your response and guidance.

I wanted to let you know that I have already opened a pull request (PR) to contribute to the project. My goal is to enhance Uptime Kuma by adding SIP-related features, and I fully understand that my deployment method is not the standard approach.

However, I'm currently running Uptime Kuma using PM2 in a Linux environment, not Docker, and I aim to migrate from v1 to v2 without relying on Docker. I realize this isn't officially supported

[msaleemCinnova]

@Wraaath ,
I’m currently running Uptime Kuma using PM2 in a Linux environment instead of Docker, and my goal is to migrate from v1 to v2 while continuing to use PM2. I prefer not to rely on a custom Docker image with patches for long-term use. However, if creating a custom image is necessary to complete the migration process, I’m open to exploring that option temporarily.

Could you please clarify what specific patch or adjustments you are referring to? Your advice will help me ensure a smooth migration while aligning with the project’s standards.

[CommanderStorm]

@msaleemCinnova
Sorry, did not have the time to look into said PR. I am currently quite overloaded with Uni and other work.

Since you already wrote the code, please just post it. Having a discussion is not quite relevant to adding notification providers or basic (cannot be implemented using current monitors) monitors like the one you are trying to add.

I don't know what a SIP is => please preferably include a test case that it works (see the mqtt test cases for examples how this can be done).

[msaleemCinnova]

@CommanderStorm ,
Thank you for taking the time to respond despite your busy schedule—I completely understand how demanding university and other work commitments can be.

I will add the code to submitted the PR. I’ll respect your preference to keep the process straightforward and will avoid extensive discussions.