v2.1.0-rc.0

[ruslandoga]

The highlights of this release candidate are:

• Introduction of Community Edition [see below]
• TOTP for 2FA [docs]
• Sites page got a new look with spark charts
  
• The new WordPress integration now works better thanks to the Plugin API
• Hosting documentation (now at plausible/community-edition) also got an update and now includes a more thorough Google integration guide
• Even though not present in this release candidate, importing the data between self-hosted and cloud will be available in v2.1.0

Introduction of Community Edition

As of version 2.1.0 this project will split into Plausible Community Edition and Plausible Enterprise Edition. The Community Edition (CE) will remain completely open source under the AGPLv3 license. The Enterprise Edition (EE) will contain extra features that will be published as source-available on Github, but we do not grant rights for anyone else to use, distribute or otherwise exploit these features. Read more here.

As part of this change, we are also rebranding the project. The new logos for CE and EE will be different so as to clarify which version is being referred to or used at any given time.

And finally as a contributor, you will have to sign a Contributor License Agreement (CLA) in order for you PRs to be considered for merging.

Upgrading to v2.1.0-rc.0

⚠️ This guide assumes you are using v2.0.0 ⚠️

Update config

Add a secret key for the new required TOTP_VAULT_KEY environment variable

_console

$ openssl rand -base64 32
aihU7k+GSBKbcVFuX9VEPyUhuwlKEomqA94/SQQ0NR4=

_{plausible-conf.env}

  BASE_URL=https://plausible.example.com
  SECRET_KEY_BASE=PkVcxRgQDfQyhPETlog3vvCrj5LdYFSv4ejPEJHJO+i/37w6RZfRjeVCpJayjUjJMfXsNurcv5upPhTRoD3KgQ==
+ TOTP_VAULT_KEY=aihU7k+GSBKbcVFuX9VEPyUhuwlKEomqA94/SQQ0NR4=

This new key is used to encrypt TOTP secrets at rest.

Update images

Update the images used for plausible and plausible_events_db (thank you @lordcris #3817 (comment))

_{docker-compose.yml}

plausible:
- image: plausible/analytics:v2.0.0
+ image: ghcr.io/plausible/community-edition:v2.1.0-rc.0

plausible_events_db:
- image: clickhouse/clickhouse-server:22.6-alpine
+ image: clickhouse/clickhouse-server:23.3.7.5-alpine

and restart the containers

_console

$ cd hosting
$ docker compose stop plausible plausible_events_db
$ docker compose rm plausible plausible_events_db
$ docker compose up -d

This will boot up the new version of the app.

Changelog

Following changes have been made since v2.0.0:

Added

• IP Block List in Site Settings
• Allow filtering with contains/matches operator for Sources, Browsers and Operating Systems.
• Allow filtering by multiple custom properties
• Wildcard and member filtering on the Stats API event:goal property
• Allow filtering with contains/matches operator for custom properties
• Add referrers.csv to CSV export
• Add a new Properties section in the dashboard to break down by custom properties
• Add custom_props.csv to CSV export (almost the same as the old prop_breakdown.csv, but has different column headers, and includes props for pageviews too, not only custom events)
• Add referrers.csv to CSV export
• Improve password validation in registration and password reset forms
• Adds Gravatar profile image to navbar
• Enforce email reverification on update
• Add Plugins API Tokens provisioning UI
• Add searching sites by domain in /sites view
• Add last 24h plots to /sites view
• Add site pinning to /sites view
• Add support for JSON logger, via LOG_FORMAT=json environment variable
• Add support for 2FA authentication
• Add 'browser_versions.csv' to CSV export
• Add CLICKHOUSE_MAX_BUFFER_SIZE_BYTES env var which defaults to 100000 (100KB)

Removed

• Removed the nested custom event property breakdown UI when filtering by a goal in Goal Conversions
• Removed the prop_names returned in the Stats API event:goal breakdown response
• Removed the prop-breakdown.csv file from CSV export
• Deprecated CLICKHOUSE_MAX_BUFFER_SIZE

Changed

• Limit the number of Goal Conversions shown on the dashboard and render a "Details" link when there are more entries to show
• Show Outbound Links / File Downloads / 404 Pages / Cloaked Links instead of Goal Conversions when filtering by the corresponding goal
• Require custom properties to be explicitly added from Site Settings > Custom Properties in order for them to show up on the dashboard
• GA/SC sections moved to new settings: Integrations
• Replace CLICKHOUSE_MAX_BUFFER_SIZE with CLICKHOUSE_MAX_BUFFER_SIZE_BYTES

Fixed

• Stop returning custom events in goal breakdown with a pageview goal filter and vice versa
• Only return (none) values in custom property breakdown for the first page (pagination) of results
• Fixed weekly/monthly e-mail report rendering issues
• Fix broken interval selection in the all time view Fix the interval selection in the all time view #3110
• Fixed IPv6 problems in data migration add transport_opts to data migration #3179
• Fixed long URLs display in Outbound Link breakdown view
• Fixed Sentry reports for ingestion requests derive jason encoder for request #3182
• Fix breakdown pagination bug in the dashboard details view when filtering by goals
• Update bot detection (matomo 6.1.4, ua_inspector 3.4.0)
• Improved the Goal Settings page (search, autcompletion etc.)
• Log mailer errors log mailer errors #3336
• Allow custom event timeseries in stats API Allow custom event timeseries in stats API #3505
• Fixes for sites with UTF characters in domain Fix domains that start with UTF character #3560
• Fix crash when using special characters in filter Escape special regex characters #3634

---

This discussion was created from the release v2.1.0-rc.0.

[lordcris]

I had to update the clickhouse server to make it work.

plausible_events_db:
- image: clickhouse/clickhouse-server:22.6-alpine
+ image: clickhouse/clickhouse-server:23.3.7.5-alpine

[ruslandoga]

Oh, right. I forgot about this! I'll add a note about this in the notes.

Thank you!

[rossdav70]

Thanks @lordcris updating the clickhouse container worked for me too.

[raabmar]

At the following step: docker compose rm plausible plausible_events_db

I get the question:
going to remove plausible_plausible_1, plausible_plausible_events_db_1?

I assume, that will delete my analytics data as well, what I don't want.
I also think, now I am half trough the steps to update, so I can't just stop in the middle

[ruslandoga]

This would remove the containers only, not the volumes. So your data is safe :)

[raabmar]

Do you know if it need any additional time to be up again? I remember one time I did this already (there was no update available so nothing changed). I think after it was up, it didn't show immediately but required a few minutes:

Still showing now:
502 Bad Gateway
nginx/1.18.0 (Ubuntu)

[ruslandoga]

plausible service sleeps for 10 seconds by default before doing anything to let PostgreSQL and ClickHouse initialize. So it shouldn't be much longer than that.

You'd need to check docker compose logs to see what's going wrong.

[raabmar]

too bad, then I messed it up, glad that the snapshot restore worked without a problem, however it seems it didn't work for me

[mdotk]

Aren't the commands "docker-compose..." not "docker compose..."

[ruslandoga]

Depending on Docker Compose version you are using, either of these commands might be correct. Please see:

• https://stackoverflow.com/questions/66514436/difference-between-docker-compose-and-docker-compose
• https://docs.docker.com/compose/

I started using docker compose in the docs since it's been more than two years that Docker Compose v2 (i.e. docker compose) had been released.

[digital88]

What are the supported versions of clickhouse in this release, I can see 23.3.x.x is supported, what about 23.8 and 24.1?

[ruslandoga]

I haven't tried any other versions but I'd think they are supported as well. I don't remember ClickHouse having any breaking changes in the past.

[digital88]

There is some problems with 24.2 #3855
Should probably explicitly note about recommended version of ch 23.3 in release notes or docs.

[ruslandoga]

The problem with 24.2 was temporary and has already been fixed.

Both release notes and example docker-compose.yml use clickhouse/clickhouse-server:23.3.7.5-alpine right now. So it should be obvious that this version should work :) But I don't think of it as the recommended version and I wouldn't want to scare self-hosters from trying out the newer ones.

[GeorgBNM]

Email is not working anymore after upgrading from 2.0.0.

Fault seems to be:
(ArgumentError) raise/1 and reraise/2 expect a module name, string or exception as the first argument, got: {:network_failure, ~c\"smtp.domain.com\", {:error, {:options, :incompatible, [verify: :verify_peer, cacerts: :undefined]}}}

Also, the new feature with spark charts is not working.

Any ideas?

[GeorgBNM]

I'm using standard docker without any proxy, I assume.

[ruslandoga]

Would you be able to share your configs? Are you serving Plausible over HTTP?

[GeorgBNM]

This is my docker-compose.yml:

`version: "3.3"
services:
#mail:
#image: bytemark/smtp
#restart: always

plausible_db:
# supported versions are 12, 13, and 14
image: postgres:12
restart: always
volumes:
- db-data:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=postgres

plausible_events_db:
image: clickhouse/clickhouse-server:23.3.7.5-alpine
restart: always
volumes:
- event-data:/var/lib/clickhouse
- ./clickhouse/clickhouse-config.xml:/etc/clickhouse-server/config.d/logging.xml:ro
- ./clickhouse/clickhouse-user-config.xml:/etc/clickhouse-server/users.d/logging.xml:ro
ulimits:
nofile:
soft: 262144
hard: 262144

plausible:
image: ghcr.io/plausible/community-edition:v2.1.0-rc.0
restart: always
command: sh -c "sleep 10 && /entrypoint.sh db createdb && /entrypoint.sh db migrate && /entrypoint.sh run"
depends_on:
- plausible_db
- plausible_events_db
#- mail
ports:
- 8000:8000
env_file:
- plausible-conf.env

volumes:
db-data:
driver: local
event-data:
driver: local
geoip:
driver: local`

plausible-conf.env:

SECRET_KEY_BASE=
TOTP_VAULT_KEY=XXXX
BASE_URL=https://analytics.domain.com

Quite standard, I think.

[ruslandoga]

BASE_URL=https://analytics.domain.com/

What's terminating the SSL connection? Please check that WS connections are allowed there.

[GeorgBNM]

Thanks, @ruslandoga found now a solution for this, because of your hint. 🙌

Now, only the SMTP is not working with rc.0.
So, I'll wait for the release.

[zell-mbc]

Just did my update as well. From 2.0 to v2.1.0-rc.0
The upgrade itself went through without problems. What I notice is that the badges on the "My Sites" landing page are now empty.
All I see is the badge itself but no chart.

I can click on the badge to get to the details just fine.
Whether email is working I'll be able to confirm next Monday when the automated reports are due.

[ruslandoga]

👋 @zell-mbc

Please see #3817 (reply in thread) above about WS connections blocked by proxies.

[zell-mbc]

Sorry, yes, adding the below to my nginx config did the trick :-)

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";

[flipzoom]

Plausible with Plesk + NGINX + WS (Websocket) Error

For all those who run the Plausible CE version on a server with Plesk, here is the way to get WS running in Plesk.

1. Go to /var/www/vhosts/system/<yourdomainname>/conf/nginx.conf
2. Find the block

#extension docker begin
location ~ ^/.* {
	proxy_pass http://0.0.0.0:8000;
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $scheme;
}
#extension docker end

3. Copy this block
4. Go to Your Domain > Websites & Domains > Dashboard > Docker Proxy Rules and remove the rule created for Plausible

5. Go to Your Domain > Websites & Domains > Hosting DNS > Apache & nginx
6. Deactivate the proxy mode and the Serve static files directly by nginx

7. Add the following and extended rule set under "Additional nginx directives"

location ~ ^/.* {
	proxy_pass http://0.0.0.0:8000;
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $scheme;
	proxy_set_header Connection 'upgrade';
	proxy_set_header Upgrade $http_upgrade;
	proxy_http_version 1.1;
}

8. Save your changes and restart nginx (via SSH sudo systemctl restart nginx or directly in Plesk Tools & Settings > Services Management > Reverse Proxy Server (nginx))

The dashboard should now work with Websocket. When copying the Docker rules, make sure that the port is correct if you have specified a different one and that the IP address is 0.0.0.0. Do not make any changes to the vHost template directly, these will be overwritten by Plesk.

[SaurabhGtr]

Hi there,

Adding the custom property doesn't appear in the display.

[ruslandoga]

@SaurabhGtr I think I've beed able to reproduce it. I'll open an issue to see what core team thinks :) Thank you!

[ruslandoga]

@SaurabhGtr In my case I was sending the properties incorrectly. Once I fixed that, the properties started showing up. Example (the property name is author): https://who.edify.space/stats.copycat.fun

[ruslandoga]

Please share more of your setup, how you encode and send the properties, etc. :)

[SaurabhGtr]

@ruslandoga Actually, I am not sure which encoding part you are referring to.

I am simply sending the event.

Am I missing something?

I do see the domain name is wrong.

[SaurabhGtr]

@ruslandoga It's Quite dumb of me; fixing the domain name worked, and it started showing.

Any idea on what we can do for not_set data?

The number is quite significant and this being none is weird.

[JeroenHaverkort]

anyone having trouble logging in after the update? even the register button redirects me to the login page, and every time i log in I get a 403 page

[ruslandoga]

Also would you be able to decode the contents of the _plausible_key cookie? Please see #3762

[rossdav70]

This worked. Thank you!

[JeroenHaverkort]

The Secret key base value didnt change. I might just delete the containers and set it up again, as the database only contained test data

[JeroenHaverkort]

Adding the table worked for me as well, so thats sorted!

[acolle]

@rossdav70 Which solution worked for you?
I tried adding the custom_domains table but that didn't do the trick for me

[Tony-MF]

Thanks a lot @ruslandoga .

Does importing the data between self-hosted and cloud will be available in v2.1.0 mean it will be possible to migrate data between two self-hosted servers using this feature?

[ruslandoga]

👋 @Tony-MF

I think it would be similar to migrating from Google Analytics to Plausible. Some details get lost, and hourly charts don't show much data. And due to how Plausible is calculating session data, visitor counts get over-estimated in exports. Here're some screenshots of my WIP for plausible.io (staging) data after export/import flow.

^{Views per visit is 0 in imported.com dashboards but won't be once #3792 is merged}

Last year:

December 2023:

December 27, 2023

[Tony-MF]

Excellent! Do you have any ETA when v2.1.0 will be available? It will help me move to another server as I could not find anyone who would migrate it for me.

[martinverbic]

I followed the instructions and updated my Plausible installation. All works well, except I can't seem to find the "IP Block List" in site settings. Is that not a part of CE?

[ruslandoga]

👋 @martinverbic

In v2.1.0-rc.0 Shields were still behind a feature flag. You can access them "directly" at /:website/settings/shields (e.g. https://who.edify.space/stats.edify.space/settings/shields)

[martinverbic]

Perfect! Thanks.

[raabmar]

I tried the update to 2.1. when it got out, but I had some troubles or did something wrong during the process, so I reverted back (snapshot of my VPS)

However, I read some others tried to update as well and they reported some issues (mostly in this thread I think).

I wonder about a timeline. Will there be another update (eg. to rc.1 or to the final v2.1.0) that addresses the topics mentioned by users e.g. there was an issue with email, but also other things

I have no feeling if this is weeks or months as somehow it's not the same pace and transparent communication I am used to from other tools I use. Does someone who is longer working with Plausible have some insights or experience from the past with those processes?

[ruslandoga]

👋 @raabmar

The main blocker right now is the exports/imports feature. Once it's done, we'll cut another release candidate. There haven't been any major changes since v2.1.0-rc.0 other than that.

e.g. there was an issue with email, ...

Ah, actually, the email issue would probably be solved with the new SMTP adapter. #3654

I'll check if it can be released in v2.1.0-rc.1 soon-ish.

but also other things

What other things? :) I'll need to double check, but I feel like most issues posted in this thread have been resolved.

[raabmar]

Thanks for the info - I didn't go over the entire thread anymore yet, I just remember over the last weeks reading about feedback / reports from others. Basically when I update, I don't want to make things worse.

It's different for everyone, in WordPress for instance I put a tremendous amount in troubleshooting, reporting, feedback, etc - but I can't do that everywhere. I try with Plausible to be a bit more of a user, without having to know the ins and outs of it. So I kind of wait in that case till a few others reported "good to go"

I try to estimate when I will be able to run the next version that doesn't break things - it feels I will wait a few more weeks then

[zell-mbc]

Are weekly email reports working for everybody else? Mine stopped at some point and I can't find a reason why.
Docker-compose log doesn't show anything email related, nor does my email server show any rejected connection attempts. So at first glance it looks like Plausible didn't even try to send something.

What would help troubleshooting is a "send report now" button somewhere in the UI. Is there maybe a way to connect to the docker container and force an email to be sent?

[zell-mbc]

Thanks a lot for your swift reaction. Settings look good in my opinion, and sending the email doesn't return an error. But still nothing seems to happen.

The only thing I notice is that in allowed_tls_versions tlsv1 doesn't have the "
Looks like I may have to wait for the next release candidate then.

[ruslandoga]

You didn't send any emails in your screenshot. You need to

Plausible.Mailer.deliver_now(email)

Judging just by the configuration, it would likely fail with peer verification for STARTTLS.

[zell-mbc]

Oops, sorry, I indeed missed the send command.
Now I get proper errors.

Plausible:
iex(plausible@de0239b42eb0)3> Plausible.Mailer.deliver_now(email) {:error, {:network_failure, ~c"mail.mydomain.com", {:error, :closed}}} iex(plausible@de0239b42eb0)4>

And on my mail server
postfix/submission/smtpd[2514451]: SSL_accept error from …pools.vodafone-ip.de[95.208.72.15]: -1 Mar 25 12:11:49 mail.mydomain.com postfix/submission/smtpd[2514451]: warning: TLS library problem: error:0A00010B:SSL routines::wrong version number:../ssl/record/ssl3_record.c:354: Mar 25 12:11:49 mail.mydomain.com postfix/submission/smtpd[2514451]: lost connection after STARTTLS from …pools.vodafone-ip.de[myIp

Looks like I am hitting the mail server with an old TLS version? Or, given that sending emails was working already and my mail server config hasn't changed, maybe it is indeed be the missing " from this allowed_tls_versions: [:tlsv1, :"tlsv1.1", :"tlsv1.2"], setting?
Is there a way to remove tlsv1 and tlsv1.1 from the allowed list?

[ruslandoga]

Quotes don't matter in this case

iex(1)> :tlsv1 == :"tlsv1"
warning: found quoted atom "tlsv1" but the quotes are not required. Atoms made exclusively of ASCII letters, numbers, underscores, beginning with a letter or underscore, and optionally ending with ! or ? do not require quotes
└─ iex:1:11

true

Is there a way to remove tlsv1 and tlsv1.1 from the allowed list?

Yes, you can mount a custom config: #3799 (reply in thread)

Or update it at runtime (but it won't be persisted across restarts):

prev_env = Application.get_env(:plausible, Plausible.Mailer)
new_env = Keyword.replace!(prev_env, :allowed_tls_versions, [:"tlsv1.2"])
Application.put_env(:plausible, Plausible.Mailer, new_env)

[zell-mbc]

Great! Unfortunately changing the protocols didn't help, after turning on debug mode on my mail server I think I can confirm that I am running into the STARTTLS issue, as you predicted.
I can see that the mail server is trying to match hostnames until it finally gives up.

So I will wait for the new release and test again. Thanks for helping along

[raabmar]

Every few weeks I look for this thread to see if there is an official release (not rc.0, rc.1. rc.2) of v2.1.0 as I only want to consider updating then. I am experimenting a lot with other tools, with Plausible I try to just be more of a normal user.

How to stay in the loop about it? After a few months of being in the Plausible space, I still have no idea at all how it works here with development, pace, releases, etc

[ruslandoga]

@raabmar you can subscribe to receive notifications of new releases by clicking the Watch button on the repo and then selecting Custom -> Releases :)

[raabmar]

Understood - done that - thanks!

[iwishiwasaneagle]

It seems that clickhouse/clickhouse-server:23.3.7.5-alpine doesn't work with Raspberry Pi 4 since it requires at least ARMv8.2-A.

[iwishiwasaneagle]

It looks like clickhouse is on v24 now. It'd be good to know what the minimum supported version is. The plausible/community-edition does mention that clickhouse/clickhouse-server:23.3.7.5-alpine should be used even with plausible v2.0 but this is not possible for RPi4 users like myself.

[ruslandoga]

👋 @iwishiwasaneagle

The solution seems to be to build your own image. Please see ClickHouse/ClickHouse#50852

[D13410N3]

1. Upgraded existing 1.5.0 installation to 2.0.0
2. Perform manual database migration as required
3. Trying to update installation to 2.1.0-rc.0 (with clickhouse updating)
4. Getting this error on start:

** (Postgrex.Error) ERROR 42P01 (undefined_table) relation "shield_rules_ip" does not exist
    query: SELECT s0."id", s0."inet", s0."action", s1."domain", s1."domain_changed_from" FROM "shield_rules_ip" AS s0 INNER JOIN "sites" AS s1 ON s1."id" = s0."site_id" WHERE (s0."updated_at" > $1::timestamp + ($2::numeric * interval '1 minute')) ORDER BY s0."updated_at"

I tried to perform manual table creation based on elixir-code from your migrations list - it can't create constraint with sites table because of column type mismatching (uuid vs bigint)
If I create table and index without constraint, I'm getting this error:

19:32:56.004 [error] GenServer Plausible.Shield.IPRuleCache.RecentlyUpdated terminating
** (Postgrex.Error) ERROR 42883 (undefined_function) operator does not exist: bigint = uuid
    query: SELECT s0."id", s0."inet", s0."action", s1."domain", s1."domain_changed_from" FROM "shield_rules_ip" AS s0 INNER JOIN "sites" AS s1 ON s1."id" = s0."site_id" WHERE (s0."updated_at" > $1::timestamp + ($2::numeric * interval '1 minute')) ORDER BY s0."updated_at"
    hint: No operator matches the given name and argument types. You might need to add explicit type casts.

What am I doing wrong?

[ruslandoga]

v2.1.0-rc.1 has been released: #4084

[ruslandoga]

Compared to v2.1.0-rc.0, it includes a new SMTP client, GA4 imports, and CSV exports and imports.

[ruslandoga]

v2.1.0 has been released: #4125