Anonymous usage tracking

[laszlocph]

with consent of course

[AkiraNorthstar]

Which data should be transmitted?
If you need a tester, then I am happy to help.

[laszlocph]

Thanks for volunteering!

The full set of data that we are going to transfer is

• one part vanity metrics to have things to celebrate. Like how many people run Woodpecker, with what kind of build volume, etc
• and one part data to support product decisions: version control systems used, or if a certain feature is used etc. Woodpecker needs to form an identity over time and we need to focus efforts on things that are used, or things that are strategic. And we need data to evaluate the strategic decisions.

This issue will be updated with the initial set of metrics, and I pledge to keep the list transparent at all times.

[davidak]

I like the UX of Syncthing. At first start, you are asked for consent to collect usage data with a preview of that data.

There is also a public page with a great visualization of the data: https://data.syncthing.net/

But don't make the mistakes Muse Group did with Audacity like not asking for consent and using Google/Yandex for analytics.

audacity/audacity#835
audacity/audacity#889

I'm not sure if OpenTelemetry would be a usable tool for that.

[anbraten]

Home Assistant is doing a similar thing. This could be a good starting point:
https://www.home-assistant.io/integrations/analytics
https://github.com/home-assistant/analytics.home-assistant.io

or Octoprint

https://github.com/OctoPrint/OctoPrint/blob/027d8f8069b86a7f5e8c185a2d8f294b631c2f08/src/octoprint/plugins/tracking/__init__.py
https://tracking.octoprint.org/
https://data.octoprint.org/

[anbraten]

Data which would be interesting to collect:

every 24h (first on start)

• version
• users counter
• active repo counter
• used forge
• activated features?
• executed pipelines counter
• total pipeline execution time
• connected agents counter
• used agent backends
• server and agent OSes

[6543]

we need some server that do collect it & verify it's a legit request ... by do a callback and see if site exist?

[anbraten]

we need some server that do collect it & verify it's a legit request ... by do a callback and see if site exist?

That would require us to "collect" the public address of the instance, but we don't want that as it should be anonymous I guess and it would require them to be public.

I recently had a a look at Grafana and InfluxDB for this.

I think we should simply create a small GO server which takes HTTP requests and inserts that data into a connected db like InfluxDB. I would create an anonymous id at the first start of a server and save it to the server database. This id (do we even need that 🤔) would be used to send a request every x hours to our server, which simply adds an entry to the database (maybe directly aggregating it in the long term).

If we want to "protect" against abuse we could add an ip based limit, like you can only create a tracking id 10 times a day and each tracking id is only allowed to report data every x hours. Somehow like letsencrypt does it.

[anbraten]

Maybe we can also add some popup shown to an admin on the first login asking if he wants to send usage tracking and generate the tracking id after that

[qwerty287]

I'd like to ask some things about this again:

There's https://github.com/woodpecker-ci/analytics without real activity - do we still want to add usage tracking?

To be honest, I don't really see a value in it.
From @anbraten's comment about the data that should be sent:

version
users counter
active repo counter
executed pipelines counter
total pipeline execution time
connected agents counter

I don't really see how these can be used to improve development.

used forge
activated features?
used agent backends
server and agent OSes

For these, I can see a value, but this is data that should only be sent once. We can easily do a poll to find out how many users use which backend, which os etc.

[anbraten]

I think it would be still pretty helpful to get more insights about our users. For example we always have to consider if we need to add options on the repo level or the instance level. A user with an instance running on a PI is totally fine updating pipline configs or env vars, for larger companies or communities like codeberg that's often not possible to force all users to a specific way directly and needs a completely different update approach where features like config versions could help. Or things like quotas / user-provided agents would be really helpful for large instance, but probably no pi user cares about limits. We for sure have to provide both, but insides could help us to focus here. We could even do specific things like counting the amount of repos having pipeline option x set, allowing us to decide on actual data if we drop that option or have to keep it / provide an alternative.

I however expect the issue that quite a lot of users of the community are against any kind of tracking wherever it be totally public or not, which could make it pretty unreliable for us.
Using surveys isn't an option for me. Just think if you would want to fill it out yourself or take the who is using wp discussion as a reference.