Improve the reports' expiration notification

[evilaliv3]

Proposal

This ticket is to discuss the possible improvements to the reports expiration notification.

A simple proposal could be to extend configuration enabling not only to configure the number of days before expiration on which starting to send expiration notifications but also the period of these notifications.

Motivation and context

As reported by many users and as well recently pointed out by @giorgiofraschini on #4278, the current reports's expiration notification is pretty unsatisfying.

Current mechanism was designed for short living reports with 30 days of data retention, where notifying users starting 72 hours before and sending an email every day was working perfectly sending a total of 3 emails that was very appropriate.

In current whistleblowing systems, in relation to the EU directive this simple implementation is not anymore appropriate for example when the data retention policy is set to 1 year;

• sending an email only 72 hours before the expiration and then every day would lead recipients to loose their report before checking in.
• sending an email starting from 1 month before and every day would lead users to loose their reports because confused of receiving an email every day for 30 days.

Before changing anything we should well consider that the new algorithm should:

• work well when multiple expiring reports exists on different dear deadlines
• work well when users are enabled to delete a report.

It is in fact better to annoy users every day with a repeated email that communicates that something is expiring in the next days than optimizing emails and risking to not inform users about a near expiration.

\cc @susannaferro @giorgiofraschini @danielvaknine @elbill @larrykind

[giorgiofraschini]

I agree that the 3days setting is non appropriate. reminders are too close to the expiration date.
Daily reminders are annoying, too. New algorithm should consider the possibility not to send daily reminders and the possibility to set an earlier reminder (14 days?)

[evilaliv3]

\cc @elbill @larrykind @susannaferro @msmannan02

[danielvaknine]

We would suggest simply extending 3 days to 7 days to make it as simple as possible, yet still effective. In addition, maximum 1 email per day (even if there are multiple reports soon to be expired)

If you want more flexibility on this, we suggest a simple "mathematical" formula to automatically calculate the notification timeframe, so that the admins don't have yet another parameter to change/adjust.

(also, the expiration date for most EU countries is 2 years)

[larrykind]

Hi,
I think this behaviour is related to a very low probability case in wich the receiver never check emails for a long time and then checks them just few days before the expiration

From our point of view to manage the case effectively I think the the reminder could be sent once per week from the report's date, keeping active the mechanism to send reminders in the last 72 hours before the deletion.

In cases where the receiver can't check email for a long time it should arrange in the way that another fellow will check emails and manage reports, or to have another receiver in the system. Otherwise it could be deducted that wistleblowing is not a critical asset for the Company and the receiver doesn't do his job correctly.

[giorgiofraschini]

Hi all. I think we there may be some confusion.
One thing is the weekly reminder for unread reports/comments on report. One reminder per week is reasonable.
Another thing is the reminder of the expiry of the report. The recipient may want to extend the expiration date or extract some information before the expiration; if he gets a reminder with only thre days on a Friday afternoon, the report is lost forever to him.
7 days is a low number of days, too, in case he is on leave for example.

[danielvaknine]

What about a simplified version like this, e-mailing the user(s) on format "X days before the expiry".
14
7
4
3
2
1

Just an idea in order to satisfy people's requests. In our opinion though, the Expiration Date is normally set to 2 years since that's the EU Whistleblowing Directive standard timeframe. But this would also work for anyone setting it to e.g. 90 days or 5 years.

[evilaliv3]

@danielvaknine : 14, 7, 4, 3, 2, 1 in relation to what?
Supposed the system has only one expiring report, the system is always up and there is no downtime in any of the mail servers involved has downtime the answer is clear: the timing should be related to the expiration of the single report.

The situation is more complicated if you consider:

• the system may have multiple reports expiring (e.g. after operation of 2 years there are systems that have different reports expiring one after the other in different subsequent days)
• you may need to perform maintainance and turn off the system and you want that as soon that the system is turned up eventually not sent emails are sent
• the system may do their evaluations in timing but if the external mailserver are not available to deliver the email or miss to deliver the email sent 14 days before will lead to send a new email only 7 days before expiration. Right know the attempt of sending an email every day was designed considering this situation. It is inefficient and noisy but more resilient.

[danielvaknine]

What we meant was that the recipient would receive emails all of these days:
14 days before the expiry
7 days before the expiry
4 days before the expiry
3 days before the expiry
2 days before the expiry
1 days before the expiry

Of course, it's just a very rough sketch and would need to be adapted to fit the requirements you put out here

[evilaliv3]

Thank you @danielvaknine , actually writing a resilient alorithm that does this is not simple.

Currently, the algorithms fast, resilient and simple because it uses only the expiration date as information to execute without keeping any state of when the email was sent.

Consider as an example to have a policy of 30 days and to have reports expiring on December 1, 2, 4, 6
Current algorithm works checking if there is at least a report expiring and send an email.
It works if I keep the system off 3 days and I restart it, and the system will check just that there are reports expiring and send an email.
It works if someone changes the expiration of the any report, eventually anticipate it.

In your proposal:

• is it enough for running the procedure of mailing that you think to use just read the expiration date?
• if now which data would you need to check and annotate?

For example:

• for sure we need to keep the date of the last email? probably information about which report it was related? probably the current expiration deadline of each report when that email was sent to check if something varied?
  Try to think to these aspects and formulate which is the data we need to keep (like on a spreadsheet) and how would you do the check on paper to see if you needed send an email