Broken web links

[cofinoa]

Topic for discussion

Moving the discussion about broken web links, from issue cf-convention/cf-convention.github.io#318

Two workflows/actions has been created to check links:

1. check_jekyll_build.yml: An action with 2 main jobs triggered when a PR it's created :
   A. to check links in Markdown files (./**/*.md)
   B. to check that Jekyll can build the Website
2. check_links_cron.yml: The other action that runs on Mondays, and has also 2 main jobs:
   C. to check that Jekyll can build the Website
   D. to check links on the site built on job C, and if it fails a new issue, it's open: #490

The exclusion rules are at .lychee/config.toml which are being used for both actions (1 and 2), but we can create different ones for each action, in case it's needed.

Currently, the following URL are being excluded:

exclude = [
    # Data/cf-standard-names/
    "http://glossary.ametsoc.org/wiki",                   
    "https://www.unidata.ucar.edu/software/udunits/udunits-current/doc/udunits",
    "https://www.unidata.ucar.edu/software/udunits/udunits-2.2.28/udunits2.html", 
    "https://www.sciencedirect.com/science/article/pii/0967063793901018",
    "https://www.ipcc.ch/ipccreports/tar/wg1/273.htm",
    "http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata",
    "http://gcmd.nasa.gov/Resources/valids",
    #
    "cfeditor.ceda.ac.uk",                                # standard_name_rule, vocabularies, discussion
    "https://mailman.cgd.ucar.edu/pipermail/cf-metadata", # discussion, governance
    "http://mmisw.org/ont", # faq (TIMEOUT)
    "https://mmisw.org/ont", # faq (TIMEOUT)
    "http://www.cgd.ucar.edu/cms/eaton/cf-metadata/clivar_article.pdf", # Data/cf-documents/cf-governance/cf2_whitepaper_final.html
    "http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-current.html", # Data/cf-documents/requirements-recommendations
    "https://www.usbr.gov/lc/socal/reports/SMappend_C.pdf", # Data/area-type-table/**/build/area-type-table.html
    "https://cf-trac.llnl.gov/trac/",                     # 2018-Workshop, 2019-Workshop
    "http://mailman.cgd.ucar.edu/pipermail/cf-metadata",  # 2019-Workshop
    "https://www.wonder.me",                              # 2021-Workshop
    "https://figshare.com/account/articles/24633939",     # 2023-Workshop
    "https://figshare.com/account/articles/24633894",     # 2023-Workshop
]

Some of the excluded URL, are spurious broken links, which are temporarily broken.

Other, are permanently broken, and we need to decided what to do [1].

Also, paths are been excluded, mainly because they contain some documents with invalid encoding, or many broken relatives links (i.e. Trac-tickets):

exclude_path = [
    "_site/Data/cf-standard-names/docs/guidelines.html",
    "_site/Data/cf-conventions/",
    "_site/Data/Trac-tickets/",
]

[1]
For example, for the
https://www.ipcc.ch/ipccreports/tar/wg1/273.htm

we could link to a capture from the Wayback Machine:
https://web.archive.org/web/20181104000136/http://www.ipcc.ch/ipccreports/tar/wg1/273.htm

[cofinoa]

The issue cf-convention/cf-convention.github.io#493, is an example of a issue with broken link report has opened, and updated with new comments added to the issue from checker cron job:
cf-convention/cf-convention.github.io#493 (comment)

When the action (i.e. cron job ) it's re-run manually and "new" error appear, and the others disappear. The issue has been updated with the report for this "manual" check:
cf-convention/cf-convention.github.io#493 (comment)

IMO, there are 2 pending actions that we need to discuss:

1. How to proceed with this spurious check errors? How long an error must persist and/or how many times appears, to consider an action on it, i.e. to be excluded?
2. There are already some link and exclusions that need to review similarly as previous point

It maybe would be useful to add this to the next meeting for the Information Management Team @cf-convention/info-mgmt

[JonathanGregory]

Dear Antonio @cofinoa

I agree that this issue would be good to discuss at the information management team meeting. Meanwhile, thanks for the work you've already done to improve it. I like the more informative report. Here are some thoughts:

• Would it be better if we had just one "broken links" issue and kept it open, so that the weekly job can simply update it if there is anything to report? Updating it brings it to the top of the default list, so it will be easy to find. Having a new issue each time there's a fault means we have to keep on closing them manually. Having a single issue would also make it easy to compare reports from different weeks.

• Some links are intermittently broken. Even though they sometimes work, it would probably be better to deactivate them. We could keep them as text only in the web pages, I presume, with some warning e.g. "See this_URL for further information, but be aware that it's a bit flaky."

• Some links take a long time to respond, and the checker is impatient. Can it be asked to wait longer for sites that we know are sometimes slow? For example, http://vocab.nerc.ac.uk/collection/P01/current, which timed out in your last report, is working, but it's very slow. It took the best part of a minute to produce the page when I tried it just now.

• I'm sure that with some bulk editing we could deal with broken relative links in archives like the Trac copy.

• Other cases need manual attention. You gave the example of https://www.ipcc.ch/ipccreports/tar/wg1/273.htm. Where do we link that, by the way? This is section 7.2.2.3 of the 2001 IPCC report. The up-to-date "official" web version is https://www.ipcc.ch/site/assets/uploads/2018/03/TAR-07.pdf, which is the entire chapter, so we'd have to alter the text to say something like "See section 7.2.2.3 on p12 of https://www.ipcc.ch/site/assets/uploads/2018/03/TAR-07.pdf."

• It will take time to work through such things. I think it's fine to exclude a link from the check, so it doesn't keep on bothering us, once we've decided we want to keep it but that it needs to be updated.

Best wishes

Jonathan

[cofinoa]

Dear @JonathanGregory, I think this requires more discussion, and it's a long-lasting action.

w.r.t.

Would it be better if we had just one "broken links" issue and kept it open, so that the weekly job can simply update it if there is anything to report? Updating it brings it to the top of the default list, so it will be easy to find. Having a new issue each time there's a fault means we have to keep on closing them manually. Having a single issue would also make it easy to compare reports from different weeks.

I agree. Just for now, I have re-opened cf-convention/cf-convention.github.io#493, to be updated with the forthcoming cron to check links.

Just needs more logic on the action workflow, but I can update the action, and find the last issue without the restriction to be just in one state (open or close), and if error is found, the issue it's updated with the report, and re-open if status is closed; or close if no-error and status is open.

[cofinoa]

@JonathanGregory w.r.t.:

Some links take a long time to respond, and the checker is impatient. Can it be asked to wait longer for sites that we know are sometimes slow? For example, http://vocab.nerc.ac.uk/collection/P01/current, which timed out in your last report, is working, but it's very slow. It took the best part of a minute to produce the page when I tried it just now.

I have increased the TIMEOUT and number of RETRIES:

timeout = 300 # maximum HTTP request timeout (defaut is 20 seconds)
max-retries = 10 # increase from default (3 retries)
retry-wait-time = 2 # Minimum wait time in seconds between retries of failed requests (default: 1)

But this is for all links, because the checker been used doesn't have a configuration per URL/link.

[JonathanGregory]

Thanks, @cofinoa. Tolerating a long wait for all of the links is fine.

[ethanrd]

Hi Antonio @cofinoa and Jonathan @JonathanGregory,

I agree it would be good to discuss this at the next CF info-mgmt team meeting. I will add it to the agenda for that meeting.

[JonathanGregory]

In this week's link checker output, with the more tolerant timeout thanks to Antonio, there is only one broken link, namely http://coastwatch.pfeg.noaa.gov/erddap/convert/units.html. It's working now, but I've noticed that we quite often have problems with it. It's referred to in the FAQ, thus:

A units conversion page on the ERDDAP site lets you try different unit strings, and provides additional context on UDUNITS (and UCUM units) further down the page.

It's potentially useful information, so it seems a shame to remove it, but since it persistently goes wrong, I suggest we could deactivate it as a link e.g.

A units conversion page on the ERDDAP site (http://coastwatch.pfeg.noaa.gov/erddap/convert/units.html; note that this link does not always work) lets you try different unit strings, and provides additional context on UDUNITS (and UCUM units) further down the page.

Is that a reasonable approach for this and other similar problems detected by the link checker?

Antonio @cofinoa, why does the link checker append two reports to the issue, not just one?

[cofinoa]

Antonio @cofinoa, why does the link checker append two reports to the issue, not just one?

I have run manually run the check, and for each run, a comment it's added, with the report.

We can improve the process, adding some logic. For example, if the ISSUE is in status open, and the check pass without errors, it can be closed. Also, if the ISSUE is in status closed, and the check is giving some error, we can re-open the ISSUE.

what do you think?

[larsbarring]

Regarding the ERDDAP site I have several times checked shortly after receiving the link checker report, and it always work without problems. Could it be that the site blocks web spiders or something similar?

Further down that web page there is an alternative way for using the service from within software tools. Would it be possible to keep the link as is in the document, but actually have the link checker using one of the provided examples (e.g. this) to test the availability ?

[cofinoa]

@larsbarring, I think you are pointing in the right direction. I can add 2 additional configuration options:

1. Sent a HTTP header to indicate a more "common" agent browser
2. Accept 403 HTTP status (i.e. forbidden) as a valid check

both, or just 1 can be applied, but both has been to been applied to all URLs, because checker doesn't have a check confguration per URL

[JonathanGregory]

@cofinoa @larsbarring. Do I understand correctly that the ERDDAP site gives 403 because it has somehow detected the request is coming from an automated process? How does it know? I see that we could accept 403 as indicating that the link is valid. What are the implications of (1) i.e. sending an http header? Do mean, rather than https, perhaps?

[larsbarring]

@JonathanGregory : I am no expert on these matter, but well-behaved web spiders/crawlers/robots can be stopped by a small file robots.txt, but there are many ways to circumvent this.

[JonathanGregory]

Antonio @cofinoa, would it be possible to suppress the output completely when there are no timeouts, errors or "unknown"? That would be helpful because there would be no addition to the issue when nothing is wrong, and we would not be alerted.

[rmendels]

@cofinoa @larsbarring Can someone send me information on which ERDDAP site is being accessed, the request being made, and the IP that it is coming from? Also when it failed (date and time if available) so I can look in the appropriate log files.

[JonathanGregory]

Dear Roy @rmendels

Thanks for your help. The missing link is http://coastwatch.pfeg.noaa.gov/erddap/convert/units.html, with error 403, shown in the link checker report in cf-convention/cf-convention.github.io#493 (comment). According to what is says there, it reported on 27th May at 0941 UTC.

Best wishes

Jonathan

[rmendels]

@JonathanGregory I will look, but first can you change the URL to be https. It would still help if I have the IP, we have gotten much more aggressive in blocking, and particularly if you are coming from a data center (Digital Ocean, AWS, Google, Microsoft) there is a chance you have been blocked because of the actions of others. These days about 80%-90% of our "bad actors" are coming from data centers.

[JonathanGregory]

@rmendels, the job is run by GitHub somehow. Perhaps @cofinoa knows how to find the IP. We can change the protocol from http to https, certainly.

[JonathanGregory]

I've done it.

[cofinoa]

@rmendels, the link checker is running as a Github Action at Github-hosted runner (i.e. Azure).

Some discussion about whitelisting Github IP ranges can be found at:
https://github.com/orgs/community/discussions/26719

and the conclusion is that, IPs range for GH Action runners are "dynamic" and no just a short list of IP ranges.

More info about this:
https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#ip-addresses

I think it will be difficult to whitelist Github's IP range and alternative workaround will be required.

As @JonathanGregory has mentioned, the scheme will be updated from HTTP to HTTPS, and hope this mitigates some errors.

[rmendels]

@cofinoa @JonathanGregory @larsbarring Our ERDDAP has been getting hit extremely hard (the price of success), and we are learning the hard way that Java 17 made a lot of changes in memory allocation that affects the settings used when launching Tomcat. We have been working on fine tuning the options until we can get more memory for the server (we are the government this can take forever), but the failure you saw is because we had just done a restart of the service. Looking at the logs, your other requests worked. But please do use https, not http. HTH.

[JonathanGregory]

Thanks for your help, Roy @rmendels

[larsbarring]

And right now it dawned to me that we should check all problematic links to see if they are http: and then try changing to https:(maybe even all links irrespective of whether they fail or not). For the conventions document we did something similar in cf-conventions/463. Thanks @rmendels for pointing out the obvious :-)

[JonathanGregory]

Yes, I agree, when we next get one we should check that. Could the wrong protocol cause a timeout?

[rmendels]

@JonathanGregory In our case the timeout was because the service was rebooting. You just happened to hit that time. But these days different sites have different policies about http, depending on IT. Some may refuse you, some, like us, redirect you. Usually something is put in the header also.

[cofinoa]

@cf-convention/info-mgmt

I have updated the workflow for the link check cron job to:

1. If failure and issue is closed, reopen
2. If failure and issue exists, add comment

you can found more details at the proper workflow:
https://github.com/cf-convention/cf-convention.github.io/blob/5114facd3b03d727cd93a35b52a45c8e63463326/.github/workflows/check_links_cron.yml

As agreed, if the link check is successful, it will not be reported, regardless of the status of the issue, and the issue should be manually closed

[JonathanGregory]

Thanks, Antonio @cofinoa. We didn't hear from the link checker this morning, and nothing has been added to issue #493. No news is good news, I presume!

[cofinoa]

Yes!, check was OK!

For check report, see: https://github.com/cf-convention/cf-convention.github.io/actions/runs/9544516346

I closed issue #493.

I'm working on action about excluded links/files.

[JonathanGregory]

Dear Roy @rmendels and Antonio @cofinoa

At 0835 UTC today the CF link-checker reported

[403] https://coastwatch.pfeg.noaa.gov/erddap/convert/units.html | Failed: Network error: Forbidden

NB it has https now. Was there a problem at that time with the server, do you know, Roy?

It's good to see that the existing issue was automatically reopened, according to plan, Antonio - thanks!

Jonathan

[JonathanGregory]

https://coastwatch.pfeg.noaa.gov/erddap/convert/units.html didn't want to talk to CF again today. Unless there are other ways round this problem, I suggest that we should either deactivate the link (we can leave the URL in the text, but not as a link) or exclude it from the checker. What do you think, @cofinoa and @rmendels?

[larsbarring]

I have checked the link manually several times and it always works, I do not think that I ever have had problems using it. Hence I suggest that we keep the link but exclude it from the checker.

[ChrisBarker-NOAA]

All -- has this been all concluded? If so, let's close this. If not, then maybe make an issue out the remaining, issues?

[cofinoa]

I have made a last modification to include 429 status code, as "valid"

I have updated the valid status code to 200, 429 and 403 hope this avoids false positive errors

and closed the associated issue #493.

Yesterday's cron link check was OK:
https://github.com/cf-convention/cf-convention.github.io/actions/runs/12348913355

Summary

Status	Count
🔍 Total	31715
✅ Successful	31056
⏳ Timeouts	0
🔀 Redirected	0
👻 Excluded	485
❓ Unknown	0
🚫 Errors	0

When errors are detected the related above issue will re-opened (:crossed_fingers:)

[cofinoa]

I'm closing this discussion with the following summary of status.

Summary of Decisions

• Use a single "broken link" issue (#493) that is automatically updated.
• More tolerant link checker with higher timeout, retries and status codes (200, 429 and 403)
• Logic for re-opening issue #493.
• Track and manage exclusions for links that frequently fail but remain useful (like ERDDAP links).
• Update links to HTTPS where possible to avoid failures caused by outdated HTTP protocol.

Pending:

• How to handle persistently broken but useful links (like ERDDAP).
• Discuss quarantined links
• Add logic to close issues automatically if no errors are detected.
• Verify if all possible links are updated from HTTP to HTTPS

Note: lychee config file

verbose = "error"
no_progress = true
timeout = 300 # maximum HTTP request timeout (defaut is 20 seconds)
max-retries = 10 # increase from default (3 retries)
retry-wait-time = 2 # Minimum wait time in seconds between retries of failed requests (default: 1)

accept = ["200", "429", "403"]

exclude = [
    "cfeditor.ceda.ac.uk",                                # standard_name_rule, vocabularies, discussion
    "https://mailman.cgd.ucar.edu/pipermail/cf-metadata", # discussion, governance
    #BEGIN Data/cf-standard-names/
    "http://glossary.ametsoc.org/wiki",                   
    "https://www.unidata.ucar.edu/software/udunits/udunits-current/doc/udunits",
    "https://www.unidata.ucar.edu/software/udunits/udunits-2.2.28/udunits2.html", 
    "https://www.sciencedirect.com/science/article/pii/0967063793901018",
    "https://www.ipcc.ch/ipccreports/tar/wg1/273.htm",
    "http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata",
    "http://gcmd.nasa.gov/Resources/valids",
    #END Data/cf-standard-names/
#    "http://mmisw.org/ont", # faq (TIMEOUT)
#    "https://mmisw.org/ont", # faq (TIMEOUT)
    "http://www.cgd.ucar.edu/cms/eaton/cf-metadata/clivar_article.pdf", # Data/cf-documents/cf-governance/cf2_whitepaper_final.html
    "http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-current.html", # Data/cf-documents/requirements-recommendations
    "https://www.usbr.gov/lc/socal/reports/SMappend_C.pdf", # Data/area-type-table/**/build/area-type-table.html
    "https://cf-trac.llnl.gov/trac/",                     # 2018-Workshop, 2019-Workshop
    "http://mailman.cgd.ucar.edu/pipermail/cf-metadata",  # 2019-Workshop
    "https://www.wonder.me",                              # 2021-Workshop
    "https://figshare.com/account/articles/24633939",     # 2023-Workshop
    "https://figshare.com/account/articles/24633894",     # 2023-Workshop
#    "https://github.com/",                               # Uncomment if you hit GitHub Rate Limit: https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api
### QUARANTINE
     "https://coastwatch.pfeg.noaa.gov/erddap/convert/units.html", # faq
     "https://github.com/orgs/cf-convention/projects/1",    # Meetings/2020-Workshop.md
     "Data/cf-standard-names/current/build/kwic_index_for_cf_standard_names.html", # vocabularies (temporal issue with KWIC generator)
     "Data/cf-standard-names/86/build/kwic_index_for_cf_standard_names.html", # vocabularies (temporal issue with KWIC generator)
### 

]
exclude_path = [
    # Jekyll post build directory (i.e. _site)
    "_site/Data/cf-standard-names/docs/guidelines.html",
    "_site/Data/cf-conventions/",
    "_site/Data/Trac-tickets/",
    "_site/GDT/", # some HTML docs are invalid input encoded, choking the link checker
    "_site/CF-beta/", # some HTML docs are invalid input encoded, choking the link checker
]