Large lists fail to update on slow i/o

[xRuffKez]

Prerequisites

• I have checked the Wiki and Discussions and found no answer

• I have searched other issues and found no duplicates

• I want to report a bug and not ask a question or ask for help

• I have set up AdGuard Home correctly and configured clients to use it. (Use the Discussions for help with installing and configuring clients.)

Platform (OS and CPU architecture)

Linux, AMD64 (aka x86_64)

Installation

GitHub releases or script from README

Setup

On one machine

AdGuard Home version

latest

Action

Adding huge lists, such as NRDs will cause timeout on add or refresh due to i/o error.

Expected result

Add huge lists even with slow or averange drives i/o.

Actual result

Unless data folder is not stored as ramdisk, huge lists will fail to load due to i/o timeout.

Additional information and/or screenshots

Every architecture is behaving like this. Pihole and Technitium does not have that issue.

[GrumpyGiuseppe]

I just added your 'New Domains' list and I got the same issue at the 3rd refresh.
System:
ZimaBoard 216 with Sata 3 and Crucial MX500 SSD.

[xRuffKez]

I just added your 'New Domains' list and I got the same issue at the 3rd refresh. System: ZimaBoard 216 with Sata 3 and Crucial MX500 SSD.

Best is to store data folder af agh in ramdisk until its fixed.
You can archive this to add a ramdisk to fstab pointing to /opt/AdGuardHome/data for example

[Dynamic5912]

Same here.. likewise I have issues with slow cache responses when large lists are used (you commented on my post about this I think) :)

[xRuffKez]

@ainar-g that's a serious issue :)

[ainar-g]

I'm not sure what you want me to do here. I've filed #7044, since we've already had a TODO in the code about it, but that will only increase the timeout on AGH's side, and the server could have its own timeout. You haven't provided any logs, so it's hard to know which timeout is hit.

[GrumpyGiuseppe]

@ainar-g that's a serious issue :)

Slightly OT - are you having slow cache issues as well?

I can only need to enabled your NRD list and it's enough to bring it to a crawl with regards to response times.

I still have 2ms average (24h stats) and 1ms at recent requests.

[Dynamic5912]

@ainar-g that's a serious issue :)

Slightly OT - are you having slow cache issues as well?
I can only need to enabled your NRD list and it's enough to bring it to a crawl with regards to response times.

I still have 2ms average (24h stats) and 1ms at recent requests.

With how many blocklists and rule counts?

[GrumpyGiuseppe]

10 lists and 4.526.135 rules.
Most of HaGeZi's recommended lists.

[Dynamic5912]

10 lists and 4.526.135 rules.

Most of HaGeZi's recommended lists.

Interesting. What's your cached response times for some entries from the query log?

[GrumpyGiuseppe]

10 lists and 4.526.135 rules.
Most of HaGeZi's recommended lists.

Interesting. What's your cached response times for some entries from the query log?

Sry, I am an idiot.
I forgot to mention that I disabled AGHs cache and only use Unbound.
Most are 1ms, rare 2-3ms.

[xRuffKez]

I use AGH with unbound as resolver, unbound uses root servers and handles caching.
I have around 7mil domains in total.
My NRD List 30day, Hagezi Pro++, TIF, OISD small and some others.
With ramdisk or without (server nvme on PCIe it takes only around 10secs to refresh the lists.
But i heard of issues on "normal harddrives/ssds" where AGH aboards or even crashes (!!!) due to i/o timeout. Thats definetly NOT an http timeout even it shows when the error occurs!

[xRuffKez]

@ainar-g that's a serious issue :)

Slightly OT - are you having slow cache issues as well?

I can only need to enabled your NRD list and it's enough to bring it to a crawl with regards to response times.

Where is that off topic? Everything works fine till agh sees a big list on common hardware and cease to fetch.

[GrumpyGiuseppe]

Yeah my AGH also crashed at/after the timeout.
(on dietpi x64, no container)

[ainar-g]

@xRuffKez, what makes you think that it's not an HTTP timeout? And once again, please provide the verbose logs of the operation. Because I'm still not sure how this is an AdGuard Home issue.

@Dynamic5912, please don't post off-topic messages, as they don't help finding the original issue. If you want to discuss something, there are Discussions for that. Issues are not a forum, and must stay on-topic.

[GrumpyGiuseppe]

I found a good final solution for this, with the suggestion of @xRuffKez
Example on my dietpi system:

• auto download filters with cron [curl -o /mnt/dietpi_userdata/downloads/pro.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/pro.txt]
• put them in agh [/mnt/dietpi_userdata/downloads/pro.txt]
• make a ramdisk for agh's filter location [tmpfs /mnt/dietpi_userdata/adguardhome/data/filters tmpfs size=100M,noatime,lazytime,nodev,nosuid]

[hagezi]

@ainar-g

Lists for testing:
https://github.com/xRuffKez/NRD/raw/main/nrd-30day_adblock_part1.txt
https://github.com/xRuffKez/NRD/raw/main/nrd-30day_adblock_part2.txt

Steps to reproduce:

• Add the above-mentioned lists to your existing lists. Adding them works without any problems.
• Make an overall update of all lists via the GUI, then you will receive the following error message.
  

It looks to me that there is a timeout (GUI timeout?) for the overall update that is triggered. I have also put the lists on a local web server so that they load faster, but this also causes the "network error". The error occurs after 180 sec.

Swapping the AGH filters folder to RAM fixes the error, as @GrumpyGiuseppe has described.

Please reopen, thanks!

EDIT:
Verboselog: aghlog.txt

I don't see anything conspicuous in the log, no filters were updated because there were no updates. But the network error still appeared in the GUI, a GUI timeout?

[ainar-g]

@hagezi, thanks for the tip. After adding a few more lists I was able to reproduce it. Seems like the AGH's HTTP server timeout is hit, not the client one. I've filed #7060 about proper configuration, and for now it's likely that we'll just increase the hardcoded one to match the timeout of the client.

[hagezi]

Thanks @ainar-g

[ainar-g]

@xRuffKez, please test version v0.108.0-a.912+4392255d on the Edge channel.

[hagezi]

@ainar-g I have tested version v0.108.0-a.912+4392255d. The adjustments fix the error.

Thanks!

[GrumpyGiuseppe]

Nice,
I got two additional recommendations for people, who want to use the ramdisk fix:

• if you download manually use my script in post#7050 to avoid unneccesary bandwidth usage
• with such big filter lists it may be useful to set a garbage collection memory limit for AGH. For systemd this can be achieved with environment variables like this:

[Service]
Environment=GOMEMLIMIT=1024MiB