watcher afk / windows behind proxy (squid)

[OlivierMary]

Describe the bug

With my personal PC no problem, but at work with proxy for afk and windows watcher I have this error:

2019-01-22 12:02:24 [INFO ]: aw-watcher-afk started  (aw_watcher_afk.afk:53)
2019-01-22 12:02:25 [WARNING]: 503 Server Error: Service Unavailable for url: http://localhost:5600/api/0/buckets/  (aw_client.client:72)

no problem with firefox plugin or other plugins

To Reproduce

Steps to reproduce the behavior:

1. Setup squid proxy
2. setup windows proxy to squid
3. Start afk or windows proxy

Expected behavior

No bug

Environment

• ActivityWatch version: 0.8.0b7
• OS: Windows 10

[johan-bjareholt]

Ah, good point. I used to have that same issue but was able to circumvent it with a small tweak as you can see in this stackoverflow post. I have only tried that quick solution on Linux though, but I assume that it works on windows as well.

https://stackoverflow.com/a/8287752/2127148

In the long run we should probably add a option to set proxy in the aw-client.ini configuration file in aw-client as that would be a simpler fix.

[OlivierMary]

Hi @johan-bjareholt

I already have this env var for all my system:

[johan-bjareholt]

@OlivierMary Try to set them in lower-case as well

[OlivierMary]

@johan-bjareholt On windows env var are case insensitive..., but in code if you search it with case sensitive maybe... will try !

[OlivierMary]

Same,

If that can help you, more log when I start watcher manually (.exe vs tray)

[johan-bjareholt]

That error is different so something makes the code go a different path from your initial report at least. An error occurs while an error is occuring, there's clearly badly written code there which needs to be fixed. So this code is not the root of the problem, but it needs to be fixed to understand what the root of the problem is.

https://github.com/ActivityWatch/aw-client/blob/0cf3de54dd60ded57832b0e6a88649f6bf8f67fd/aw_client/client.py#L74

@ErikBjare If I understand this correctly, we should handle the exception json.decoder.JSONDecodeError instead of json.JSONDecodeError.

[ErikBjare]

I think that last error is probably due to the proxy returning something that's not JSON, and it seems to me the exception is handled correctly. The documentation states it's called json.JSONDecodeError (but is probably located in json.decode, although shouldn't matter as the local variable 'json' referenced before assignment is the issue, which makes no sense to me in the latest version of the code.

I've recently rewritten that code anyway, and this issue is mostly a curiosity and does not seem to be an actual issue with ActivityWatch, so closing.

[johan-bjareholt]

How to properly use ActivityWatch with a proxy is not clear though, so I'll re-open this issue until we create a new issue where we describe that we need to document it and possibly also add a configuration in aw-client.ini for it.

[ErikBjare]

@johan-bjareholt Why not just use an SSH tunnel? Works fine. Using it with a proxy is probably not at all secure so we shouldn't put any effort into supporting it anyway.

Edit: Using squid as a proxy to localhost seems to need special configuration anyway: https://stackoverflow.com/questions/33169382/allow-access-to-localhost-from-localhost-by-squid

[OlivierMary]

We cannot all use ssh tunnel ... The problem is the watcher don't use correctly proxy or internet configuration. And why ? Actually it's only in localhost.... Just one try 127.0.0.1 and another localhost to connect to server. Maybe need to have a common configuration between server / watchers / client GUI. Will be hard if we have to configure proxy or other thing on all watcher / client ... Boring for users.... And when server will be online... A common configuration will be great !

[OlivierMary]

I use already squid + corkscrew. But if I can't put the proxy conf in worker that cant be ok..

[ErikBjare]

Proxy configuration should just work if you've declared the HTTP_PROXY env variable (as said in the documentation).

The problem is the watcher don't use correctly proxy or internet configuration. And why ? Actually it's only in localhost.... Just one try 127.0.0.1 and another localhost to connect to server. Maybe need to have a common configuration between server / watchers / client GUI. Will be hard if we have to configure proxy or other thing on all watcher / client ... Boring for users.... And when server will be online... A common configuration will be great !

I'm really having difficulty understanding what you're trying to say here. As far as I can tell it is using the proxy configuration correctly, otherwise you wouldn't get the 503 Service Unavailable error. The issue is most likely the proxy not handling it correctly, as the StackOverflow question I linked above suggested.

This question on SO also suggests it could be a consequence of a proxy configuration issue: https://superuser.com/questions/124897/503-service-unavailable-what-really-it-means

What I don't get is why you'd need a proxy for something that runs locally in the first place? Using ActivityWatch with a proxy should never be needed (unless you're trying to run the server on a different machine, which is highly unsecure and therefore not supported). Maybe you could try and reset the environment variables specifically for ActivityWatch so it doesn't use the proxy?

[ErikBjare]

@johan-bjareholt I really don't see why you reopened the issue. The requests library should (and seems to) pick up the proxy configuration correctly from environment variables and the 503 suggests it actually did. What is there for us to do here? (apart from helping him debug his proxy config)

[ErikBjare]

If it's a issue with Squid not liking the use of localhost (for some DNS reason) when connecting to aw-server, try changing hostname in aw-client.ini to 127.0.0.1 instead of localhost. (As the proxy might not like the use of localhost)

You could also try setting the environment variable NO_PROXY=localhost which should make the requests lib ignore the proxy settings correctly (as suggested in this SO answer). Should be a better option than resetting the environment variable that I wrote about earlier.

[OlivierMary]

Ok, will try that next week not at work this week... Http_proxy is ok. No_proxy maybe. Will check that.

[johan-bjareholt]

Why not just use an SSH tunnel? Works fine. Using it with a proxy is probably not at all secure so we shouldn't put any effort into supporting it anyway.

I'm proposing to document how to make activitywatch avoid using the system-default proxy.

Proxy configuration should just work if you've declared the HTTP_PROXY env variable (as said in the documentation).

We should have this note in our documentation, people shouldn't have to dig through the request library documentation to find a solution.

Also, some people don't know how to set ENV vars, so it might also be a good idea to add a option to aw-client.ini which can override the default system proxy as that will simplify things a lot.

[ErikBjare]

We should have this note in our documentation, people shouldn't have to dig through the request library documentation to find a solution.

This is a very common practice for configuring a proxy, and not the issue here, rather the cause of the problem.

Also, some people don't know how to set ENV vars, so it might also be a good idea to add a option to aw-client.ini which can override the default system proxy as that will simplify things a lot.

If anything, we should configure aw-client to ignore proxies from env variables altogether (at least if the address of the server is 127.0.0.1/localhost).

[johan-bjareholt]

I made an issue for aw-client here, let's discuss the rest there and I'll close this issue.

ActivityWatch/aw-client#41

If anything, we should configure aw-client to ignore proxies from env variables altogether (at least if the address of the server is 127.0.0.1/localhost).

That might be a good idea, but for now I added a notice in the docs. If that solution solves it nicely we can remove this notice.

ActivityWatch/docs@2cb481c

[OlivierMary]

Hi there, I changed all .ini + change to last released and I have now the buckets, but no entries for afk / window...

Any idea?

[ErikBjare]

@OlivierMary Something is probably wrong with your aw-client configuration, causing the two watchers that use it to fail. Check the logs.

[OlivierMary]

Hi, Already check no error... Just persistent DB in queues directory are very huge 100mb for one...

[ErikBjare]

Just persistent DB in queues directory are very huge 100mb for one...

Then they are probably in the process of emptying... Could take a while. Check the raw data tab to see if any new events arrive to the buckets.

[OlivierMary]

@ErikBjare Yes maybe that, Windows is ok now just AFK no new event count... but a very long AFK timer is growing.... will see. DB size didn't grow down.

[johan-bjareholt]

It is also a known bug that consuming the queue is much slower on windows than on macOS/Linux. The pre-merge events a few releases ago should have improved this a bit, but if the queue has days of events it might still take hours to be fully worked through. Exactly why it's so slow on windows though I don't know.

[OlivierMary]

All is good now, that take 4-5h to consume ~1 month.