Add Log Viewer to Modern UI

[andrewbaldwin44]

Proposal

• Add custom log handler LogReader to capture all logs
• Add a route to web.py to return the logs in JSON format
• Add tests to ensure correct behaviour from /logs
• Add a new tab in the modern UI to fetch from /logs and display the out

• Add an exception handler to catch and log uncaught exceptions

Fixes #2437

[cyberw]

Something off about python 3.8? Havent tried it out just yet, but it looks very promising!

[andrewbaldwin44]

Strange, I'm not sure why Python 3.8 specifically was failing with the last build but seems everything is passing now

[cyberw]

This is already a great improvement, but can we also show unhandled exceptions?

For example, if I use an invalid host url (asdfasdf) this is logged on command line, but nothing shown in the UI.

[2023-10-31 14:53:34,306] svenskaspel.se/INFO/locust.runners: All users spawned: {"QuickstartUser": 1} (1 total users)
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
File "/Users/lafp/git/locust/locust/user/users.py", line 177, in run_user
user.run()
File "/Users/lafp/git/locust/locust/user/users.py", line 143, in run
self.on_start()
File "/Users/lafp/git/locust/examples/locustfile.py", line 20, in on_start
self.client.post("/login", json={"username": "foo", "password": "bar"})
File "/Users/lafp/.pyenv/versions/3.11.1/lib/python3.11/site-packages/requests/sessions.py", line 635, in post
return self.request("POST", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lafp/git/locust/locust/clients.py", line 135, in request
response = self._send_request_safe_mode(method, url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lafp/git/locust/locust/clients.py", line 180, in _send_request_safe_mode
return super().request(method, url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lafp/.pyenv/versions/3.11.1/lib/python3.11/site-packages/requests/sessions.py", line 573, in request
prep = self.prepare_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lafp/.pyenv/versions/3.11.1/lib/python3.11/site-packages/requests/sessions.py", line 484, in prepare_request
p.prepare(
File "/Users/lafp/.pyenv/versions/3.11.1/lib/python3.11/site-packages/requests/models.py", line 368, in prepare
self.prepare_url(url, params)
File "/Users/lafp/.pyenv/versions/3.11.1/lib/python3.11/site-packages/requests/models.py", line 439, in prepare_url
raise MissingSchema(
requests.exceptions.MissingSchema: Invalid URL 'asdfasdf/login': No scheme supplied. Perhaps you meant https://asdfasdf/login?
2023-10-31T13:53:34Z <Greenlet at 0x10374b060: run_user(<locustfile.QuickstartUser object at 0x1052c5350>)> failed with MissingSchema

[andrewbaldwin44]

@cyberw How were you able to reproduce this exactly? When I ran this the error would be caught by the greenlet_exception_logger and logged using the logger with the level set as CRITICAL

locust --modern-ui -f locust.py -H asdfasdf --autostart

If we want errors to be captured in a more general sense I believe we would have to do so using app.errorhandler

[cyberw]

I set the url in the ui, maybe that is the difference?

Or maybe it is because it happened in the on_start method? in that case that is another bug :)

Could you have a go at doing it (using app.errorhandler or whatever)? Showing almost all errors in the UI is kinda confusing :)

[cyberw]

app.errorhandler doesnt seem to catch this. Even with that change, I dont see it in the UI.

I'm running locust -H https://foo.com --modern-ui -f examples/locustfile.py (and then setting host to a bad url in the UI)

Traceback (most recent call last):
File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
File "/Users/lafp/git/locust/locust/user/users.py", line 177, in run_user
user.run()
File "/Users/lafp/git/locust/locust/user/users.py", line 143, in run
self.on_start()
File "/Users/lafp/git/locust/examples/locustfile.py", line 20, in on_start
self.client.post("/login", json={"username": "foo", "password": "bar"})
File "/Users/lafp/.pyenv/versions/3.11.1/lib/python3.11/site-packages/requests/sessions.py", line 635, in post
return self.request("POST", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lafp/git/locust/locust/clients.py", line 135, in request
response = self._send_request_safe_mode(method, url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lafp/git/locust/locust/clients.py", line 180, in _send_request_safe_mode
return super().request(method, url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lafp/.pyenv/versions/3.11.1/lib/python3.11/site-packages/requests/sessions.py", line 573, in request
prep = self.prepare_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lafp/.pyenv/versions/3.11.1/lib/python3.11/site-packages/requests/sessions.py", line 484, in prepare_request
p.prepare(
File "/Users/lafp/.pyenv/versions/3.11.1/lib/python3.11/site-packages/requests/models.py", line 368, in prepare
self.prepare_url(url, params)
File "/Users/lafp/.pyenv/versions/3.11.1/lib/python3.11/site-packages/requests/models.py", line 439, in prepare_url
raise MissingSchema(
requests.exceptions.MissingSchema: Invalid URL 'asdf/login': No scheme supplied. Perhaps you meant https://asdf/login?

[cyberw]

If you want, I can investigate it "from the other end" (correctly handling unhandled exceptions in on_start).

[andrewbaldwin44]

Ah I see the issue, the on_start is being called outside the context of the Flask application, so naturally the Flask app will not catch the error. And I was previously not able to reproduce the issue because my locustfile did not have an on_start

[andrewbaldwin44]

@cyberw Adding logs to _send_request_safe_mode would allow us to capture any application errors that can happen while requests are being made. Any other relevant exceptions should be caught by the Flask error handler (given that they are within the context of the Flask app) or the greenlet_exception_logger

[cyberw]

That is an improvement, but not perfect. It will duplicate the error in the console output (which isnt terrible) but more importantly it doesnt handle other types of exceptions happening in on_start.

Anyways this isnt really your fault. I can investigate it later on, unless you want to dig deeper. Ok to merge?

[andrewbaldwin44]

Yeah do you prefer me to remove the last commit before merging? I'd be happy to dig into this more but I think fixing the error handling for the on_start is a ticket in and of itself. I'm already not clear as to why errors from the on_start are not being captured by the greenlet_exception_logger and it will take me some time to understand the full flow here as I don't have the same level of context in the application here as you :)

[cyberw]

Yea, lets remove the commit. It could theoretically cause log spam for someone (if they are catching the exception themselves)

[cyberw]

I went ahead and fixed it right away :) #2442

[andrewbaldwin44]

Nice! Thanks for taking care of that :)