[🐛 Bug]: Python API reference doc for 4.20.0 is missing some pages

[emanlove]

What happened?

The v4.20.0 Python API reference doc is missing some pages. Generally it reflects the docstrings for all the Python Modules. Currently though several modules are missing their pages (ie there is no link to their pages).

How can we reproduce the issue?

1. Goto https://www.selenium.dev/selenium/docs/api/py/api.html 

2. Scroll down to the modules of "WebDriver.firefox" or "Webdriver.safari", "Webdriver.webkitgtk" or "Webdriver.ie" as well as "selenium.webdriver.common.html5.application_cache" (near the top under "Webdriver.common").

3. Note several of the modules are not linked. These should be linked.

For example I should be able to click on "selenium.webdriver.firefox.webdriver" and be able to read about the "get_full_page_screenshot_as_file" function. Or as one can click on "selenium.webdriver.edge.webdriver" (just above the "Webdriver.firefox" section) and see the documentation for edge the same should happen for these other browsers.

Relevant log output

N/A

Operating System

Chrome

Selenium version

N/A

What are the browser(s) and version(s) where you see this issue?

Chrome, Firefox, Safari

What are the browser driver(s) and version(s) where you see this issue?

N/A

Are you using Selenium Grid?

N/A

[github-actions]

@emanlove, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[iampopovich]

@diemol hi , Am I correct that we need to update the list of classes in the file /selenium/py/docs/source/api.rst to create documentation for them?

i tried to build documentation locally and got some errors like

ons.rst, webdriver_webkitgtk/selenium.webdriver.webkitgtk.service.rst, webdriver_webkitgtk/selenium.webdriver.webkitgtk.webdriver.rst
WARNING: Failed to import selenium.webdriver.firefox.extension_connection.
Possible hints:

there is no class selenium.webdriver.firefox.extension_connection anymore
so shall we remove all the incorrect imports ?

[titusfortner]

I haven't updated api docs for 4.22,going to do that today. I'll get back to you with results.

[titusfortner]

Yes, something isn't working properly with Python documentation.

Executing tox -c py/tox.ini -e docs throws a bunch of warnings.
https://github.com/SeleniumHQ/selenium/actions/runs/9618006350/job/26530983788#step:7:13

Comparing 4.21 to 4.22 shows a number of files getting deleted that shouldn't be
This is what I'm seeing: 58fa379

If you have ideas for what is wrong and how to fix, please help. :)

[iampopovich]

I haven't fully solved the problem yet, but I have some thoughts on a solution:

1. I searched for similar errors and found that there are issues with Sphinx when adding the source code directory for documentation. I checked conf.py - if it worked before, it shouldn't break now. Example of a similar problem: Sphinx issue.

2. When updating the documentation, warnings are thrown such as the failure to import selenium.webdriver.firefox.extension_connection. There's no such class, but autodoc tries to import it. Removing the .rst file from the documentation directory will make the warning disappear.

3. I deleted all previously generated documentation and ran ./go py:docs. The .rst documents are generated, but I'm getting module import errors like websocket or trio, which suggests it's just an issue with my unconfigured environment. However, I don't receive any errors importing already generated documents.

4. I will open a PR with an updated list of classes in api.rst, leaving only those classes that exist in the project under py/selenium directory. This can be considered a small step towards improving the current documentation.

Based on all of the above, I suggest deleting all current documentation from docs and regenerating it with an up-to-date list of classes. Deleted classes should also be removed from the documentation in the docs directory in the future to avoid generating new warnings.

[emanlove]

@iampopovich I think you are close with the assumption on the list of classes within api.rst. I have long had the feeling that at some level these should be auto generated from the code, that is both docstrings as well as package directory structure. I think to a degree what is within api.rst is generated, possibly using sphinx-autogen [doc]. In addition as they should be auto generated the shouldn't be checked in but regenerated each time. Trying some things out around this thought ..

[emanlove]

Also the sphinx version (currently pinned at 1.8.2) really should be bumped up to something much more recent (latest is 7.3.7) along with Jinja2.

[iampopovich]

Sphinx has not been updated since the release of Selenium 4.1 for some reason. I also saw through git blame that version 1.8.2 was set in the tox.ini and requirements.txt files in the docs directory in 2023, which I consider a "recent" change in the context of dependencies for generating documentation. But yes, I agree that Sphinx is already outdated.

[emanlove]

There appears to be an upper limit on Sphinx which still supports Python 3.8 (sphinx ver 7.1.4). I suspect there is other things going on there as well. Now as I wanted to use sphinx-autogen to create those .rst files beyond just the base api.rst file I did want to update sphinx. I was seeing various references to functionality that required newer version of sphinx. Even though these minimum requirements and references seemed to be "if you do not want to create stub pages with sphinx-autogen" I didn't want to mess with using such an old version of sphinx.

I have a rough working example of generating code. I spell it out all here in case someone wants to walk though it. Strong Cautionary Note There is some dependency conflict here with either cryppto or the like with either sphinx/jinja or possible even pip. I am trying to narrow this down as well but just caution to use virtualenv so as not to get one into state of a nasty cleanup. The update of pip and the order of install below works fine. I can also switch this over into the build model of selenium using tox and will do so shortly. This just allows one to start experimenting now ..

The key below is this line PYTHONPATH=. sphinx-autogen docs/source/api.rst which just generates all those autosumary files.

mkdir selenium
cd selenium
git clone [email protected]:SeleniumHQ/selenium.git se
# update se/py/docs/requirements.txt with Jinja==3.1.4 and sphinx==7.1.4 and added -r requirements.txt
virtualenv doc-py38-env
source doc-py38-env/bin/activate
python -m pip install -U pip
cd se/py
pip install -r requirements.txt
pip install -r docs/requirements.txt
PYTHONPATH=. sphinx-autogen docs/source/api.rst
cd docs/
make html

[titusfortner]

really should be bumped up to something much more recent

The first time I worked with Python docs (a few years ago?) updating Sphinx broke things, so I set it at the previous version that worked and backed away slowly. If people who know what they are doing want a later version and it does the right thing, yes, let's do it.

[emanlove]

Working another PR that updates Sphinx and brings in some of the changes @iampopovich makes within #14172 ..

[titusfortner]

Fixed with #14173 as soon as I do the 4.22 update to the docs

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. Please open a new issue for related bugs.