27.0-rc2 sys:get_status/2 documentation update and ex_doc use

[okeuday]

Describe the bug
The documentation at lib/stdlib-6.0/doc/html/sys.html#get_status/2 references the deprecated format_status/2 callback functions, so it needs to be updated to reference the format_status/1 callback functions.

Attempting to build the documentation with make release_docs depends on the remote dependency ex_doc which depends on the remote dependency elixir. The makefile does provide the output below when ex_doc is missing:
Could not find ex_doc! Do you want to download latest ex_doc from github? (y/n)?

Entering y didn't work for me, likely due to the versions I have available. A circular dependency like this is brittle, especially when the latest ex_doc version is being used. To ensure documentation can always be built, it would be good to use a specific version of ex_doc (which can then enforce a version constraint on the elixir it is using). I would assume OS package managers of Erlang/OTP packages would want the documentation dependencies to be local so they can be part of checksums and verification of the source creating the same build output (at any point in time), but they could always reference the html tarball from the specific release instead of attempting to generate the documentation.

To Reproduce
Documentation can be obtained at https://github.com/erlang/otp/releases/tag/OTP-27.0-rc2 to avoid ex_doc problems.

Expected behavior
I expected the documentation to require only local dependencies, to ensure the documentation can be built in a dependable way.

Affected versions
27.0

Additional context
Updating the sys:status/2 documentation is related to the 27.0-rc2 compiler warning the callback MODULE:format_status(_,_) is deprecated; use format_status/1 instead.

[michalmuskala]

FWIW the ex_doc escript has precompiled elixir embedded in it, so it doesn't depend on locally installed elixir version, or even having elixir installed for that matter. It should only depend on erlang.

[garazdawi]

Entering y didn't work for me, likely due to the versions I have available.

I don't know why entering y did not work for you, but (as Michał wrote), Elixir is embedded into the ex_doc escript so that is not the issue.

What happens if you do ./otp_build download_ex_doc before building the docs? Does that work?

I would assume OS package managers of Erlang/OTP packages would want the documentation dependencies to be local so they can be part of checksums and verification of the source creating the same build output (at any point in time), but they could always reference the html tarball from the specific release instead of attempting to generate the documentation.

The debian package maintainer has decided to create packages for each of the ex_doc dependencies and build the docs from source and that seems to be working well from what I can tell.

[okeuday]

@garazdawi downloading ex_doc succeeds:

$ ./otp_build download_ex_doc
Downloaded ex_doc to ...otp_src_27.0-rc2/bin/ex_doc

Using ex_doc fails for me, but I was assuming it was due to having an older elixir in my path

$ elixir -v
Erlang/OTP 27 [RELEASE CANDIDATE 2] [erts-14.3] [source] [64-bit] [smp:24:24] [ds:24:24:10] [async-threads:1] [jit:ns]

Elixir 1.15.0 (compiled with Erlang/OTP 25)

The ex_doc failure has the output below:

$ make release_docs
cd ...otp_src_27.0-rc2/lib && \
  PATH=...otp_src_27.0-rc2/bin:"${PATH}" ERL_TOP=...otp_src_27.0-rc2 \
  make BUILD_ALL=1 TESTROOT="" release_docs
make[1]: Entering directory '...otp_src_27.0-rc2/lib'
make[2]: Entering directory '...otp_src_27.0-rc2/lib/stdlib'
=== Entering application stdlib
make[3]: Entering directory '...otp_src_27.0-rc2/lib/stdlib/src'
make -w RELEASE_PATH="...otp_src_27.0-rc2/release/x86_64-pc-linux-gnu" \
          release_docs_spec 
make[4]: Entering directory '...otp_src_27.0-rc2/lib/stdlib/src'
make[4]: Nothing to be done for 'release_docs_spec'.
make[4]: Leaving directory '...otp_src_27.0-rc2/lib/stdlib/src'
make[3]: Leaving directory '...otp_src_27.0-rc2/lib/stdlib/src'
make[3]: Entering directory '...otp_src_27.0-rc2/lib/stdlib/doc'
make -w RELEASE_PATH="...otp_src_27.0-rc2/release/x86_64-pc-linux-gnu" \
          release_docs_spec
make[4]: Entering directory '...otp_src_27.0-rc2/lib/stdlib/doc'
 GEN    html/index.html
** (RuntimeError) could not find source or debug info for :argparse
    (ex_doc 0.31.2) lib/ex_doc/language/source.ex:136: ExDoc.Language.Source.get_basedir/2
    (ex_doc 0.31.2) lib/ex_doc/language/erlang.ex:29: ExDoc.Language.Erlang.module_data/3
    (ex_doc 0.31.2) lib/ex_doc/retriever.ex:85: ExDoc.Retriever.get_module/2
    (ex_doc 0.31.2) lib/ex_doc/retriever.ex:54: anonymous fn/3 in ExDoc.Retriever.docs_from_modules/3
    (elixir 1.16.0) lib/enum.ex:2528: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ex_doc 0.31.2) lib/ex_doc/retriever.ex:23: ExDoc.Retriever.docs_from_dir/2
    (ex_doc 0.31.2) lib/ex_doc.ex:24: ExDoc.generate_docs/3
    (ex_doc 0.31.2) lib/ex_doc/cli.ex:69: anonymous fn/6 in ExDoc.CLI.generate/3make[4]: *** [...otp_src_27.0-rc2/make/doc.mk:62: html/index.html] Error 1
make[4]: Leaving directory '...otp_src_27.0-rc2/lib/stdlib/doc'
make[3]: *** [...otp_src_27.0-rc2/make/otp_release_targets.mk:35: release_docs] Error 2
make[3]: Leaving directory '...otp_src_27.0-rc2/lib/stdlib/doc'
make[2]: *** [...otp_src_27.0-rc2/make/otp_subdir.mk:29: release_docs] Error 2
make[2]: Leaving directory '...otp_src_27.0-rc2/lib/stdlib'
make[1]: *** [...otp_src_27.0-rc2/make/otp_subdir.mk:29: release_docs] Error 2
make[1]: Leaving directory '...otp_src_27.0-rc2/lib'
make: *** [Makefile:439: release_docs] Error 2

@michalmuskala Somehow it has trouble above due to its use of argparse.

[okeuday]

@garazdawi If you ignore the ex_doc execution failure I happen to get, the problem is using an opaque binary that exists outside the repository when you consider the project as an open source project that is meant to provide transparency and the potential for security problems in the future. The debian package maintainer may not be as concerned about the security aspect of ex_doc use but I would expect Gentoo Linux, Alpine Linux and OpenBSD package maintainers would be. I also understand that often the use of remote binaries is meant to avoid extra development effort, so it typically represents cutting corners (if it doesn't relate to a profit motive).

I am describing this to provide more context to the "checksums and verification of the source creating the same build output (at any point in time)" concept (i.e., making the build deterministic to have reliable operation).

[garazdawi]

The ex_doc that you download through ./otp_build download_ex_doc is checked against a sha256 to make sure that it is the one expected and we also note the ex_doc version used. From the ex_doc version anyone can build their own ex_doc and create as reproduceable builds as ex_doc allows.

I had forgotten, but I actually added a note about which version should be used in the installation guide for the docs. I've opened #8304 to make that information a bit more easily accessible.

As for your build error, I don't know what is going on, but it almost seems like you are using the bootstrap build to build the docs. Did you do make before doing make release_docs?

[okeuday]

@garazdawi Yes, make was done as part of ./otp_build setup -a --prefix=PREFIX --enable-deterministic-build

The sha256 is never checked if the sha1sum executable exists based on the source code

otp/otp_build

Lines 1018 to 1032 in e31d35c

	if command -v sha1sum > /dev/null; then
	if ! (cd "$ERL_TOP/make" && sha1sum --status -c "ex_doc.sha1sum"); then
	echo "The sha1sum check of $ERL_TOP/bin/ex_doc failed!" >&2
	exit 1
	fi
	elif command -v sha26sum > /dev/null; then
	if ! (cd "$ERL_TOP/make" && sha256sum --status -c "ex_doc.sha256sum"); then
	echo "The sha256sum check of $ERL_TOP/bin/ex_doc failed!" >&2
	exit 1
	fi
	else
	echo "Neither sha1sum nor sha256sum found to verify $ERL_TOP/bin/ex_doc" >&2
	echo "Please check manually that the correct ex_doc was downloaded." >&2
	exit 1
	fi

. That verification could easily break in the future, or the ex_doc source code could have a new problem that is never part of testing due to its separate release process. Using a binary blob in a repository can always lead to problems in new and interesting ways.

It would be better if ex_doc source code was converted to Erlang to live inside the repository, allowing the cyclic dependency to be eliminated. The source code could be converted manually or an automatic source code conversion could be created similar to ex2erl, though that would require replicating Elixir's macro processing source code and Elixir's internal source code to fully convert all the source code to Erlang.

[garazdawi]

Yes, make was done as part of ./otp_build setup -a --prefix=PREFIX --enable-deterministic-build

Aha, you did a deterministic build, that would also do it. It is currently not possible to build html/epub docs using ex_doc when a modules are compiled using +deterministic. I opened an issue to track this fault: elixir-lang/ex_doc#1881.

The sha256 is never checked if the sha1sum executable exists based on the source code

Thanks for highlighting the misstake I made there. I'll fix that to use sha256sum to verify.

It would be better if ex_doc source code was converted to Erlang to live inside the repository, allowing the cyclic dependency to be eliminated. The source code could be converted manually or an automatic source code conversion could be created similar to ex2erl, though that would require replicating Elixir's macro processing source code and Elixir's internal source code to fully convert all the source code to Erlang.

You and I have very different views on what better means in this context :)

I'm not sure wherein the cyclic dependency lies though? As far as I can tell there is this dependency:

Erlang compiler -> Elixir -> ExDoc and all deps -> Erlang/OTP docs

So while the Erlang/OTP compiler and docs can come from the same repository, they do not have to. In fact, they don't have to even be the same release, as long as the version dependencies work. For instance the ex_doc escript for 27-rc2 that is downloaded by ./otp_build download_ex_doc was built using the Erlang 26 compiler.

The future will probably prove me wrong, but I think the current solution is good enough for now. There are bound to be some quirks that need to be ironed out, but without reports of any actual issues I don't want to rush to a solution that might not work.

[okeuday]

@garazdawi Please keep in mind the original purpose for opening this issue was the sys:get_status/2 documentation related to the format_status/1 callback functions. That documentation likely still needs to be updated, to be current with the format_status compiler warning in 27.0-rc2.

[garazdawi]

#8307