Write literate (executable) documentation

[cwgoes]

Let's figure out a way to actually run our documentation (the commands therein) in CI, so we notice when it's wrong. I think that will help a lot with keeping the docs up to date since it will be immediately noticeable (and surrounding text can easily be updated when the commands themselves are updated).

A quick-and-dirty way would be something like:

1. Parse out sequences of shell commands from markdown files in the docs directory (only ones marked as literate)
2. Spin up a build container with the standard environment (gaiad, gaiacli) and run through the command sequence
3. Fail on any failed command (unless the command is marked as intended-to-fail)

[alexanderbez]

Its a good idea, but seems somewhat fragile as it depends on the integrity of the docs, no? I.e. what if the docs miss certain steps? I guess that'll expose flaws in our documentation too.

[cwgoes]

I.e. what if the docs miss certain steps?

Yes, that's true - I think this should only apply to some files, but for those it does apply to having the steps be linearly sequenceable for anyone testing them themselves is probably prudent anyways.

[ValarDragon]

I'm not sure this is worth doing, unless its a quick fix. This is only a problem because we keep breaking the CLI.

Once we launch, the CLI should have some sense of stability, so we can just mandate updating the docs at each breaking change. Instead, Golang provides testable examples. We can manually ensure compatibility between example and docs. (e.g. when the examples change, review will require updating the docs)

The above is much easier than trying to figure out a way to parse the code out of the docs imo.

[zramsay]

is the CLI approaching stability whereby we can write this executable doc and not have to completely overhaul it in a few weeks?

[alexanderbez]

not quite yet...

[cwgoes]

Hmm, the idea was more that such an executable doc could be overhauled gradually as changes are made, in the PRs which they are made - and that due to this it would be unnecessary to overhaul docs in large chunks periodically when they get extremely out of date.

I don't feel too strongly about this though, just a concept.

[rigelrozanski]

this seems a bit crazy for now honestly. HOWEVER note that I actually did once upon-a-time create a way to do this with bash. https://github.com/rigelrozanski/shelldown

However I don't think we should be testing the CLI in bash - we should be testing the cobra command directly. See this issue: #2706

I would be really into some basic code-gen from the markdown comments (could be a similar technique to shelldown - but this codegen should be for testing the cobra command directly in golang.

[cwgoes]

I think we've decided not to do this.