Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload pre-built docs #1083

Open
marscher opened this issue Dec 12, 2014 · 39 comments
Open

Upload pre-built docs #1083

marscher opened this issue Dec 12, 2014 · 39 comments
Labels
Accepted Accepted issue on our roadmap Feature New feature

Comments

@marscher
Copy link

marscher commented Dec 12, 2014

I wonder if that would be possible? Would be extremely useful, since I'm building and deploying my packages with travis, and could perform the doc building also during this runs.

You may ask why this is necessary, since its not obvious that I can not use your build environment for documentation. I've to convert some ipython notebooks with cabal to rst and include them in my sphinx build. This aint possible on RTD and probably will never be.

Thanks for your effort!

Front logo Front conversations

@agjohnson agjohnson added this to the APIv2.1 milestone Dec 19, 2014
@agjohnson
Copy link
Contributor

We agree this would be an awesome feature, we've already talked about how this might work and it's vaguely on our roadmap already.

@marscher
Copy link
Author

marscher commented Feb 9, 2015

what is the status of APIv2.1?

@felixc
Copy link

felixc commented Mar 1, 2015

👍 to this. I could really use a "dumb file upload" API, where I can just provide a zip/tar/whatever of HTML+CSS files, along with some basic metadata like what version it's for.

@thedillonb
Copy link

I'm also very interested in this. Since I try to generate as much documentation via the code comments it would be very easy to, during CI, generate the pages, and upload to RTD - without the need for checking-in these generated files!

@embray
Copy link

embray commented Jun 3, 2015

IIRC this used to work, or at least, there was a file upload option but it was removed?

What needs to be done to overcome the original problems with this feature? It would be nice to have again, and could even allow us to offload some build overhead from RTD.

@marscher
Copy link
Author

marscher commented Jun 4, 2015

Maybe it has been removed, so people don't misuse rtfd as a free hosting service?

@agjohnson
Copy link
Contributor

I believe it was once a feature of RTD, but it was removed. We'd like to have this supported with an overhaul of the API -- a task we haven't added to our roadmap yet.

The more pressing issue we need to take a stance on is what are we accepting via html upload. RTD doesn't aim to be just a file host -- you could output and host documentation almost anywhere on the internet. Our goal is making the documentation authoring and viewing experience great, with Sphinx being our primary focus. The documentation experience would go south if we just took any old html and hosted it. We don't have the resources or community support to improve the experience of al the various documentation output folks use.

We're still shaping our opinion here, but we look to support some form of integration with externally built sphinx documentation, via some local command wrappers that you could run on external CI services/etc. We don't have an ETA on when we might drop that on our roadmap yet, but it is on our radar.

@embray
Copy link

embray commented Jun 4, 2015

It seems the biggest issue is then just ensuring a certain level of quality, and now allowing spam and or non-docs to be hosted. Perhaps the feature could be enabled only for validated users. Or maybe even paying (?) users? I can see how it would be difficult to handle that resource-wise though.

@agjohnson agjohnson added the Needed: design decision A core team decision is required label Jun 8, 2015
jantman added a commit to jantman/awslimitchecker that referenced this issue Jun 13, 2015
@gregmuellegger gregmuellegger added the Improvement Minor improvement to code label Sep 14, 2015
@luketunmer
Copy link

This feature would be great, even it was turned off on the public RTD server, but available for corporate-hosted ones in which the concerns over hosting generic HTML would not be an issue. The builds of our repos are just to contorted to expect a RTD server to check out and build, whereas our Jenkins CI server does all of that already.

@mistercrunch
Copy link

I can't get RTD to compile my project because of external dependencies that require OS modules. I'd love to be able to upload my compiled docs.

Since this issue isn't moving I have to rely on pythonhosted and kill the RTD entry for my project, loosing some SEO love....

@agjohnson
Copy link
Contributor

@mistercrunch if you don't use these dependencies within your documentation at all, you can mock them out or use conda.

@cpascual
Copy link

I would love to see this too. We are having hard time mocking our deps in RTD and we already build on Travis. Conda may help here, but still it seems like duplicating efforts given that we already have everything in Travis...

Alternatively, if we could run in RTD the same docker containers that we already use in Travis, that would also be great.

@humitos
Copy link
Member

humitos commented Jan 22, 2019

Hi all! This topic was raised again and we started discussing about it and having a brainstorming of ideas. Although, we don't have a clear direction on what to support and/or how and/or where (community/corporate site).

We have some things to consider for anything that we implement:

  • how to deal with spam/non-docs projects
  • upload PDF/epub/singlehtml versions
  • once a project upload docs what to do with "building docs on RTD"
  • output is valid for RTD (contains our custom JS files that serve ads, or the flyout menu, for example)
  • generate search index
  • support multiple versions for uploaded docs
  • support translations

On the other hand, we need to consider other things around the workflow,

  • write the logic as a Sphinx extension to be able to get metadata from inside and rely on the user to install all the dependencies and deal with setup, latex, etc or,
  • give users a Docker image (dealing with root permissions of output files) with everything pre-installed to build the docs and a guide to extend it or,
  • ... another option

Please, if you have some ideas to share about this topic, write them down here to help us to take the right direction on this. We still don't know where this feature fits better, though.

@humitos humitos changed the title upload docs by public API? Upload pre-built docs Jan 22, 2019
@marscher
Copy link
Author

The docker option also sounds very interesting as it would make mocking c-extensions completely obsolete.

@drewcassidy
Copy link

My project requires building the docs from a C++ library that requires GCC-10 or llvm+libc++ to build, so it cant be built on the RTD docker images. this would be nice to see

@marscher
Copy link
Author

marscher commented May 9, 2021

What about allowing certain docker images for building (on RTD of course), validating the build output outside of that and then, if a "stack"-reviewer accepts it, release it into the wild wild internet.
I guess there must be a duplicate already suggesting this.

@agjohnson
Copy link
Contributor

👋 So, just an update here for anyone following along:

This is loosely going on our roadmap for sometime this year. However, we have a number of technical pieces we need to handle first. There are some big question marks here still.


That said, a number of the issues in comments here already have solutions available, without requiring upload of pre-built documentation. In the better part of a decade this issue has been open, we've addressed a number of build and dependency issues with new features:

My project requires special OS packages

You can use build.apt_packages to hopefully install what you need now:

My project needs to execute special commands before build

We'll have a beta in the next couple weeks for build.jobs configuration option, enabling commands pre/post build steps. This should solve a number of issues users have with building their documentation outside their local dev environment.

My project executes different commands than Read the Docs

We are gathering feedback for this feature. Perhaps build.jobs implemented in #9016 works for most projects already.


If you have other use cases that are not well covered by our build infrastructure now, feel free to open a separate issue.

I'll keep this issue open, but we'll be putting discrete issues on our roadmap. I'll remove the design decision designation, not to communicate we have a concrete plan here, but we'll have discussion on individual issues as they come up. There are certainly a lot of questions here still.

@agjohnson
Copy link
Contributor

We haven't had any serious progress here yet this year, but I am currently experimenting with a solution for this using build.jobs and magic-wormhole for file transfer towards RTD. So far, a script starts a new build and then sends files to the build process directly, while the build process waits for these files.

Noting for later, my proof of concept is at:
https://github.com/readthedocs/test-builds/tree/wormhole

I wouldn't recommend this for use yet, there are a few things that will likely change here. I think a one-time codeword would be a good addition and wormhole relays seem unstable. A drop in replacement would be croc or maybe even running a private relay

@stefanseefeld
Copy link

I'm also very interested in this feature. Our build processes are quite complex and involve a substantial amount of work to complete. Documentation is extracted partly from the compiled code (notably Python extension modules built using C++ and Boost.Python), so running those build processes within RTD is not really an option.
The best solution would really be to have a (REST ?) API to push the generated docs directly into RTD, rather than building them on-site.

@humitos
Copy link
Member

humitos commented Jan 25, 2023

@stefanseefeld

I'm also very interested in this feature.

Have you tested the solution proposed by @agjohnson in #1083 (comment). We would appreciate your feedback on that.

@wrinch
Copy link

wrinch commented Oct 8, 2023

Hi, I'm interested in this as well, but from a slightly different Standpoint. We are using OctopusDeploy to release our python modules to multiple environments. I don't want to release the Documentation until the code hits a specific environment.

So I want to build the docs once, then deploy them to each environment as the release proceeds though out testing process.

Also, we have a complicated build pipeline with a bunch of C++ dependencies and CMake etc. So I would really just like a HTTP REST API to upload a documentation package ideally...

Just putting this here to add a little different perspective. GitHub branches in this case aren't our ideal workflow. I could probably get around it with tagging... But not ideal. And then there is the whole build steps...

@stefanseefeld
Copy link

Just FYI, I have found https://github.com/Polyconseil/dokang which provides a simple documentation hub to which one can push any documentation html tree. This works reasonably well for me, but isn't actually maintained (and much less developed), so I would very much welcome any effort to offer this reduced feature set as part of the readthedocs project.

@g3n35i5
Copy link

g3n35i5 commented Apr 22, 2024

Is there any update on this topic?

@humitos
Copy link
Member

humitos commented Apr 22, 2024

Unfortunately, not too many relevant updates here. However, we are going to discuss about this topic again in the near future and figure it out if there is a potential way to integrate this idea into Read the Docs without breaking the versions pattern we use.

@ericholscher
Copy link
Member

Next steps here are to scope this work. There's some discussion ongoing here: https://github.com/readthedocs/meta/discussions/147#discussioncomment-11308657

@g3n35i5
Copy link

g3n35i5 commented Nov 21, 2024

Next steps here are to scope this work. There's some discussion ongoing here: https://github.com/readthedocs/meta/discussions/147#discussioncomment-11308657

Thanks for the update! Unfortunately, your link doesn't work.

@ericholscher
Copy link
Member

Yea, that's on an internal repo for now, we've had a number of conversations about this in various contexts over the years 🙃 . The plan is to take the discussion there and write up a minimal v1 of the product that we'll work on here.

@humitos
Copy link
Member

humitos commented Dec 4, 2024

I'm moving this issue out from our roadmap since we already discussed it and we are not thinking about implementing HTML upload for now. We will start allowing users to update the Version.identifier and Version.slug first and grow from there:

That feature is not exactly HTML upload but it will solves that use case as well. Users can build their docs outside Read the Docs and point a version to a specific branch that executes build.commands to download and unzip pre-built documentation. Example:

build:
  commands:
    - wget https://your-organization.com/assets/docs.html.zip
    - unzip docs.zip --output=$READTHEDOCS_OUTPUT/html

This idea can be combined with a GitHub Action (or similar) that builds your docs and triggers a build for this version on Read the Docs using the APIv3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Accepted Accepted issue on our roadmap Feature New feature
Projects
Status: Planned
Development

No branches or pull requests