Document checkpoints without make_checkpoint()

[dberenbaum]

Our documentation so far in https://dvc.org/doc/command-reference/exp/run (and maybe elsewhere?) assumes that checkpoints only work with make_checkpoint() or a signal file. However, checkpoints still work without make_checkpoint() if the stage is designed to make a single checkpoint instead of multiple checkpoints. The final output will still be saved for the following iteration as long as it is marked as checkpoint: true in the dvc.yaml.

This would be great to document because:

1. It is language agnostic (without resorting to clunky workarounds) and therefore more consistent with the rest of DVC
2. It provides a simpler starting point for using checkpoints

See https://github.com/iterative/dvc-checkpoints-mnist/tree/python_agnostic for an example.

• Address also specific docs feedback in Decouple checkpoints and make_checkpoint #2319

@pmrowla @jorgeorpinel @dmpetrov

[pmrowla]

This is just using a regular output as a checkpoint. I guess the distinction is that in order for the "resume" feature in exp run to work, you still have to specify the checkpoint flag, but otherwise this is just doing the regular

dvc repro
git add .
git commit
<repeat>

workflow (with the results stored in an experiment ref instead of a regular git branch).

Maybe we should just consider having an explicit flag to extend ("resume") an existing experiment branch (that may or may not have checkpoint outputs) instead?

edit: although in this case exp run --reset behavior would not work the same as it does now (unless the user still specifies checkpoint outs)

[dberenbaum]

This is just using a regular output as a checkpoint.

Exactly, which makes it easier to grasp for existing DVC users and more consistent with the rest of DVC (no need to inject DVC API functions into the user code). Not that make_checkpoint() isn't useful, it just seems like we are jumping straight into the advanced use case.

[iesahin]

So make_checkpoint() is for creating checkpoints manually, but without it dvc exp run already saves a checkpoint after running the command. Right?

If checkpoint: true in dvc.yaml, it's as if running the experiments in a loop. We can

while true ; do 
   dvc exp run
done

to run the experiments continuously.

BTW, yes, this works as I expected but quitting from the loop is very difficult. I had to close tmux pane. 😅

[pmrowla]

So make_checkpoint() is for creating checkpoints manually, but without it dvc exp run already saves a checkpoint after running the command. Right?

Yes, DVC will always generate a final checkpoint commit after running the command (assuming the workspace state has actually changed and there is actually changes to commit)

[jorgeorpinel]

checkpoints still work without make_checkpoint() if the stage is designed to make a single checkpoint (per exp run)
great to document because: ... It is language agnostic ... provides a simpler starting point

Yeah I didn't even realize this was possible. Sure!

just using a regular output as a checkpoint

But the experiments (checkpoints) are added into a branch this way, which is quite different from regular experiments. Kind of the main difference with checkpoints!

it's as if running the experiments in a loop

BTW that connects with iterative/dvc/issues/5608

[dberenbaum]

We also probably need an example in the docs and/or in https://github.com/iterative/dvc-checkpoints-mnist that shows how to use signal files to generate checkpoints.

[jorgeorpinel]

how to use signal files to generate checkpoints

Meaning withouot make_checkpoint(), or (both that and) without Python?

[dberenbaum]

I mean manually doing the steps in https://dvc.org/doc/api-reference/make_checkpoint#description. Torn on whether to do it in Python for consistency or in another language to show how that works. Maybe we can start with Python and possibly translate into another language later.

[dberenbaum]

I mean manually doing the steps in https://dvc.org/doc/api-reference/make_checkpoint#description

I put an example in https://github.com/iterative/dvc-checkpoints-mnist/tree/signal_file.

Next step is probably to reference this repo in the docs or develop a more robust tutorial out of this or another scenario. Seems like checkpoints in general are not featured that prominently yet, and the different ways to implement them are not laid out clearly.

Any thoughts on an approach here?

[jorgeorpinel]

Agree. Just not sure what the priority is. We can wait and see if people seem to need this guidance (those who don't find the signal_file branch of https://github.com/iterative/dvc-checkpoints-mnist first).

[dberenbaum]

IMHO a first step would be to reference https://github.com/iterative/dvc-checkpoints-mnist in the docs (I think it's only mentioned in dvclive now) and explain the different branches.

[dberenbaum]

We can wait and see if people seem to need this guidance (those who don't find the signal_file branch of https://github.com/iterative/dvc-checkpoints-mnist first).

I'm worried that users aren't very aware of checkpoints or how to use them until we better highlight and document them. As a first step, what about referencing the mnist examples in https://dvc.org/doc/user-guide/experiment-management#checkpoints-in-source-code?

[jorgeorpinel]

Makes sense to wait a little before assigning a priority.

what about referencing the mnist examples in https://dvc.org/doc/user-guide/experiment-management#checkpoints-in-source-code?

Sure, we can add a paragraph. Or would it be more useful in #2373? (I can contribute to the branch).

[jorgeorpinel]

UPDATE: for now please see #2381 and iterative/dvc-checkpoints-mnist#3

[dberenbaum]

#2381 seems sufficient for now. Closing this one.