Review: dockerized experiments scenario

[shcheklein]

https://katacoda.com/dvc/courses/get-started/experiments

Issues:

• ERROR: unexpected error - 'ascii' codec can't encode characters in position 462-464: ordinal not in range(128) on dvc exp show Step 3
• Update available 2.0.15 -> 2.0.17 for every command - we need to run an update before? Run CI builds all those containers regularly?
• ERROR: failed to reproduce 'dvc.yaml': failed to run: python src/featurization.py data/prepared data/features, exited with 137 on dvc exp run --run-all OOM?
• 📖 See Experiment Management for more information on DVC's approach. - link is broken

[iesahin]

• ERROR: failed to reproduce 'dvc.yaml': failed to run: python src/featurization.py data/prepared data/features, exited with 137 on dvc exp run --run-all OOM?

Yep, this is OOM. It's not reliable, sometimes happens, sometimes not. This is the reason I want to start updating the Experiments with Tensorflow/MNIST first.

@shcheklein

[iesahin]

After running

bin/run-in-container.py -c dvcorg/doc-katacoda:start-experiments -k execute ../katacoda-scenarios/get-started/06-experiments/*.md 

I get the following. All other problems are Katacoda specific.

### RUNNING ###
$ dvc exp --help
###############
usage: dvc experiments [-h] [-q | -v]
                       {show,apply,diff,run,gc,branch,list,push,pull,remove}
                       ...

Commands to run and compare experiments.
Documentation: <https://man.dvc.org/exp>

positional arguments:
  {show,apply,diff,run,gc,branch,list,push,pull,remove}
                        Use `dvc experiments CMD --help` to display command-
                        specific help.
    show                Print experiments.
    apply               Apply the changes from an experiment to your
                        workspace.
    diff                Show changes between experiments in the DVC
                        repository.
    run                 Reproduce complete or partial experiment pipelines.
    gc                  Garbage collect unneeded experiments.
    branch              Promote an experiment to a Git branch.
    list                List local and remote experiments.
    push                Push a local experiment to a Git remote.
    pull                Pull an experiment from a Git remote.
    remove              Remove local experiments.

optional arguments:
  -h, --help            show this help message and exit
  -q, --quiet           Be quiet.
  -v, --verbose         Be verbose.
### RUNNING ###
$ dvc exp run --set-param featurize.max_features=1500  -S featurize.ngrams=2
###############
### RUNNING ###
$ git diff params.yaml
###############
diff --git a/params.yaml b/params.yaml
index 8f34d08..bd501e9 100644
--- a/params.yaml
+++ b/params.yaml
@@ -3,8 +3,8 @@ prepare:
   seed: 20170428
 
 featurize:
-  max_features: 500
-  ngrams: 1
+  max_features: 1500
+  ngrams: 2
 
 train:
   seed: 20170428
### RUNNING ###
$ dvc exp diff
###############
Path         Param                   Value    Change
params.yaml  featurize.max_features  1500     1000
params.yaml  featurize.ngrams        2        1

### RUNNING ###
$ dvc exp run --queue -n exp-1 -S train.n_est=50
###############
Queued experiment 'abce64c' for future execution.
### RUNNING ###
$ dvc exp run --queue -n exp-2 -S train.n_est=100
###############
Queued experiment '2fd7df5' for future execution.
### RUNNING ###
$ dvc exp run --queue -n exp-3 -S train.n_est=150
###############
Queued experiment '75f4350' for future execution.
### RUNNING ###
$ dvc exp run --queue -n exp-4 -S train.n_est=200
###############
Queued experiment '6f5c221' for future execution.
### RUNNING ###
$ dvc exp run --run-all
###############
Verifying data sources in stage: 'data/data.xml.dvc'
Verifying data sources in stage: 'data/data.xml.dvc'
Verifying data sources in stage: 'data/data.xml.dvc'
Verifying data sources in stage: 'data/data.xml.dvc'


Running stage 'prepare':
> python src/prepare.py data/data.xml
Running stage 'prepare':
> python src/prepare.py data/data.xml


Running stage 'prepare':
> python src/prepare.py data/data.xml
Running stage 'prepare':
> python src/prepare.py data/data.xml
### RUNNING ###
$ dvc exp show --no-timestamp  --include-params train.n_est  --no-pager
###############
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Experiment       ┃ avg_prec ┃ roc_auc ┃ train.n_est ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━┩
│ workspace        │  0.52048 │  0.9032 │ 50          │
│ katacoda-project │  0.52048 │  0.9032 │ 50          │
│ ├── *6f5c221     │        - │       - │ 200         │
│ ├── *75f4350     │        - │       - │ 150         │
│ ├── *2fd7df5     │        - │       - │ 100         │
│ └── *abce64c     │        - │       - │ 50          │
└──────────────────┴──────────┴─────────┴─────────────┘
### RUNNING ###
$ dvc exp apply exp-2
###############
### RUNNING ###
$ git add dvc.lock params.yaml prc.json roc.json scores.json
###############
### RUNNING ###
$ git commit -m "Preserve best Avg. Prec. experiment"
###############
[detached HEAD 06c52aa] Preserve best Avg. Prec. experiment
 1 file changed, 2 insertions(+), 2 deletions(-)
### RUNNING ###
$ dvc exp show --no-timestamp  --include-params train.n_est  --no-pager
###############
┏━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Experiment ┃ avg_prec ┃ roc_auc ┃ train.n_est ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━┩
│ workspace  │  0.52048 │  0.9032 │ 50          │
│ 06c52aa    │  0.52048 │  0.9032 │ 50          │
└────────────┴──────────┴─────────┴─────────────┘
### RUNNING ###
$ dvc exp show -n 2 --no-timestamp  --include-params train.n_est  --no-pager
###############
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Experiment       ┃ avg_prec ┃ roc_auc ┃ train.n_est ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━┩
│ workspace        │  0.52048 │  0.9032 │ 50          │
│ 06c52aa          │  0.52048 │  0.9032 │ 50          │
│ katacoda-project │  0.52048 │  0.9032 │ 50          │
│ ├── *6f5c221     │        - │       - │ 200         │
│ ├── *75f4350     │        - │       - │ 150         │
│ ├── *2fd7df5     │        - │       - │ 100         │
│ └── *abce64c     │        - │       - │ 50          │
└──────────────────┴──────────┴─────────┴─────────────┘
### RUNNING ###
$ dvc exp gc  --workspace 
###############
### RUNNING ###
$ dvc exp show -n 2 --no-timestamp  --include-params train.n_est  --no-pager
###############
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Experiment       ┃ avg_prec ┃ roc_auc ┃ train.n_est ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━┩
│ workspace        │  0.52048 │  0.9032 │ 50          │
│ 06c52aa          │  0.52048 │  0.9032 │ 50          │
│ katacoda-project │  0.52048 │  0.9032 │ 50          │
│ ├── *6f5c221     │        - │       - │ 200         │
│ ├── *75f4350     │        - │       - │ 150         │
│ ├── *2fd7df5     │        - │       - │ 100         │
│ └── *abce64c     │        - │       - │ 50          │
└──────────────────┴──────────┴─────────┴─────────────┘

[iesahin]

I didn't close this review issue, feel free to update with a new review or close if the scenario is OK now. @shcheklein Thank you 🙏🏼

[shcheklein]

Closing this for now. Thanks @iesahin . I'll create a new one if I see something else.