ray-integration

Ray provides a simple, universal API for building distributed applications, read more about ray here.
Ray integration with LSF enables users to start up a Ray cluster on LSF and run DL workloads through that either in a batch or interactive mode.

Configuring Conda

• Before you begin make sure you have conda install on your machine, details about installing conda on linux machine is here.
• For reference sample conda env yml is present here, to create sample conda env that will run GPU and CPU workloads run, it has mix of conda and pip dependencies:

  conda env create -f sample_conda_env/sample_ray_env.yml
  

• To test if you have ray installed with version number run:

   conda activate ray
   pip install -U ray
   ray --version
   ray, version 1.4.0
  

Running ray as interactive LSF job

• Run the below bsub command to get multiple GPUs (i.e. 2 GPUs in this example) on multiple nodes (i.e. 2 hosts in this example) from LSF scheduler with 20GB hardlimit on memory

  bsub -Is -M 20GB! -n 2 -R "span[ptile=1]" -gpu "num=2" bash
  

• Sample workloads are present in sample_workload directory, sample_code_for_ray.py is CPU only workload and cifar_pytorch_example.py will work on CPU as well as GPU.
• Start the script by running the following command:

  ./ray_launch_cluster.sh -c "python <full_path_of_sample_workload>/cifar_pytorch_example.py --use-gpu --num_epochs 5 --num-workers 4" -n "ray" -m 20000000000
  

  Where:
  -c is the user command that needs to be scaled under ray
  -n is the conda namespace that will be activate before the cluster is spawned
  -m is object store memory size in bytes as required by ray

Acessing ray dashboard in interactive job mode:

• Get ray head node and dashboard port, please find below log lines on the console

  Starting ray head node on:  ccc2-10
  The size of object store memory in bytes is:  20000000000
  2021-06-07 14:19:11,441 INFO services.py:1269 -- View the Ray dashboard at http://127.0.0.1:3752
  

  Where:
  - head node name: ccc2-10
  - dashboard port: 3752
• Run the below set of commands on the terminal to port forward dashboard from cluster to your local machine:

  export PORT=3752
  export HEAD_NODE=ccc2-10.sl.cloud.ibm.com
  ssh -L $PORT:localhost:$PORT -N -f -l <username> $HEAD_NODE
  

• Access the dashboard at your laptop on:

    http://127.0.0.1:3752
  

Running ray as a batch job

• Run the below command to run ray as batch job

    bsub -o std%J.out -e std%J.out -M 20GB! -n 2 -R "span[ptile=1]" -gpu "num=2"  ./ray_launch_cluster.sh -c "python <full_path_of_sample_workload>/cifar_pytorch_example.py --use-gpu --num-workers 4 --num_epochs 5" -n "ray" -m 20000000000
  

• To access the dashboard please refer to log file generated for batch job and perform port forwarding referring to commands described above.