Skip to content

Commit

Permalink
Add guidance on using profilers from within Jupyter notebooks.
Browse files Browse the repository at this point in the history
  • Loading branch information
Robadob committed Mar 22, 2024
1 parent 400ee2c commit b1be5c1
Show file tree
Hide file tree
Showing 7 changed files with 233 additions and 1 deletion.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"id": "e80c86cb-d60a-4e45-a34c-191aab11acf0",
"metadata": {},
"outputs": [],
"source": [
"!pip install line_profiler\n",
"%load_ext line_profiler"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "de5e278a-6826-4f42-b1af-496f5aabf996",
"metadata": {},
"outputs": [],
"source": [
"def fizzbuzz(n):\n",
" for i in range(1, n + 1):\n",
" if i % 3 == 0 and i % 5 == 0:\n",
" print(\"FizzBuzz\")\n",
" elif i % 3 == 0:\n",
" print(\"Fizz\")\n",
" elif i % 5 == 0:\n",
" print(\"Buzz\")\n",
" else:\n",
" print(i)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c54f2642-617e-4a51-b3e8-1ff1ba0e8426",
"metadata": {},
"outputs": [],
"source": [
"%lprun -f fizzbuzz fizzbuzz(100)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"id": "83b45371-9cd9-4a8f-b0ff-ceb1bfc2ecc2",
"metadata": {},
"outputs": [],
"source": [
"import time\n",
"\n",
"def a_1():\n",
" for i in range(3):\n",
" b_1()\n",
" time.sleep(1)\n",
" b_2()\n",
" \n",
"def b_1():\n",
" c_1()\n",
" c_2()\n",
"\n",
"def b_2():\n",
" time.sleep(1)\n",
" \n",
"def c_1():\n",
" time.sleep(0.5)\n",
"\n",
"def c_2():\n",
" time.sleep(0.3)\n",
" d_1()\n",
"\n",
"def d_1():\n",
" time.sleep(0.1)\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0fc38195-3dc4-4a7c-82f6-1167f54d3182",
"metadata": {},
"outputs": [],
"source": [
"!pip install snakeviz"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c47affa0-23cc-43f8-a4f8-e73ab2f44afd",
"metadata": {},
"outputs": [],
"source": [
"%load_ext snakeviz"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "58040218-8578-47fc-9a18-476f4b27d25a",
"metadata": {},
"outputs": [],
"source": [
"%snakeviz a_1()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
43 changes: 43 additions & 0 deletions episodes/profiling-functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,42 @@ By clicking a box within the diagram, it will "zoom" making the selected box the

As you hover each box, information to the left of the diagram updates specifying the location of the method and for how long it ran.

::::::::::::::::::::::::::::::::::::: callout

## snakeviz Inside Notebooks

If you're more familiar with writing Python inside Jupyter notebooks, you might not feel comfortable using the command line for profiling. Instead, `snakeviz` can be called directly from inside notebooks, and it will automatically call `cProfile` for you.

First `snakeviz` must be installed and it's extension loaded.

```py
!pip install snakeviz
%load_ext snakeviz
```

Following this, you can either call `%snakeviz` to profile a function defined earlier in the notebook.

```py
%snakeviz my_function()
```

Or, you can create a `%%snakeviz` cell, to profile the python executed within it.

```py
%%snakeviz

def my_function():
print("Hello World!")

my_function()
```

In both cases, the full `snakeviz` profile visualisation will appear as an output within the notebook!

*You may wish to right click the top of the output, and select "Disable Scrolling for Outputs" to expand it's box if it begins too small.*

:::::::::::::::::::::::::::::::::::::::::::::

## Worked Example

:::::::::::::::::::::::::::::::::: instructor
Expand Down Expand Up @@ -310,6 +346,13 @@ The sunburst visualisation displays less text on the boxes, so it can be harder

:::::::::::::::::::::::::::::::::::::::::::::

If you followed along inside a notebook it might look like this:

![The worked example inside a notebook.](episodes/fig/snakeviz-worked-example-notebook.png){alt="A Jupyter notebook showing the worked example profiled with snakeviz." width=80%}

Because notebooks operate by creating temporary Python files, the filename (shown `1378276351.py` above) and line numbers are not too useful, but function names should still be helpful.


## Exercises

The following exercises allow you to review your understanding of what has been covered in this episode.
Expand Down
37 changes: 36 additions & 1 deletion episodes/profiling-lines.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,42 @@ Therefore it can be seen in this example, how the time spent executing each line

The `-r` argument passed to `kernprof` (or `line_profiler`) enables rich output, if you run the profile locally it should look similar to this. *This requires the optional package `rich`, it will have been installed if `[all]` was specified when installing `line_profiler` with `pip`.*

![Rich (highlighted) console output provided by `line_profiler` for the above FizzBuzz profile code.](episodes/fig/line_profiler-worked-example.png){alt="A screenshot of the `line_profiler` output from the previous code block, where the code within the line contents column has basic highlighting."}
![Rich (highlighted) console output provided by `line_profiler` for the above FizzBuzz profile code.](episodes/fig/line_profiler-worked-example-rich.png){alt="A screenshot of the `line_profiler` output from the previous code block, where the code within the line contents column has basic highlighting."}

:::::::::::::::::::::::::::::::::::::::::::::

::::::::::::::::::::::::::::::::::::: callout

## line_profiler Inside Notebooks

If you're more familiar with writing Python inside Jupyter notebooks, you might not feel comfortable using the command line for profiling. Instead, `line_profiler` can be called directly from inside notebooks. It is still necessary for the code to be profiled to be placed within a function.

First `line_profiler` must be installed and it's extension loaded.

```py
!pip install line_profiler
%load_ext line_profiler
```

Following this, you call `line_profiler` with `%lprun`.

```py
%lprun -f profiled_function_name entry_function_call()
```

The functions to be line profiled are specified with `-f <function name>`, this is repeated for each individual function that you would otherwise apply the `@profile` decorator to.

This is followed by calling the function which runs the full code to be profiled.

For the above fizzbuzz example it would be:

```py
%lprun -f fizzbuzz fizzbuzz(100)
```

This will then create an output cell with any output from the profiled code, followed by the standard output from `line_profiler`. *It is not currently possible to get the rich/coloured output from `line_profiler` within notebooks.*

![Output provided by `line_profiler` inside a Juypter notebook for the above FizzBuzz profile code.](episodes/fig/line_profiler-worked-example-notebook.png){alt="A screenshot of the line_profiler output from the previous code block inside a Jupyter notebook."}

:::::::::::::::::::::::::::::::::::::::::::::

Expand Down

0 comments on commit b1be5c1

Please sign in to comment.