Jupyter Notebook

The last modifications of this post were around 1 year ago, some information may be outdated!

If you use VSCode, you should use its Jupyter Notebook extension, it's quick, clean and very easy to use.

Installation

Jupyter notebook

# BY PIP
pip install --upgrade pip
pip install --upgrade ipython jupyter

# BY CONDA
conda install ipython jupyter

Or read more in this note.

If you meet error OSError: [Errno 99] Cannot assign requested address, try

jupyter notebook --ip=127.0.0.1 --port=8080
# or
jupyter notebook --ip=127.0.0.1 --port=8080 --allow-root

Setting up a password

# create a juputer notebook config file
# it can be used for other settings
# https://jupyter-notebook.readthedocs.io/en/stable/public_server.html#prerequisite-a-notebook-configuration-file
jupyter notebook --generate-config

# create a new password
# note: sha1 cannot be reverted!!
jupyter notebook password

Inside notebook:

from notebook.auth import passwd
passwd()

With docker

# create a sha1 password
# download file create_sha1.py from https://github.com/dinhanhthi/scripts
# run ./create_sha1.py

# docker-compose.yml
environment:
  - PASSWD='sha1:d03968479249:319e92302e68d601392918f011d6c9334493023f'

# Dockerfile
CMD /bin/bash -c 'jupyter lab --no-browser --allow-root --ip=0.0.0.0 --NotebookApp.password="$PASSWD" "$@"'

R with jupyter notebook

Read more here.

# install jupyter
sudo apt-get install libzmq3-dev libcurl4-openssl-dev libssl-dev jupyter-core jupyter-client

# install R on linux
sudo apt install r-base

# R kernel for Jupyter Notebook
R # enter R environnement
# install R kernel
install.packages(c('repr', 'IRdisplay', 'IRkernel'), type = 'source')
# or
install.packages(c('repr', 'IRkernel'), type = 'source')
# make jupyter see r kernel
IRkernel::installspec() # current user
IRkernel::installspec(user = FALSE) # global

# embedded R
# use by cell magic %%R
pip install rpy2

# in a notebook
%load_ext rpy2.ipython

# then use
%%R
# R's codes

Other tips

• Running 2 tasks in the same cell TAKE LONGER TIME than running each on different cells.
• Download a folder in jupyter notebook:

  • Inside notebook, use:

    %%bash
    tar -czf archive.tar.gz foldername

  • Or using nbzip (only working on current server).

Check the info

# function's info
?<func-name>

# function's shortcode
??<func-name>

# get the list of current variables
whos

Check where command executed from (in your $path)?

!type python

python is /Users/thi/anaconda/envs/python3.6/bin/python

Multiline commands

# Using '\'
df.columns = df.columns.str.replace('.', ' ')\
                       .str.replace('\s+', ' ')\
                       .str.strip().str.upper()

You CANNOT put # comments at the end of each line break!

Hotkeys / Shortcuts

There are 2 modes: command mode (pres ESC to activate) and edit mode (Enter to activate). Below are the most useful ones (for me).

You can edit / add more shortcuts in Help > Edit Keyboard Shortcuts.

Jupyter notebook on remote server

Open jupyter notebook in local browser but the backend-server is on remote.

• If jupyter server is already running on remote at http://192.168.0.155:9889,

  ssh -N -L localhost:9888:192.168.0.155:9899 <username-remote>@<remote-host> -p <port>
  # if there is no port, remove `-p <port>`

  Open browser: http://localhost:9888 (type password if needed).

• If jupyter server is not running on remote yet,

  # connect to remote
  ssh <username-remote>@<remote-host> -p <port>
  # if there is no port, remove `-p <port>`

  On remote,

  # run juputer with custom port
  jupyter notebook --no-browser --port=9899
  
  # if there is error `OSError: [Errno 99] Cannot assign requested address`
  jupyter notebook --ip=0.0.0.0 --no-browser --port=9899
  
  # if there is error `Running as root is not recommended`
  jupyter notebook --ip=0.0.0.0 --no-browser --port=9899 --alow-root

  It's running and there are somethings like that,

  http://127.0.0.1:9889/?token=717d9d276f0537a9...831793df6319ad389accd

  Open another terminal window and type,

  ssh -N -L localhost:9888:localhost:9889 <username-remote>@<remote-host> -p <port>
  # if there is no port, remove `-p <port>`
  # there is nothing but it's running

  Open browser:

  http://localhost:9888/?token=717d9d276f0537a9...831793df6319ad389accd

You can choose any port number you wanna instead of 9888 and 9889 (they can be the same), note that, you need to use a port number GREATER THAN 8000!

Install new python package inside Jupyter Notebook

Using conda^[ref],

# Install a conda package in the current Jupyter kernel
import sys
!conda install --yes --prefix {sys.prefix} numpy

# DON'T DO THIS
!conda install --yes numpy

Using pip,

# Install a pip package in the current Jupyter kernel
import sys
!{sys.executable} -m pip install numpy

# DON'T DO THIS
!pip install numpy

Check version and update/upgrade,

!pip show pandas

Display dataframes side-by-side

from IPython.display import display_html
def display_side_by_side(*args):
    html_str=''
    for df in args:
        html_str+=df.to_html()
    display_html(html_str.replace('table','table style="display:inline; margin-right: 5px;"'),raw=True)

display_side_by_side(df1,df2,df1)

Get previous outputs

_ # previous output
__ # second-to-last output
___ # third-to-last output

Display 2 figures side-by-side markdown cell

Put below codes in the markdown cell of Jupyter Notebook.

<tr>
  <td> <img src="Nordic_trails.jpg" alt="Drawing" style="width: 250px;"/> </td>
  <td> <img src="Nordic_trails.jpg" alt="Drawing" style="width: 250px;"/> </td>
</tr>

Magic Functions

• Check the full list (in examples) here or their docs here.
• You can define your custom magic functions here.

Auto update the new updated modules (put at the beginning of the notebook)

%load_ext autoreload
%autoreload 2 # Reload all modules every time before executing

%autoreload 0 # disable autoreloader

Check more settings of %autoreload here.

Show the plots inside the notebook:

%matplotlib inline

Get the commands from 1 to 4:

%history -n 1-4 # get commands 1 to 4

With the bash command line + and using also curl,

👉 Note: curl.

%%bash -s $APP_NAME

APP_NAME=$1

cat > ./predictor/instances.json <<END
{
   "instances": [
     {
       "data": {
         "b64": "WW91IGFyZW4ndCBraW5kLCBpIGhhdGUgeW91Lg=="
       }
     }
   ]
}
END

curl -s -X POST \
  -H "Content-Type: application/json; charset=utf-8" \
  -d @./predictor/instances.json \
  http://localhost:7080/predictions/$APP_NAME/

%%bash -s $VAR_1 $VAR_2

VAR_1 = $1
VAR_2 = $2

We can run bash script inside a cell with ! pip install numpy.

Extensions

Table of contents

1. Install npm and nodejs.
2. Install this extension.
3. Enable in jupyter lab view.
4. Refresh the page.

# errors
# UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 23: ordinal not in range(128)
npm config set unicode false

Debugger

1. Install xeus-python, jupyterlab

   pip install xeus-python
   pip install jupyterlab

2. Install this extension.

3. Refresh the page, you have to choose kernel xpython (instead of Python 3) to use the debugger.

Convert notebook to HTML

pip install jupyterlab
jupyter nbconvert --to html <notebook>