Baking Brad

Setting Up and Using Jupyter Notebooks for Data Science

January 15, 2024

Setting Up and Using Jupyter Notebooks for Data Science

Introduction

Jupyter Notebook is an open-source web application that allows users to create and share documents that contain live code, equations, visualizations, and narrative text. It is an interactive computing environment that enables users to execute code and see results in real-time right inside their browser.

Jupyter Notebook was created to make data science and scientific computing more accessible, reproducible, and collaborative. It has since found widespread adoption across domains like data science, machine learning, academic research, and teaching programming.

There are several key reasons why Jupyter Notebook has become a popular platform:

Integrates Code and Narrative - Notebooks interleave code cells that can be executed with markdown cells that contain text, images, and visualizations. This allows you to provide context and explanations alongside the code. The outputs from running code cells are also visible to tell a complete data narrative.

Supports Data Science Workflows - Jupyter is optimized for interactive data exploration and analysis workflows commonly used in data science. You can load in data, preprocess it, visualize, build models, and share the results in a single notebook document.

Reproducible Research - Notebooks can be shared with others, allowing them to easily reproduce computations and findings. This makes it great for scientific research and collaborations. Notebooks can also be exported to HTML/PDF formats for publishing.

Collaboration - Multiple people can work on the same notebook file together in real-time using Jupyter's built-in collaboration tools.

Jupyter Notebook is valued for fostering reproducible, collaborative data science projects with an integrated environment that combines code, visualizations, and narrative in a sharable format. It removes much of the boilerplate effort associated with setting up coding projects, enabling users to focus on the research.

Dive into the world of interactive computing with "Learning Jupyter 5" by Dan Toomey. Perfect for developers and data scientists, this book is a treasure trove for mastering Jupyter Notebook and transforming your data science projects.


Installation and Setup


System requirements for Jupyter Notebook

Before installing Jupyter Notebook, there are some system requirements to consider:

Python Version

Jupyter Notebook requires Python 3.6 or later.

Specifically, the latest Python 3.x version is recommended for the best experience. Older Python 3.x versions are still compatible, but may miss out on new features or fixes.

Python 2.7 is also supported, but Python 2 reaches end-of-life in 2020, so Python 3 is recommended.

See the Documentation for full details.

Optional: Anaconda Distribution

The Anaconda Distribution is an optional recommendation to install alongside Jupyter Notebook.

Anaconda includes Python, hundreds of common data science packages, the conda package manager, and Jupyter Notebook itself in a single install. This allows you to bypass the need to setup other dependencies separately.

Using Anaconda is fully optional, but can simplify setup for data science use cases. Make sure to download the latest Python 3.x version.

Installation Methods

There are a couple options to install Jupyter Notebook:

Using Anaconda

If you install the Anaconda Distribution, Jupyter Notebook is included by default in the base install.

To verify, launch Anaconda Navigator and ensure you see "Jupyter Notebook" in the list of programs. Then you can launch directly from Navigator.

Using pip

If you have Python setup already, you can use the pip package manager to install Jupyter Notebook.

On Windows, open Command Prompt and run:

pip install notebook

On MacOS/Linux, open Terminal and run:

pip3 install notebook

If you have both Python 2.x and 3.x installed, you may need to use pip3 specifically to install packages for Python 3.

See the Documentation for full details on using pip to install Jupyter Notebook on each operating system.

Launching Jupyter Notebook

Once installed successfully, you can open a terminal/command prompt window and enter:

jupyter notebook

This will start the Jupyter server and launch your default web browser to the Jupyter dashboard.

Creating Your First Notebook

In the Jupyter dashboard, click the "New" button and select a kernel to create a blank notebook document. This will allow you to start writing Python code using that kernel.

The dashboard interface allows you to navigate directories on your system. You can create new notebooks or open existing ones.

Key Features

Jupyter Notebook has several key features that enable an interactive environment for data analysis and computation.

Code Cells

Code cells are the building blocks of Jupyter Notebooks. They allow you to write and execute code in a wide range of languages.

Supported Languages - Out of the box, Jupyter supports popular data science languages like Python, R, Julia and Scala. There are also hundreds of community-maintained kernels for languages like Java, C#, GO, Javascript, Ruby, SQL and more.

Writing Code - Type your code into a code cell just as you would in a regular editor. Jupyter uses syntax highlighting for many languages to help with readability. The cell has two modes: edit mode for typing code and command mode for executing code.

Executing Code - Use the ▶ button or Shift+Enter to execute code in a cell. The kernel associated with that notebook will run the code and output any variables, print statements, plots, errors etc back into the notebook beneath the cell.

Output Captured - All code outputs are captured and displayed within the notebook, including print statements. The last statement in a cell is automatically printed. Explicit printing allows multiple outputs from a cell.

Execution Order Matters - Note that cells run sequentially from top to bottom. So one cell's execution can depend on a prior cell. This differs from regular script files and requires thinking about dependencies as you build notebooks.

Markdown Cells

While code cells contain executable code, markdown cells provide text, images and more for narrative.

Text Formatting - Markdown provides shortcuts for formatting text with headers, bold, italics, lists, quotes and more to author rich content. It converts cleanly to HTML and other document formats.

Embedding Media - Images can be embedded with markdown syntax and will be displayed right within the final notebooks. The same goes for linking out URLs.

Notebook Conversions - Entire notebooks can be exported as HTML, PDF, Python scripts, Markdown and more using Jupyter's nbconvert tool. This provides options for sharing and publishing notebooks with others.

Notebook Organization - Markdown allows use of hierarchical heading tags to help structure and outline notebooks as computational narratives for better storytelling with code.

The Kernel

The kernel is the processing engine that executes the code inside notebooks and returns the results.

Kernel Specifications - When you create a notebook, you select a kernel that matches the programming language you want to use. The kernel runs in a separate process outside of the notebook itself. Popular kernels include Python, R, Julia and Scala.

Interrupt and Restart - If code in a notebook is taking too long to run, you can interrupt the kernel to stop it. Restarting it then clears all variable memory and starts it fresh. These features are helpful when debugging or re-running from scratch.

Reusable Environments - Kernels map notebooks to a reusable programming environment (languages, packages, variables). So notebooks can be reliably executed and shared without having to copy paste code into separate IDEs.

Notebooks for Reproducible Research

One major benefit of Jupyter Notebooks is enabling reproducible data science research.

Complete Environment - A notebook file contains the full history of code, outputs, visualizations and narrative in a single shareable document. This provides context often lost when sharing only raw code files.

Publishing Environments - Entire notebooks can be exported to HTML/PDF formats for clean publishing with other researchers, colleagues or the public. All code and outputs are visible. GitHub also renders notebooks nicely.

Enabling Collaboration - Notebooks can be shared directly for easy reproducibility. Jupyter also enables live collaboration - allowing multiple users to edit a notebook in real time like Google Docs.

This reproducibility and smooth sharing of the coding environment is why notebooks are widely used for data science research and collaboration.

Extending Notebooks

One of the great things about Jupyter is that the environment is extensible to enhance notebooks in many ways.

Installing Python & R Libraries - Any libraries available via pip/conda (Python) or CRAN (R) can be installed to extend notebook capabilities. For example, popular Python data science libraries like NumPy, Pandas, Matplotlib, Scikit-Learn etc. R has libraries like ggplot2, dplyr etc.

Notebook Extensions - There are also browser-based Jupyter Notebook extensions that add functionality without any coding. Some useful extensions include:

  • Table of Contents generation
  • Code completion intelligence
  • Variable inspector
  • Spell checking
  • Notebook shortcuts
  • Data visualization toolkits

JupyterLab IDE - For heavy notebook users, JupyterLab provides a full integrated development environment (IDE). It includes a file browser, code console, documentation viewer, terminals, notebook editor, text editor and more for working with Jupyter notebooks and code. It's like a souped up Jupyter dashboard.

JupyterHub & Jupyter Enterprise Gateway - For larger team and enterprise usage, JupyterHub allows multi-user access and collaboration on a central Jupyter server. Jupyter Enterprise Gateway provides execution kernels on remote clusters/cloud resources accessible from a central location.

Connecting to Data & Models - There are also libraries to connect Jupyter notebooks to external data sources and environments:

  • Access files on AWS S3, Google Cloud Storage from within the notebook
  • Query databases like PostgreSQL, MySQL and MongoDB from a notebook
  • Execute Scala & Spark code on a cluster (using Apache Toree or sparkmagic)
  • Run notebooks as parameterized jobs on Kubernetes

The community has built all sorts of extensions and libraries to connect notebooks to new systems.

Advanced Features

Beyond the basics, Jupyter Notebook has some really helpful advanced features from exporting formats to extensions and more.

Exporting Notebooks - Entire notebooks can be exported into various formats for sharing and publishing using Jupyter's nbconvert tool:

  • HTML - A clean standalone HTML page with all cell outputs formatted and displayed nicely. Great for publishing to a blog or website.
  • PDF - Export as a PDF report. This formats all the markdown and code cell outputs into printable PDF pages. Handy for printable notebooks or slides.
  • Python Scripts - The code cells can be exported as a raw .py Python file for reusing code outside the notebook environment.
  • Slideshows - Notebooks can be exported as Jupyter slideshow (RISE) or Reveal.js slide decks for presentations. This keeps data visualizations linked live from code outputs.

Version Control Integration - Jupyter notebooks stored as files can integrate with Version Control systems like Git/GitHub. This allows tracking changes and history and collaboration:

  • Notebook changes can be committed and pushed to GitHub repositories.
  • Multiple users can then pull, merge and commit changes to the central notebooks.

Real-time Collaboration - Jupyter supports Google Docs style live collaboration with multiple cursors simultaneously editing a notebook file. Great for teams.

Notebook Extensions - There are community-shared browser extensions that add helpful features to the notebook web interface:

  • Table of contents generation
  • Keyboard shortcuts
  • Data visualization widgets for Python
  • Connecting notebooks to external environments
  • and more!

Browse the extensions at https://jupyterlab.readthedocs.io/en/stable/user/extensions.html

Connect to External Data & Compute - It's common to use notebooks as an interface connecting to external systems:

  • Python & R libraries to connect and query SQL/NoSQL databases
  • Apache Spark using PySpark/SparkR/Sparkly shells to scale-out across clusters
  • Bindings to run Scala & R code against Big Data platforms like Apache Hadoop

This allows powerful compute resources and big data to be leveraged from right within notebooks using Python & Scala, bringing insights to your fingertips.

Use Cases

Jupyter notebooks are used across many domains - from education to industry applications. Here are some common use cases:

Data Science & Machine Learning

  • Analysis Workflows - Iterate through data transformation, visualization and statistical analysis interactively during exploratory data analysis.
  • Model Building - Prototype, evaluate, compare and tune machine learning models with visibility into the code and outputs in a single notebook.
  • Reporting & Dashboards - Parameterize notebooks to generate updated reports, KPIs and dashboards pulling latest production data. Effects of new code changes are visible.

Academic Research

  • Computational Essays - Publish academic papers supplementing the narrative with code, equations, diagrams and more with contextual explanations. Enables reproducibility.
  • Research Sharing - Share full interactive environment used to reach research conclusions with other academics for transparency, reproducibility and collaboration.

Data Journalism

  • Publish reports, stories and analyses while enabling visibility into the code and data behind charts, key findings and more. Promotes transparency.

Teaching Programming

  • Instructors can create annotated lesson material with interspersed code snippets and demonstrations easily.
  • Students use notebooks for hands-on coding assignments and projects with compute resources provided.

For virtually any scenario involving data analysis, computation and explanatory narrative, Jupyter Notebooks likely have an application to enable efficiency and collaboration.

Conclusion

Jupyter Notebook is an incredibly useful tool for interactive computing across many disciplines and use cases.

Key Benefits

Some of the key benefits that have made Jupyter Notebooks a ubiquitous data science platform:

  • Integrates live code, outputs, visualizations and narrative in shareable documents
  • Supports iterative data science workflows for analysis and modeling
  • Promotes reproducible research by sharing executable computing environments
  • Enables collaborative editing for teams like a computational Google Doc
  • Extensible to interface with external data, models, libraries and clusters
  • Scales from individuals to teams to enterprises (with JupyterHub)

Notebooks help reduce boilerplate code complexity so users can focus on high value data science tasks.

Additional Resources

Here are some great starting points for further learning:

Tags: Data Science, Jupyter Notebooks, Programming, Python, Scientific Research, Machine Learning