Use Magic Commands

You're deep in an analysis, iterating on a DataFrame, and you need to check the memory usage, time a cell, or see all your variables. Typing the same three lines of Python over and over is not just tedious — it breaks your flow and slows you down. Jupyter's magic commands are the hidden accelerators that let you do these tasks in a fraction of a second, keeping your hands on the keyboard and your mind on the data.

In this lesson, you'll learn what magic commands are, how they differ between line and cell magics, and how to apply them to make your data science workflow significantly more efficient. You'll see practical examples, compare options, and troubleshoot common pitfalls.

The problem this lesson solves

If you've ever found yourself writing import time and wrapping code in t0 = time.time() just to see how long a cell takes to run, you know the pain. Or you've thrown %matplotlib inline at the top of a notebook without fully understanding it, and wondered why your plots sometimes don't show. These are symptoms of the same problem: you're working in Python, but not with the environment Jupyter gives you.

Magic commands are built into IPython, the kernel that powers Jupyter. They're prefixed with % (line magic) or %% (cell magic) and provide shortcuts for common tasks — timing, profiling, debugging, shell access, and more. Knowing them means you spend less time on boilerplate and more time on analysis.

For a data scientist, efficiency isn't just about speed; it's about reducing cognitive load. When you don't have to remember the five lines of code to load a CSV with the right encoding, or the three lines to time a cell, you free up mental space for the actual data problems. This lesson exists to give you that efficiency boost.

Core concept / mental model

Think of magic commands as superpowers for your notebook. They're not part of standard Python — they're special instructions that the IPython kernel understands and executes before Python runs. This means they can do things Python alone can't, like interact with the kernel, alter notebook state, or access shell commands.

Think of the difference between order at a restaurant vs. cooking in the kitchen. Normal Python code is like the kitchen: you measure, mix, and put things away. A magic command is like calling your waiter: one word gets you exactly what you want, already prepared.

There are two types:

• Line magics start with a single % and apply to one line. Example: %time times the next line.
• Cell magics start with %% and apply to the entire cell. Example: %%time times the whole cell.

Pro tip: You can often omit the % for line magics if you're at the start of a cell, but it's better to be explicit, especially if you're mixing in variable assignments.

Some of the most commonly used magics include:

• %time and %%time — measure execution time
• %timeit and %%timeit — run code many times for more accurate timing
• %matplotlib inline — render plots directly in the notebook
• %load — load code from a file or URL into the cell
• %run — run a Python script
• %who / %whos — list your variables
• %lsmagic — list all available magics
• %%writefile — write the cell contents to a file

How it works step by step

Let's break down the process of using a magic command effectively. The key is to remember that the magic command is interpreted by IPython, not by Python itself. That's why they can do things like modify the notebook's namespace or run shell commands.

1. Start the cell or line with the magic prefix ( % or %%).
2. Choose the right magic for the task: timing, debugging, running external code, etc.
3. Include any arguments (like -n for number of iterations in %timeit).
4. Observe the output — magic commands often print meaningful info (time, memory, variable list).
5. Integrate with your normal Python code. For example, you can use %matplotlib inline at the top to set up plotting, then use plt.plot() in the same notebook.

Here's a typical workflow when you're exploring data:

• First, you might %load a previous analysis script to reuse it.
• Then use %%time to see how long a heavy data transformation takes.
• Later, %whos shows the objects you've created so you don't accidentally overwrite something.
• Finally, %matplotlib inline lets your plots appear inline so you can quickly visualize results.

Hands-on walkthrough

Let's get our hands dirty with a practical Data Science example. We'll work with a small DataFrame, time its operations, and use magic commands to be more productive.

First, let's create a simple dataset using pandas (make sure you have it installed):

import pandas as pd
import numpy as np

# Sample data: 100,000 rows
df = pd.DataFrame({
    'x': np.random.randn(100000),
    'y': np.random.randn(100000),
    'group': np.random.choice(['a', 'b', 'c'], 100000)
})

Now, let's time how long a groupby operation takes using %%time:

%%time
result = df.groupby('group')['x'].mean()
print(result)

Expected output:

CPU times: user 38 ms, sys: 6 ms, total: 44 ms
Wall time: 43 ms
group
a    0.001234
b    0.000567
c   -0.000891
Name: x, dtype: float64

Now, let's use %timeit for a more precise benchmark (it runs the code multiple times and gives you average and standard deviation):

%timeit df.groupby('group')['x'].sum()

Expected output:

35.5 ms ± 1.2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Next, check which variables you have in your notebook with %whos:

%whos

Expected output (truncated):

Variable   Type        Data/Info
-------------------------------
df         DataFrame   100000x3
np         module      <module 'numpy' from ...>
pd         module      <module 'pandas' from ...>
result     Series      3 elements

Now let's try a cell magic that writes a reusable function to a file:

%%writefile my_analysis.py

import pandas as pd

def get_means(df, col):
    return df.groupby('group')[col].mean()

Expected output:

Writing my_analysis.py

You can then import this function to reuse later:

from my_analysis import get_means
print(get_means(df, 'y'))

Compare options / when to choose what

Different magic commands suit different needs. Here's a comparison table to help you choose:

Rule of thumb: - Use %time for a quick sense of speed; %timeit when you need reliable numbers. - Use %%time or %%timeit when your timing includes cell-level operations (like pandas transformations). - Use %who when you're unsure what's in your namespace — this prevents accidental overwrites. - Use %%writefile for turning exploratory code into reproducible scripts.

Troubleshooting & edge cases

Even magic commands can trip you up. Here are common issues and how to fix them.

1. Magic command not found

If you type %mycommand and get Error: Magic function 'mycommand' not found, you've probably made a typo or the magic isn't available in your kernel. Check available magics with %lsmagic.

2. %% at the top of a cell causes SyntaxError

If you use %% and then put other code on the same line, it fails. The %% must be the first thing in the cell, and the entire cell's content is the argument to the magic. Example:

%%time
# This is fine
print('hi')

But this fails:

%%time print('hi')  # WRONG

3. %timeit gives different results than %time

That's normal! %timeit runs the code multiple times and displays stats, while %time gives just one run. Use %timeit for accurate benchmarks, but remember it could be slower due to many runs.

4. Plots not showing inline

If your plots don't appear, ensure you ran %matplotlib inline in the notebook (or used %matplotlib notebook for interactive). If plots still don't show, restart the kernel and re-run.

5. %%writefile overwrites existing files

By default, %%writefile overwrites the file. To append, use %%writefile -a filename.py.

What you've learned & what's next

You now have a solid grasp of magic commands and how to use them for efficiency. You can time code with %time and %%time, benchmark with %timeit, list variables with %who, write files with %%writefile, and set up inline plots with %matplotlib inline.

To connect this to your data science workflow: after you've cleaned and analyzed data, you'll often need to share reproducible notebooks. In the next lesson, we'll explore how to turn your notebooks into clean, shareable reports — using magic commands to keep your code lean and your story compelling.

Now try experimenting with these magics on your own data. The more you use them, the more natural they'll feel.