Python Pandas Tutorial: A Complete Introduction for Beginners (2024)

Moreover, for those of you looking to do a data science bootcamp or some other accelerated data science education program, it's highly recommended you start learning pandas on your own before you start the program.

Even though accelerated programs teach you pandas, better skills beforehand means you'll be able to maximize time for learning and mastering the more complicated material.

Pandas is an easy package to install. Open up your terminal program (for Mac users) or command line (for PC users) and install it using either of the following commands:

conda install pandas

OR

pip install pandas

Alternatively, if you're currently viewing this article in a Jupyter notebook you can run this cell:

!pip install pandas
 Learn Data Science with 

The ! at the beginning runs cells as if they were in a terminal.

To import pandas we usually import it with a shorter name since it's used so much:

import pandas as pd
 Learn Data Science with 

The primary two components of pandas are the Series and DataFrame.

A Series is essentially a column, and a DataFrame is a multi-dimensional table made up of a collection of Series.

DataFrames and Series are quite similar in that many operations that you can do with one you can do with the other, such as filling in null values and calculating the mean.

You'll see how these components work when we start working with data below.

Creating DataFrames right in Python is good to know and quite useful when testing new methods and functions you find in the pandas docs.

There are many ways to create a DataFrame from scratch, but a great option is to just use a simple dict.

Let's say we have a fruit stand that sells apples and oranges. We want to have a column for each fruit and a row for each customer purchase. To organize this as a dictionary for pandas we could do something like:

data = { 'apples': [3, 2, 0, 1], 'oranges': [0, 3, 7, 2]}
 Learn Data Science with 

purchases = pd.DataFrame(data)purchases
 Learn Data Science with 

How did that work?

Each (key, value) item in data corresponds to a column in the resulting DataFrame.

The Index of this DataFrame was given to us on creation as the numbers 0-3, but we could also create our own when we initialize the DataFrame.

Let's have customer names as our index:

purchases = pd.DataFrame(data, index=['June', 'Robert', 'Lily', 'David'])purchases
 Learn Data Science with 

purchases.loc['June']
 Learn Data Science with 

Out:

apples 3oranges 0Name: June, dtype: int64
 Learn Data Science with 

There's more on locating and extracting data from the DataFrame later, but now you should be able to create a DataFrame with any random data to learn on.

Let's move on to some quick methods for creating DataFrames from various other sources.

df = pd.read_csv('purchases.csv')df
 Learn Data Science with 

df = pd.read_csv('purchases.csv', index_col=0)df
 Learn Data Science with 

Here we're setting the index to be column zero.

You'll find that most CSVs won't ever have an index column and so usually you don't have to worry about this step.

df = pd.read_json('purchases.json')df
 Learn Data Science with 

Notice this time our index came with us correctly since using JSON allowed indexes to work through nesting. Feel free to open data_file.json in a notepad so you can see how it works.

Pandas will try to figure out how to create a DataFrame by analyzing structure of your JSON, and sometimes it doesn't get it right. Often you'll need to set the orient keyword argument depending on the structure, so check out read_json docs about that argument to see which orientation you're using.

If you’re working with data from a SQL database you need to first establish a connection using an appropriate Python library, then pass a query to pandas. Here we'll use SQLite to demonstrate.

First, we need pysqlite3 installed, so run this command in your terminal:

pip install pysqlite3

Or run this cell if you're in a notebook:

!pip install pysqlite3
 Learn Data Science with 

sqlite3 is used to create a connection to a database which we can then use to generate a DataFrame through a SELECT query.

So first we'll make a connection to a SQLite database file:

import sqlite3con = sqlite3.connect("database.db")
 Learn Data Science with 

SQL Tip

If you have data in PostgreSQL, MySQL, or some other SQL server, you'll need to obtain the right Python library to make a connection. For example, psycopg2 (link) is a commonly used library for making connections to PostgreSQL. Furthermore, you would make a connection to a database URI instead of a file like we did here with SQLite.

For a great course on SQL check out The Complete SQL Bootcamp on Udemy

In this SQLite database we have a table called purchases, and our index is in a column called "index".

By passing a SELECT query and our con, we can read from the purchases table:

df = pd.read_sql_query("SELECT * FROM purchases", con)df
 Learn Data Science with 

df = df.set_index('index')df
 Learn Data Science with 

df.to_csv('new_purchases.csv')df.to_json('new_purchases.json')df.to_sql('new_purchases', con)
 Learn Data Science with 

When we save JSON and CSV files, all we have to input into those functions is our desired filename with the appropriate file extension. With SQL, we’re not creating a new file but instead inserting a new table into the database using our con variable from before.

Let's move on to importing some real-world data and detailing a few of the operations you'll be using a lot.

DataFrames possess hundreds of methods and other operations that are crucial to any analysis. As a beginner, you should know the operations that perform simple transformations of your data and those that provide fundamental statistical analysis.

Let's load in the IMDB movies dataset to begin:

movies_df = pd.read_csv("IMDB-Movie-Data.csv", index_col="Title")
 Learn Data Science with 

movies_df.head()
 Learn Data Science with 

.head() outputs the first five rows of your DataFrame by default, but we could also pass a number as well: movies_df.head(10) would output the top ten rows, for example.

To see the last five rows use .tail(). tail() also accepts a number, and in this case we printing the bottom two rows.:

movies_df.tail(2)
 Learn Data Science with 

Typically when we load in a dataset, we like to view the first five or so rows to see what's under the hood. Here we can see the names of each column, the index, and examples of values in each row.

You'll notice that the index in our DataFrame is the Title column, which you can tell by how the word Title is slightly lower than the rest of the columns.

movies_df.info()
 Learn Data Science with 

Out:

<class 'pandas.core.frame.DataFrame'>Index: 1000 entries, Guardians of the Galaxy to Nine LivesData columns (total 11 columns):Rank 1000 non-null int64Genre 1000 non-null objectDescription 1000 non-null objectDirector 1000 non-null objectActors 1000 non-null objectYear 1000 non-null int64Runtime (Minutes) 1000 non-null int64Rating 1000 non-null float64Votes 1000 non-null int64Revenue (Millions) 872 non-null float64Metascore 936 non-null float64dtypes: float64(3), int64(4), object(4)memory usage: 93.8+ KB
 Learn Data Science with 

.info() provides the essential details about your dataset, such as the number of rows and columns, the number of non-null values, what type of data is in each column, and how much memory your DataFrame is using.

Notice in our movies dataset we have some obvious missing values in the Revenue and Metascore columns. We'll look at how to handle those in a bit.

Seeing the datatype quickly is actually quite useful. Imagine you just imported some JSON and the integers were recorded as strings. You go to do some arithmetic and find an "unsupported operand" Exception because you can't do math with strings. Calling .info() will quickly point out that your column you thought was all integers are actually string objects.

Another fast and useful attribute is .shape, which outputs just a tuple of (rows, columns):

movies_df.shape
 Learn Data Science with 

Out:

(1000, 11)
 Learn Data Science with 

Note that .shape has no parentheses and is a simple tuple of format (rows, columns). So we have 1000 rows and 11 columns in our movies DataFrame.

You'll be going to .shape a lot when cleaning and transforming data. For example, you might filter some rows based on some criteria and then want to know quickly how many rows were removed.

This dataset does not have duplicate rows, but it is always important to verify you aren't aggregating duplicate rows.

To demonstrate, let's simply just double up our movies DataFrame by appending it to itself:

temp_df = movies_df.append(movies_df)temp_df.shape
 Learn Data Science with 

Out:

(2000, 11)
 Learn Data Science with 

Using append() will return a copy without affecting the original DataFrame. We are capturing this copy in temp so we aren't working with the real data.

Notice call .shape quickly proves our DataFrame rows have doubled.

Now we can try dropping duplicates:

temp_df = temp_df.drop_duplicates()temp_df.shape
 Learn Data Science with 

Out:

(1000, 11)
 Learn Data Science with 

Just like append(), the drop_duplicates() method will also return a copy of your DataFrame, but this time with duplicates removed. Calling .shape confirms we're back to the 1000 rows of our original dataset.

It's a little verbose to keep assigning DataFrames to the same variable like in this example. For this reason, pandas has the inplace keyword argument on many of its methods. Using inplace=True will modify the DataFrame object in place:

temp_df.drop_duplicates(inplace=True)
 Learn Data Science with 

Now our temp_df will have the transformed data automatically.

Another important argument for drop_duplicates() is keep, which has three possible options:

• first: (default) Drop duplicates except for the first occurrence.
• last: Drop duplicates except for the last occurrence.
• False: Drop all duplicates.

Since we didn't define the keep arugment in the previous example it was defaulted to first. This means that if two rows are the same pandas will drop the second row and keep the first row. Using last has the opposite effect: the first row is dropped.

keep, on the other hand, will drop all duplicates. If two rows are the same then both will be dropped. Watch what happens to temp_df:

temp_df = movies_df.append(movies_df) # make a new copytemp_df.drop_duplicates(inplace=True, keep=False)temp_df.shape
 Learn Data Science with 

Out:

(0, 11)
 Learn Data Science with 

Many times datasets will have verbose column names with symbols, upper and lowercase words, spaces, and typos. To make selecting data by column name easier we can spend a little time cleaning up their names.

Here's how to print the column names of our dataset:

movies_df.columns
 Learn Data Science with 

Out:

Index(['Rank', 'Genre', 'Description', 'Director', 'Actors', 'Year', 'Runtime (Minutes)', 'Rating', 'Votes', 'Revenue (Millions)', 'Metascore'], dtype='object')
 Learn Data Science with 

Not only does .columns come in handy if you want to rename columns by allowing for simple copy and paste, it's also useful if you need to understand why you are receiving a Key Error when selecting data by column.

We can use the .rename() method to rename certain or all columns via a dict. We don't want parentheses, so let's rename those:

movies_df.rename(columns={ 'Runtime (Minutes)': 'Runtime', 'Revenue (Millions)': 'Revenue_millions' }, inplace=True)movies_df.columns
 Learn Data Science with 

Out:

Index(['Rank', 'Genre', 'Description', 'Director', 'Actors', 'Year', 'Runtime', 'Rating', 'Votes', 'Revenue_millions', 'Metascore'], dtype='object')
 Learn Data Science with 

movies_df.columns = ['rank', 'genre', 'description', 'director', 'actors', 'year', 'runtime', 'rating', 'votes', 'revenue_millions', 'metascore']movies_df.columns
 Learn Data Science with 

Out:

Index(['rank', 'genre', 'description', 'director', 'actors', 'year', 'runtime', 'rating', 'votes', 'revenue_millions', 'metascore'], dtype='object')
 Learn Data Science with 

movies_df.columns = [col.lower() for col in movies_df]movies_df.columns
 Learn Data Science with 

Out:

Index(['rank', 'genre', 'description', 'director', 'actors', 'year', 'runtime', 'rating', 'votes', 'revenue_millions', 'metascore'], dtype='object')
 Learn Data Science with 

list (and dict) comprehensions come in handy a lot when working with pandas and data in general.

It's a good idea to lowercase, remove special characters, and replace spaces with underscores if you'll be working with a dataset for some time.

When exploring data, you’ll most likely encounter missing or null values, which are essentially placeholders for non-existent values. Most commonly you'll see Python's None or NumPy's np.nan, each of which are handled differently in some situations.

There are two options in dealing with nulls:

1. Get rid of rows or columns with nulls
2. Replace nulls with non-null values, a technique known as imputation

Let's calculate to total number of nulls in each column of our dataset. The first step is to check which cells in our DataFrame are null:

movies_df.isnull()
 Learn Data Science with 

Notice isnull() returns a DataFrame where each cell is either True or False depending on that cell's null status.

To count the number of nulls in each column we use an aggregate function for summing:

movies_df.isnull().sum()
 Learn Data Science with 

Out:

rank 0genre 0description 0director 0actors 0year 0runtime 0rating 0votes 0revenue_millions 128metascore 64dtype: int64
 Learn Data Science with 

.isnull() just by iteself isn't very useful, and is usually used in conjunction with other methods, like sum().

We can see now that our data has 128 missing values for revenue_millions and 64 missing values for metascore.

Data Scientists and Analysts regularly face the dilemma of dropping or imputing null values, and is a decision that requires intimate knowledge of your data and its context. Overall, removing null data is only suggested if you have a small amount of missing data.

Remove nulls is pretty simple:

movies_df.dropna()
 Learn Data Science with 

This operation will delete any row with at least a single null value, but it will return a new DataFrame without altering the original one. You could specify inplace=True in this method as well.

So in the case of our dataset, this operation would remove 128 rows where revenue_millions is null and 64 rows where metascore is null. This obviously seems like a waste since there's perfectly good data in the other columns of those dropped rows. That's why we'll look at imputation next.

Other than just dropping rows, you can also drop columns with null values by setting axis=1:

movies_df.dropna(axis=1)
 Learn Data Science with 

Intuition

What's with this axis=1parameter?

It's not immediately obvious where axis comes from and why you need it to be 1 for it to affect columns. To see why, just look at the .shape output:

movies_df.shape

Out: (1000, 11)

As we learned above, this is a tuple that represents the shape of the DataFrame, i.e. 1000 rows and 11 columns. Note that the rows are at index zero of this tuple and columns are at index one of this tuple. This is why axis=1 affects columns. This comes from NumPy, and is a great example of why learning NumPy is worth your time.

Imputation is a conventional feature engineering technique used to keep valuable data that have null values.

There may be instances where dropping every row with a null value removes too big a chunk from your dataset, so instead we can impute that null with another value, usually the mean or the median of that column.

Let's look at imputing the missing values in the revenue_millions column. First we'll extract that column into its own variable:

revenue = movies_df['revenue_millions']
 Learn Data Science with 

Using square brackets is the general way we select columns in a DataFrame.

If you remember back to when we created DataFrames from scratch, the keys of the dict ended up as column names. Now when we select columns of a DataFrame, we use brackets just like if we were accessing a Python dictionary.

revenue now contains a Series:

revenue.head()
 Learn Data Science with 

Out:

TitleGuardians of the Galaxy 333.13Prometheus 126.46Split 138.12Sing 270.32Suicide Squad 325.02Name: revenue_millions, dtype: float64
 Learn Data Science with 

Slightly different formatting than a DataFrame, but we still have our Title index.

We'll impute the missing values of revenue using the mean. Here's the mean value:

revenue_mean = revenue.mean()revenue_mean
 Learn Data Science with 

Out:

82.95637614678897
 Learn Data Science with 

revenue.fillna(revenue_mean, inplace=True)
 Learn Data Science with 

movies_df.isnull().sum()
 Learn Data Science with 

Out:

rank 0genre 0description 0director 0actors 0year 0runtime 0rating 0votes 0revenue_millions 0metascore 64dtype: int64
 Learn Data Science with 

Imputing an entire column with the same value like this is a basic example. It would be a better idea to try a more granular imputation by Genre or Director.

For example, you would find the mean of the revenue generated in each genre individually and impute the nulls in each genre with that genre's mean.

Let's now look at more ways to examine and understand the dataset.

movies_df.describe()
 Learn Data Science with 

Understanding which numbers are continuous also comes in handy when thinking about the type of plot to use to represent your data visually.

.describe() can also be used on a categorical variable to get the count of rows, unique count of categories, top category, and freq of top category:

movies_df['genre'].describe()
 Learn Data Science with 

Out:

count 1000unique 207top Action,Adventure,Sci-Fifreq 50Name: genre, dtype: object
 Learn Data Science with 

This tells us that the genre column has 207 unique values, the top value is Action/Adventure/Sci-Fi, which shows up 50 times (freq).

.value_counts() can tell us the frequency of all values in a column:

movies_df['genre'].value_counts().head(10)
 Learn Data Science with 

Out:

Action,Adventure,Sci-Fi 50Drama 48Comedy,Drama,Romance 35Comedy 32Drama,Romance 31Action,Adventure,Fantasy 27Comedy,Drama 27Animation,Adventure,Comedy 27Comedy,Romance 26Crime,Drama,Thriller 24Name: genre, dtype: int64
 Learn Data Science with 

Relationships between continuous variables

By using the correlation method .corr() we can generate the relationship between each continuous variable:

movies_df.corr()
 Learn Data Science with 

Correlation tables are a numerical representation of the bivariate relationships in the dataset.

Positive numbers indicate a positive correlation — one goes up the other goes up — and negative numbers represent an inverse correlation — one goes up the other goes down. 1.0 indicates a perfect correlation.

So looking in the first row, first column we see rank has a perfect correlation with itself, which is obvious. On the other hand, the correlation between votes and revenue_millions is 0.6. A little more interesting.

Examining bivariate relationships comes in handy when you have an outcome or dependent variable in mind and would like to see the features most correlated to the increase or decrease of the outcome. You can visually represent bivariate relationships with scatterplots (seen below in the plotting section).

For a deeper look into data summarizations check out Essential Statistics for Data Science.

Let's now look more at manipulating DataFrames.

Up until now we've focused on some basic summaries of our data. We've learned about simple column extraction using single brackets, and we imputed null values in a column using fillna(). Below are the other methods of slicing, selecting, and extracting you'll need to use constantly.

It's important to note that, although many methods are the same, DataFrames and Series have different attributes, so you'll need be sure to know which type you are working with or else you will receive attribute errors.

Let's look at working with columns first.

genre_col = movies_df['genre']type(genre_col)
 Learn Data Science with 

Out:

pandas.core.series.Series
 Learn Data Science with 

genre_col = movies_df[['genre']]type(genre_col)
 Learn Data Science with 

pandas.core.frame.DataFrame
 Learn Data Science with 

subset = movies_df[['genre', 'rating']]subset.head()
 Learn Data Science with 

For rows, we have two options:

• .loc - locates by name
• .iloc- locates by numerical index

Remember that we are still indexed by movie Title, so to use .loc we give it the Title of a movie:

prom = movies_df.loc["Prometheus"]prom
 Learn Data Science with 

Out:

rank 2genre Adventure,Mystery,Sci-Fidescription Following clues to the origin of mankind, a te...director Ridley Scottactors Noomi Rapace, Logan Marshall-Green, Michael Fa...year 2012runtime 124rating 7votes 485820revenue_millions 126.46metascore 65Name: Prometheus, dtype: object
 Learn Data Science with 

prom = movies_df.iloc[1]
 Learn Data Science with 

loc and iloc can be thought of as similar to Python list slicing. To show this even further, let's select multiple rows.

How would you do it with a list? In Python, just slice with brackets like example_list[1:4]. It's works the same way in pandas:

movie_subset = movies_df.loc['Prometheus':'Sing']movie_subset = movies_df.iloc[1:4]movie_subset
 Learn Data Science with 

One important distinction between using .loc and .iloc to select multiple rows is that .locincludes the movie Sing in the result, but when using .iloc we're getting rows 1:4 but the movie at index 4 (Suicide Squad) is not included.

Slicing with .iloc follows the same rules as slicing with lists, the object at the index at the end is not included.

We’ve gone over how to select columns and rows, but what if we want to make a conditional selection?

For example, what if we want to filter our movies DataFrame to show only films directed by Ridley Scott or films with a rating greater than or equal to 8.0?

To do that, we take a column from the DataFrame and apply a Boolean condition to it. Here's an example of a Boolean condition:

condition = (movies_df['director'] == "Ridley Scott")condition.head()
 Learn Data Science with 

Out:

TitleGuardians of the Galaxy FalsePrometheus TrueSplit FalseSing FalseSuicide Squad FalseName: director, dtype: bool
 Learn Data Science with 

Similar to isnull(), this returns a Series of True and False values: True for films directed by Ridley Scott and False for ones not directed by him.

We want to filter out all movies not directed by Ridley Scott, in other words, we don’t want the False films. To return the rows where that condition is True we have to pass this operation into the DataFrame:

movies_df[movies_df['director'] == "Ridley Scott"]
 Learn Data Science with 

You can get used to looking at these conditionals by reading it like:

Select movies_df where movies_df director equals Ridley Scott.

Let's look at conditional selections using numerical values by filtering the DataFrame by ratings:

movies_df[movies_df['rating'] >= 8.6].head(3)
 Learn Data Science with 

We can make some richer conditionals by using logical operators | for "or" and & for "and".

Let's filter the the DataFrame to show only movies by Christopher Nolan OR Ridley Scott:

movies_df[(movies_df['director'] == 'Christopher Nolan') | (movies_df['director'] == 'Ridley Scott')].head()
 Learn Data Science with 

We need to make sure to group evaluations with parentheses so Python knows how to evaluate the conditional.

Using the isin() method we could make this more concise though:

movies_df[movies_df['director'].isin(['Christopher Nolan', 'Ridley Scott'])].head()
 Learn Data Science with 

Let's say we want all movies that were released between 2005 and 2010, have a rating above 8.0, but made below the 25th percentile in revenue.

Here's how we could do all of that:

movies_df[ ((movies_df['year'] >= 2005) & (movies_df['year'] <= 2010)) & (movies_df['rating'] > 8.0) & (movies_df['revenue_millions'] < movies_df['revenue_millions'].quantile(0.25))]
 Learn Data Science with 

If you recall up when we used .describe() the 25th percentile for revenue was about 17.4, and we can access this value directly by using the quantile() method with a float of 0.25.

So here we have only four movies that match that criteria.

It is possible to iterate over a DataFrame or Series as you would with a list, but doing so — especially on large datasets — is very slow.

An efficient alternative is to apply() a function to the dataset. For example, we could use a function to convert movies with an 8.0 or greater to a string value of "good" and the rest to "bad" and use this transformed values to create a new column.

First we would create a function that, when given a rating, determines if it's good or bad:

def rating_function(x): if x >= 8.0: return "good" else: return "bad"
 Learn Data Science with 

movies_df["rating_category"] = movies_df["rating"].apply(rating_function)movies_df.head(2)
 Learn Data Science with 

The .apply() method passes every value in the rating column through the rating_function and then returns a new Series. This Series is then assigned to a new column called rating_category.

You can also use anonymous functions as well. This lambda function achieves the same result as rating_function:

movies_df["rating_category"] = movies_df["rating"].apply(lambda x: 'good' if x >= 8.0 else 'bad')movies_df.head(2)
 Learn Data Science with 

import matplotlib.pyplot as pltplt.rcParams.update({'font.size': 20, 'figure.figsize': (10, 8)}) # set font and plot size to be larger
 Learn Data Science with 

Plotting Tip

For categorical variables utilize Bar Charts* and Boxplots.

For continuous variables utilize Histograms, Scatterplots, Line graphs, and Boxplots.

movies_df.plot(kind='scatter', x='rating', y='revenue_millions', title='Revenue (millions) vs Rating');
 Learn Data Science with 

What's with the semicolon? It's not a syntax error, just a way to hide the <matplotlib.axes._subplots.AxesSubplot at 0x26613b5cc18> output when plotting in Jupyter notebooks.

If we want to plot a simple Histogram based on a single column, we can call plot on a column:

movies_df['rating'].plot(kind='hist', title='Rating');
 Learn Data Science with 

movies_df['rating'].describe()
 Learn Data Science with 

Out:

count 1000.000000mean 6.723200std 0.945429min 1.90000025% 6.20000050% 6.80000075% 7.400000max 9.000000Name: rating, dtype: float64
 Learn Data Science with 

movies_df['rating'].plot(kind="box");
 Learn Data Science with 

movies_df.boxplot(column='revenue_millions', by='rating_category');
 Learn Data Science with 

Exploring, cleaning, transforming, and visualization data with pandas in Python is an essential skill in data science. Just cleaning wrangling data is 80% of your job as a Data Scientist. After a few projects and some practice, you should be very comfortable with most of the basics.

To keep improving, view the extensive tutorials offered by the official pandas docs, follow along with a few Kaggle kernels, and keep working on your own projects!

Resources

Applied Data Science with Python — Coursera

Covers an intro to Python, Visualization, Machine Learning, Text Mining, and Social Network Analysis in Python. Also provides many challenging quizzes and assignments to further enhance your learning.

Complete SQL Bootcamp — Udemy

An excellent course for learning SQL. The instructor explains everything from beginner to advanced SQL queries and techniques, and provides many exercises to help you learn.