To test out code changes, you’ll need to build pandas from source, whichrequires a C/C++ compiler and Python environment. If you’re making documentationchanges, you can skip to contributing to the documentation but if you skipcreating the development environment you won’t be able to build the documentationlocally before pushing your changes. It’s recommended to also install the pre-commit hooks.
Step 1: install a C compiler#
How to do this will depend on your platform. If you choose to use Docker or GitPodin the next step, then you can skip this step.
Windows
You will need Build Tools for Visual Studio 2022.
Note
You DO NOT need to install Visual Studio 2022.You only need “Build Tools for Visual Studio 2022” found byscrolling down to “All downloads” -> “Tools for Visual Studio”.In the installer, select the “Desktop development with C++” Workloads.
Alternatively, you can install the necessary components on the commandline usingvs_BuildTools.exe
Alternatively, you could use the WSLand consult the Linux instructions below.
macOS
To use the mamba-based compilers, you will need to install theDeveloper Tools using xcode-select --install.
If you prefer to use a different compiler, general information can be found here:https://devguide.python.org/setup/#macos
Linux
For Linux-based mamba installations, you won’t have to install anyadditional components outside of the mamba environment. The instructionsbelow are only needed if your setup isn’t based on mamba environments.
Some Linux distributions will come with a pre-installed C compiler. To find outwhich compilers (and versions) are installed on your system:
# for Debian/Ubuntu:dpkg --list | grep compiler# for Red Hat/RHEL/CentOS/Fedora:yum list installed | grep -i --color compiler
GCC (GNU Compiler Collection), is a widely usedcompiler, which supports C and a number of other languages. If GCC is listedas an installed compiler nothing more is required.
If no C compiler is installed, or you wish to upgrade, or you’re using a differentLinux distribution, consult your favorite search engine for compiler installation/updateinstructions.
Let us know if you have any difficulties by opening an issue or reaching out on our contributorcommunity Slack.
Step 2: create an isolated environment#
Before we begin, please:
Make sure that you have cloned the repository
cdto the pandas source directory you just created with the clone command
Option 1: using mamba (recommended)#
Install miniforge to get mamba
Make sure your mamba is up to date (
mamba update mamba)Create and activate the
pandas-devmamba environment using the following commands:
mamba env create --file environment.ymlmamba activate pandas-dev
Option 2: using pip#
You’ll need to have at least the minimum Python version that pandas supports.You also need to have setuptools 51.0.0 or later to build pandas.
Unix/macOS with virtualenv
# Create a virtual environment# Use an ENV_DIR of your choice. We'll use ~/virtualenvs/pandas-dev# Any parent directories should already existpython3 -m venv ~/virtualenvs/pandas-dev# Activate the virtualenv. ~/virtualenvs/pandas-dev/bin/activate# Install the build dependenciespython -m pip install -r requirements-dev.txt
Unix/macOS with pyenv
Consult the docs for setting up pyenv here.
# Create a virtual environment# Use an ENV_DIR of your choice. We'll use ~/Users/<yourname>/.pyenv/versions/pandas-devpyenv virtualenv <version> <name-to-give-it># For instance:pyenv virtualenv 3.9.10 pandas-dev# Activate the virtualenvpyenv activate pandas-dev# Now install the build dependencies in the cloned pandas repopython -m pip install -r requirements-dev.txt
Windows
Below is a brief overview on how to set-up a virtual environment with Powershellunder Windows. For details please refer to theofficial virtualenv user guide.
Use an ENV_DIR of your choice. We’ll use ~\\virtualenvs\\pandas-dev where~ is the folder pointed to by either $env:USERPROFILE (Powershell) or%USERPROFILE% (cmd.exe) environment variable. Any parent directoriesshould already exist.
# Create a virtual environmentpython -m venv $env:USERPROFILE\virtualenvs\pandas-dev# Activate the virtualenv. Use activate.bat for cmd.exe~\virtualenvs\pandas-dev\Scripts\Activate.ps1# Install the build dependenciespython -m pip install -r requirements-dev.txt
Option 3: using Docker#
pandas provides a DockerFile in the root directory to build a Docker imagewith a full pandas development environment.
Docker Commands
Build the Docker image:
# Build the imagedocker build -t pandas-dev .
Run Container:
# Run a container and bind your local repo to the container# This command assumes you are running from your local repo# but if not alter ${PWD} to match your local repo pathdocker run -it --rm -v ${PWD}:/home/pandas pandas-devEven easier, you can integrate Docker with the following IDEs:
Visual Studio Code
You can use the DockerFile to launch a remote session with Visual Studio Code,a popular free IDE, using the .devcontainer.json file.See https://code.visualstudio.com/docs/remote/containers for details.
PyCharm (Professional)
Enable Docker support and use the Services tool window to build and manage images as well asrun and interact with containers.See https://www.jetbrains.com/help/pycharm/docker.html for details.
Option 4: using Gitpod#
Gitpod is an open-source platform that automatically creates the correct developmentenvironment right in your browser, reducing the need to install local developmentenvironments and deal with incompatible dependencies.
If you are a Windows user, unfamiliar with using the command line or building pandasfor the first time, it is often faster to build with Gitpod. Here are the in-depth instructionsfor building pandas with GitPod.
Step 3: build and install pandas#
There are currently two supported ways of building pandas, pip/meson and setuptools(setup.py).Historically, pandas has only supported using setuptools to build pandas. However, this methodrequires a lot of convoluted code in setup.py and also has many issues in compiling pandas in paralleldue to limitations in setuptools.
The newer build system, invokes the meson backend through pip (via a PEP 517 build).It automatically uses all available cores on your CPU, and also avoids the need for manual rebuilds byrebuilding automatically whenever pandas is imported (with an editable install).
For these reasons, you should compile pandas with meson.Because the meson build system is newer, you may find bugs/minor issues as it matures. You can report these bugshere.
To compile pandas with meson, run:
# Build and install pandas# By default, this will print verbose output# showing the "rebuild" taking place on import (see section below for explanation)# If you do not want to see this, omit everything after --no-build-isolationpython -m pip install -ve . --no-build-isolation --config-settings editable-verbose=true
Note
The version number is pulled from the latest repository tag. Be sure to fetch the latest tags from upstreambefore building:
# set the upstream repository, if not done already, and fetch the latest tagsgit remote add upstream https://github.com/pandas-dev/pandas.gitgit fetch upstream --tags
Build options
It is possible to pass options from the pip frontend to the meson backend if you would like to configure yourinstall. Occasionally, you’ll want to use this to adjust the build directory, and/or toggle debug/optimization levels.
You can pass a build directory to pandas by appending --config-settings builddir="your builddir here" to your pip command.This option allows you to configure where meson stores your built C extensions, and allows for fast rebuilds.
Sometimes, it might be useful to compile pandas with debugging symbols, when debugging C extensions.Appending --config-settings setup-args="-Ddebug=true" will do the trick.
With pip, it is possible to chain together multiple config settings (for example specifying both a build directoryand building with debug symbols would look like--config-settings builddir="your builddir here" --config-settings=setup-args="-Dbuildtype=debug".
Compiling pandas with setup.py
Note
This method of compiling pandas will be deprecated and removed very soon, as the meson backend matures.
To compile pandas with setuptools, run:
python setup.py develop
Note
If pandas is already installed (via meson), you have to uninstall it first:
python -m pip uninstall pandas
This is because python setup.py develop will not uninstall the loader script that meson-pythonuses to import the extension from the build folder, which may cause errors such as anFileNotFoundError to be raised.
Note
You will need to repeat this step each time the C extensions change, for exampleif you modified any file in pandas/_libs or if you did a fetch and merge from upstream/main.
Checking the build
At this point you should be able to import pandas from your locally built version:
$ python>>> import pandas>>> print(pandas.__version__) # note: the exact output may differ2.0.0.dev0+880.g2b9e661fbb.dirty
At this point you may want to tryrunning the test suite.
Keeping up to date with the latest build
When building pandas with meson, importing pandas will automatically trigger a rebuild, even when C/Cython files are modified.By default, no output will be produced by this rebuild (the import will just take longer). If you would like to see meson’soutput when importing pandas, you can set the environment variable MESONPY_EDTIABLE_VERBOSE. For example, this would be:
# On Linux/macOSMESONPY_EDITABLE_VERBOSE=1 python# Windowsset MESONPY_EDITABLE_VERBOSE=1 # Only need to set this once per sessionpython
If you would like to see this verbose output every time, you can set the editable-verbose config setting to true like so:
python -m pip install -ve . --config-settings editable-verbose=true
If you ever find yourself wondering whether setuptools or meson was used to build your pandas,you can check the value of pandas._built_with_meson, which will be true if meson was usedto compile pandas.
