Extending pandas — pandas 2.2.1 documentation (2024)

ExtensionDtype#

A pandas.api.extensions.ExtensionDtype is similar to a numpy.dtype object. It describes thedata type. Implementers are responsible for a few unique items like the name.

One particularly important item is the type property. This should be theclass that is the scalar type for your data. For example, if you were writing anextension array for IP Address data, this might be ipaddress.IPv4Address.

See the extension dtype source for interface definition.

pandas.api.extensions.ExtensionDtype can be registered to pandas to allow creation via a string dtype name.This allows one to instantiate Series and .astype() with a registered string name, forexample 'category' is a registered string accessor for the CategoricalDtype.

See the extension dtype dtypes for more on how to register dtypes.

ExtensionArray#

This class provides all the array-like functionality. ExtensionArrays arelimited to 1 dimension. An ExtensionArray is linked to an ExtensionDtype via thedtype attribute.

pandas makes no restrictions on how an extension array is created via its__new__ or __init__, and puts no restrictions on how you store yourdata. We do require that your array be convertible to a NumPy array, even ifthis is relatively expensive (as it is for Categorical).

They may be backed by none, one, or many NumPy arrays. For example,pandas.Categorical is an extension array backed by two arrays,one for codes and one for categories. An array of IPv6 addresses maybe backed by a NumPy structured array with two fields, one for thelower 64 bits and one for the upper 64 bits. Or they may be backedby some other storage type, like Python lists.

See the extension array source for the interface definition. The docstringsand comments contain guidance for properly implementing the interface.

ExtensionArray operator support#

By default, there are no operators defined for the class ExtensionArray.There are two approaches for providing operator support for your ExtensionArray:

1. Define each of the operators on your ExtensionArray subclass.

2. Use an operator implementation from pandas that depends on operators that are already definedon the underlying elements (scalars) of the ExtensionArray.

For the first approach, you define selected operators, e.g., __add__, __le__, etc. thatyou want your ExtensionArray subclass to support.

The second approach assumes that the underlying elements (i.e., scalar type) of the ExtensionArrayhave the individual operators already defined. In other words, if your ExtensionArraynamed MyExtensionArray is implemented so that each element is an instanceof the class MyExtensionElement, then if the operators are definedfor MyExtensionElement, the second approach will automaticallydefine the operators for MyExtensionArray.

A mixin class, ExtensionScalarOpsMixin supports this secondapproach. If developing an ExtensionArray subclass, for example MyExtensionArray,can simply include ExtensionScalarOpsMixin as a parent class of MyExtensionArray,and then call the methods _add_arithmetic_ops() and/or_add_comparison_ops() to hook the operators intoyour MyExtensionArray class, as follows:

from pandas.api.extensions import ExtensionArray, ExtensionScalarOpsMixinclass MyExtensionArray(ExtensionArray, ExtensionScalarOpsMixin): passMyExtensionArray._add_arithmetic_ops()MyExtensionArray._add_comparison_ops()

For arithmetic operations, this implementation will try to reconstruct a newExtensionArray with the result of the element-wise operation. Whetheror not that succeeds depends on whether the operation returns a resultthat’s valid for the ExtensionArray. If an ExtensionArray cannotbe reconstructed, an ndarray containing the scalars returned instead.

For ease of implementation and consistency with operations between pandasand NumPy ndarrays, we recommend not handling Series and Indexes in your binary ops.Instead, you should detect these cases and return NotImplemented.When pandas encounters an operation like op(Series, ExtensionArray), pandaswill

1. unbox the array from the Series (Series.array)

2. call result = op(values, ExtensionArray)

3. re-box the result in a Series

NumPy universal functions#

Series implements __array_ufunc__. As part of the implementation,pandas unboxes the ExtensionArray from the Series, applies the ufunc,and re-boxes it if necessary.

If applicable, we highly recommend that you implement __array_ufunc__ in yourextension array to avoid coercion to an ndarray. Seethe NumPy documentationfor an example.

As part of your implementation, we require that you defer to pandas when a pandascontainer (Series, DataFrame, Index) is detected in inputs.If any of those is present, you should return NotImplemented. pandas will take care ofunboxing the array from the container and re-calling the ufunc with the unwrapped input.

Testing extension arrays#

We provide a test suite for ensuring that your extension arrays satisfy the expectedbehavior. To use the test suite, you must provide several pytest fixtures and inheritfrom the base test class. The required fixtures are found inpandas-dev/pandas.

To use a test, subclass it:

from pandas.tests.extension import baseclass TestConstructors(base.BaseConstructorsTests): pass

See pandas-dev/pandasfor a list of all the tests available.

Compatibility with Apache Arrow#

An ExtensionArray can support conversion to / from pyarrow arrays(and thus support for example serialization to the Parquet file format)by implementing two methods: ExtensionArray.__arrow_array__ andExtensionDtype.__from_arrow__.

The ExtensionArray.__arrow_array__ ensures that pyarrow knowns howto convert the specific extension array into a pyarrow.Array (also whenincluded as a column in a pandas DataFrame):

class MyExtensionArray(ExtensionArray): ... def __arrow_array__(self, type=None): # convert the underlying array values to a pyarrow Array import pyarrow return pyarrow.array(..., type=type)

The ExtensionDtype.__from_arrow__ method then controls the conversionback from pyarrow to a pandas ExtensionArray. This method receives a pyarrowArray or ChunkedArray as only argument and is expected to return theappropriate pandas ExtensionArray for this dtype and the passed values:

class ExtensionDtype: ... def __from_arrow__(self, array: pyarrow.Array/ChunkedArray) -> ExtensionArray: ...

See more in the Arrow documentation.

Those methods have been implemented for the nullable integer and string extensiondtypes included in pandas, and ensure roundtrip to pyarrow and the Parquet file format.

Override constructor properties#

Each data structure has several constructor properties for returning a newdata structure as the result of an operation. By overriding these properties,you can retain subclasses through pandas data manipulations.

There are 3 possible constructor properties to be defined on a subclass:

• DataFrame/Series._constructor: Used when a manipulation result has the same dimension as the original.

• DataFrame._constructor_sliced: Used when a DataFrame (sub-)class manipulation result should be a Series (sub-)class.

• Series._constructor_expanddim: Used when a Series (sub-)class manipulation result should be a DataFrame (sub-)class, e.g. Series.to_frame().

Below example shows how to define SubclassedSeries and SubclassedDataFrame overriding constructor properties.

class SubclassedSeries(pd.Series): @property def _constructor(self): return SubclassedSeries @property def _constructor_expanddim(self): return SubclassedDataFrameclass SubclassedDataFrame(pd.DataFrame): @property def _constructor(self): return SubclassedDataFrame @property def _constructor_sliced(self): return SubclassedSeries

>>> s = SubclassedSeries([1, 2, 3])>>> type(s)<class '__main__.SubclassedSeries'>>>> to_framed = s.to_frame()>>> type(to_framed)<class '__main__.SubclassedDataFrame'>>>> df = SubclassedDataFrame({"A": [1, 2, 3], "B": [4, 5, 6], "C": [7, 8, 9]})>>> df A B C0 1 4 71 2 5 82 3 6 9>>> type(df)<class '__main__.SubclassedDataFrame'>>>> sliced1 = df[["A", "B"]]>>> sliced1 A B0 1 41 2 52 3 6>>> type(sliced1)<class '__main__.SubclassedDataFrame'>>>> sliced2 = df["A"]>>> sliced20 11 22 3Name: A, dtype: int64>>> type(sliced2)<class '__main__.SubclassedSeries'>

Define original properties#

To let original data structures have additional properties, you should let pandas know what properties are added. pandas maps unknown properties to data names overriding __getattribute__. Defining original properties can be done in one of 2 ways:

1. Define _internal_names and _internal_names_set for temporary properties which WILL NOT be passed to manipulation results.

2. Define _metadata for normal properties which will be passed to manipulation results.

Below is an example to define two original properties, “internal_cache” as a temporary property and “added_property” as a normal property

class SubclassedDataFrame2(pd.DataFrame): # temporary properties _internal_names = pd.DataFrame._internal_names + ["internal_cache"] _internal_names_set = set(_internal_names) # normal properties _metadata = ["added_property"] @property def _constructor(self): return SubclassedDataFrame2

>>> df = SubclassedDataFrame2({"A": [1, 2, 3], "B": [4, 5, 6], "C": [7, 8, 9]})>>> df A B C0 1 4 71 2 5 82 3 6 9>>> df.internal_cache = "cached">>> df.added_property = "property">>> df.internal_cachecached>>> df.added_propertyproperty# properties defined in _internal_names is reset after manipulation>>> df[["A", "B"]].internal_cacheAttributeError: 'SubclassedDataFrame2' object has no attribute 'internal_cache'# properties defined in _metadata are retained>>> df[["A", "B"]].added_propertyproperty