Pandas 中文参考指南

Frequently Asked Questions (FAQ)

DataFrame memory usage

调用` info()时会显示 DataFrame的内存使用情况（包括索引）。一个配置选项`display.memory_usage（参见` the list of options）指定调用 info()方法时是否显示 DataFrame`内存使用情况。

The memory usage of a DataFrame (including the index) is shown when calling the info(). A configuration option, display.memory_usage (see the list of options), specifies if the DataFrame memory usage will be displayed when invoking the info() method.

例如，调用` info()时会显示以下 DataFrame`的内存使用情况：

For example, the memory usage of the DataFrame below is shown when calling info():

In [1]: dtypes = [
   ...:     "int64",
   ...:     "float64",
   ...:     "datetime64[ns]",
   ...:     "timedelta64[ns]",
   ...:     "complex128",
   ...:     "object",
   ...:     "bool",
   ...: ]
   ...:

In [2]: n = 5000

In [3]: data = {t: np.random.randint(100, size=n).astype(t) for t in dtypes}

In [4]: df = pd.DataFrame(data)

In [5]: df["categorical"] = df["object"].astype("category")

In [6]: df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 8 columns):
 #   Column           Non-Null Count  Dtype
---  ------           --------------  -----
 0   int64            5000 non-null   int64
 1   float64          5000 non-null   float64
 2   datetime64[ns]   5000 non-null   datetime64[ns]
 3   timedelta64[ns]  5000 non-null   timedelta64[ns]
 4   complex128       5000 non-null   complex128
 5   object           5000 non-null   object
 6   bool             5000 non-null   bool
 7   categorical      5000 non-null   category
dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), timedelta64[ns](1)
memory usage: 288.2+ KB

`+`符号表示真实内存使用情况可能更高，因为 pandas 不会计算含有`dtype=object`列中值所使用的内存。

The + symbol indicates that the true memory usage could be higher, because pandas does not count the memory used by values in columns with dtype=object.

传递`memory_usage='deep'`会启用更准确的内存使用情况报告，占包含对象总使用量。这是可选的，因为执行更深入的内省可能代价高昂。

Passing memory_usage='deep' will enable a more accurate memory usage report, accounting for the full usage of the contained objects. This is optional as it can be expensive to do this deeper introspection.

In [7]: df.info(memory_usage="deep")
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 8 columns):
 #   Column           Non-Null Count  Dtype
---  ------           --------------  -----
 0   int64            5000 non-null   int64
 1   float64          5000 non-null   float64
 2   datetime64[ns]   5000 non-null   datetime64[ns]
 3   timedelta64[ns]  5000 non-null   timedelta64[ns]
 4   complex128       5000 non-null   complex128
 5   object           5000 non-null   object
 6   bool             5000 non-null   bool
 7   categorical      5000 non-null   category
dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), timedelta64[ns](1)
memory usage: 424.7 KB

默认情况下，显示选项设置为`True`，但通过调用` info()`时传递`memory_usage`参数可以明确地覆盖该设置。

By default the display option is set to True but can be explicitly overridden by passing the memory_usage argument when invoking info().

可以通过调用` memory_usage()方法查找每列的内存使用情况。这会返回一个 Series，其中索引由列名表示，而每列的内存使用情况以字节为单位显示。对于上述的 DataFrame，可以使用 memory_usage()`方法查找每列的内存使用情况以及总内存使用情况：

The memory usage of each column can be found by calling the memory_usage() method. This returns a Series with an index represented by column names and memory usage of each column shown in bytes. For the DataFrame above, the memory usage of each column and the total memory usage can be found with the memory_usage() method:

In [8]: df.memory_usage()
Out[8]:
Index                128
int64              40000
float64            40000
datetime64[ns]     40000
timedelta64[ns]    40000
complex128         80000
object             40000
bool                5000
categorical         9968
dtype: int64

# total memory usage of dataframe
In [9]: df.memory_usage().sum()
Out[9]: 295096

默认情况下，返回的 Series 中会显示 DataFrame 索引的内存使用量，可以通过传递 index=False 参数来抑制索引的内存使用量：

By default the memory usage of the DataFrame index is shown in the returned Series, the memory usage of the index can be suppressed by passing the index=False argument:

In [10]: df.memory_usage(index=False)
Out[10]:
int64              40000
float64            40000
datetime64[ns]     40000
timedelta64[ns]    40000
complex128         80000
object             40000
bool                5000
categorical         9968
dtype: int64

info() 方法显示的内存使用量利用 memory_usage() 方法来确定 DataFrame 的内存使用量，同时也以人类可读的单位（2 的幂次方制表示法；即 1 KB = 1024 字节）设置输出的格式。

The memory usage displayed by the info() method utilizes the memory_usage() method to determine the memory usage of a DataFrame while also formatting the output in human-readable units (base-2 representation; i.e. 1KB = 1024 bytes).

另请参阅 Categorical Memory Usage。

Using if/truth statements with pandas

当你尝试将某个内容转换为 bool 时，pandas 遵循 NumPy 抛出错误的惯例。这发生在 if 语句中或者在使用布尔运算时：and、or 和 not。以下代码结果应该是什么尚不清楚：

pandas follows the NumPy convention of raising an error when you try to convert something to a bool. This happens in an if-statement or when using the boolean operations: and, or, and not. It is not clear what the result of the following code should be:

>>> if pd.Series([False, True, False]):
...     pass

它应该是 True 因为它不是零长度吗，还是因为那里有 False 个值而应该是 False？不是很清楚，因此，pandas 抛出一个 ValueError：

Should it be True because it’s not zero-length, or False because there are False values? It is unclear, so instead, pandas raises a ValueError:

In [11]: if pd.Series([False, True, False]):
   ....:     print("I was true")
   ....:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-11-5c782b38cd2f> in ?()
----> 1 if pd.Series([False, True, False]):
      2     print("I was true")

~/work/pandas/pandas/pandas/core/generic.py in ?(self)
   1575     @final
   1576     def __nonzero__(self) -> NoReturn:
-> 1577         raise ValueError(
   1578             f"The truth value of a {type(self).__name__} is ambiguous. "
   1579             "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
   1580         )

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

你需要明确选择你想使用 DataFrame 做什么，例如使用 any()、 all() 或 empty()。或者，你可能要比较 pandas 对象是否为 None：

You need to explicitly choose what you want to do with the DataFrame, e.g. use any(), all() or empty(). Alternatively, you might want to compare if the pandas object is None:

In [12]: if pd.Series([False, True, False]) is not None:
   ....:     print("I was not None")
   ....:
I was not None

以下是如何检查任意值是否为 True 的方法：

Below is how to check if any of the values are True:

In [13]: if pd.Series([False, True, False]).any():
   ....:     print("I am any")
   ....:
I am any

Bitwise boolean

位布尔运算符如 == 和 != 返回一个布尔值 Series，与标量进行逐元素比较时执行此操作。

Bitwise boolean operators like == and != return a boolean Series which performs an element-wise comparison when compared to a scalar.

In [14]: s = pd.Series(range(5))

In [15]: s == 4
Out[15]:
0    False
1    False
2    False
3    False
4     True
dtype: bool

有关更多示例，请参阅 boolean comparisons。

See boolean comparisons for more examples.

Using the in operator

对 Series 使用 Python in 运算符是为了测试索引内的隶属关系，而不是值之间的隶属关系。

Using the Python in operator on a Series tests for membership in the index, not membership among the values.

In [16]: s = pd.Series(range(5), index=list("abcde"))

In [17]: 2 in s
Out[17]: False

In [18]: 'b' in s
Out[18]: True

如果这个行为令人惊讶，请记住对 Python 字典使用 in 是为了测试键而不是值，而且 Series 类似于字典。要测试值之间的隶属关系，请使用 isin() 方法：

If this behavior is surprising, keep in mind that using in on a Python dictionary tests keys, not values, and Series are dict-like. To test for membership in the values, use the method isin():

In [19]: s.isin([2])
Out[19]:
a    False
b    False
c     True
d    False
e    False
dtype: bool

In [20]: s.isin([2]).any()
Out[20]: True

对于 DataFrame，同样，in 应用于列轴，测试列名列表中的隶属关系。

For DataFrame, likewise, in applies to the column axis, testing for membership in the list of column names.

Mutating with User Defined Function (UDF) methods

本节适用于使用 UDF 的 pandas 方法。尤其是方法 DataFrame.apply()、 DataFrame.aggregate()、 DataFrame.transform() 和 DataFrame.filter()。

This section applies to pandas methods that take a UDF. In particular, the methods DataFrame.apply(), DataFrame.aggregate(), DataFrame.transform(), and DataFrame.filter().

在编程中，一条普遍规则是当迭代一个容器时，不要改变它。更改会使迭代器失效，从而导致意外的行为。考虑以下示例：

It is a general rule in programming that one should not mutate a container while it is being iterated over. Mutation will invalidate the iterator, causing unexpected behavior. Consider the example:

In [21]: values = [0, 1, 2, 3, 4, 5]

In [22]: n_removed = 0

In [23]: for k, value in enumerate(values):
   ....:     idx = k - n_removed
   ....:     if value % 2 == 1:
   ....:         del values[idx]
   ....:         n_removed += 1
   ....:     else:
   ....:         values[idx] = value + 1
   ....:

In [24]: values
Out[24]: [1, 4, 5]

人们可能会预期结果为 [1, 3, 5]。当使用采用 UDF 的 pandas 方法时，pandas 内部通常会迭代 DataFrame 或其他 pandas 对象。因此，如果 UDF 改变（更改） DataFrame，就会出现意外的行为。

One probably would have expected that the result would be [1, 3, 5]. When using a pandas method that takes a UDF, internally pandas is often iterating over the DataFrame or other pandas object. Therefore, if the UDF mutates (changes) the DataFrame, unexpected behavior can arise.

下面是 DataFrame.apply() 的一个类似示例：

Here is a similar example with DataFrame.apply():

In [25]: def f(s):
   ....:     s.pop("a")
   ....:     return s
   ....:

In [26]: df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})

In [27]: df.apply(f, axis="columns")
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/work/pandas/pandas/pandas/core/indexes/base.py:3805, in Index.get_loc(self, key)
   3804 try:
-> 3805     return self._engine.get_loc(casted_key)
   3806 except KeyError as err:

File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc()

File index.pyx:196, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:7081, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:7089, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'a'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[27], line 1
----> 1 df.apply(f, axis="columns")

File ~/work/pandas/pandas/pandas/core/frame.py:10374, in DataFrame.apply(self, func, axis, raw, result_type, args, by_row, engine, engine_kwargs, **kwargs)
  10360 from pandas.core.apply import frame_apply
  10362 op = frame_apply(
  10363     self,
  10364     func=func,
   (...)
  10372     kwargs=kwargs,
  10373 )
> 10374 return op.apply().__finalize__(self, method="apply")

File ~/work/pandas/pandas/pandas/core/apply.py:916, in FrameApply.apply(self)
    913 elif self.raw:
    914     return self.apply_raw(engine=self.engine, engine_kwargs=self.engine_kwargs)
--> 916 return self.apply_standard()

File ~/work/pandas/pandas/pandas/core/apply.py:1063, in FrameApply.apply_standard(self)
   1061 def apply_standard(self):
   1062     if self.engine == "python":
-> 1063         results, res_index = self.apply_series_generator()
   1064     else:
   1065         results, res_index = self.apply_series_numba()

File ~/work/pandas/pandas/pandas/core/apply.py:1081, in FrameApply.apply_series_generator(self)
   1078 with option_context("mode.chained_assignment", None):
   1079     for i, v in enumerate(series_gen):
   1080         # ignore SettingWithCopy here in case the user mutates
-> 1081         results[i] = self.func(v, *self.args, **self.kwargs)
   1082         if isinstance(results[i], ABCSeries):
   1083             # If we have a view on v, we need to make a copy because
   1084             #  series_generator will swap out the underlying data
   1085             results[i] = results[i].copy(deep=False)

Cell In[25], line 2, in f(s)
      1 def f(s):
----> 2     s.pop("a")
      3     return s

File ~/work/pandas/pandas/pandas/core/series.py:5391, in Series.pop(self, item)
   5366 def pop(self, item: Hashable) -> Any:
   5367     """
   5368     Return item and drops from series. Raise KeyError if not found.
   5369
   (...)
   5389     dtype: int64
   5390     """
-> 5391     return super().pop(item=item)

File ~/work/pandas/pandas/pandas/core/generic.py:947, in NDFrame.pop(self, item)
    946 def pop(self, item: Hashable) -> Series | Any:
--> 947     result = self[item]
    948     del self[item]
    950     return result

File ~/work/pandas/pandas/pandas/core/series.py:1121, in Series.__getitem__(self, key)
   1118     return self._values[key]
   1120 elif key_is_scalar:
-> 1121     return self._get_value(key)
   1123 # Convert generator to list before going through hashable part
   1124 # (We will iterate through the generator there to check for slices)
   1125 if is_iterator(key):

File ~/work/pandas/pandas/pandas/core/series.py:1237, in Series._get_value(self, label, takeable)
   1234     return self._values[label]
   1236 # Similar to Index.get_value, but we do not fall back to positional
-> 1237 loc = self.index.get_loc(label)
   1239 if is_integer(loc):
   1240     return self._values[loc]

File ~/work/pandas/pandas/pandas/core/indexes/base.py:3812, in Index.get_loc(self, key)
   3807     if isinstance(casted_key, slice) or (
   3808         isinstance(casted_key, abc.Iterable)
   3809         and any(isinstance(x, slice) for x in casted_key)
   3810     ):
   3811         raise InvalidIndexError(key)
-> 3812     raise KeyError(key) from err
   3813 except TypeError:
   3814     # If we have a listlike key, _check_indexing_error will raise
   3815     #  InvalidIndexError. Otherwise we fall through and re-raise
   3816     #  the TypeError.
   3817     self._check_indexing_error(key)

KeyError: 'a'

要解决这个问题，可以制作一个副本，这样更改就不会应用于正在迭代的容器。

To resolve this issue, one can make a copy so that the mutation does not apply to the container being iterated over.

In [28]: values = [0, 1, 2, 3, 4, 5]

In [29]: n_removed = 0

In [30]: for k, value in enumerate(values.copy()):
   ....:     idx = k - n_removed
   ....:     if value % 2 == 1:
   ....:         del values[idx]
   ....:         n_removed += 1
   ....:     else:
   ....:         values[idx] = value + 1
   ....:

In [31]: values
Out[31]: [1, 3, 5]

In [32]: def f(s):
   ....:     s = s.copy()
   ....:     s.pop("a")
   ....:     return s
   ....:

In [33]: df = pd.DataFrame({"a": [1, 2, 3], 'b': [4, 5, 6]})

In [34]: df.apply(f, axis="columns")
Out[34]:
   b
0  4
1  5
2  6

Missing value representation for NumPy types

np.nan as the NA representation for NumPy types

由于从 NumPy 和 Python 的基础开始就缺乏 NA（缺失）支持，所以 NA 本可以使用以下方式表示：

For lack of NA (missing) support from the ground up in NumPy and Python in general, NA could have been represented with:

A masked array solution: an array of data and an array of boolean values indicating whether a value is there or is missing.
Using a special sentinel value, bit pattern, or set of sentinel values to denote NA across the dtypes.

特殊值 np.nan（非数字）被选作 NumPy 类型的值 NA，并且有诸如 DataFrame.isna() 和 DataFrame.notna() 的 API 函数可用于跨不同 dtypes 检测 NA 值。但是，此选择有一个缺点，即它会将缺失的整数数据强制转换为浮点类型，如 Support for integer NA 所示。

The special value np.nan (Not-A-Number) was chosen as the NA value for NumPy types, and there are API functions like DataFrame.isna() and DataFrame.notna() which can be used across the dtypes to detect NA values. However, this choice has a downside of coercing missing integer data as float types as shown in Support for integer NA.

NA type promotions for NumPy types

通过 reindex() 或其他某种方式向现有 Series 或 DataFrame 引入 NA 时，布尔值和整数类型将被提升到不同的 dtype 以存储 NA。提升在此表中总结：

When introducing NAs into an existing Series or DataFrame via reindex() or some other means, boolean and integer types will be promoted to a different dtype in order to store the NAs. The promotions are summarized in this table:

类型类

Typeclass

用于存储 NA 的提升 dtype

Promotion dtype for storing NAs

floating

无变化

no change

object

无变化

no change

integer

转换为 float64

cast to float64

boolean

转换为 object

cast to object

Support for integer NA

由于 NumPy 缺乏从头内置的高性能 NA 支持，因此首要牺牲便是能够表示整数数组中的 NA。例如：

In the absence of high performance NA support being built into NumPy from the ground up, the primary casualty is the ability to represent NAs in integer arrays. For example:

In [35]: s = pd.Series([1, 2, 3, 4, 5], index=list("abcde"))

In [36]: s
Out[36]:
a    1
b    2
c    3
d    4
e    5
dtype: int64

In [37]: s.dtype
Out[37]: dtype('int64')

In [38]: s2 = s.reindex(["a", "b", "c", "f", "u"])

In [39]: s2
Out[39]:
a    1.0
b    2.0
c    3.0
f    NaN
u    NaN
dtype: float64

In [40]: s2.dtype
Out[40]: dtype('float64')

这一权衡在很大程度上是出于内存和性能方面的考虑，并且也使最终的 Series 持续保持“数字”状态。

This trade-off is made largely for memory and performance reasons, and also so that the resulting Series continues to be “numeric”.

如果您需要表示可能具有缺失值的大整数，请使用 pandas 或 pyarrow 提供的可空整数扩展 dtype 之一

If you need to represent integers with possibly missing values, use one of the nullable-integer extension dtypes provided by pandas or pyarrow

In [41]: s_int = pd.Series([1, 2, 3, 4, 5], index=list("abcde"), dtype=pd.Int64Dtype())

In [42]: s_int
Out[42]:
a    1
b    2
c    3
d    4
e    5
dtype: Int64

In [43]: s_int.dtype
Out[43]: Int64Dtype()

In [44]: s2_int = s_int.reindex(["a", "b", "c", "f", "u"])

In [45]: s2_int
Out[45]:
a       1
b       2
c       3
f    <NA>
u    <NA>
dtype: Int64

In [46]: s2_int.dtype
Out[46]: Int64Dtype()

In [47]: s_int_pa = pd.Series([1, 2, None], dtype="int64[pyarrow]")

In [48]: s_int_pa
Out[48]:
0       1
1       2
2    <NA>
dtype: int64[pyarrow]

有关更多信息，请参阅 Nullable integer data type 和 PyArrow Functionality。

See Nullable integer data type and PyArrow Functionality for more.

Why not make NumPy like R?

许多人认为 NumPy 理应简单地模拟更特定于领域的统计编程语言 R 中现有的 NA 支持。部分原因在于 NumPy 类型层次结构：

Many people have suggested that NumPy should simply emulate the NA support present in the more domain-specific statistical programming language R. Part of the reason is the NumPy type hierarchy:

类型类

Typeclass

Dtypes

numpy.floating

float16, float32, float64, float128

numpy.integer

int8, int16, int32, int64

numpy.unsignedinteger

uint8, uint16, uint32, uint64

numpy.object_

object_

numpy.bool_

bool_

numpy.character

bytes,_ str_

与之相反，R 语言只有极少数的内置数据类型：integer、numeric（浮点数）、character 和 boolean。NA 类型是通过为每个类型保留特殊的比特模式作为缺失值来实现的。虽然使用完整的 NumPy 类型层次结构也可以做到这一点，但这将是一项更大规模的权衡（尤其对于 8 位和 16 位数据类型）和实施工作。

The R language, by contrast, only has a handful of built-in data types: integer, numeric (floating-point), character, and boolean. NA types are implemented by reserving special bit patterns for each type to be used as the missing value. While doing this with the full NumPy type hierarchy would be possible, it would be a more substantial trade-off (especially for the 8- and 16-bit data types) and implementation undertaking.

不过，现在可以通过使用掩码 NumPy 类型（例如 Int64Dtype）或 PyArrow 类型 ( ArrowDtype）来获得 R NA 语义。

However, R NA semantics are now available by using masked NumPy types such as Int64Dtype or PyArrow types (ArrowDtype).

Differences with NumPy

对于 Series 和 DataFrame 对象， var() 按 N-1 归一化以生成 unbiased estimates of the population variance，而 NumPy 的 numpy.var() 按 N 归一化，后者用来衡量样本的差异性。请注意，在 pandas 和 NumPy 中， cov() 均按 N-1 归一化。

For Series and DataFrame objects, var() normalizes by N-1 to produce unbiased estimates of the population variance, while NumPy’s numpy.var() normalizes by N, which measures the variance of the sample. Note that cov() normalizes by N-1 in both pandas and NumPy.

Thread-safety

pandas 并不是 100% 线程安全的。已知的问题与 copy() 方法有关。如果您要大量复制线程之间共享的 DataFrame 对象，我们建议在发生数据复制的线程内保持锁定。

pandas is not 100% thread safe. The known issues relate to the copy() method. If you are doing a lot of copying of DataFrame objects shared among threads, we recommend holding locks inside the threads where the data copying occurs.

请参阅 this link 了解更多信息。

See this link for more information.

Byte-ordering issues

您有时需要处理在字节顺序与您运行 Python 所在的计算机不同的机器上创建的数据。此问题的常见症状是类似于以下内容的错误：

Occasionally you may have to deal with data that were created on a machine with a different byte order than the one on which you are running Python. A common symptom of this issue is an error like:

Traceback
    ...
ValueError: Big-endian buffer not supported on little-endian compiler

为处理此问题，您应在使用类似于以下内容将基础 NumPy 阵列转换为本机系统字节顺序后，才将其传递给 Series 或 DataFrame 构造函数：

To deal with this issue you should convert the underlying NumPy array to the native system byte order before passing it to Series or DataFrame constructors using something similar to the following:

In [49]: x = np.array(list(range(10)), ">i4")  # big endian

In [50]: newx = x.byteswap().view(x.dtype.newbyteorder())  # force native byteorder

In [51]: s = pd.Series(newx)

请参阅 the NumPy documentation on byte order 了解更多详情。

See the NumPy documentation on byte order for more details.