Allow for the pyarrow type system for _pandas_like backends

[tswast]

Context: I'd like to make a "bigframes" (BigQuery DataFrames) Narwhals backend (work-in-progress here: https://github.com/tswast/narwhals/commits/bigframes/)

While implementing this, the biggest roadblock I hit is with regards to dtypes. In general, BigFrames defaults to pyarrow as the default dtypes for multiple reasons.

1. It avoids data loss. BigQuery treats NULL and NaN as separate values, whereas this isn't possible with the numpy-based dtypes.
2. It seems to be the direction of pandas (though this is hotly debated: Moving to PyArrow dtypes by default pandas-dev/pandas#61618)

I'm curious if it would be possible to make _pandas_like backends that use the pyarrow type system? Alternatively, maybe there's a way for a _pandas_like backend to supply its own mapping from narwhals dtypes to pandas-y dtypes?

[dangotbanned]

Thanks for opening this discussion, @tswast (apologies nobody has seen it yet)

It seems like you're very clued in on the woes of pandas' various type systems!

1. It avoids data loss. BigQuery treats NULL and NaN as separate values, whereas this isn't possible with the numpy-based dtypes.

Yep, we're kinda dealing with this at the moment - but it is somewhat of a moving target (#3160 (comment))

I'm curious if it would be possible to make _pandas_like backends that use the pyarrow type system?

These are all the pandas APIs I know of that may be helpful:

import pandas as pd

# NOTE: `pandas>=3` only
pd.options.mode.nan_is_na = False

# NOTE: Not in `pandas-stubs`
pd.options.future.infer_string = True
pd.options.future.no_silent_downcasting = True

pd.options.mode.string_storage = "pyarrow"

pd.DataFrame().convert_dtypes(dtype_backend="pyarrow")
pd.Series().convert_dtypes(dtype_backend="pyarrow")

Publicly we have narwhals.maybe_convert_dtypes, which wraps the last two methods.

Internally, this search will point you to where you can specify dtype_backend="pyarrow".

One last thing, while testing bigframes you might find it helpful to compare against these two 🙂

pytest --constructors='pandas[pyarrow],modin[pyarrow]'