Pandas 3.0 downgrade

[swinter1]

Issue

Upgrade to pandas 3.0 caused existing code where pandas uses string to crash (specifically found through pd.json_normalize)

Details and reproduction (Claude)

"""Minimal reproduction of segfault: torch (2.4.1+cu124) + pyarrow (24.0.0) import order.

Root cause: When torch is imported BEFORE pyarrow (or pandas 3.0+ which uses pyarrow strings
by default), creating a pyarrow string array segfaults. This manifests as a crash on any pandas
operation that infers string dtype (e.g. pd.Series(['a']), pd.DataFrame({'col': ['a']}),
pd.json_normalize with string values).

Versions: torch==2.4.1+cu124, pyarrow==24.0.0, pandas==3.0.2, numpy==1.26.4, python==3.11.9
"""

# %% Demonstrate the root cause: torch + pyarrow import order
import pyarrow as pa
import torch

print(f"torch {torch.__version__}, pyarrow {pa.__version__}")

# This segfaults because torch was imported first
print("Creating pyarrow string array (will segfault)...")
arr = pa.array(["a", "b"])  # SEGFAULT HERE
print(f"OK: {arr}")

# %% The same issue surfaces through pandas 3.0+ (which uses pyarrow strings by default)
# import torch
# import pandas as pd
# print(f"pandas {pd.__version__}")
# pd.Series(["a", "b"])  # SEGFAULT - pandas 3.0 infers pyarrow string dtype
#
# %% Workarounds:
# 1. Import pyarrow (or pandas) BEFORE torch
# 2. Pin pandas<3.0 (avoids pyarrow string backend)
# 3. pd.set_option("future.infer_string", False) before any string operations
# 4. Upgrade torch to a version compatible with pyarrow 24