Python pyarrow not found when install with pip

[franze_m]

Hello,
i have installed pyarrow with pip and it was not found by python.
My python-programm was running in Pycharm with installed pyarrow well.

I thing that a path was not correct , but i cant find it.

Statement with error:

df = pd.read_csv(
CSV_FILENAME,
engine="pyarrow",
header=0,
sep=";",
usecols=["DateTime", "Verdichter_laeuft_seit", "Temp_EQ_Eintritt"],
parse_dates=["DateTime"],
dtype={"Verdichter_laeuft_seit": "float", "Temp_EQ_Eintritt": "float"}
)

Traceback (most recent call last):
File "/home/user/./plotwp-test.py", line 54, in <module>
main()
File "/home/user/./plotwp-test.py", line 28, in main
df = pd.read_csv(
File "/usr/lib/python3/dist-packages/pandas/io/parsers.py", line 688, in read_csv
return _read(filepath_or_buffer, kwds)
File "/usr/lib/python3/dist-packages/pandas/io/parsers.py", line 454, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "/usr/lib/python3/dist-packages/pandas/io/parsers.py", line 948, in __init__
self._make_engine(self.engine)
File "/usr/lib/python3/dist-packages/pandas/io/parsers.py", line 1187, in _make_engine
raise ValueError(
ValueError: Unknown engine: pyarrow (valid options are "c", "python", or "python-fwf")
user@mx:~
$ pip install pyarrow
Defaulting to user installation because normal site-packages is not writeable
Collecting pyarrow
Using cached pyarrow-14.0.2-cp39-cp39-manylinux_2_28_x86_64.whl (38.0 MB)
Requirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.9/dist-packages (from pyarrow) (1.26.2)
Installing collected packages: pyarrow
Successfully installed pyarrow-14.0.2

whatis wrong ?

[nopes]

You should start to use venvs, maybe pycharm already did that for you and you have to start the venv if you want to launch your application without pycharm

[franze_m]

I would use pyarrow not only for one project. Is ist possible to install it global ?

[nopes]

Yes this is possible, running pip outside an venv and as root should do that.

You might want to check if you have installed python2 on you system (fe debian still do this [at least last time I checked]) in this case you might have to use pip3 instead of pip (fe at debian systems)

[franze_m]

It does not help!

$ sudo pip3 install pyarrow
Collecting pyarrow
Downloading pyarrow-14.0.2-cp39-cp39-manylinux_2_28_x86_64.whl (38.0 MB)
|████████████████████████████████| 38.0 MB 6.3 MB/s
Requirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.9/dist-packages (from pyarrow) (1.26.2)
Installing collected packages: pyarrow
Successfully installed pyarrow-14.0.2
user@mx:~
$ ./plotwp-test.py
Traceback (most recent call last):
File "/home/user/./plotwp-test.py", line 54, in <module>
main()
File "/home/user/./plotwp-test.py", line 28, in main
df = pd.read_csv(
File "/usr/lib/python3/dist-packages/pandas/io/parsers.py", line 688, in read_csv
return _read(filepath_or_buffer, kwds)
File "/usr/lib/python3/dist-packages/pandas/io/parsers.py", line 454, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "/usr/lib/python3/dist-packages/pandas/io/parsers.py", line 948, in __init__
self._make_engine(self.engine)
File "/usr/lib/python3/dist-packages/pandas/io/parsers.py", line 1187, in _make_engine
raise ValueError(
ValueError: Unknown engine: pyarrow (valid options are "c", "python", or "python-fwf")

[Huhn Hur Tu]

please check if your pyarrow package is in the $PATH by

find /usr -name "pyarrow*"

[Huhn Hur Tu]

Mixinng packages with different packagemanagers like pip/apt is a bad idea, better use a venv for packages not provides by your system

Debian has good warning when running pip as root

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead

[franze_m]

I have inserted a pd.show_versions() form more Informations and it looks good for me. The python-program is running well when i comment out pyarrow, but i have big files and make
plots from it and the performace with pyarrow is much better. I will got it running without any special enviroment for a project.

$ find /usr -name "pyarrow*"
/usr/local/lib/python3.9/dist-packages/pyarrow
/usr/local/lib/python3.9/dist-packages/pyarrow/src/arrow/python/pyarrow_api.h
/usr/local/lib/python3.9/dist-packages/pyarrow/src/arrow/python/pyarrow.cc
/usr/local/lib/python3.9/dist-packages/pyarrow/src/arrow/python/pyarrow.h
/usr/local/lib/python3.9/dist-packages/pyarrow/src/arrow/python/pyarrow_lib.h
/usr/local/lib/python3.9/dist-packages/pyarrow/tests/pyarrow_cython_example.pyx
/usr/local/lib/python3.9/dist-packages/pyarrow/include/arrow/python/pyarrow_api.h
/usr/local/lib/python3.9/dist-packages/pyarrow/include/arrow/python/pyarrow.h
/usr/local/lib/python3.9/dist-packages/pyarrow/include/arrow/python/pyarrow_lib.h
/usr/local/lib/python3.9/dist-packages/pyarrow-14.0.2.dist-info

$ ./plotwp-test.py

INSTALLED VERSIONS
------------------
commit : b5958ee1999e9aead1938c0bba2b674378807b3d
python : 3.9.2.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.0-26-amd64
Version : #1 SMP Debian 5.10.197-1 (2023-09-29)
machine : x86_64
processor :
byteorder : little
LC_ALL : de_DE.UTF-8
LANG : de_DE.UTF-8
LOCALE : de_DE.UTF-8

pandas : 1.1.5
numpy : 1.26.2
pytz : 2021.1
dateutil : 2.8.1
pip : 23.0.1
setuptools : 52.0.0
Cython : None
pytest : 6.0.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.3
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.4
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 14.0.2
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

INSTALLED VERSIONS
------------------
commit : b5958ee1999e9aead1938c0bba2b674378807b3d
python : 3.9.2.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.0-26-amd64
Version : #1 SMP Debian 5.10.197-1 (2023-09-29)
machine : x86_64
processor :
byteorder : little
LC_ALL : de_DE.UTF-8
LANG : de_DE.UTF-8
LOCALE : de_DE.UTF-8

pandas : 1.1.5
numpy : 1.26.2
pytz : 2021.1
dateutil : 2.8.1
pip : 23.0.1
setuptools : 52.0.0
Cython : None
pytest : 6.0.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.3
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.4
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 14.0.2
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

Source-Statement with Error:

df = pd.read_csv(
CSV_FILENAME,
#engine="pyarrow",
header=0,
sep=";",
usecols=["DateTime", "Verdichter_laeuft_seit", "Temp_EQ_Eintritt"],
parse_dates=["DateTime"],
dtype={"Verdichter_laeuft_seit": "float", "Temp_EQ_Eintritt": "float"}
)

[nopes]

I suspect that the system is already in a problematic state (precisely because no venvs were used). A quick search suggests that there are multiple installations of PyArrow that are interfering sa https://github.com/pandas-dev/pandas/issues/24976

To verify this I would either create a clean venv or a fresh system and check if I still run into that issue or not