Supported Python Packages
If the following list does not include a package that you need to run a job, you can specify it in the Additional Packages box on the Job Settings page when you create a job. Use the <package_name>==<version_number> syntax.
Note
LiveRamp advises Analytics Environment customers to migrate their code to a new Dataproc 2.0 or 2.1 cluster and test it if they have not already done so. Dataproc 1.4 is no longer supported by Google.
The following list includes the packages that are already available in the Dataproc 2.1 clusters.
ansiwrap==0.8.4
asttokens==2.4.1
async-generator==1.10
async-lru==2.0.4
binaryornot==0.4.4
category-encoders==2.6.3
certifi==2023.7.22
certipy==0.1.3
click-plugins==1.1.1
conda==22.9.0
contourpy==1.2.1
dash==2.18.1
dash-bootstrap-components==1.6.0
dash-core-components==2.0.0
dash-daq==0.5.0
dash-html-components==2.0.0
dash-renderer==1.8.3
dash-table==5.0.0
db-dtypes==1.3.0
descartes==1.1.0
executing==2.1.0
Flask==2.2.5
fqdn==1.5.1
google==3.0.0
google-api-core==2.17.1
google-api-python-client==2.119.0
google-auth==2.28.1
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.0
google-cloud==0.34.0
google-cloud-appengine-logging==1.3.1
google-cloud-audit-log==0.2.5
google-cloud-bigquery==3.17.2
google-cloud-bigquery-storage==2.15.0
google-cloud-bigtable==2.11.3
google-cloud-container==2.11.2
google-cloud-core==2.4.1
google-cloud-datacatalog==3.9.3
google-cloud-dataproc==5.0.3
google-cloud-datastore==2.8.3
google-cloud-language==2.5.2
google-cloud-logging==3.2.5
google-cloud-monitoring==2.11.3
google-cloud-pubsub==2.13.12
google-cloud-redis==2.9.3
google-cloud-spanner==3.19.0
google-cloud-speech==2.15.1
google-cloud-storage==2.14.0
google-cloud-texttospeech==2.12.3
google-cloud-translate==3.8.4
google-cloud-vision==3.1.4
google-crc32c==1.5.0
google-resumable-media==2.7.0
googleapis-common-protos==1.62.0
grpc-google-iam-v1==0.12.6
grpcio==1.57.0
grpcio-status==1.57.0
htmlmin==0.1.12
inequality==1.0.0
ipykernel==6.29.2
ipython==8.22.1
ipython-genutils==0.2.0
isoduration==20.11.0
itsdangerous==2.2.0
jsonpointer==2.0
jupyter-events==0.10.0
jupyter-lsp==2.2.5
jupyter_client==8.6.0
jupyter_core==5.7.1
jupyter_server==2.14.2
jupyter_server_terminals==0.5.3
jupyterlab==4.0.4
koalas==0.22.0
lime==0.2.0.1
llvmlite==0.38.1
matplotlib==3.8.4
mccabe==0.6.1
missingno==0.4.2
munkres==1.1.4
notebook==7.0.0
numpy==1.26.4
oauth2client==4.1.3
opencv-python==4.6.0
overrides==7.7.0
packaging==24.1
pandas==2.2.1
plotly==5.19.0
ply==3.11
prompt_toolkit==3.0.47
proto-plus==1.22.3
protobuf==4.21.12
pure_eval==0.2.3
py4j==0.10.9.7
pyarrow==17.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.7
pycurl==7.45.1
pydata-google-auth==1.8.2
PyQt5==5.15.7
PyQt5-sip==12.11.0
PyQtWebEngine==5.15.4
pyspark==3.5.1
requests==2.31.0
retrying==1.3.4
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
scikit-learn==1.1.0
scipy==1.9.3
shap==0.44.1
shapash==2.4.3
slicer==0.0.7
snuggs==1.4.7
spvcm==0.3.0
stack-data==0.6.3
text-unidecode==1.3
textwrap3==0.9.2
toree==0.5.0
tornado==6.4.1
traitlets==5.14.3
tzdata==2024.1
uri-template==1.3.0
urllib3==2.2.1
wcwidth==0.2.13
webcolors==24.8.0
webencodings==0.5.1
websocket-client==1.8.0
Werkzeug==3.0.4
xgboost==1.5.0
zstandard==0.19.0
The following list includes the packages that are already available in the Dataproc 2.0 clusters.
absl-py==2.1.0
alabaster==0.7.12
ansiwrap==0.8.4
arviz==0.12.1
astunparse==1.6.3
async-generator==1.10
bcolz==1.2.1
binaryornot==0.4.4
blinker==1.4
cachetools==4.2.4
certifi==2024.2.2
certipy==0.1.3
cftime==1.6.3
click==7.1.2
click-plugins==1.1.1
cloudpickle==3.0.0
conda==4.9.2
contourpy==1.1.1
cycler==0.12.1
cytoolz==0.11.2
descartes==1.1.0
dill==0.3.8
fastprogress==1.0.3
fonttools==4.49.0
future==0.18.3
gast==0.3.3
GDAL==3.5.0
google-api-core==1.31.6
google-auth==1.35.0
google-auth-oauthlib==0.4.6
google-cloud-bigquery==2.34.4
google-cloud-bigquery-storage==2.1.0
google-cloud-bigtable==1.6.1
google-cloud-container==2.3.1
google-cloud-core==2.4.1
google-cloud-datacatalog==3.0.0
google-cloud-dataproc==2.2.0
google-cloud-datastore==2.1.6
google-cloud-language==2.0.0
google-cloud-logging==2.1.1
google-cloud-monitoring==2.0.1
google-cloud-pubsub==2.2.0
google-cloud-redis==2.0.0
google-cloud-spanner==2.1.1
google-cloud-speech==2.0.1
google-cloud-storage==2.7.0
google-cloud-texttospeech==2.2.0
google-cloud-translate==3.0.2
google-cloud-vision==2.0.0
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.7.0
googleapis-common-protos==1.62.0
graphframes==0.6
grpc-google-iam-v1==0.12.4
grpcio==1.60.1
grpcio-status==1.48.2
h5py==2.10.0
HeapDict==1.0.1
htmlmin==0.1.12
hyperopt==0.2.7
importlib-metadata==4.8.3
importlib-resources==5.4.0
inequality==1.0.0
intervaltree==3.0.2
ipython-genutils==0.2.0
joblib==1.3.2
jupyter-contrib-core==0.3.3
jupyterlab==3.2.5
keras==2.6.0
Keras-Preprocessing==1.1.2
kiwisolver==1.4.5
libcst==0.4.5
llvmlite==0.36.0
MarkupSafe==2.1.5
matplotlib==3.7.5
mccabe==0.6.1
missingno==0.4.2
munch==2.5.0
netCDF4==1.6.5
networkx==3.1
nose==1.3.7
numpy==1.22.1
oauth2client==4.1.3
oauthlib==3.2.2
opt-einsum==3.3.0
pamela==1.0.0
pandas==1.5.3
parso==0.7.0
patsy==0.5.6
pexpect==4.8.0
pillow==10.2.0
poyo==0.5.0
proto-plus==1.23.0
protobuf==3.20.3
py4j==0.10.9.5
pyarrow==2.0.0
pyasn1==0.4.8
pyasn1-modules==0.3.0
pycurl==7.45.1
pyflakes==2.2.0
pymc3==3.11.4
PyQt5==5.12.3
PyQt5_sip==4.19.18
PyQtChart==5.12
PyQtWebEngine==5.12.1
pyspark==3.2.2
pytest==7.1.2
python-dateutil==2.8.2
pytz==2024.1
requests==2.27.1
rsa==4.9
sasl==0.3.1
scikit-learn==1.2.2
scipy==1.4.1
semver==3.0.2
snuggs==1.4.7
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
spvcm==0.3.0
spylon==0.3.0
spylon-kernel==0.4.1
tensorboard==2.2.2
tensorboard-plugin-wit==1.8.1
tensorflow==2.2.0
tensorflow-estimator==2.2.0
termcolor==2.4.0
text-unidecode==1.3
textwrap3==0.9.2
Theano-PyMC==1.1.2
threadpoolctl==3.3.0
tqdm==4.66.2
traitlets==5.14.1
typing-extensions==3.7.4.3
typing-inspect==0.7.1
tzdata==2022.1
uritemplate==3.0.1
urllib3==1.26.18
webencodings==0.5.1
Werkzeug==3.0.3
wrapt==1.16.0
xarray==2023.1.0
xarray-einstats==0.2.2
Note
LiveRamp advises Analytics Environment customers to migrate their code to a new Dataproc 2.0 or 2.1 cluster and test it, if they have not already done so. Dataproc 1.4 is no longer supported by Google.
The following list includes the packages that are available in the Dataproc 1.4 clusters.
absl-py==0.15.0
aiohttp==3.8.4
aiosignal==1.2.0
alembic==1.7.7
ansi2html==1.8.0
apyori==1.1.2
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arviz==0.11.4
asn1crypto==0.24.0
astunparse==1.6.3
async-generator==1.10
async-timeout==4.0.2
asynctest==0.13.0
attrs==22.2.0
autograd==1.6.2
backcall==0.2.0
bleach==4.1.0
blis==0.7.9
bokeh==2.3.3
Boruta==0.3
boto==2.49.0
branca==0.5.0
cached-property==1.5.2
cachetools==4.2.4
catalogue==2.0.8
causalml==0.12.3
certifi==2018.4.16
cffi==1.11.5
cftime==1.6.0
ChannelAttribution==2.1.4
chardet==3.0.4
charset-normalizer==2.0.12
clang==5.0
click==8.0.4
cloudpickle==2.2.1
cmdstanpy==0.9.68
colour==0.1.5
conda==4.5.4
confection==0.1.0
contextvars==2.4
convertdate==2.3.2
cryptography==2.2.2
cycler==0.11.0
cymem==2.0.7
Cython==0.29.35
dash==2.10.2
dash-auth==2.0.0
dash-bootstrap-components==0.13.1
dash-core-components==2.0.0
dash-html-components==2.0.0
dash-table==5.0.0
dask==2021.3.0
databricks-cli==0.17.7
dataclasses==0.8
DateTime==4.9
dcborow-mmlspark==0.14
dev1 decorator==4.4.2
defusedxml==0.7.1
dill==0.3.4
distributed==2021.3.0
distro==1.8.0
dnspython==2.2.1
docker==5.0.3
docopt==0.6.2
dowhy==0.8
dtreeviz==1.3.7
entrypoints==0.4
ephem==4.1.4
evidently==0.2.1
explainerdashboard==0.3.8.2
extratools==0.8.2.1
fastprogress==1.0.3
findspark==2.0.1
Flask==2.0.3
flask-simplelogin==0.1.1
Flask-WTF==0.15.1
flatbuffers==1.12
folium==0.13.0
frozenlist==1.2.0
fsspec==2022.1.0
future==0.18.3
fuzzywuzzy==0.18.0
gast==0.4.0
gcsfs==2022.1.0
gensim==4.2.0
gitdb==4.0.9
GitPython==3.1.18
google-api-core==2.8.2
google-auth==1.35.0
google-auth-oauthlib==0.4.6
google-cloud-bigquery==2.34.4
google-cloud-bigquery-datatransfer==3.6.2
google-cloud-bigquery-storage==2.13.2
google-cloud-core==2.3.1
google-cloud-dataproc==4.0.3
google-cloud-pubsub==2.13.0
google-cloud-storage==2.0.0
google-compute-engine==2.8.13
google-crc32c==1.3.0
google-pasta==0.2.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.3
gower==0.1.2
graphframes==0.6
graphviz==0.19.1
graphx==0.0.4
greenlet==2.0.2
grpc-google-iam-v1==0.12.4
grpcio==1.48.2
grpcio-status==1.48.2
gunicorn==20.1.0
h2o==3.40.0.4
h5py==3.1.0
handyspark==0.2.2a1
hdbscan==0.8.27
HeapDict==1.0.1
hijri-converter==2.2.4
holidays==0.13
html5lib==1.1
htmlmin==0.1.12
hyperopt==0.2.7
ibis-framework==1.3.0
idna==2.6
idna-ssl==1.1.0
ImageHash==4.3.1
imbalanced-learn==0.8.1
immutables==0.19
importlib-metadata==4.8.3
importlib-resources==5.4.0
iniconfig==1.1.1
ipykernel==5.5.6
ipython==7.16.3
ipython-genutils==0.2.0
ipywidgets==7.7.5
itsdangerous==2.0.1
jedi==0.17.2
Jinja2==3.0.3
joblib==1.0.1
jsonpickle==2.2.0
jsonschema==3.2.0
jupyter-client==7.1.2
jupyter-core==4.9.2
jupyter-dash==0.4.2
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.1.4
keras==2.6.0
Keras-Preprocessing==1.1.2
kiwisolver==1.3.1
kmodes==0.12.2
koalas==1.8.2
korean-lunar-calendar==0.3.1
langcodes==3.3.0
Levenshtein==0.21.1
Lifetimes==0.11.3
lightgbm==3.3.5
llvmlite==0.36.0
locket==1.0.0
LunarCalendar==0.0.9
lxml==4.9.2
Mako==1.1.6
Markdown==3.3.7
MarkupSafe==2.0.1
matplotlib==3.3.4
missingno==0.5.2
mistune==0.8.4
mlflow==1.23.1
mlxtend==0.19.0
modin==0.8.3.
post4 mpmath==1.3.0
msgpack==1.0.5
multidict==5.2.0
multimethod==1.5
multipledispatch==1.0.0
murmurhash==1.0.9
mypy-extensions==1.0.0
nbclient==0.5.9
nbconvert==6.0.7
nbformat==5.1.3
nest-asyncio==1.5.6
netCDF4==1.6.1
networkx==2.5.1
nltk==3.6.7
node2vec==0.4.3
nose==1.3.7
notebook==6.4.10
numba==0.53.1
numpy==1.19.5
oauthlib==3.2.2
openfoodfacts==0.1.3
opt-einsum==3.3.0
oyaml==1.0
packaging==21.3
pandas==1.1.5
pandas-gbq==0.14.1
pandas-profiling==3.1.0
pandasql==0.7.3
pandera==0.6.5
pandocfilters==1.5.0
parso==0.7.1
partd==1.2.0
pathy==0.10.2
patsy==0.5.3
pexpect==4.8.0
pgeocode==0.3.0
phik==0.12.0
pickleshare==0.7.5
Pillow==8.4.0
plotly==5.15.0
pluggy==1.0.0
pmdarima==1.8.4
prefixspan==0.5.2
preshed==3.0.8
progressbar2==3.55.0
prometheus-client==0.17.0
prometheus-flask-exporter==0.22.4
prompt-toolkit==3.0.36
prophet==1.0.1
proto-plus==1.22.3
protobuf==3.19.6
psutil==5.9.5
ptyprocess==0.7.0
py==1.11.0
py4j==0.10.9.5
pyarrow==6.0.1
pyasn1==0.5.0
pyasn1-modules==0.3.0
pyclustering==0.10.1.2
pycosat==0.6.3
pycparser==2.18
pycryptodome==3.18.0
pycspade==0.6.6
pydantic==1.9.2
pydata-google-auth==1.8.0
pydot==1.4.2 pydotplus==2.0.2
pygam==0.8.0
Pygments==2.14.0
PyJWT==2.4.0
pymc3==3.10.0
PyMeeus==0.5.12
pyOpenSSL==18.0.0
pyparsing==3.0.7
pyro-api==0.1.2
pyro-ppl==1.8.0
pyrsistent==0.18.0
PySocks==1.6.8
pyspark==3.2.4
pyspark-pandas==0.0.7
pystan==2.19.1.1
pytest==7.0.1
python-dateutil==2.8.2
python-http-client==3.3.7
python-Levenshtein==0.21.1
python-utils==3.5.2
pytz==2023.3
pyvis==0.3.2
PyWavelets==1.1.1
PyYAML==5.4.1
pyzmq==25.1.0
querystring-parser==1.2.4
rapidfuzz==2.11.1
regex==2023.6.3
requests==2.27.1
requests-oauthlib==1.3.1
retrying==1.3.4
rfpimp==1.3.7
rsa==4.9
ruamel_yaml==0.15.37
scikit-learn==0.24.2
scikit-learn-extra==0.3.0
scipy==1.5.4
seaborn==0.11.2
Send2Trash==1.8.2
sendgrid==6.10.0
setuptools-git==1.2
shap==0.41.0
six==1.15.0
sklearn==0.0
slicer==0.0.7
smart-open==6.3.0
smmap==5.0.0
sortedcontainers==2.4.0
spacy==3.5.4
spacy-legacy==3.0.12
spacy-loggers==1.0.4
SQLAlchemy==1.4.48
sqlparse==0.4.4
srsly==2.4.6
starkbank-ecdsa==2.2.0
statsmodels==0.12.2
sweetviz==2.1.4
sympy==1.9
tableaudocumentapi==0.11
tableauserverclient==0.18.0
tabulate==0.8.10
tangled-up-in-unicode==0.1.0
tblib==1.7.0
tenacity==8.2.2
tensorboard==2.6.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.6.2
tensorflow-estimator==2.6.0
termcolor==1.1.0
terminado==0.12.1
testpath==0.6.0
Theano-PyMC==1.0.11
thinc==8.1.10
threadpoolctl==3.1.0
tomli==1.2.3
toolz==0.12.0
torch==1.10.2
tornado==6.1
tqdm==4.64.1
traitlets==4.3.3
typer==0.9.0
typing-extensions==3.7.4.3
typing-inspect==0.9.0
ujson==4.3.0
urllib3==1.26.16
visions==0.7.4
waitress==2.0.0
wasabi==1.1.2
wcwidth==0.2.6
webencodings==0.5.1
websocket-client==1.3.1
Werkzeug==2.0.3
widgetsnbextension==3.6.4
wrapt==1.12.1
WTForms==3.0.0
xarray==0.16.2
xgboost==1.5.2
xlrd==2.0.1
XlsxWriter==3.1.2
yarl==1.7.2
zict==2.1.0
zipp==3.6.0
zope.interface==5.5.2