python3: can't open file 'notebook': errno 2 no such file or directory

I am facing issues with running spark in Jupyter notebook. I have following variables set in /.bashrc

export SPARK_HOME=~/Development/Spark/spark-2.4.4-bin-hadoop2.7
export PATH=$SPARK_HOME/bin:$PATH
export PATH=~/anaconda3/bin:$PATH
export PATH=$PATH:~/.local/bin/
export PYTHONPATH=$SPARK_HOME/python:$PYTHONPATH
export PYSPARK_DRIVER_PYTON=ipython
export PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark
export PYSPARK_PYTHON=python3

When I type pyspark, I am getting the error

python3: can't open file 'notebook': errno 2 no such file or directory

For me 'jupyter notebook' is opening up a notebook in browser.

How can I fix this?

1 Answer

TL;DR Check that the environment variables PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON are not set in spark-env.sh.

I ran into a similar issue after setting up Spark using a book before taking a Pyspark Udemy course.

After scouring StackOverflow and troubleshooting, it turned out that I had defined the environment variables PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON in both my ~/.bashrc and spark-env.sh, as shown below.

~/.bashrc

export PYSPARK_PYTHON=~/anaconda3/bin/python
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook

spark-env.sh

PYSPARK_PYTHON=python3
PYSPARK_DRIVER_PYTHON=python3

The solution for me was to remove the lines in the spark-env.sh. I was then able to start jupyter notebook by running the pyspark command and utilize pyspark within the notebook. It is expected that the jupyter notebook command should open jupyter notebooks in a browser.

Hope this helps!