Skip to content

pyspark is not required when install koalas #2221

Description

@wubing101

koalas is a great package.

when I install the package, all requirements are as below:
pip install koalas==1.8.2 Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Collecting koalas==1.8.2 Using cached https://pypi.tuna.tsinghua.edu.cn/packages/28/9a/d69cf12ea62116873b427e5843be8ae8431b18f2a0714d6f4eec3ee4cda6/koalas-1.8.2-py3-none-any.whl (390 kB) Requirement already satisfied: numpy>=1.14 in /Users/celential-bing/.pyenv/versions/3.8.12/envs/time_machine/lib/python3.8/site-packages (from koalas==1.8.2) (1.21.5) Requirement already satisfied: pandas>=0.23.2 in /Users/celential-bing/.pyenv/versions/3.8.12/envs/time_machine/lib/python3.8/site-packages (from koalas==1.8.2) (1.3.5) Requirement already satisfied: pyarrow>=0.10 in /Users/celential-bing/.pyenv/versions/3.8.12/envs/time_machine/lib/python3.8/site-packages (from koalas==1.8.2) (7.0.0) Requirement already satisfied: pytz>=2017.3 in /Users/celential-bing/.pyenv/versions/3.8.12/envs/time_machine/lib/python3.8/site-packages (from pandas>=0.23.2->koalas==1.8.2) (2021.1) Requirement already satisfied: python-dateutil>=2.7.3 in /Users/celential-bing/.pyenv/versions/3.8.12/envs/time_machine/lib/python3.8/site-packages (from pandas>=0.23.2->koalas==1.8.2) (2.8.2) Requirement already satisfied: six>=1.5 in /Users/celential-bing/.pyenv/versions/3.8.12/envs/time_machine/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas>=0.23.2->koalas==1.8.2) (1.16.0) Installing collected packages: koalas Successfully installed koalas-1.8.2

but it also needs pyspark, for example when I start a service:
`ImportError: Unable to import pyspark - consider doing a pip install with [spark] extra to install pyspark with pip
Traceback (most recent call last):
File "/Users/celential-bing/.pyenv/versions/time_machine/lib/python3.8/site-packages/databricks/koalas/init.py", line 49, in assert_pyspark_version
import pyspark
ModuleNotFoundError: No module named 'pyspark'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/celential-bing/time-machine/timemachine/app.py", line 1, in
from timemachine import app, dapp
File "/Users/celential-bing/time-machine/timemachine/init.py", line 301, in
app, dapp, schema = create_app()
File "/Users/celential-bing/time-machine/timemachine/init.py", line 57, in create_app
raise e
File "/Users/celential-bing/time-machine/timemachine/init.py", line 54, in create_app
return TimeMachineInitializer(app).init_app()
File "/Users/celential-bing/time-machine/timemachine/init.py", line 212, in init_app
self.init_app_in_ctx()
File "/Users/celential-bing/time-machine/timemachine/init.py", line 176, in init_app_in_ctx
self.init_views()
File "/Users/celential-bing/time-machine/timemachine/init.py", line 69, in init_views
from timemachine.views.base import (
File "/Users/celential-bing/time-machine/timemachine/views/base.py", line 13, in
from timemachine.models.base import Module, Lambda
File "/Users/celential-bing/time-machine/timemachine/models/base.py", line 14, in
from timemachine.engines import current_engine, DF
File "/Users/celential-bing/time-machine/timemachine/engines/init.py", line 9, in
from databricks.koalas import DataFrame as SparkDataFrame
File "/Users/celential-bing/.pyenv/versions/time_machine/lib/python3.8/site-packages/databricks/koalas/init.py", line 72, in
assert_pyspark_version()
File "/Users/celential-bing/.pyenv/versions/time_machine/lib/python3.8/site-packages/databricks/koalas/init.py", line 51, in assert_pyspark_version
raise ImportError(
ImportError: Unable to import pyspark - consider doing a pip install with [spark] extra to install pyspark with pip`

so I suggest adding the pyspark in the requirement.txt. I didn't find the file, so pull an issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions