3

I created an exe file for my pySpark script using pyinstaller. But when i run the exe, I am facing "No module named 'py4j.java_collections'" error.

pyinstaller version - 3.5

python version - Python 3.7.1

Spark version - 2.4.3

OS - Windows 10

I am creating the exe using pyinstaller -F myscript.py command to create the exe. I looked up on other threads and added below path variables in Windows Environment Variables

PYTHONPATH = $SPARK_HOME/python/:$PYTHONPATH;$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH

SPARK_HOME = C:\Users\\Desktop\spark\spark-2.4.3-bin-hadoop2.7

My script is having wx components as well for input and message dialog box.

import glob
from pyspark.sql import SparkSession

#Spark Session
spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .config("spark.some.config.option", "some-value") \
    .config("spark.sql.session.timeZone", "UTC") \
    .config("spark.sql.execution.pandas.respectSessionTimeZone", "False") \
    .getOrCreate()

try:
    app = wx.App()
    app.MainLoop()

    fileLocationBox = wx.TextEntryDialog(None, 'Enter File Directory :', 'Convert To CSV', 'Enter Parquet File Location')

    if fileLocationBox.ShowModal() == wx.ID_OK:
        fileLoc = fileLocationBox.GetValue()

    fileLocationBox.Destroy()

    files = [f for f in glob.glob(fileLoc + "**/*.parquet", recursive=True)]

    t = spark.read.load(files)

    t.coalesce(1).write.csv(fileLoc+'\\OutputCSV', header=True)

    dlg = wx.MessageDialog(None, "Output CSV generated", "Output", wx.OK)
    if dlg.ShowModal() == wx.ID_OK:
        dlg.Destroy()

except Exception as e:
    print ("ERROR:")
    print (type(e))     # the exception instance
    print (e.args)

I have run out of options. Please help on this issue.

KS17
  • 71
  • 3
  • May this would help ypu to get rid from this https://stackoverflow.com/questions/57713994/modulenotfounderror-no-module-named-crypto-math – Mathan Aug 30 '19 at 10:47
  • @KS17 I also have similar error. did you find any workaround ? thank u – user1 Jan 26 '20 at 20:18

0 Answers0