Pyspark Outofmemoryerrors When Performing Many Dataframe Joins

There's many posts about this issue, but none have answered my question. I'm running into OutOfMemoryErrors in PySpark while attempting to join many different dataframes together.

Solution 1:

I faced similar issue and it worked with: Spark Submit:

spark-submit --driver-memory3g\
            --executor-memory14g\
            *.py

Code:

sc = SparkContext().getOrCreate()