Python Manual

I configured a new system, spark 2.3.0, python 3.6.0, dataframe read and other operations working a… Read more Rdd Collect Issue

I am running a PySpark job that calls udfs. I know udfs are bad with memory and slow due to seriali… Read more Error Pythonudfrunner: Python Worker Exited Unexpectedly (crashed)

When I try to execute this command line at pyspark arquivo = sc.textFile('dataset_analise_senti… Read more Sparkexception: Python Worker Failed To Connect Back When Execute Spark Action

I'm using spark version 2.0.1 & python 2.7. I'm running following code # This will retu… Read more Typeerror: 'groupeddata' Object Is Not Iterable In Pyspark

There's many posts about this issue, but none have answered my question. I'm running into O… Read more Pyspark Outofmemoryerrors When Performing Many Dataframe Joins

I have a pyspark.sql.dataframe.DataFrame like the following df.show() +--------------------+----+--… Read more Pyspark: Count On Pyspark.sql.dataframe.dataframe Takes Long Time