Python Manual

I configured a new system, spark 2.3.0, python 3.6.0, dataframe read and other operations working a… Read more Rdd Collect Issue

I am running a PySpark job that calls udfs. I know udfs are bad with memory and slow due to seriali… Read more Error Pythonudfrunner: Python Worker Exited Unexpectedly (crashed)

When I try to execute this command line at pyspark arquivo = sc.textFile('dataset_analise_senti… Read more Sparkexception: Python Worker Failed To Connect Back When Execute Spark Action

I'm using spark version 2.0.1 & python 2.7. I'm running following code # This will retu… Read more Typeerror: 'groupeddata' Object Is Not Iterable In Pyspark

There's many posts about this issue, but none have answered my question. I'm running into O… Read more Pyspark Outofmemoryerrors When Performing Many Dataframe Joins

I have a pyspark.sql.dataframe.DataFrame like the following df.show() +--------------------+----+--… Read more Pyspark: Count On Pyspark.sql.dataframe.dataframe Takes Long Time

I tried taking a schema as a common schema by df.schema() and load all the CSV files to it .But fai… Read more Spark - Merge / Union Dataframe With Different Schema (column Names And Sequence) To A Dataframe With Master Common Schema

I am trying to distribute Keras training on a cluster and use Elephas for that. But, when running t… Read more Elephas Not Loaded In Pyspark: No Module Named Elephas.spark_model