Python Tutorial for Beginners

I have the following DataFrame: +----------+ | date| +----------+ |2017-01-25| |2017-01-21| |… Read more How To Convert Date To The First Day Of Month In A Pyspark Dataframe Column?

I am trying to measure the performance impact on having to copy a dataframe from scala to python an… Read more Pyspark. Transformer That Generates A Random Number Generates Always The Same Number

I'm loading many versions of JSON files to spark DataFrame. some of the files holds columns A,B… Read more Spark - Set Null When Column Not Exist In Dataframe

I have 2 dataframes A(35 Million records) and B(30000 records) A |Text | ------- | pqr | ------- |… Read more Pyspark Crossjoin Between 2 Dataframes With Millions Of Records

This solution in theory, works perfectly for what I need, which is to create a new copied version o… Read more How To Use To_json And From_json To Eliminate Nested Structfields In Pyspark Dataframe?

I am having 100,000+ of records in dataframe. I want to create a file dynamically and push 1000 rec… Read more How To Get 1000 Records From Dataframe And Write Into A File Using Pyspark?

I have the following CSV (sample) id timestamp routeid creationdate parameter… Read more Converting Complex Rdd To A Flatten Rdd With Pyspark

I have a StructField in a dataframe that is not nullable. Simple example: import pyspark.sql.functi… Read more Can I Change The Nullability Of A Column In My Spark Dataframe?