python (12.9k questions)

javascript (9.2k questions)

reactjs (4.7k questions)

java (4.2k questions)

java (4.2k questions)

c# (3.5k questions)

c# (3.5k questions)

html (3.3k questions)

Questions - pyspark

Is it possible to define recursive DataType in PySpark Dataframe?

I want to create a schema like this example: friendSchema = StructType([ \ StructField("firstname",StringType(),True), \ StructField("middlename",StringType(),True), \ ...
test-img

Omid

apache-spark

pyspark

apache-spark-sql

Votes: 0

Answers: 4

Latest Answer

Yes, it's possible. What you're looking to do is called a nested struct. A StructType schema can itself include StructType fields, which will do what you want. So for example: def test_build_nested_sc...
test-img

nirb89

GroupBy based on condition Pyspark

I have a peculiar problem to which I don't see an easy solution (and efficient one). I have a list of addresses (more than 10M) with different user_time_active. I also have count per user_time_active ...
test-img

Martin Walczyński

pyspark

group-by

pandas-groupby

aggregation

Votes: 0

Answers: 1

Latest Answer

For anyone wondering the solution, I found one! End table looks a bit different, but it works! I use: df = df.withColumn("status", f.when(f.col("user_time_active") == 1, 1) \ ...
test-img

Martin Walczyński

How to use spark.DataFrameReader from Foundry Transforms

I have a file with non-classic formatting so I need to use the spark.DataFrameReader (spark.read.csv) on the raw file directly so that I can set the appropriate parsing configurations. How can I do th...
test-img

Dovid Braverman

apache-spark

pyspark

palantir-foundry

foundry-code-repositories

Votes: 0

Answers: 1

Latest Answer

You'll want to follow the methodology over here. Strongly recommend using unit test based methods to iterate on your code to recover the file contents. Your compute function code will look like: from...
test-img

vanhooser

Prevent commas in open text columns from messing up spark read csv

I have a directory of CSVs that has some text columns that DO NOT have quotes around them (I cannot control this as these are from an external source). These open text columns have commas in them and...
test-img

ben890

csv

apache-spark

pyspark

apache-spark-sql

delimiter

Votes: 0

Answers: 1

Latest Answer

I agree with @blackbishop. There is no magic to read this type of data without any string manipulation. That's sort of why I asked what is after Note column. Only way to handle is to either change the...
test-img

Emma

Posts

Questions

Blogs

Jobs