Questions about spark-structured-streaming
Read more about spark-structured-streaming
python (12.9k questions)
javascript (9.2k questions)
reactjs (4.7k questions)
java (4.2k questions)
java (4.2k questions)
c# (3.5k questions)
c# (3.5k questions)
html (3.3k questions)
Is there a monitoring endpoint for Spark Structured streaming?
In Spark's official docs, we see that there are monitoring endpoints for DStream like
/streaming/statistics
However, there does not seem to be ones for structured streaming mentioned here. I'm look...
yeskarthik
Votes: 0
Answers: 0
Convert spark streaming dataframe column value into new dataframe
I have a nested json in one of the columns in a spark streaming dataframe. I would like to extract the column value, which is a json string, create a new data frame with it and write to a json file.
I...
RCT
Votes: 0
Answers: 2
How to get Kafka header's value to Spark Dataset as a single column?
We have a Kafka stream that has headers enabled
.option("includeHeaders", true)
thus making them stored as a high level dataset's colum, bearing the inside array of structs with key and v...
Eljah
Votes: 0
Answers: 1
Spark structured streaming job failing after running for 3-4 days with java.io.FileNotFoundException: File does not exist
We have a spark structured streaming job written in scala running in production which reads from a kafka topic and writes to HDFS sink. Triggertime is 60 Seconds. The job has been deployed 4months bac...
sam.ban
Votes: 0
Answers: 0