I have my delta table, which can be read from Athena. When I try to get the data through a query from spark I get the following error: Caused by: org.apache.spark.SparkException: Job aborted due to s...

We have ingested CDC data to S3 raw layer. This CDC data in JSON file and has DML records (delete, update etc). We used spark streaming with delta lake to de-dup S3- raw layer data and move to standar...

My Java application uses a Spark Structured Streaming connected to a socket server continuously getting sensor measurement records (IoT) wrapped in an RDMessage object that records the message type fo...

My code used to work. Why does my code not work anymore? I updated to the newer Databricks runtime 10.2 so I had to change some earlier code to use pandas on pyspark. # Drop customer ID for AutoML aut...