python (12.9k questions)

javascript (9.2k questions)

reactjs (4.7k questions)

java (4.2k questions)

java (4.2k questions)

c# (3.5k questions)

c# (3.5k questions)

html (3.3k questions)

Questions - mini-batch

Databricks Autoloader is getting stuck and does not pass to the next batch

I have a simple job scheduled every 5 min. Basically it listens to cloudfiles on storage account and writes them into delta table, extremely simple. The code is something like this: df = (spark .rea...
test-img

Maksym

spark-streaming

databricks

autoload

rocksdb

mini-batch

Votes: 0

Answers: 1

Latest Answer

I resolved it by using .option('cloudFiles.useIncrementalListing', 'false') My filenames are composed of flowname + timestamp, like this: flow_name_2022-01-18T14-19-50.018Z.json So my guess is: some c...
test-img

Maksym

Posts

Questions

Blogs

Jobs