2 years ago

#33163

test-img

jsrathnayake

How to rollback to previous state if any failure happened while parallelly loading data into Snowflake

While loading data into Snowflake tables parallelly, How can I rollback to previous states in tables if any failure happened.

Data loading parallelly

This is the scenario:

  1. I have loaded data into staging layer and now I want to move data into modeling layer
  2. There are several batches run sequentially and in each batch there are tables load parallelly.
  3. If one batch passed successfully, it will continue with next batch execution (Ex : Batch 1 to Batch 2)
  4. Assume that, in batch 2 "Table 7" loading failed due to data type issues or something.
  5. Now the batch 3 execution will be stopped and entire operation will be halted.

How can I restore to the previous states in tables in batch 2 (Table 5,6) and batch 1 (Table 1,2,3,4). Means I want to rollback the entire execution and successfully loaded tables should not have the new changes(Inserts/Updates).

Programmatic Time travel procedure? or any mechanism. Hope your suggestions here.

datatable

parallel-processing

snowflake-cloud-data-platform

restore

rollback

0 Answers

Your Answer

Accepted video resources