2 years ago
#33163
jsrathnayake
How to rollback to previous state if any failure happened while parallelly loading data into Snowflake
While loading data into Snowflake tables parallelly, How can I rollback to previous states in tables if any failure happened.
This is the scenario:
- I have loaded data into staging layer and now I want to move data into modeling layer
- There are several batches run sequentially and in each batch there are tables load parallelly.
- If one batch passed successfully, it will continue with next batch execution (Ex : Batch 1 to Batch 2)
- Assume that, in batch 2 "Table 7" loading failed due to data type issues or something.
- Now the batch 3 execution will be stopped and entire operation will be halted.
How can I restore to the previous states in tables in batch 2 (Table 5,6) and batch 1 (Table 1,2,3,4). Means I want to rollback the entire execution and successfully loaded tables should not have the new changes(Inserts/Updates).
Programmatic Time travel procedure? or any mechanism. Hope your suggestions here.
datatable
parallel-processing
snowflake-cloud-data-platform
restore
rollback
0 Answers
Your Answer