2 years ago

#12040

test-img

captainslocum

Is there a way to configure date-partitioned folders for AWS DMS endpoint target S3?

I'm using terraform in order to configure this DMS migration task that migrates (full-load+cdc) the data from a MySQL instance to a S3 bucket.

The problem is that the configuration seems not to take effect and no partition-folder is created. All the migrated files are created in the same directory within the bucket.

In the documentation they say the endpoint s3 setting DatePartitionEnabled, introduced in the version 3.4.2, is supported both for CDC and FullLoad+CDC.

My terraform configuration spec:

resource "aws_dms_endpoint" "example" {
  endpoint_id                 = "example"
  endpoint_type               = "target"
  engine_name                 = "s3"

  s3_settings {
    bucket_name = "example"
    bucket_folder = "example-folder"
    compression_type = "GZIP"
    data_format = "parquet"
    parquet_version = "parquet-2-0"
    service_access_role_arn = var.service_access_role_arn
    date_partition_enabled = true
  }

  tags = {
    Name = "example"
  }
}

But in the respective s3 bucket I get no folders, but sequential files as if this option wasn't there.

LOAD00000001.parquet
LOAD00000002.parquet
...

I'm using terraform 1.0.7, aws provider 3.66.0 and a DMS Replication Instance 3.4.6.

Does anyone know what could be this issue?

amazon-web-services

amazon-s3

terraform

dms

aws-dms

0 Answers

Your Answer

Accepted video resources