2 years ago
#64547
friartuck
Changing the number of partitions in AWS MSK with no downtime
Still fairly new to kafka. Trying to change the number of partitions without any downtime/data loss. Currently have 1 partition with 2 brokers and I want to move to, say 10 partitions with 3 brokers (happy to have 10 partitions on each broker). Wondering is there a way for me to do this without any downtime/lost data.
I read
When you update your existing MSK cluster with a custom MSK configuration, Amazon MSK does rolling restarts when necessary, using best practices to minimize customer downtime. For example, after Amazon MSK restarts each broker, it tries to let the broker catch up on data that the broker might have missed during the configuration update before it moves to the next broker.
from https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-properties.html
Is it saying, "yes there is always going to be downtime", or "sometimes there may be downtime under certain conditions"?
I'm not sure I understand why data loss may occur - I would expect existing events in the existing paritions to be distributed across the new + old partitions... I don't see why that would need to cause data loss. Perhaps there would be some downtime to the consumers are kafka re-organizes the data into new partitions? Can we still push events into kafka during this period?
I see here Kafka - is it possible to alter Topic's partition count while keeping the change transparent to Producers and Consumers? that in Kafka (presumably that includes MSK?), that only NEW data added to kafka is put into the new partitions, which understandably could affect things if consumers depend upon this behavior but mine don't.
TL;DR Can I add more partitions without (i) missing new events (ii) loosing existing events
Thanks
amazon-web-services
apache-kafka
aws-msk
0 Answers
Your Answer