We are looking to start using Apache Kafka, and we are studying the best ways to create processes to consume data in Kafka. When I say process, I mean the creation of a pipeline like Airflow, but when we use Kafka (from what I've read), the process keeps running (listening to the topic) without stopping.

The question is how to execute these processes in production?

We thought of creating our python classes and scheduling in crontab (some processes will be "infinite", without stopping), or containing and using kubernetes. However, it is still not clear to us, which would be the best way, and if there are others.

Could someone help? After creating our classes in python what would be the best way to put this in production to consume Kafka data?