Kafka Data Engineer_5+ years

We are looking for an experienced Kafka Data Engineer with strong hands-on expertise in Apache Kafka, Kafka Topic/Cluster Migration, Data Streaming, and Distributed Systems. The candidate will be responsible for planning and executing Kafka migrations, cluster-to-cluster replication, data validation, performance tuning, and ensuring minimal downtime during migration activities.

Key Responsibilities

• Design and execute Kafka topic and cluster migrations across environments, clusters, or cloud platforms.
• Perform Kafka cluster upgrades, broker migrations, and data replication using MirrorMaker 2.0, Confluent Replicator, Cluster Linking, or similar technologies.
• Migrate and validate topics, schemas, consumer groups, offsets, ACLs, and configurations.
• Monitor Kafka throughput, latency, consumer lag, partition health, and overall cluster performance.
• Troubleshoot Kafka connectivity, replication, partition balancing, and consumer lag issues.
• Develop automation scripts using Python, Shell scripting, Java, or Scala.
• Work with application teams to validate producers, consumers, and streaming pipelines after migration.
• Implement Kafka security using SSL/TLS, SASL, Kerberos, RBAC, and ACLs.
• Collaborate with DevOps, Cloud, and Data Engineering teams for infrastructure and deployment activities.
• Prepare migration plans, rollback strategies, technical documentation, and post-migration reports.

Mandatory Skills

• Strong hands-on experience with Apache Kafka Administration and Development.
• Proven experience in Kafka Topic Migration / Cluster Migration / Cross-Cluster Replication.
• Hands-on experience with MirrorMaker 2.0, Confluent Replicator, or Cluster Linking.
• Strong knowledge of Kafka Connect, Kafka Streams, Schema Registry, and Confluent Platform.
• Good understanding of Kafka partitions, brokers, replication, consumer groups, offsets, and consumer lag.
• Experience working in Linux/Unix environments.
• Proficiency in Python, Java, or Scala.
• Strong understanding of distributed systems, event-driven architecture, and messaging systems.
• Experience with at least one cloud platform such as AWS MSK, Azure Event Hubs/Kafka, or GCP Pub/Sub.

Good to Have

• Experience with Databricks, Spark Streaming, Apache Flink, or other real-time processing technologies.
• Knowledge of Docker and Kubernetes.
• Exposure to Terraform / Infrastructure as Code and CI/CD pipelines.
• Experience with monitoring tools such as Prometheus, Grafana, Splunk, or ELK.
• Kafka or Confluent certification is an added advantage.

Key Skills

Kafka | Kafka Migration | MirrorMaker 2.0 | Kafka Administration | Confluent Platform | Kafka Connect | Cluster Linking | Kafka Replication | Python/Java | Distributed Systems