Data Engineer 2

Key Responsibilities

Data Architecture & Solution Development

• Translate business and functional requirements into robust, scalable data solutions aligned with overall architecture.
• Design and implement data integrations and data quality frameworks.
• Participate in full development lifecycle including design, implementation, testing, documentation, delivery, support, and maintenance.

Pipeline & API Development

• Develop and maintain scalable data pipelines.
• Build new API integrations to support data flow and analytics requirements.
• Implement large-scale, high-volume, low-latency applications for efficient data processing.

Big Data & ETL Development

• Design, develop, and maintain ETL pipelines.
• Apply dimension modeling and data warehousing best practices.
• Write complex SQL queries for data transformation, extraction, and validation.

Streaming & Advanced Processing

• Build stream processing platforms using Kafka and Spark Streaming.
• Optimize data pipelines for performance and scalability.
• Independently deliver end-to-end projects with minimal supervision.

Required Skills & Experience

• 5+ years of development experience in big data using Spark.
• Strong programming skills in Python.
• Expertise in SQL and complex query writing.
• Experience with dimension modeling, data warehousing, and ETL pipelines.
• Knowledge of streaming platforms like Kafka and Spark Streaming.

Nice to Have

• Experience with job orchestration frameworks like Airflow, Oozie, or Luigi.
• Familiarity with AWS services such as S3, EMR, RDS.
• Understanding of cloud data warehouses like Snowflake.
• Knowledge of SQL distribution engines like Presto, Druid.
• Familiarity with streaming frameworks like Flink.
• Exposure to NoSQL databases such as HBase or Cassandra.