Data Engineer - AWS + Databricks_104560

Company:IT Services Organization

Key Skills: Pyspark, AWS, Databricks, SQL, Data Engineer, AWS Lambda

Roles and Responsibilities

• Design, develop, and maintain end-to-end data pipelines using Databricks, PySpark, SQL, and AWS services.
• Build scalable data processing solutions using Delta Lake, Unity Catalog, and Apache Spark capabilities.
• Develop SQL queries, stored procedures, data marts, views, and reporting-layer components for relational data platforms.
• Analyze Spark and Databricks workloads, reduce job runtimes, and implement cost and performance optimization measures.
• Participate in solution design, code reviews, production support, proof-of-concept initiatives, and technical documentation.
• Support CI/CD pipelines and infrastructure automation using Terraform and GitHub or equivalent version-control platforms.
• Provide technical mentoring to junior engineers and coordinate delivery for midsized engineering teams.

Skills Required

• 5 to 8 years of IT experience with strong hands-on experience in Databricks on AWS.
• Proficiency in PySpark and Python for data engineering and distributed data processing.
• Strong SQL development experience, including stored procedures, data marts, and relational database views.
• Experience designing ETL workflows, end-to-end data pipeline architectures, data warehouse layers, and reporting layers.
• Working knowledge of AWS data and orchestration services, including Amazon S3, AWS Glue, Amazon SNS, and AWS Step Functions.
• Experience with Delta Lake, Unity Catalog, Spark troubleshooting, workload tuning, and Databricks cost optimization.
• Exposure to CI/CD pipelines, Terraform-based infrastructure automation, GitHub, Agile delivery, and SDLC practices.

Good to Have

• Experience using AWS Lambda in data engineering or cloud integration workflows.

Education: Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.