Job Description
- Bachelor's degree in Computer Science, Information Technology, or a related field.
- Apache Spark and Delta Lake expertise
- Hands-on experience with big data pipelines.
- Programming skills : Proficiency in Python, Scala, or SQL.
- Databricks platform experience and its features and capabilities.
- Data engineering background : Solid experience in data modeling, warehousing, and ETL/ELT processes.
- Cloud computing knowledge : Experience with AWS, Azure, GCP, and their integration with Databricks.
- Data security and compliance : Knowledge of encryption, access controls, and data privacy regulations (e.g., GDPR, HIPAA).
- Ability to analyze and optimize system performance.
Job Responsibilities
- Project management experience : Skilled in managing timelines, resources, and stakeholder expectations.
- Communication and collaboration : Ability to work effectively with cross-functional teams and explain technical concepts to non-technical stakeholders.
- Commitment to staying current with big data technologies, Databricks updates, and industry best practices.
- Lead the setup and configuration of the Databricks environment within the organization's cloud infrastructure (AWS, Azure, GCP).
- Ensure the integration of Databricks with existing data sources, data warehouses, and other analytics tools.
- Design, develop, and maintain robust data pipelines within Databricks, leveraging Apache Spark and Delta Lake for efficient data processing and storage.
- Implement ETL/ELT processes that cater to the needs of diverse analytics and machine learning projects.
- Monitor the performance of data processes and queries within Databricks, identifying and resolving bottlenecks.
- Optimize Spark jobs and Databricks clusters for cost efficiency and computational performance.
- Implement and maintain security measures
Location
Bangalore Urban, Karnataka, India