Still Struggling with Tech Hiring? Discover Olibr's Solution Now!
Login / RegisterDeveloperClient
Login/Register

DHRUMIL SHAH

senior Data engineer

  • Ahmedabad, GJ
  • 7 years

Available From: Available

    Candidate's About

    Experienced IT professional with 7.2 years of expertise in Data Engineering, specializing in HDFS, Spark, Scala, and Python. Proficient in Airflow and AWS services like S3 and Lambda. Skilled in Hadoop and Spark architectures, performance tuning, and clean code writing. Familiar with deployment pipelines and Kafka. Strong team player with excellent communication skills and a track record of meeting organizational expectations and deadlines. Quick learner, problem-solver, and adaptable to different

    Work & Experience

    S

    Senior Data engineer

    Jul 22 - PRESENT

    Technology

    • Icon
      Java
    • Icon
      Python
    • Icon
      Kafka
    • Icon
      AWS

    Tools

    • Icon
      Jira
    • Icon
      Visual-Studio
    • Icon
      Linux
    • Develop and maintain data pipelines using HDFS, Spark, Scala, and Python.
    • Optimize data processing performance through caching and broadcast techniques.
    • Collaborate with cross-functional teams to design and implement scalable data solutions.
    • Provide technical guidance and mentorship to junior team members.
    • Ensure timely delivery of high-quality data engineer...
    Read More
    D

    Data Engineer

    Jun 20 - Jul 22

    Technology

    • Icon
      AWS
    • Icon
      Kafka
    • Icon
      Python
    • Icon
      Java

    Tools

    • Icon
      Jira
    • Icon
      Linux
    • Icon
      Visual-Studio
    • Developed and maintained data pipelines using HDFS, Spark, Scala, and Python.
    • Optimized data processing performance through caching and broadcast techniques.
    • Collaborated with cross-functional teams to design and implement scalable data solutions.
    • Provided technical guidance and mentorship to junior team members.
    • Ensured timely delivery of high-quality data ...
    Read More
    I

    IT Analyst

    Jun 18 - May 20

    Technology

    • Icon
      AWS
    • Icon
      Java
    • Icon
      Python
    • Icon
      Kafka

    Tools

    • Icon
      Jira
    • Icon
      Linux
    • Icon
      Visual-Studio
    • Analyzed IT systems, identified areas for improvement, and developed solutions.
    • Assisted in system design, configuration, and implementation.
    • Conducted testing and quality assurance to ensure system functionality.
    • Provided technical support and troubleshooting for end-users.
    • Collaborated with team members and stakeholders to gather requirements and deliver I...
    Read More

    Enrollment data analysis

    Big Data developer

    • Analyzed Hadoop cluster using Hive for data processing.
    • Loaded and transformed large volumes of structured and unstructured data.
    • Implemented User Defined Functions in Hive for aggregating data from HDFS.
    • Utilized Spark with Scala and Spark SQL for faster data processing.
    • Collaborated with testing team for bug fixes and prepared documentation for data records and release cycle notes.
    • Conducted unit testing and system testing for the project.
    • MS-SQL
    • JIRA
    • Hive
    • Hadoop
    • Spark
    • Scala
    View Details

    Trade Finance

    Big Data Tester

    • Prepared Hive test scripts for data validation in Staging, DL, and Dimension layers.
    • Created test cases and tracked them in JIRA.
    • Implemented external tables for validating target data.
    • Executed test cases and performed data validation.
    • Provided additional support to meet project timelines and resolved various issues.
    • Reported daily and weekly status updates to stakeholders.
    • Identified and tracked defects until closure.
    • Participated in client and defect calls.
    • MS-SQL
    • JIRA
    • Hive
    • Hadoop
    View Details
    S

    Software Engineer

    Jul 12 - Feb 15

    Technology

    • Icon
      AWS
    • Icon
      Java
    • Icon
      Kafka
    • Icon
      Python

    Tools

    • Icon
      Jira
    • Icon
      Visual-Studio
    • Icon
      Linux
    • Developed software solutions using various technologies such as Java, Python, and SQL.
    • Participated in the complete software development life cycle, including requirements gathering, design, coding, testing, and deployment.
    • Collaborated with cross-functional teams to deliver high-quality software products.
    • Conducted code reviews and provided technical guidance to ju...
    Read More

    Zillow

    Developer

    • Processed various feeds (visits, ispot-DMA, SEO clarity, application purchase, application submits) for Zillow Group, a leading US real estate marketplace.
    • Transformed and stored the processed data in AWS S3 bucket.
    • Implemented data processing and transformation pipelines for efficient data handling.
    • Contributed to the data management and storage strategy for Zillow Group's real estate data.
    • DB VIsualizer
    • PySpark
    • Python
    • Hive
    • Airflow
    • AWS
    View Details

    Rating Direct on Market Intelligence

    Big Data developer

    • Conducted thorough analysis of business requirements and existing tables/files.
    • Created Hive tables based on design and requirements.
    • Developed Scala code for API creation and functional programming.
    • Implemented Pyspark code for data transformation and created Airflow DAG using Python script.
    • Scheduled and executed daily Spark jobs using Airflow, storing the data in AWS S3 bucket.
    • Enhanced existing code to accommodate Business Change Requests.
    • Utilized Data Frames and Spark SQL for data analysis and transformation.
    • HDFS
    • MS-SQL
    • Spark
    • Scala
    • Hive
    • Linux
    View Details

    Education

    G

    IT engineering

    Gujarat Technological University
    2008 - 2012