Still Struggling with Tech Hiring? Discover Olibr's Solution Now!
Login / RegisterDeveloperClient
Login/Register

Ashok Pidugu

Data Engineer

  • Hyderabad, TG
  • 10 years

Available From: Available

    Candidate's About

    With over 7 years of IT industry experience, I have spent the past 5 years as an Azure data engineer, utilizing Microsoft Azure with Databricks and Databricks Workspace for business analytics. I have hands-on experience in data extraction, transformation, and production.

    Work & Experience

    D

    Data Engineer

    Dec 20 - Nov 21

    Technology

    • Icon
      SQL

    Tools

    • Icon
      Linux
    • Developed and implemented technology solutions for clients
    • Provided technical support, including 24/7 assistance, training, and guidance
    • Managed and maintained clients' infrastructure to ensure optimal performance and security
    • Offered consultation and advisory services to help clients make informed decisions
    • Ensured data security and privacy for clients' sen...
    Read More

    CDM Data Model

    Azure-Databricks-Spark Developer

    • Understood client requirements and prepared design documents Wrote hive queries based on transformation logic and filters. 
    • Developed spark applications using Spark-SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats to uncover insights into customer usage patterns.
    • Extracted, transformed, and loaded data from source systems to Azure data storage services using a combination of Azure data factory, T-SQL, and U-SQL Azure data lake analytics .
    • Ingested data to one or more Azure services (Azure data lake, Azure Storage, Azure SQL, Azure DW) and processed the data in Azure Databricks.
    • Developed spark applications using PySpark and Spark SQL for data extraction, transformation, and aggregation from multiple file formats to uncover insights into customer usage patterns.
    • Estimated the cluster size, monitored, and troubleshooted the spark Databricks cluster. 
    • Applied the spark Data Frame API to complete data manipulation within a spark session. 
    • Have a good understanding of spark architecture, including spark core, spark SQL, Data Frame, Spark streaming, Driver Node, Worker Node, Stages, Executors and Tasks, Deployment Modes, the execution hierarchy, fault tolerance and collection.
       
    • Azure Databricks
    • Azure SQL database
    • Azure SQL Datawarehouse
    View Details
    D

    Data Engineer

    Dec 19 - Nov 20

    Technology

    • Icon
      SQL

    Tools

    • Icon
      Linux
    • Developed and implemented technology solutions for clients
    • Provided technical support, including 24/7 assistance, training, and guidance
    • Managed and maintained clients' infrastructure to ensure optimal performance and security
    • Offered consultation and advisory services to help clients make informed decisions
    • Ensured data security and privacy for clients' sen...
    Read More

    Mobile One Data Analytics

    Azure-Databricks-Spark Developer

    • Understood client requirements and prepared design documents 
    • Converted Hive/SQL queries into Spark transformations using Spark RDDs & Python 
    • Analyzed the SQL scripts and designed the solution to implement using PySpark 
    • Extracted, transformed, and loaded data from source systems to Azure data storage services using a combination of Azure Data Factory, Spark SQL, and U-SQL T-SQL Azure Data Lake Analytics 
    • Ingested data to one or more Azure Services - Azure Data Lake, Azure Storage, Azure SQL, Azure DW - and processed the data in Azure Databricks 
    • Created pipelines in ADF using Linked Services/Datasets/Pipeline to extract, transform, and load data from different sources like Azure SQL, Blob Storage, Azure SQL data warehouse, write-back tool, and backwards 
    • Experienced in performance tuning of Spark Applications for setting the right batch interval time, correct level of parallelism, and memory tuning Wrote UDF’s in Pyspark to meet specific business requirements 
    • Had hands-on experience developing SQL scripts for automation purposes.
    • Azure Py-Spark
    • Spark SQL
    View Details
    D

    Data Engineer

    Dec 17 - Nov 19

    Technology

    • Icon
      SQL

    Tools

    • Icon
      Linux
    • Developed and implemented technology solutions for clients
    • Provided technical support, including 24/7 assistance, training, and guidance
    • Managed and maintained clients' infrastructure to ensure optimal performance and security
    • Offered consultation and advisory services to help clients make informed decisions
    • Ensured data security and privacy for clients' sen...
    Read More

    Data Reconciliation

    Bigdata Lead

    • Prepared design documents and provided technical solutions based on client requirements
    • Wrote Hive queries for data transformation and filtering
    • Ensured client satisfaction and effective demand planning for timely delivery
    • Designed, architected, and deployed technical solutions efficiently
    • Adopted new technologies effectively and provided guidance to offshore team
    • Utilized Spark for data processing and transformation
    • Utilized Presto and Redshift for ad-hoc queries and analysis
    • Generated reports and created data for analysis and insights
    • Participated in business and system requirement sessions and translated requirements to technical specifications
    • Planned solution roadmap for future growth, process architecture, and mapping
    • Provided inputs on solution architecture based on evaluation and understanding of alternatives, frameworks, and products
    • Estimated effort for future releases to ensure timely delivery.
    • Spark SQL
    • Azure Py-Spark
    View Details
    D

    Data Engineer

    Dec 16 - Nov 17

    Technology

    • Icon
      SQL

    Tools

    • Icon
      Linux
    • Developed and implemented technology solutions for clients
    • Provided technical support, including 24/7 assistance, training, and guidance
    • Managed and maintained clients' infrastructure to ensure optimal performance and security
    • Offered consultation and advisory services to help clients make informed decisions
    • Ensured data security and privacy for clients' sen...
    Read More

    Migration Project from ETL to Hadoop Systems

    Bigdata Lead

    • Participated in business and system requirements sessions
    • Elicited requirements and translated them into technical specifications
    • Interacted with clients to elicit architectural and non-functional requirements such as performance, scalability, reliability, availability, and maintainability
    • Performed performance optimizations on Hadoop, Oracle, and Shell Scripts
    • Designed applications based on identified architecture and supported implementation of design by resolving complex technical issues
    • Demonstrated excellent resource balancing and leadership skills with an ability to drive initiatives through efficient teamwork.
    • Hadoop – Hive, HDFS
    View Details
    D

    Data Engineer

    Dec 14 - Jun 16

    Technology

    • Icon
      SQL

    Tools

    • Icon
      Linux
    • Developed and implemented technology solutions for clients
    • Provided technical support, including 24/7 assistance, training, and guidance
    • Managed and maintained clients' infrastructure to ensure optimal performance and security
    • Offered consultation and advisory services to help clients make informed decisions
    • Ensured data security and privacy for clients' sen...
    Read More

    Analytics Platform

    Bigdata Lead

    • Created components using Mahout algorithms and wrote Hive queries to visualize data for algorithm prediction.
    • Created charts for algorithm results using hive queries with BDE tool.
    • Quickly analyzed and fixed defects in the application.
    • Identified and provided suggestions for fixing performance-related issues in the application.
    • Participated in requirements and design analysis.
    • Optimized code to reduce performance overhead.
    • Coordinated with multiple vendors from geographically distributed teams for successful development, testing, and deployment of a single project.
    • Faced challenges in understanding the functions and projects of each team member in a dynamic and changing environment.
    • Interpreted clients' needs and business problems and translated them into specific requests to the local or global databases to resolve them adequately and timely.
    • Handled a high volume of requests under high pressure on deadlines.
    • Ensured the quality of the data provided matched with official KPIs.
    • Managed a variety of variables and large volumes of data from very diverse sources.
    • Data Pipelines
    View Details
    D

    Data Engineer

    Jul 10 - Nov 12

    Technology

    • Icon
      Java
    • Collaborated with cross-functional teams to understand business requirements and design data models and solutions
    • Implemented Data Warehousing concepts such as star-schema modeling, ETL, data integration, and data management
    • Developed and deployed SQL queries, stored procedures, and scripts to automate data processing
    • Optimized SQL queries and implemented performanc...
    Read More

    Your Finance

    Bigdata Lead

    • Implemented new enhancements using Java and Oracle based on client requirements.
    • Quickly analyzed and fixed defects.
    • Identified and provided suggestions for fixing application performance related issues.
    • Participated in requirements and design analysis.
    • Optimized code, resulting in reduced performance overhead.
    • Java/J2EE
    View Details
    D

    Data Engineer

    Nov 08 - Jul 10

    Technology

    • Icon
      Java
    • Designed, developed and maintained the data architecture, including data models, database schemas, and ETL processes, for various projects.
    • Collaborated with cross-functional teams to understand their data requirements and design solutions that met those needs.
    • Created and maintained documentation related to the data architecture, including data dictionaries, ER diagrams, a...
    Read More

    BBS-BB Increment

    Bigdata Lead

    • Implemented new enhancements using Java and Oracle according to client requirements, ensuring that the applications met the desired specifications and quality standards.
    • Conducted defect analysis and resolved issues quickly, minimizing downtime and ensuring that systems were running smoothly.
    • Identified application performance-related issues, performed detailed analysis of the problems, and provided suggestions for fixing them, such as optimizing database queries, improving code quality, or redesigning system architecture.
    • Participated in requirements and design analysis, collaborating with other team members to ensure that the solutions met the client's needs and aligned with project objectives.
    • Contributed to design discussions and provided feedback on technical feasibility, scalability, and maintainability of proposed solutions.
    • Conducted code reviews and ensured that code adhered to best practices and coding standards, optimizing code quality, maintainability, and performance.
    • Worked collaboratively with cross-functional teams, such as QA, BA, and PM, to ensure successful delivery of projects and met project deadlines.
    • Created and maintained technical documentation, including functional and technical specifications, design documents, and testing plans.
    • Participated in knowledge-sharing activities, such as team meetings, training sessions, and technical presentations, to continuously improve skills and knowledge.
    • Java
    • Oracle
    View Details

    Education

    J

    Bachelor of Technology

    J.N.T.U
    2004 - 2008