Job Description

One of our leading client is looking for Lead Databricks Engineer remotely.

Responsibilities:

• Design, develop, and deploy data pipelines using Databricks, including data ingestion, transformation, and loading (ETL) processes.
• Develop and maintain high-quality, scalable, and maintainable Databricks notebooks using Python.
• Work with Delta Lake and other advanced features.
• Leverage Unity Catalog for data governance, access control, and data discovery.
• Develop and optimize data pipelines for performance and cost-effectiveness.
• Integrate with various data sources, including but not limited to databases and cloud storage (Azure Blob Storage, ADLS, Synapse), and APIs.
• Experience working with Parquet files for data storage and processing.
• Experience with data integration from Azure Data Factory, Azure Data Lake, and other relevant Azure services.
• Perform data quality checks and validation to ensure data accuracy and integrity.
• Troubleshoot and resolve data pipeline issues effectively.
• Collaborate with data analysts, business analysts, and business stakeholders to understand their data needs and translate them into technical solutions.
• Participate in code reviews and contribute to best practices within the team.

Qualifications:

• 10+ years of experience as a Data Engineer with a strong focus on Databricks.
• Strong proficiency in Python and SQL.
• Hands-on experience with Databricks features such as delta lake, clusters, notebooks, jobs, and workspaces.
• Experience with data warehousing concepts and best practices.
• Strong understanding of data modeling and data quality principles.
• Experience with Agile development methodologies.
• Excellent communication and collaboration skills.
• Ability to work independently and as part of a team.

Preferred Qualifications:

• Experience with Azure cloud services (e.g., Azure Data Factory, Azure Synapse Analytics)
• Experience with CI/CD pipelines for Databricks workloads.
• Experience with data visualization tools (e.g., Power BI, Tableau)
• Experience with data security and privacy best practices

Job Tags

Similar Jobs