Lead Databricks Engineer Job at Programmers.io, Texas

RmgvbnhTTWlCZXptNFoyeTVLR29udXJVOGc9PQ==
  • Programmers.io
  • Texas

Job Description

One of our leading client is looking for Lead Databricks Engineer remotely.

Responsibilities:

  • Design, develop, and deploy data pipelines using Databricks, including data ingestion, transformation, and loading (ETL) processes.
  • Develop and maintain high-quality, scalable, and maintainable Databricks notebooks using Python.
  • Work with Delta Lake and other advanced features.
  • Leverage Unity Catalog for data governance, access control, and data discovery.
  • Develop and optimize data pipelines for performance and cost-effectiveness.
  • Integrate with various data sources, including but not limited to databases and cloud storage (Azure Blob Storage, ADLS, Synapse), and APIs.
  • Experience working with Parquet files for data storage and processing.
  • Experience with data integration from Azure Data Factory, Azure Data Lake, and other relevant Azure services.
  • Perform data quality checks and validation to ensure data accuracy and integrity.
  • Troubleshoot and resolve data pipeline issues effectively.
  • Collaborate with data analysts, business analysts, and business stakeholders to understand their data needs and translate them into technical solutions.
  • Participate in code reviews and contribute to best practices within the team.

Qualifications:

  • 10+ years of experience as a Data Engineer with a strong focus on Databricks.
  • Strong proficiency in Python and SQL.
  • Hands-on experience with Databricks features such as delta lake, clusters, notebooks, jobs, and workspaces.
  • Experience with data warehousing concepts and best practices.
  • Strong understanding of data modeling and data quality principles.
  • Experience with Agile development methodologies.
  • Excellent communication and collaboration skills.
  • Ability to work independently and as part of a team.

Preferred Qualifications:

  • Experience with Azure cloud services (e.g., Azure Data Factory, Azure Synapse Analytics)
  • Experience with CI/CD pipelines for Databricks workloads.
  • Experience with data visualization tools (e.g., Power BI, Tableau)
  • Experience with data security and privacy best practices

Job Tags

Remote work,

Similar Jobs

Taco Bell

General Manager Job at Taco Bell

Company Summary: Who is Taco Bell? Taco Bell is a leading Mexican-inspired quick service restaurant brand that is a part of Yum! Brands, Inc. which includes Pizza Hut and KFC. Founded on innovation and passion with Glen Bell bringing tacos to the masses in a world of hot...

University of South Florida

Clinical research associate Job at University of South Florida

 ...Salary Plan: Administrative Job Code/Title: 4728 / Clinical Research Associate Hiring Salary/Salary Range: $45,000 - $61,000 Position...  ...Information for Applicants This position is subject to a Level 1 criminal background check. Job Opening Number: 38717... 

Get It - Hospitality

Event Servers/Baker's Best Catering - Remote | WFH Job at Get It - Hospitality

**Job Title: Part-Time/On-Call Event Server****Salary: Competitive** Join a team with over 35 years of experience in delivering exceptional...  ...with bending, stooping, and kneeling - Ability to follow company policies and procedures - Strong ability to take direction and... 

Terracon

Senior Project Geologist/Engineer Job at Terracon

 ...associated soil, groundwater, NAPL, and vapor data, maintaining Conceptual Site Models (CSMs), and preparing reports for a major refinery project in southern Illinois. This project also includes providing litigation support to the client and legal counsel. This position... 

Beijing Niko Catering Co., LTD

Pizza chef Job at Beijing Niko Catering Co., LTD

Pizza making