Lead Databricks Engineer Job at Programmers.io, Texas

RmgvbnhTTWlCZXptNFoyeTVLR29udXJVOGc9PQ==
  • Programmers.io
  • Texas

Job Description

One of our leading client is looking for Lead Databricks Engineer remotely.

Responsibilities:

  • Design, develop, and deploy data pipelines using Databricks, including data ingestion, transformation, and loading (ETL) processes.
  • Develop and maintain high-quality, scalable, and maintainable Databricks notebooks using Python.
  • Work with Delta Lake and other advanced features.
  • Leverage Unity Catalog for data governance, access control, and data discovery.
  • Develop and optimize data pipelines for performance and cost-effectiveness.
  • Integrate with various data sources, including but not limited to databases and cloud storage (Azure Blob Storage, ADLS, Synapse), and APIs.
  • Experience working with Parquet files for data storage and processing.
  • Experience with data integration from Azure Data Factory, Azure Data Lake, and other relevant Azure services.
  • Perform data quality checks and validation to ensure data accuracy and integrity.
  • Troubleshoot and resolve data pipeline issues effectively.
  • Collaborate with data analysts, business analysts, and business stakeholders to understand their data needs and translate them into technical solutions.
  • Participate in code reviews and contribute to best practices within the team.

Qualifications:

  • 10+ years of experience as a Data Engineer with a strong focus on Databricks.
  • Strong proficiency in Python and SQL.
  • Hands-on experience with Databricks features such as delta lake, clusters, notebooks, jobs, and workspaces.
  • Experience with data warehousing concepts and best practices.
  • Strong understanding of data modeling and data quality principles.
  • Experience with Agile development methodologies.
  • Excellent communication and collaboration skills.
  • Ability to work independently and as part of a team.

Preferred Qualifications:

  • Experience with Azure cloud services (e.g., Azure Data Factory, Azure Synapse Analytics)
  • Experience with CI/CD pipelines for Databricks workloads.
  • Experience with data visualization tools (e.g., Power BI, Tableau)
  • Experience with data security and privacy best practices

Job Tags

Remote work,

Similar Jobs

Direct Solution Marketing

Enrollment Coordinator Job at Direct Solution Marketing

 ...Enrollment Coordinator $35K+ Entry Level/$100K+ Managers We are aggressively hiring agents to enroll low income families into a free government benefit program. Agents are on location at community events, CBOs, and community centers to qualify and enroll families... 

Woundlocal

Wound Care Nurse (RN/LVN) - Houston Job at Woundlocal

 ...Now hiring experienced wound care nurses! Freenet Health Corp. is a rapidly growing healthcare management company servicing virtual and mobile medical practices. We are now hiring multiple positions for our mobile wound care practice Woundlocal . We are seeking... 

LeverEdge

Growth Marketing Intern (On Campus) Job at LeverEdge

 ...Communicating what you learn about performance marketing with a cohort of Growth Marketing Managers from other campuses (via our Slack channel) Recommending new ways to complete our mission What we're looking for: # Be innovative. Nothing stops you from finding... 

Brewer Global

BSA / AML Analyst Job at Brewer Global

 ...We are searching for a BSA and AML Analyst to provide transaction monitoring. Responsibilities: Deliver exceptional customer service. Maintain knowledge of banking rules and regulations. Perform duties related to BSA/AML/OFAC compliance, including quality control... 

Hoag Memorial Hospital Presbyterian

SECURITY OFFICER PROFESSIONAL I - SECURITY Job at Hoag Memorial Hospital Presbyterian

 ...and Responsibilities Under the direction of the Supervisor of Security Operations I, Supervisor of Security Operations II, the Manager...  ...Services License for Security Guard or Proprietary Private Security Officer within six weeks of employment Valid California...