Senior Data Engineer — Commerce, CA

We are seeking an experienced Senior Data Engineer to design, develop, and maintain scalable data architectures and engineering solutions.

This individual will play a key role in building and optimizing enterprise data platforms, supporting cloud migrations, and establishing data engineering standards and best practices.

The ideal candidate will bring strong hands-on experience with Azure Databricks, Apache Spark, Python, PySpark, and SQL, along with demonstrated technical leadership and the ability to collaborate with business and technical stakeholders.

Key Responsibilities

Analyze and define data requirements, specifications, and architecture to support business objectives.

Design, develop, test, and maintain scalable data architectures, pipelines, and data management solutions.

Evaluate data sources, assess data quality, and provide actionable recommendations based on analysis and findings.

Develop and maintain data standards, policies, procedures, and governance practices.

Manage data compilation, cataloging, caching, distribution, storage, and retrieval.

Evaluate emerging database technologies and recommend improvements to existing data architectures.

Analyze and plan for anticipated changes in data capacity and storage requirements.

Develop and implement data mining, data warehousing, and advanced analytics solutions.

Oversee the design and implementation of an Azure Databricks lakehouse architecture, including Bronze, Silver, and Gold data layers.

Plan and execute cloud migrations, including source-to-target reconciliation, controlled cutovers, and recovery procedures.

Build, maintain, and optimize data pipelines integrating information from multiple enterprise systems and business functions.

Implement Unity Catalog governance, access controls, and data lineage in partnership with IT and Security teams.

Establish engineering standards for data modeling, testing, CI/CD, monitoring, documentation, and production support.

Conduct technical design and code reviews, mentor engineers, and coordinate technical delivery.

Partner with business stakeholders, BI developers, data scientists, and engineering teams to translate requirements into reliable data products.

Develop and maintain scripts and automation using Python, SQL, and other programming languages across Windows and UNIX environments.

Support statistical analysis, machine learning, and advanced analytics initiatives as needed.

Present technical findings, architectural recommendations, and data-driven insights to technical and nontechnical audiences.

Perform other related duties as assigned.

Required Qualifications

5+ years of data engineering experience, including 3+ years of hands-on production experience with Databricks and Apache Spark.

Demonstrated technical leadership in data architecture, engineering standards, and mentoring.

Production experience with Azure Databricks, ADLS Gen2, and Azure identity and security fundamentals.

Strong Delta Lake expertise, including incremental processing, MERGE operations, schema management, time travel, and table optimization.

Hands-on experience with Unity Catalog and Databricks Workflows/Lakeflow Jobs.

Advanced SQL development skills.

Significant experience developing solutions using Python and PySpark.

Experience with medallion architecture, dimensional modeling, databases, APIs, and file-based integrations.

Experience managing large, complex datasets from multiple sources and databases.

Proficiency with Git, automated testing, CI/CD pipelines, and deployments across development, testing, and production environments.

Proven ability to implement data quality checks, reconciliation processes, monitoring, alerting, and reliable pipeline recovery.

Strong analytical and problem-solving skills with the ability to evaluate architectural decisions and technical tradeoffs.

Excellent written and verbal communication skills, including the ability to explain technical concepts to diverse audiences.

Ability to work independently, prioritize competing responsibilities, and collaborate effectively across teams.

Preferred Qualifications

Databricks Data Engineer Associate or Professional certification.

Experience with Structured Streaming and Azure Event Hubs.

Familiarity with Terraform, Databricks deployment bundles, Azure Data Factory, or dbt.

Experience with Tableau, Databricks SQL, MLflow, or machine learning data pipelines.

Experience working with sensitive enterprise data or in regulated, hospitality, gaming, or similarly complex industries.

Education

Bachelor's degree in Computer Science, Management Information Systems (MIS), or a related field.

An equivalent combination of education and relevant professional experience may be considered.

Relevant industry technical certifications are a plus.

Additional Requirements

Strong organizational skills and attention to detail.

Ability to work independently and exercise sound judgment.

Professional communication and collaboration skills.

Flexibility to accommodate business and project needs.