Software Engineer - Lakeflow PhD Candidates

Company: Databricks Inc.
Location: Mountain View
Posted on: May 17, 2025

Job Description:

Job Description: Data Engineer - Lakeflow Team at DatabricksDatabricks is transforming the data lifecycle, from ingestion to generative AI, across multiple clouds with a unified platform. We serve over 10,000 customers, processing exabytes of data daily on more than 15 million VMs, and our growth continues rapidly.The Lakeflow team is seeking recent PhD graduates to work on products like Apache Spark Structured Streaming, Delta Live Tables (DLT), and Materialized Views. Structured Streaming is one of the world's leading streaming engines. DLT simplifies the development and management of reliable batch and streaming data pipelines, providing high-quality data on the Databricks Lakehouse Platform. It streamlines ETL processes with declarative pipeline development, automatic data testing, and comprehensive monitoring and recovery features. DLT also enhances pipeline execution through logical and physical optimizations, including instance type selection and autoscaling.Additionally, we have developed Enzyme, a new catalyst optimization layer designed to accelerate ETL processes by enabling incremental computation and materialization of intermediate results. Enzyme maintains up-to-date materializations of query results stored in Delta tables by employing a cost model that chooses between various optimization techniques inspired by traditional materialized view maintenance, delta streaming, and common ETL patterns.As part of the Lakeflow DLT team, you'll have opportunities to innovate in areas such as:

• Query compilation and optimization
• Distributed query execution and scheduling
• Resource management
• Transaction coordination
• Automatic physical data optimizationWhat We Look For:
  • PhD in databases or systems
  • Strong knowledge of database systems, storage solutions, distributed systems, and performance tuning
  • Motivation to deliver value to customers and influence product developmentDatabricks offers fair and competitive compensation, with a local pay range of $142,200 to $204,600 USD. Compensation is determined based on skills, experience, certifications, and location, and may include bonuses, equity, and benefits. For more details, visit our benefits page.About Databricks:Databricks is a leading data and AI company trusted by over 10,000 organizations worldwide, including Fortune 500 companies. We are headquartered in San Francisco, founded by creators of Lakehouse, Apache Spark, Delta Lake, and MLflow. Follow us on Twitter, LinkedIn, and Facebook.Benefits and Diversity:We provide comprehensive benefits tailored to regional needs. We are committed to fostering a diverse and inclusive culture, ensuring equal opportunity for all applicants regardless of background or characteristics.Compliance:Access to export-controlled technology or source code may require a U.S. government license, and we reserve discretion in applying for such licenses.
    #J-18808-Ljbffr

Keywords: Databricks Inc., Berkeley , Software Engineer - Lakeflow PhD Candidates, IT / Software / Systems , Mountain View, California