11-30, 11:35–12:05 (Europe/Amsterdam), Ernst-Curie
Do you find it takes too long to deploy your ML models responsibly in production? Say goodbye to complex integrations of containerized training and inference, preprocessing pipelines, model registries, monitoring frameworks, feature stores, and prediction stores. As demonstrated in this talk, all of these concepts can be easily implemented within dbt using only Python and SQL, the two languages data scientists truly love to write in.
In just a few short years, dbt has taken the data engineering world by storm, becoming the de facto standard for data transformation pipelines. Its primary strength, which endures, is SQL, a ubiquitous programming language shared by data analysts, scientists, and engineers.
However, there are limitations to what SQL can achieve. With the introduction of Python models since version 1.3, data pipelines in dbt have become significantly more expressive, to the extent that you can implement and orchestrate entire batch machine learning pipelines. By harnessing the inherent power of dbt-core, classic SQL, and the richness of Python and its ecosystem, you can finally create a machine learning workflow that is accessible to everyone on your team.
In this talk, we will delve into the dos and don'ts of using Python models in dbt, illustrated through an exemplary daily-running machine learning pipeline aimed at beating football odds provided by bookmakers, utilizing the European Soccer Dataset. By the end of this session, you will have a firm grasp of key design patterns for a successful machine learning project within dbt.
Prerequisites:
- Required: python, SQL, a base understanding of ML
- Optional: dbt, JinJa, MLOps concepts
Previous knowledge expected
Dorian is a Machine Learning Engineer at Dataroots, leveraging his background in Computer Science Engineering and over 5 years of industry experience across various client projects. His forte lies in implementing machine learning models in production while adhering to robust software practices.
Outside the realm of tech, Dorian finds solace and joy in diverse interests. He was a driving force of the AI music band Beatroots, where he explored the fusion of AI and musical creativity. Additionally, he enjoys expressing himself through punk rock songs on his guitar and occasionally blitzes chess moves online.