Fundamentals of Data Observability

Andy Petrella has been in the data industry for almost 20 years, starting his career as a software engineer and data miner in the GIS space. He has evangelized big data for more than a decade, especially Apache Spark for which he created the Spark-Notebook. During his time evangelizing Spark and helping hundreds of companies in the US and in EU work on their data pipelines and models, he has witnessed the lack of visibility and control of data jobs after they are deployed in production. Since 2015, he has been talking to tech and data-savvy people to build a sustainable solution for this problem: “how to make data observable” in a way that can be adopted smoothly by any data practitioner. Today, he is regularly invited to companies to educate their data teams, whilst running Kensu, which has more than 50 years of total development time dedicated to building the set tools to help data engineers and their peers to build trust in what they deliver. Also he is in ongoing talks with advocates such as Gartner to create a definition of Data Observability that refers to all its important facets. Finally, he has written books, blogs, slides, training materials, etc. since 2013, including many materials with O’Reilly.