In the slow process of developing machine learning models, data scientists and data engineers need to work together, yet they often work at cross purposes. As ludicrous as it sounds, Ive seen models take months to get to production because the data scientists were waiting for data engineers to build production systems to suit the model, while the data engineers were waiting for the data scientists to build a model that worked with the production systems.
A previous article by VentureBeat reported that 87% of machine learning projects dont make it into production, and a combination of data concerns and lack of collaboration were primary factors. On the collaboration side, the tension between data engineers and data scientists and how they work together can lead to unnecessary frustration and delays. While team alignment and empathy building can alleviate these tensions, adopting some developing MLOps technologies can help mitigate issues at the root cause.
Before we dive into solutions, lets lay out the problem in more detail. Scientists and engineers (data and otherwise) have always been like cats and dogs, oil and water. A simple web search of scientists vs engineers will lead you to a lengthy debate about which group is more prestigious. Engineers are tasked with construction, operation and maintenance, so they focus on the simplest, most efficient and reliable systems possible. On the other hand, scientists are tasked with doing whatever it takes to build the most accurate models, so they want access to all the data, and they want to manipulate it in unique, sophisticated ways.
Instead of fixating on the differences, I find its much more productive to acknowledge theyre both immensely valuable and to think about how we can use each of their talents to the fullest capacity. By focusing on the things that unify data scientists and data engineers a dedication to timely, quality information and well-designed systems the two sides can foster a more collaborative environment. And by understanding each others pain points, the two teams can build empathy and understanding to make working together easier. There are also emerging tools and systems that can help bridge the gap between these two camps and help them meet more readily in the middle.
MLOps is an emerging area that applies the ideas and principles of DevOps practices to the data science and machine learning ecosystem. It lifts the burden of building and maintenance off of data engineers, while providing flexibility and freedom for data scientists. This is a win-win solution. Lets take a look at some common problems, and the tools that are emerging to more effectively solve them.
Model orchestration. The first major hurdle when trying to put a model into production is deployment: where to build it, how to host it, and how to manage it. This is largely an engineering problem, so when you have a team of data scientists and data engineers, it typically falls to the data engineers.
Building this system takes weeks, if not months time that the data or ML engineers could have spent improving data flows or improving models. Model orchestration platforms standardize model deployment frameworks and help make this step significantly easier. While companies like Facebook can invest resources in platforms like FBLearner to handle model orchestration, this is less feasible for smaller or emerging companies. Thankfully, open source systems have started to emerge to handle the process, namely MLFlow and KubeFlow. Both of these systems use containerization to help manage the infrastructure side of model deployment.
Feature stores. The second major hurdle to taking a model from the lab to production lies with the data. Oftentimes, models are trained using historical data housed in a data warehouse but queried with data from a production database. Discrepancies between these systems cause models to perform poorly or not at all and often require significant data engineering work to re-implement things in the production database.
Ive personally spent weeks building out and prototyping impactful features that never made it to production because the data engineers didnt have the bandwidth to productionize them. Feature stores, or data stores built specifically to support the training and productionization of machine learning models, are working to alleviate this issue by ensuring that data and features built in the lab are immediately production-ready. Data scientists have the peace of mind that their models are getting built, and data engineers dont have to worry about keeping two different systems perfectly in line. Larger corporations like Uber and Airbnb have built their own feature stores (Michelangelo and ZipLine respectively), but vendors that sell pre-built solutions have emerged. Logical Clocks, for example, offers a feature store for its Hopsworks platform. And my team at Kaskada is building a feature store for event-based data.
DataOps. Theres no experience quite like getting paged late at night because your model is behaving strangely. After briefly checking the model service, you come to the inevitable conclusion: something has changed with the data.
Ive had variations on the following conversation more times than I like to admit:
Finding the issue is like finding a needle in a haystack. Fortunately, new frameworks and tools are coming into place that set up monitoring and testing for data and data sources and can save valuable time. Great Expectations is one of these emerging tools to improve how databases are built, documented, and monitored. Databand.ai is another company entering the data pipeline monitoring space; in fact the company published a great blog post herethat explores in greater detail why traditional pipeline monitoring solutions dont work for data engineering and data science.
By using tools to reduce the complexity of asks and by growing empathy and trust between data scientists and data engineers, data scientists can be empowered to deliver without overly burdening data engineers. Both teams can focus on what they do best and what they enjoy about their jobs, instead of fighting with each other. These tools can help turn a combative relationship into a collaborative one where everyone ends up happy.
Max Boyd is a Data Science Lead at Kaskada. He has built and deployed models as a Data Scientist and Machine Learning Engineer at several Seattle-area tech startups in HR, finance and real estate.
Visit link:
How to get your data scientists and data engineers rowing in the same direction - VentureBeat
- Microsoft reveals how it caught mutating Monero mining malware with machine learning - The Next Web [Last Updated On: December 1st, 2019] [Originally Added On: December 1st, 2019]
- The role of machine learning in IT service management - ITProPortal [Last Updated On: December 1st, 2019] [Originally Added On: December 1st, 2019]
- Workday talks machine learning and the future of human capital management - ZDNet [Last Updated On: December 1st, 2019] [Originally Added On: December 1st, 2019]
- Verification In The Era Of Autonomous Driving, Artificial Intelligence And Machine Learning - SemiEngineering [Last Updated On: December 1st, 2019] [Originally Added On: December 1st, 2019]
- Synthesis-planning program relies on human insight and machine learning - Chemical & Engineering News [Last Updated On: December 1st, 2019] [Originally Added On: December 1st, 2019]
- Here's why machine learning is critical to success for banks of the future - Tech Wire Asia [Last Updated On: December 1st, 2019] [Originally Added On: December 1st, 2019]
- The 10 Hottest AI And Machine Learning Startups Of 2019 - CRN: The Biggest Tech News For Partners And The IT Channel [Last Updated On: December 1st, 2019] [Originally Added On: December 1st, 2019]
- Onica Showcases Advanced Internet of Things, Artificial Intelligence, and Machine Learning Capabilities at AWS re:Invent 2019 - PR Web [Last Updated On: December 3rd, 2019] [Originally Added On: December 3rd, 2019]
- Machine Learning Answers: If Caterpillar Stock Drops 10% A Week, Whats The Chance Itll Recoup Its Losses In A Month? - Forbes [Last Updated On: December 3rd, 2019] [Originally Added On: December 3rd, 2019]
- Amazons new AI keyboard is confusing everyone - The Verge [Last Updated On: December 5th, 2019] [Originally Added On: December 5th, 2019]
- Exploring the Present and Future Impact of Robotics and Machine Learning on the Healthcare Industry - Robotics and Automation News [Last Updated On: December 5th, 2019] [Originally Added On: December 5th, 2019]
- 3 questions to ask before investing in machine learning for pop health - Healthcare IT News [Last Updated On: December 5th, 2019] [Originally Added On: December 5th, 2019]
- Amazon Wants to Teach You Machine Learning Through Music? - Dice Insights [Last Updated On: December 5th, 2019] [Originally Added On: December 5th, 2019]
- Measuring Employee Engagement with A.I. and Machine Learning - Dice Insights [Last Updated On: December 6th, 2019] [Originally Added On: December 6th, 2019]
- The NFL And Amazon Want To Transform Player Health Through Machine Learning - Forbes [Last Updated On: December 11th, 2019] [Originally Added On: December 11th, 2019]
- Scientists are using machine learning algos to draw maps of 10 billion cells from the human body to fight cancer - The Register [Last Updated On: December 11th, 2019] [Originally Added On: December 11th, 2019]
- Appearance of proteins used to predict function with machine learning - Drug Target Review [Last Updated On: December 11th, 2019] [Originally Added On: December 11th, 2019]
- Google is using machine learning to make alarm tones based on the time and weather - The Verge [Last Updated On: December 11th, 2019] [Originally Added On: December 11th, 2019]
- 10 Machine Learning Techniques and their Definitions - AiThority [Last Updated On: December 11th, 2019] [Originally Added On: December 11th, 2019]
- Taking UX and finance security to the next level with IBM's machine learning - The Paypers [Last Updated On: December 12th, 2019] [Originally Added On: December 12th, 2019]
- Government invests 49m in data analytics, machine learning and AI Ireland, news for Ireland, FDI,Ireland,Technology, - Business World [Last Updated On: December 12th, 2019] [Originally Added On: December 12th, 2019]
- Machine Learning Answers: If Nvidia Stock Drops 10% A Week, Whats The Chance Itll Recoup Its Losses In A Month? - Forbes [Last Updated On: December 12th, 2019] [Originally Added On: December 12th, 2019]
- Bing: To Use Machine Learning; You Have To Be Okay With It Not Being Perfect - Search Engine Roundtable [Last Updated On: December 12th, 2019] [Originally Added On: December 12th, 2019]
- IQVIA on the adoption of AI and machine learning - OutSourcing-Pharma.com [Last Updated On: December 12th, 2019] [Originally Added On: December 12th, 2019]
- Schneider Electric Wins 'AI/ Machine Learning Innovation' and 'Edge Project of the Year' at the 2019 SDC Awards - PRNewswire [Last Updated On: December 12th, 2019] [Originally Added On: December 12th, 2019]
- Industry Call to Define Universal Open Standards for Machine Learning Operations and Governance - MarTech Series [Last Updated On: December 12th, 2019] [Originally Added On: December 12th, 2019]
- Qualitest Acquires AI and Machine Learning Company AlgoTrace to Expand Its Offering - PRNewswire [Last Updated On: December 12th, 2019] [Originally Added On: December 12th, 2019]
- Automation And Machine Learning: Transforming The Office Of The CFO - Forbes [Last Updated On: December 12th, 2019] [Originally Added On: December 12th, 2019]
- Machine learning results: pay attention to what you don't see - STAT [Last Updated On: December 12th, 2019] [Originally Added On: December 12th, 2019]
- The challenge in Deep Learning is to sustain the current pace of innovation, explains Ivan Vasilev, machine learning engineer - Packt Hub [Last Updated On: December 15th, 2019] [Originally Added On: December 15th, 2019]
- Israelis develop 'self-healing' cars powered by machine learning and AI - The Jerusalem Post [Last Updated On: December 15th, 2019] [Originally Added On: December 15th, 2019]
- Theres No Such Thing As The Machine Learning Platform - Forbes [Last Updated On: December 15th, 2019] [Originally Added On: December 15th, 2019]
- Global Contextual Advertising Markets, 2019-2025: Advances in AI and Machine Learning to Boost Prospects for Real-Time Contextual Targeting -... [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- Machine Learning Answers: If Twitter Stock Drops 10% A Week, Whats The Chance Itll Recoup Its Losses In A Month? - Forbes [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- Tech connection: To reach patients, pharma adds AI, machine learning and more to its digital toolbox - FiercePharma [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- Machine Learning Answers: If Seagate Stock Drops 10% A Week, Whats The Chance Itll Recoup Its Losses In A Month? - Forbes [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- MJ or LeBron Who's the G.O.A.T.? Machine Learning and AI Might Give Us an Answer - Built In Chicago [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- Amazon Releases A New Tool To Improve Machine Learning Processes - Forbes [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- AI and machine learning platforms will start to challenge conventional thinking - CRN.in [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- What is Deep Learning? Everything you need to know - TechRadar [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- Machine Learning Answers: If BlackBerry Stock Drops 10% A Week, Whats The Chance Itll Recoup Its Losses In A Month? - Forbes [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- QStride to be acquired by India-based blockchain, analytics, machine learning consultancy - Staffing Industry Analysts [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- Dotscience Forms Partnerships to Strengthen Machine Learning - Database Trends and Applications [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- The Machines Are Learning, and So Are the Students - The New York Times [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- Kubernetes and containers are the perfect fit for machine learning - JAXenter [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- Data science and machine learning: what to learn in 2020 - Packt Hub [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- What is Machine Learning? A definition - Expert System [Last Updated On: December 20th, 2019] [Originally Added On: December 20th, 2019]
- Want to dive into the lucrative world of deep learning? Take this $29 class. - Mashable [Last Updated On: December 24th, 2019] [Originally Added On: December 24th, 2019]
- Another free web course to gain machine-learning skills (thanks, Finland), NIST probes 'racist' face-recog and more - The Register [Last Updated On: December 24th, 2019] [Originally Added On: December 24th, 2019]
- TinyML as a Service and machine learning at the edge - Ericsson [Last Updated On: December 24th, 2019] [Originally Added On: December 24th, 2019]
- Machine Learning in 2019 Was About Balancing Privacy and Progress - ITPro Today [Last Updated On: December 24th, 2019] [Originally Added On: December 24th, 2019]
- Ten Predictions for AI and Machine Learning in 2020 - Database Trends and Applications [Last Updated On: December 25th, 2019] [Originally Added On: December 25th, 2019]
- The Value of Machine-Driven Initiatives for K12 Schools - EdTech Magazine: Focus on Higher Education [Last Updated On: December 25th, 2019] [Originally Added On: December 25th, 2019]
- CMSWire's Top 10 AI and Machine Learning Articles of 2019 - CMSWire [Last Updated On: December 25th, 2019] [Originally Added On: December 25th, 2019]
- Machine Learning Market Accounted for US$ 1,289.5 Mn in 2016 and is expected to grow at a CAGR of 49.7% during the forecast period 2017 2025 - The... [Last Updated On: December 27th, 2019] [Originally Added On: December 27th, 2019]
- Are We Overly Infatuated With Deep Learning? - Forbes [Last Updated On: December 27th, 2019] [Originally Added On: December 27th, 2019]
- Can machine learning take over the role of investors? - TechHQ [Last Updated On: December 27th, 2019] [Originally Added On: December 27th, 2019]
- Dr. Max Welling on Federated Learning and Bayesian Thinking - Synced [Last Updated On: December 28th, 2019] [Originally Added On: December 28th, 2019]
- 2010 2019: The rise of deep learning - The Next Web [Last Updated On: January 4th, 2020] [Originally Added On: January 4th, 2020]
- Machine Learning Answers: Sprint Stock Is Down 15% Over The Last Quarter, What Are The Chances It'll Rebound? - Trefis [Last Updated On: January 4th, 2020] [Originally Added On: January 4th, 2020]
- Sports Organizations Using Machine Learning Technology to Drive Sponsorship Revenues - Sports Illustrated [Last Updated On: January 4th, 2020] [Originally Added On: January 4th, 2020]
- What is deep learning and why is it in demand? - Express Computer [Last Updated On: January 4th, 2020] [Originally Added On: January 4th, 2020]
- Byrider to Partner With PointPredictive as Machine Learning AI Partner to Prevent Fraud - CloudWedge [Last Updated On: January 4th, 2020] [Originally Added On: January 4th, 2020]
- Stare into the mind of God with this algorithmic beetle generator - SB Nation [Last Updated On: January 5th, 2020] [Originally Added On: January 5th, 2020]
- US announces AI software export restrictions - The Verge [Last Updated On: January 5th, 2020] [Originally Added On: January 5th, 2020]
- How AI And Machine Learning Can Make Forecasting Intelligent - Demand Gen Report [Last Updated On: January 5th, 2020] [Originally Added On: January 5th, 2020]
- Fighting the Risks Associated with Transparency of AI Models - EnterpriseTalk [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- NXP Debuts i.MX Applications Processor with Dedicated Neural Processing Unit for Advanced Machine Learning at the Edge - GlobeNewswire [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- Cerner Expands Collaboration with Amazon Web as its Preferred Machine Learning Provider - Story of Future [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- Can We Do Deep Learning Without Multiplications? - Analytics India Magazine [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- Machine learning is innately conservative and wants you to either act like everyone else, or never change - Boing Boing [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- Pear Therapeutics Expands Pipeline with Machine Learning, Digital Therapeutic and Digital Biomarker Technologies - Business Wire [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- FLIR Systems and ANSYS to Speed Thermal Camera Machine Learning for Safer Cars - Business Wire [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- SiFive and CEVA Partner to Bring Machine Learning Processors to Mainstream Markets - PRNewswire [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- Tiny Machine Learning On The Attiny85 - Hackaday [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- Finally, a good use for AI: Machine-learning tool guesstimates how well your code will run on a CPU core - The Register [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- AI, machine learning, and other frothy tech subjects remained overhyped in 2019 - Boing Boing [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- Chemists are training machine learning algorithms used by Facebook and Google to find new molecules - News@Northeastern [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- AI and machine learning trends to look toward in 2020 - Healthcare IT News [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]
- What Is Machine Learning? | How It Works, Techniques ... [Last Updated On: January 7th, 2020] [Originally Added On: January 7th, 2020]