Create And Scale Complex Artificial Intelligence And Machine Learning Pipelines Anywhere With IBM CodeFlare – Forbes

Pixabay

To say that AI is complicated is an understatement. Machine learning, a subset of artificial intelligence, is a multifaceted process that integrates and scales mountains of data that comes in different forms from various sources. Data is used to train machine learning models in order to develop insights and solutions from newly acquired related data. For example, an image recognition model trained with several million dog and cat photos can efficiently classify a new image as either a cat or a dog.

A better way to build and manage machine learning models

Project Codeflare

The development of machine learning models requires the coordination of many processes linked together with pipelines. Pipelines can handle data ingestion, scrubbing, and manipulation from varied sources for training and inference.Machine learning models use end-to-end pipelines to manage input and output data collection and processing.

To deal with the extraordinary growth of AI and its ever-increasing complexity, IBM created an open-source framework calledCodeFlareto deal with AIs complex pipeline requirements. CodeFlaresimplifies the integration, scaling, and acceleration of complex multi-step analytics and machine learning pipelines on the cloud.Hybrid cloud deployment is one of the critical design points for CodeFlare, which using OpenShift can be easily deployed from on-premises to public clouds to edge.

It is important to note thatCodeFlare is not currently a generally available product, and IBM has yet to commit to a timeline for it becoming a product. Nevertheless, CodeFlare is available as an open-source project.And, as an evolving project, some aspects of orchestration and automation are still work in progress. At this stage, issues can be reported through the public GitHub project. IBM invites community engagement through issue and bug reports, which will be handled on a best effort basis.

CodeFlares main features are:

Technology

CodeFlare is built on top of Ray, an open-source distributed computing framework for machine learning applications. According to IBM, CodeFlare extends the capabilities of Ray by adding specific elements to make scaling workflows easier. CodeFlare pipelines run on a serverless platform using IBM Cloud Code Engine and Red Hat OpenShift. This platform providesCodeFlare the flexibility to be deployed just about anywhere.

Emerging workflows

Emerging AI/ML workflows pose new challenges

CodeFlare can integrate emerging workflows with complex pipelines that require integration and coordination of different tools and runtimes. It is designed also to scale complex pipelines such as multi-step NLP, complex time series and forecasting, reinforcement learning, and AI-Workbenches. The framework can integrate, run, and scale heterogenous pipelines that use data from multiple sources and require different treatments.

How much difference does CodeFlare make?

According to theIBM Research blog, CodeFlare significantly increases the efficiency of machine learning. The blog states that a user used the framework to analyze and optimize approximately 100,000 pipelines for training machine learning models. CodeFlare cut the time it took to execute each pipeline from 4 hours to 15 minutes - an 18x speedup provided by CodeFlare.

The research blog also indicates that CodeFlare can save scientists months of work on large pipelines, providing the data team more time for productive and development work.

Wrapping up

Studies show that about75%of prototype machine learning models fail to transition to production status despite large investments in artificial intelligence. Several reasons for low conversion rates range from poor project planning to weak collaboration and communications between AI data team members.

CodeFlare is a purpose-built platform that provides complete end-to-end pipeline visibility and analytics for a broad range of machine learning models and workflows. It providesa more straightforward way to integrate and scale full pipelines while offering a unified runtime and programming interface.

For those reasons, despite the historical high AI model failure rates, Moor Insights & Strategy believes that machine learning models using CodeFlare pipelines will have a high percentage of machine learning models transition from experimental status to production status.

Analyst Notes:

Note: Moor Insights & Strategy writers and editors may have contributed to this article.

Moor Insights & Strategy, like all research and analyst firms, provides or has provided paid research, analysis, advising, or consulting to many high-tech companies in the industry, including 8x8, Advanced Micro Devices, Amazon, Applied Micro, ARM, Aruba Networks, AT&T, AWS, A-10 Strategies,Bitfusion, Blaize, Box, Broadcom, Calix, Cisco Systems, Clear Software, Cloudera,Clumio, Cognitive Systems, CompuCom, Dell, Dell EMC, Dell Technologies, Diablo Technologies, Digital Optics,Dreamchain, Echelon, Ericsson, Extreme Networks, Flex, Foxconn, Frame (now VMware), Fujitsu, Gen Z Consortium, Glue Networks, GlobalFoundries, Google (Nest-Revolve), Google Cloud, HP Inc., Hewlett Packard Enterprise, Honeywell, Huawei Technologies, IBM, Ion VR,Inseego, Infosys, Intel, Interdigital, Jabil Circuit, Konica Minolta, Lattice Semiconductor, Lenovo, Linux Foundation,MapBox, Marvell,Mavenir, Marseille Inc, Mayfair Equity, Meraki (Cisco),Mesophere, Microsoft, Mojo Networks, National Instruments, NetApp, Nightwatch, NOKIA (Alcatel-Lucent), Nortek,Novumind, NVIDIA, Nuvia, ON Semiconductor, ONUG, OpenStack Foundation, Oracle, Poly, Panasas,Peraso, Pexip, Pixelworks, Plume Design, Poly,Portworx, Pure Storage, Qualcomm, Rackspace, Rambus,RayvoltE-Bikes, Red Hat,Residio, Samsung Electronics, SAP, SAS, Scale Computing, Schneider Electric, Silver Peak, SONY,Springpath, Spirent, Splunk, Sprint, Stratus Technologies, Symantec, Synaptics, Syniverse, Synopsys, Tanium, TE Connectivity,TensTorrent,TobiiTechnology, T-Mobile, Twitter, Unity Technologies, UiPath, Verizon Communications,Vidyo, VMware, Wave Computing,Wellsmith, Xilinx, Zebra,Zededa, and Zoho which may be cited in blogs and research.

See original here:

Create And Scale Complex Artificial Intelligence And Machine Learning Pipelines Anywhere With IBM CodeFlare - Forbes