Daily Archives: September 20, 2021

The Transform Technology Summits start October 13th with Low-Code/No Code: Enabling Enterprise Agility. Register now!

When it comes to AI, algorithmic innovations are substantially more important than hardware at least where the problems involve billions to trillions of data points. Thats the conclusion of a team of scientists at MITs Computer Science and Artificial Intelligence Laboratory (CSAIL), who conducted what they claim is the first study on how fast algorithms are improving across a broad range of examples.

Algorithms tell software how to make sense of text, visual, and audio data so that they can, in turn, draw inferences from it. For example, OpenAIs GPT-3 was trained on webpages, ebooks, and other documents to learn how to write papers in a humanlike way. The more efficient the algorithm, the less work the software has to do. And as algorithms are enhanced, less computing power should be needed in theory. But this isnt settled science. AI research and infrastructure startups like OpenAI and Cerberus are betting that algorithms will have to increase in size substantially to reach higher levels of sophistication.

The CSAIL team, led by MIT research scientist Neil Thompson, who previously coauthored a paper showing that algorithms were approaching the limits of modern computing hardware, analyzed data from 57 computer science textbooks and more than 1,110 research papers to trace the history of where algorithms improved. In total, they looked at 113 algorithm families, or sets of algorithms that solved the same problem, that had been highlighted as most important by the textbooks.

The team reconstructed the history of the 113, tracking each time a new algorithm was proposed for a problem and making special note of those that were more efficient. Starting from the 1940s to now, the team found an average of eight algorithms per family of which a couple improved in efficiency.

For large computing problems, 43% of algorithm families had year-on-year improvements that were equal to or larger than the gains from Moores law, the principle that the speed of computers roughly doubles every two years. In 14% of problems, the performance improvements vastly outpaced those that came from improved hardware, with the gains from better algorithms being particularly meaningful for big data problems.

The new MIT study adds to a growing body of evidence that the size of algorithms matters less than their architectural complexity. For example, earlier this month, a team of Google researchers published a study claiming that a model much smaller than GPT-3 fine-tuned language net (FLAN) bests GPT-3 by a large margin on a number of challenging benchmarks. And in a 2020 survey, OpenAI found that since 2012, the amount of compute needed to train an AI model to the same performance on classifying images in a popular benchmark, ImageNet, has been decreasing by a factor of two every 16 months.

Theres findings to the contrary. In 2018, OpenAI researchers released a separate analysis showing that from 2012 to 2018, the amount of compute used in the largest AI training runs grew more than 300,000 times with a 3.5-month doubling time, exceeding the pace of Moores law. But assuming algorithmic improvements receive greater attention in the years to come, they could solve some of the other problems associated with large language models, like environmental impact and cost.

In June 2020, researchers at the University of Massachusetts at Amherst released a report estimating that the amount of power required for training and searching a certain model involves the emissions of roughly 626,000 pounds of carbon dioxide, equivalent to nearly 5 times the lifetime emissions of the average U.S. car. GPT-3 alone used 1,287 megawatts during training and produced 552 metric tons of carbon dioxide emissions, a Google study found the same amount emitted by 100 average homes electricity usage over a year.

On the expenses side, a Synced report estimated that the University of Washingtons Grover fake news detection model cost $25,000 to train; OpenAI reportedly racked up $12 million training GPT-3; and Google spent around $6,912 to train BERT. While AI training costs dropped 100-fold between 2017 and 2019, according to one source, these amounts far exceed the computing budgets of most startups and institutions let alone independent researchers.

Through our analysis, we were able to say how many more tasks could be done using the same amount of computing power after an algorithm improved, Thompson said in a press release. In an era where the environmental footprint of computing is increasingly worrisome, this is a way to improve businesses and other organizations without the downside.

Read more from the original source:

Improved algorithms may be more important for AI performance than faster hardware - VentureBeat

Welcome toAI book reviews, a series of posts that explore the latest literature on artificial intelligence.

Recent advances in deep learning have rekindled interest in the imminence of machines that can think and act like humans, or artificial general intelligence. By following the path of building bigger and better neural networks, the thinking goes, we will be able to get closer and closer to creating a digital version of the human brain.

But this is a myth, argues computer scientist Erik Larson, and all evidence suggests that human and machine intelligence are radically different. Larsons new book, The Myth of Artificial Intelligence: Why Computers Cant Think the Way We Do, discusses how widely publicized misconceptions about intelligence and inference have led AI research down narrow paths that are limiting innovation and scientific discoveries.

And unless scientists, researchers, and the organizations that support their work dont change course, Larson warns, they will be doomed to resignation to the creep of a machine-land, where genuine invention is sidelined in favor of futuristic talk advocating current approaches, often from entrenched interests.

From a scientific standpoint, the myth of AI assumes that we will achieve artificial general intelligence (AGI) by making progress on narrow applications, such as classifying images, understanding voice commands, or playing games. But the technologies underlying these narrow AI systems do not address the broader challenges that must be solved for general intelligence capabilities, such as holding basic conversations, accomplishing simple chores in a house, or other tasks that require common sense.

As we successfully apply simpler, narrow versions of intelligence that benefit from faster computers and lots of data, we are not making incremental progress, but rather picking the low-hanging fruit, Larson writes.

The cultural consequence of the myth of AI is ignoring the scientific mystery of intelligence and endlessly talking about ongoing progress on deep learning and other contemporary technologies. This myth discourages scientists from thinking about new ways to tackle the challenge of intelligence.

We are unlikely to get innovation if we choose to ignore a core mystery rather than face it up, Larson writes. A healthy culture for innovation emphasizes exploring unknowns, not hyping extensions of existing methods Mythology about inevitable success in AI tends to extinguish the very culture of invention necessary for real progress.

You step out of your home and notice that the street is wet. Your first thought is that it must have been raining. But its sunny and the sidewalk is dry, so you immediately cross out the possibility of rain. As you look to the side, you see a road wash tanker parked down the street. You conclude that the road is wet because the tanker washed it.

This is an example inference, the act of going from observations to conclusions, and is the basic function of intelligent beings. Were constantly inferring things based on what we know and what we perceive. Most of it happens subconsciously, in the background of our mind, without focus and direct attention.

Any system that infers must have some basic intelligence, because the very act of using what is known and what is observed to update beliefs is inescapably tied up with what we mean by intelligence, Larson writes.

AI researchers base their systems on two types of inference machines: deductive and abductive. Deductive inference uses prior knowledge to reason about the world. This is the basis of symbolic artificial intelligence, the main focus of researchers in the early decades of AI. Engineers create symbolic systems by endowing them with a predefined set of rules and facts, and the AI uses this knowledge to reason about the data it receives.

Inductive inference, which has gained more traction among AI researchers and tech companies in the past decade, is the acquisition of knowledge through experience. Machine learning algorithms are inductive inference engines. An ML model trained on relevant examples will find patterns that map inputs to outputs. In recent years, AI researchers have used machine learning, big data, and advanced processors to train models on tasks that were beyond the capacity of symbolic systems.

A third type of reasoning, abductive inference, was first introduced by American scientist Charles Sanders Peirce in the 19th century. Abductive inference is the cognitive ability to come up with intuitions and hypotheses, to make guesses that are better than random stabs at the truth.

For example, there can be numerous reasons for the street to be wet (including some that we havent directly experienced before), but abductive inference enables us to select the most promising hypotheses, quickly eliminate the wrong ones, look for new ones and reach a reliable conclusion. As Larson puts it in The Myth of Artificial Intelligence, We guess, out of a background of effectively infinite possibilities, which hypotheses seem likely or plausible.

Abductive inference is what many refer to as common sense. It is the conceptual framework within which we view facts or data and the glue that brings the other types of inference together. It enables us to focus at any moment on whats relevant among the ton of information that exists in our mind and the ton of data were receiving through our senses.

The problem is that the AI community hasnt paid enough attention to abductive inference.

Abduction entered the AI discussion with attempts at Abductive Logic Programming in the 1980s and 1990s, but those efforts were flawed and later abandoned. They were reformulations of logic programming, which is a variant of deduction, Larson told TechTalks.

Abduction got another chance in the 2010s as Bayesian networks, inference engines that try to compute causality. But like the earlier approaches, the newer approaches shared the flaw of not capturing true abduction, Larson said, adding that Bayesian and other graphical models are variants of induction. In The Myth of Artificial Intelligence, he refers to them as abduction in name only.

For the most part, the history of AI has been dominated by deduction and induction.

When the early AI pioneers like [Alan] Newell, [Herbert] Simon, [John] McCarthy, and [Marvin] Minsky took up the question of artificial inference (the core of AI), they assumed that writing deductive-style rules would suffice to generate intelligent thought and action, Larson said. That was never the case, really, as should have been earlier acknowledged in discussions about how we do science.

For decades, researchers tried to expand the powers of symbolic AI systems by providing them with manually written rules and facts. The premise was that if you endow an AI system with all the knowledge that humans know, it will be able to act as smartly as humans. But pure symbolic AI has failed for various reasons. Symbolic systems cant acquire and add new knowledge, which makes them rigid. Creating symbolic AI becomes an endless chase of adding new facts and rules only to find the system making new mistakes that it cant fix. And much of our knowledge is implicit and cannot be expressed in rules and facts and fed to symbolic systems.

Its curious here that no one really explicitly stopped and said Wait. This is not going to work! Larson said. That would have shifted research directly towards abduction or hypothesis generation or, say, context-sensitive inference.

In the past two decades, with the growing availability of data and compute resources, machine learning algorithmsespecially deep neural networkshave become the focus of attention in the AI community. Deep learning technology has unlocked many applications that were previously beyond the limits of computers. And it has attracted interest and money from some of the wealthiest companies in the world.

I think with the advent of the World Wide Web, the empirical or inductive (data-centric) approaches took over, and abduction, as with deduction, was largely forgotten, Larson said.

But machine learning systems also suffer from severe limits, including the lack of causality, poor handling of edge cases, and the need for too much data. And these limits are becoming more evident and problematic as researchers try to apply ML to sensitive fields such as healthcare and finance.

Some scientists, including reinforcement learning pioneer Richard Sutton, believe that we should stick to methods that can scale with the availability of data and computation, namely learning and search. For example, as neural networks grow bigger and are trained on more data, they will eventually overcome their limits and lead to new breakthroughs.

Larson dismisses the scaling up of data-driven AI as fundamentally flawed as a model for intelligence. While both search and learning can provide useful applications, they are based on non-abductive inference, he reiterates.

Search wont scale into commonsense or abductive inference without a revolution in thinking about inference, which hasnt happened yet. Similarly with machine learning, the data-driven nature of learning approaches means essentially that the inferences have to bein the data, so to speak, and thats demonstrably not true of many intelligent inferences thatpeople routinelyperform, Larson said. We dont just look to the past, captured, say, in a large dataset, to figure out what to conclude or think or infer about the future.

Other scientists believe that hybrid AI that brings together symbolic systems and neural networks will have a bigger promise of dealing with the shortcomings of deep learning. One example is IBM Watson, which became famous when it beat world champions at Jeopardy! More recent proof-of-concept hybrid models have shown promising results in applications where symbolic AI and deep learning alone perform poorly.

Larson believes that hybrid systems can fill in the gaps in machine learningonly or rules-basedonly approaches. As a researcher in the field of natural language processing, he is currently working on combining large pre-trained language models like GPT-3 with older work on the semantic web in the form of knowledge graphs to create better applications in search, question answering, and other tasks.

But deduction-induction combos dont get us to abduction, because the three types of inference are formally distinct, so they dont reduce to each other and cant be combined to get a third, he said.

In The Myth of Artificial Intelligence, Larson describes attempts to circumvent abduction as the inference trap.

Purely inductively inspired techniques like machine learning remain inadequate, no matter how fast computers get, and hybrid systems like Watson fall short of general understanding as well, he writes. In open-ended scenarios requiring knowledge about the world like language understanding, abduction is central and irreplaceable. Because of this, attempts at combining deductive and inductive strategies are always doomed to fail The field needs a fundamental theory of abduction. In the meantime, we are stuck in traps.

The AI communitys narrow focus on data-driven approaches has centralized research and innovation in a few organizations that have vast stores of data and deep pockets. With deep learning becoming a useful way to turn data into profitable products, big tech companies are now locked in a tight race to hire AI talent, driving researchers away from academia by offering them lucrative salaries.

This shift has made it very difficult for non-profit labs and small companies to become involved in AI research.

When you tie research and development in AI to the ownership and control of very large datasets, you get a barrier to entry for start-ups, who dont own the data, Larson said, adding that data-driven AI intrinsically creates winner-take-all scenarios in the commercial sector.

The monopolization of AI is in turn hampering scientific research. With big tech companies focusing on creating applications in which they can leverage their vast data resources to maintain the edge over their competitors, theres little incentive to explore alternative approaches to AI. Work in the field starts to skew toward narrow and profitable applications at the expense of efforts that can lead to new inventions.

No one at present knows how AI would look in the absence of such gargantuan centralized datasets, so theres nothing really on offer for entrepreneurs looking to compete by designing different and more powerful AI, Larson said.

In his book, Larson warns about the current culture of AI, which is squeezing profits out of low-hanging fruit, while continuing to spin AI mythology. The illusion of progress on artificial general intelligence can lead to another AI winter, he writes.

But while an AI winter might dampen interest in deep learning and data-driven AI, it can open the way for a new generation of thinkers to explore new pathways. Larson hopes scientists start looking beyond existing methods.

In The Myth of Artificial Intelligence, Larson provides an inference framework that sheds light on the challenges that the field faces today and helps readers to see through the overblown claims about progress toward AGI or singularity.

My hope is that non-specialists have some tools to combat this kind of inevitability thinking, which isnt scientific, and that my colleagues and other AI scientists can view it as a wake-up call to get to work on the very real problems the field faces, Larson said.

View original post here:

Abductive inference: The blind spot of artificial intelligence - TechTalks

Thesurveycommittee, which receives input from a host of smaller panels, takes into account a gargantuan amount of information to create research strategies. Although the Academies wont release the committees final recommendation to NASA for a few more weeks, scientists are itching to know which of their questions will make it in, and which will be left out.

The Decadal Survey really helps NASA decide howtheyregoing to lead the future of human discovery in space, soitsreally important thattheyrewell informed, saysBrant Robertson, a professor of astronomy and astrophysics at UC Santa Cruz.

One teamof researcherswants to useartificial intelligenceto make this process easier. Their proposal isnt for a specific mission or line of questioning; rather, they say,their AI can help scientists make tough decisions about which other proposals to prioritize.

The idea is that by training an AI to spot research areas that are either growing or declining rapidly, the tool could make it easier for survey committees and panels to decide what should make the list.

What we wanted was to have a system that would do a lot of the work that the Decadal Survey does, and let the scientists working on the Decadal Survey do what they will do best, saysHarley Thronson, a retired senior scientist at NASAs Goddard Space Flight Center and lead authorof the proposal.

Although members of each committee are chosen for their expertise in their respective fields,itsimpossible for every member to grasp the nuance of every scientific theme. The number of astrophysics publications increases by 5%every year, according to the authors. Thats a lot for anyone to process.

Thats where Thronsons AI comes in.

It took just over a year to build, but eventually, Thronsons team was able to train it on more than 400,000 pieces of research published in the decade leading up to the Astro2010 survey. They were also able to teach the AI to sift through thousands of abstracts toidentifyboth low-and high-impact areasfromtwo-and three-word topic phrases likeplanetary systemorextrasolar planet.

According to theresearcherswhitepaper, the AI successfullybackcastedsix popular research themesofthe last 10 years, including a meteoric rise in exoplanet research and observation of galaxies.

One of the challenging aspects of artificial intelligence is that they sometimes will predict, or come up with, or analyze things that are completely surprising to the humans, says Thronson. And we saw this a lot.

Thronson and his collaborators think the steering committee should use their AI to help review and summarize the vast amounts of text the panel must sift through, leaving human experts to makethe final call.

Their research isnt the first to try to use AI to analyze and shape scientific literature. Other AIs have already been usedto help scientistspeer-reviewtheircolleagueswork.

But could it be trusted with a task as important and influential as the DecadalSurvey?

Read the rest here:

This AI could predict 10 years of scientific prioritiesif we let it - MIT Technology Review

Venture Capital concept image with business icons and copyspace.For more variation of this image ... [+] please visit my portfolio

According to data from PitchBook, the funding for AI deals has continued its furious pace.In the latest quarter, the amount invested came to a record $31.6 billion.Note that there were 11 deals the closed more than $500 million.

Granted, plenty of these startups will fade away or even go bust.But of course, some will ultimately disrupt industries and change the landscape of the global economy.

To be disrupted, you have to believe the AI is going to make 10x better recommendations than whats available today, said Eric Vishria, who is a General Partner at Benchmark.I think that is likely to happen in really complex, high dimensional spaces, where there are so many intermingled factors at play that finding correlations via standard analytical techniques is really difficult.

So then what are some of the industries that are vulnerable to AI disruption?Well, lets see where some of the top VCs are investing today:

Software Development:There have been advances in DevOps and IDEs.Yet software development remains labor intensive.And it does not help that its extremely difficult to recruit qualified developers.

But AI can make a big difference.Advancements in state-of-the-art natural language processing algorithms could revolutionize software development, initially by significantly reducing the boilerplate code that software developers write today and in the long-run by writing entire applications with little assistance from humans, said Nnamdi Iregbulem, who is a Partner at Lightspeed Venture Partners.

Consider the use of GPT-3, which is a neural network that trains models to create content.Products like GitHub Copilot, which are also based on GPT-3, will also disrupt software development, said Jai Das, who is the President and Partner at Sapphire Ventures.

Cybersecurity:This is one of the biggest software markets.But the technologies really need retooling.After all, there continues to be more and more breaches and hacks.

Cybersecurity is likely to turn into an AI-vs-AI game very soon, said Deepak Jeevankumar, who is a Managing Director at Dell Technologies Capital.Sophisticated attackers are already using AI and bots to get over defenses.

Construction:This is a massive industry and will continue to grow, as the global population continues to increase.Yet construction has seen relatively small amounts of IT investment.But AI could be a game changer.

An incremental 1% increase in efficiency can mean millions of dollars in cost savings, said Shawn Carolan, who is a Managing Partner at Menlo Ventures.There are many companies, like Openspace.ai, doing transformative work using AI in the construction space. Openspace leverages AI and machine vision to essentially become a photographic memory for job sites. It automatically uploads and stitches together images of a job site so that customers can do a virtual walk-through and monitor the project at any time.

Talent Management:HR has generally lagged with innovation.The fact is that many of the processes are manual and inefficient.

But AI can certainly be a solution. In fact, AI startups like Eightfold.ai have been able to post substantial growth in the HR category.In June, the company announced funding of $220 million, which was led by the SoftBank Vision Fund 2.

Every single company is talking about talent as a key priority, and the companies that embrace AI to find better candidates faster, cheaper, at scale, they have a true competitive advantage, said Kirthiga Reddy, who is a Partner at SoftBank.Understanding how to use AI to amplify the interactions in the talent lifecycle is a differentiator and advantage for these businesses."

Drug Discovery: The development of the Covid-19 vaccinesfrom companies like Pfizer, Moderna and BioNTechhas highlighted the power of innovation in the healthcare industry.But despite this, there is still much be done.The fact is that drug development is costly and time-consuming.

It's becoming impossible to process these large datasets without using the latest AI/ML technologies, said Dusan Perovic, who is a partner at Two Sigma Ventures.Companies that are early adopters of these data science tools and thereby are able to analyze larger datasets are going to make faster progress than companies that rely on older data analytics tools.

Tom (@ttaulli) is an advisor/board member to startups and the author of Artificial Intelligence Basics: A Non-Technical Introduction, The Robotic Process Automation Handbook: A Guide to Implementing RPA Systems and Implementing AI Systems: Transform Your Business in 6 Steps. He also has developed various online courses, such as for the COBOL.

See the rest here:

AI Disruption: What VCs Are Betting On - Forbes