Explainable AI – Times of India

AI is transforming engineering in nearly every industry and application area. With that, comes requirements for highly accurate AI models. Indeed, AI models can often be more accurate as they replace traditional methods, yet this can sometimes come at a price: how is this complex AI model making decisions, and how can we, as engineers, verify the results are working as expected?

Enter explainable AI a set of tools and techniques that help us to understand model decisions and uncover problems with black-box models like bias or susceptibility to adversarial attacks. Explainability can help those working with AI to understand how machine learning models arrive at predictions, which can be as simple as understanding which features drive model decisions but more difficult when trying to explain complex models.

Evolution of AI models

Why the push for explainable AI? Models werent always this complex. In fact, lets start with a simple example of a thermostat in winter. The rule-based model is as follows:

Is the thermostat working as expected? The variables are current room temperature and whether the heater is working, so it is very easy to verify based on the temperature in the room.

Certain models, such as temperature control, are inherently explainable due to either the simplicity of the problem, or an inherent, common sense understanding of the physical relationships. In general, for applications where black-box models arent acceptable, using simple models that are inherently explainable may work and be accepted as valid if they are sufficiently accurate.

However, moving to more advanced models has advantages:

Figure 1: Evolution of AI models. A simple model may be more transparent, while a more sophisticated model can improve performance.

Why Explainability?

AI models are often referred to as black-boxes, with no visibility into what the model learned during training, or how to determine whether the model will work as expected in unknown conditions. The focus on explainable models aims to ask questions about the model to uncover any unknowns and explain their predictions, decisions, and actions.

Complexity vs. Explainability

For all the positives about moving to more complex models, the ability to understand what is happening inside the model becomes increasingly challenging. Therefore, engineers need to arrive at new approaches to make sure they can maintain confidence in the models as predictive power increases.

Figure 2: The tradeoff between explainablility and predictive power. In general, more powerful models tend to be less explainable, and engineers will need new approaches to explainability to make sure they can maintain confidence in the models as predictive power increases.

Using explainable models can provide the most insight without adding extra steps to the process. For example, using decision trees or linear weights can provide exact evidence as to why the model chose a particular result.

Engineers who require more insight into their data and models and are driving explainability research for:

Current Explainability Methods

Explainable methods fall into two categories:

Figure 3: The difference between global and local methods. Local methods focus on a single prediction, while global methods focus on multiple predictions.

Understanding feature influence

Global methods include feature ranking, which sorts features by their impact on model predictions, and partial dependence plots, which home in on one specific feature and indicate its impact on model predictions across the whole range of its values.

The most popular local methods are:

Visualizations

When building models for image processing or computer vision applications, visualizations are one of the best ways to assess model explainability.

Model visualizations: Local methods like Grad-CAM and occlusion sensitivity can identify locations in images and text that most strongly influenced the prediction of the model.

Figure 4: Visualizations that provide insight into the incorrect prediction of the network.

Feature comparisons and groupings: The global method T-SNE is one example of using feature groupings to understand relationships between categories. T-SNE does a good job of showing high-dimensional data in a simple two-dimensional plot.

These are only a few of the many techniques currently available to help model developers with explainability. Regardless of the details of the algorithm, the goal is the same: to help engineers gain a deeper understanding about the data and model. When used during AI modeling and testing, these techniques can provide more insight and confidence into AI predictions.

Beyond Explainability

Explainability helps overcome an important drawback of many advanced AI models and their black-box nature. But overcoming stakeholder or regulatory resistance against black-box models is only one step towards confidently using AI in engineered systems. AI used in practice requires models that can be understood, that were constructed using a rigorous process, and that can operate at a level necessary for safety-critical and sensitive applications.

Continuing areas of focus and improvement include:

Is Explainability Right for Your Application?

The future of AI will have a strong emphasis on explainability. As AI is incorporated into safety-critical and everyday applications, scrutiny from both internal stakeholders and external users is likely to increase. Viewing explainability as essential benefits everyone. Engineers have better information to use to debug their models to ensure the output matches their intuition. They gain more insight to meet requirements and standards. And, theyre able to focus on increased transparency for systems that keep getting more complex.

Views expressed above are the author's own.

END OF ARTICLE

Link:

Explainable AI - Times of India