What does your voice say about you?

Your accent can nod to where you come from; the pace of your speech can reveal your emotional state; your voiceprint can be used to identify you.

Linguists, companies and governments are now parsing our voices for these details, using them as biometric tools to uncover more and more information about us.

While a lot of this information is used to make our lives easier, it has also been used to controversial and worrying effect.

And the next frontier of voice technology means we may not be able to trust what we hear even if we appear to have said it ourselves.

Much like a fingerprint, we all have a unique voiceprint.

Its distinctiveness is a product of our differently-shaped bodies, our throats, our larynxes, mouths and tongues.

These days it's common to confirm your identity with your voice to log into your bank, access your ATO file, or unlock your phone.

But the use of voiceprints has a longer history than you might think.

"By the early 20th century, vocal portraits were added to archives of criminal records in police departments across Europe and the United States," says Xiaochang Li, assistant professor of communication at Stanford University.

"These recordings were used for a number of forensic purposes, from identification to physiognomy, which is tied to eugenics."

The portraits were used not so much to identify a single person, but to try and identify characteristics of criminality across a population.

Researchers believed the voice could reveal age and class but also uncertainty, obfuscation or lies.

But use of these portraits failed to take hold, in part due to the technical constraints of reproducing the recordings.

Fast forward to 1944 and the mighty R&D company Bell Labs credited with developing radio astronomy, the transistor, the laser and the photovoltaic cell was researching the sound spectrograph.

It's a visual representation of sound that maps frequencies and their intensities.

They were attempting to map qualities of the voice pitch, rate of speech, missing harmonics to identity.

They couldn't conclusively prove which features signalled markers of identity, but the sound spectrograph would become instrumental in the use of the voiceprint for identification because it's much faster for a computer program to look at an image of sound than to listen in real time.

And now voice ID systems such as Nuance, Verint and Pindrop are replacing customer service representatives in banks, call centres and government agencies around the world.

But what if you are looking to identify qualities in the voice that don't present in a sound snapshot? Qualities that shift and change over your lifetime like your accent?

In the early 2000s, as part of the asylum seeker application process to Australia, applicants who arrived without papers were asked to take a language test.

Not a test to confirm their ability to speak English, but a linguistics test to confirm their country of origin.

Tim McNamara, a professor of applied linguistics at the University of Melbourne, says it was first introduced in Sweden in 1993.

"The Swedish government decided to use the way people spoke as part of the process the likelihood that the person is from where they claim to be from," he says.

If done properly, a Language Analysis for the Determination of Origin (LADO) test would employ a trained linguist to elicit speech samples from the applicant that would place them on one side of a border or another.

Your phone and other personal computing devices know an awful lot about you.

However, in Australia the test was often conducted by another asylum seeker who could speak the language, or a translator who spoke a similar dialect.

"I think language verification results are appalling," says Sanmati Verma, accredited immigration law specialist at Clothier Anderson Immigration Lawyers.

"In the context of displaced refugee communities folks have not been to school, they are living in ethnocised ghettos, they haven't been interacting with members of local population there is no way to determine through language where they are truly from."

Professor McNamara, with a group of international linguists, wrote a set of guidelines that say if LADO is to be used, there are ways to sharpen the tool: use qualified linguists, better understand how language moves across borders, and allow for uncertainty.

"The real problem with LADO is that it is done badly, and it's done cheaply. And the courts want black and white, but there has to be room for doubt," he says.

LADO is still being used by several governments in Europe and the UK, but Australia has exchanged the test in favour of constructing a person's narrative, a process that has also attracted criticisms.

Nonetheless, the decisions within the application process are still being made by humans. What happens when we hand over the reins to automated systems?

Public services such medical systems and government departments are turning to automation to find efficiencies and to meet increasing demand.

Dial the ATO and your voiceprint can be matched in less than 30 seconds by a computer algorithm, forgoing minutes of security questions.

Beyond the simple voiceprint, companies at the frontier of voice tech suggest they can sort for all kinds of different traits, such as our emotional states.

Earshot is about people, places, stories and ideas, in all their diversity.

The next time you dial a call centre you could be triaged by an AI voice for how stressed you sound over the phone.

One company, Clearspeed, even promises to be able to hear risk in the voice, through an automated questionnaire.

CEO Alex Martin says its algorithms are looking for the "acute threat response, in concert with what is known as cognitive effort".

"These are the things that collide and impact speech phonation, resonating impacts of speech and that's what we're detecting and measuring."

For instance, a job applicant answers a set of tailored yes/no questions over the phone, and they are flagged low to high risk. Low risk, they go onto an interview; high risk, their application is reviewed.

"Everyone should be able to say no and have absolutely no problem in their response. Their no looks normal in our model and we score it as low to average risk. But if there is neurophysiological reaction, depending on the amount of reaction it could be scored as potential risk or high risk," Mr Martin explains.

Clearspeed is concerned with clearance problems, assessing hundreds of applications quickly, automatically, and says its tool should be used in constellation with standard practises.

But as more systems are automated, the role of humans and human judgement is diminishing.

Is artificial intelligence sexist?

"Judgement is what happens when we face a decision that doesn't have a pre-coded answer," says Mark Andrejevic, professor of communication at Monash University.

"Most of the meaningful decisions we make in our lives are not syllogistic, [meaning] they cannot be reduced to a set of logical principles building on a few givens."

He cautions against automating too many decisions in our lives.

"Automation is really good for those syllogistic decisions. But we want to avoid handing over the non-syllogistic decisions because it reduces the capacities we need to live a life in common."

And then there's next frontier in voice technology, where we do away with our bodies altogether.

In the past couple of weeks, a new voice synthesis product was released.

Overdub proposes to create ultra-realistic text-to-speech of your own voice.

Using 10 minutes of recorded audio of your voice, it creates a clone of your voice, which you can make say anything you want.

All you have to do is type the desired sentence to the web app, and out it comes your voice, as if you'd said it using your throat, larynx, mouth and tongue.

The implications of text-to-speech synthesis are manifold, exciting and terrifying.

Photo editors made us doubt images, video editors resulted in deep fakes soon we will not be able to trust what we hear. Even if we appear to have said it ourselves.

I created a voice clone to test whether "she" would pass as the narrator of my Earshot documentary.

And the process revealed the elegance and the clumsiness of the tech. To train the AI, I had to repeat prescribed sentences and inflect them with joy or anger or sadness, as if the AI needed emotional training too.

When my voice had been cloned and I heard the first utterances of "her", I was amazed at the reproduction but listening more closely the odd vowels and strange phrasings came into focus, and the uncanniness I first heard started to come undone.

But, in terms of this technology, we are only at the beginning.

Our voices are shifting from representations of us to technological tools. Tools that we can learn how to use to redefine ourselves, for us and for who is listening.

Get more stories that go beyond the news cycle with our weekly newsletter.

Link:
Your voice says a lot about you and AI is listening - ABC News