Government algorithms are out of control and ruin lives – Open Democracy

In addition to the individual hardship, there are also significant structural issues at play. There are more than 200 government blacklists in use in the Netherlands. Blacklists like these could soon be turbo-charged through technology: in December, a legislative proposal was adopted by the Dutch parliament that would give the government algorithmic-profiling powers that exceed the preventative fraud detection system (SyRI), which was designed to detect false benefit claims and which the courts declared in violation of human rights only a year ago. Allowing for data sharing between government and private companies, it would be an Orwellian nightmare come true. A parliamentary inquiry has been scheduled for the summer of 2022, which will likely be too late for any learnings from the tax benefits scandal to feed into the discussion on the draft law currently ongoing in the Senate.

The automation and further entrenchment of institutional racism is not just happening in the Netherlands, but across Europe and the world. It is happening in all areas of life imaginable. From social welfare to racist predictive policing tools to assigning organs for transplant on the basis of race, sex and class, and the classist, prejudiced system that was used to assign A-Level test results to students in the UK after their exams were cancelled due to COVID: technology is a part of every aspect of our lives.

The urgency for civil society to further step up its efforts to address issues of automating racism and discrimination is therefore greater than ever. Public awareness raising, advocacy, and litigation will need to go hand in hand with in-depth engagement with policy makers and legislators. An instrument like the recent EU draft regulation on Artificial Intelligence, for example, does not sufficiently take into account that context matters just as much as the technology itself when it comes to the detrimental impact it can have. No matter how much we are promised that the tech is accurate, its deployment in and of itself can have a negative impact on especially marginalised and racialised individuals. Even a perfectly working tool of oppression is still a tool of oppression.

When this negative impact materialises, the groups and individuals that are affected will need to be able to seek redress directly. This means that concrete enforcement actions for human rights violations resulting from the use of technology currently missing from many regulatory initiatives, including the draft regulation should be possible. This will make possible the kinds of strategic legal cases that civil society can unite around to bring about change and lead to increased protection of human rights in the digital context for everyone.

While technological development can sometimes feel like an unstoppable rollercoaster, this is a false narrative. Technology is developed by humans. We decide what we create, where we deploy it, and how we regulate it. The only urgency we should feel is in the realisation that the decisions we make now will have consequences for generations to come.

Continue reading here:

Government algorithms are out of control and ruin lives - Open Democracy