ExtremeTech explains: All about the dark web, and how to use it – ExtremeTech

If youve paid any attention to online marketplaces for illegal goods like the now-defunct Silk Road or the FBIs investigations into criminal in cyberspace, chances are youve heard the term dark web. Curious about what it means? Youve come to the right place.

The dark web is sometimes called onionland because of its content accessible only using services like Tor. The rest of the internet is simply referred to as the clearweb, since it isnt generally encrypted.

The dark web works just about the same as the regular internet: it uses the same TCP/IP framework to transmit HTTP and FTP traffic within and between networks, over the same phone, cable or FiOS lines that carry regular internet traffic. Content on the dark web consists of HTML webpages and their assets, just like it does on the rest of the web. In fact, under the hood, the dark web is the same as the regular web, with two important exceptions that also distinguish the dark web from the deep web.

First: the dark web isnt indexed by search engines. Second, content on the dark web cant be accessed with regular web browsing software alone; additional software is required to make the networks talk to one another.

This is because content on the dark web is hosted on overlay networks, which are physically connected to the internet but arent accessible to web crawlers. That relative inaccessibility is because the dark web uses a complete, but fundamentally different, network addressing system than the web addresses most of us know and use. Browsers like Chrome and Firefox are programmed to access website files using the DNS index, which turns a files unique address on its unique server into a string of text that you can type into your address bar. Sites indexed by the DNS registry are accessible via top-level domains like .com and .org, among others. After ICANN opened up the suffixing system to other strings of text, we started to see web addresses that look like home.cern and bit.ly but you can still type those into your address bar and get to a website, because theyre in the official DNS registry. Dark websites dont participate in the DNS system, and web crawlers dont have the software to get onto the dark web, so the dark web and the clearweb dont really cross-pollinate.

Content obscured in this way can still be accessed, but you need the right software. Its a bit like a Wi-Fi network that doesnt broadcast its SSID: you can only get access if you already know exactly how to find it. Some content accessible only through Tor is hosted at a .onion pseudo-top-level domain, which means that in the right software, you might type in foobar.onion and get to the Foobar dark website.

Such software, including the Tor browser bundle, is capable of bridging the differences in network behavior between the dark web and the clearweb. But that only works when youre using a compatible browser and have the right encryption. Tor, Freenet and I2P are the most commonly cited examples of software capable of accessing the dark web. Typing a .onion address into your Chrome address bar wont get you anywhere. Furthermore, many if not most .onion sites are generated sixteen-character non-mnemonic alphanumeric strings, rather than being composed of words like most clearweb URLs.

There also exists a difference in the path web traffic takes on the clearnet versus the dark web. Tor is valuable because it sends your own web traffic through multiple different network nodes, masking its origin and destination. Theres significant overlap between VPNs and the dark web; both services use encryption and multiple network nodes to anonymize traffic. But VPNs deal with clearweb sites that participate in the DNS system, while dark web browsers deal with domains not recognized by ICANN.

The structure of the dark web makes it anonymizing, which means that first and foremost, its used for anonymous communication and web browsing. This accounts for the vast majority of network traffic through Tor. Why seek out anonymity? To read and write about things that might get you in trouble, like political dissent or whistleblowing. The same technology that enables Tor is capable of tunneling out from behind the Great Firewall of China, and the US government contributes to the development of such software.

Anonymity also brings out those who wish to do illegal things. A 2014 study found that of the different kinds of sites on the dark net, there are more markets devoted to drugs and guns than any other kind of dark site, including forums, bitcoin laundering, hacking, fraud, whistleblowing and even regular old porn.

To paraphrase Jim Jeffries, if you want to murder someone, you cant just walk up to Pier 31 and shout GUNS, WHO WANTS TO SELL ME SOME GUNS!? But with a website like an evil eBay that lists weapons and other contraband for sale, all of a sudden you dont have to know someone with black market connections. You just have to be able to install some software.

Tor hidden services are the other thing the dark web does, and theyre what gives the dark web its shady reputation. Hidden services refers to dark sites where both the host and the visitor are anonymous to one another. That technology enables dark web sites that host illegal content to persist. Hidden services account for only 1.5% of the Tor network volume. But the overwhelming majority of resources requested over Tor hidden services fully 80% of that traffic were requests from child abuse sites. Outgoing traffic from the dark web flowed mainly between botnets and their hidden control servers. More detail on Tors traffic patterns and how much of its total bandwidth is used for illegal activities is available in a blog post by the Tor project.

The dark web is notoriously dodgy territory for both buyers and sellers. Law enforcement has been chipping away at the nominal anonymity afforded by software like Tor, and anything of interest on the dark web is as likely to be a scam as it is to be a honeypot. Between social engineering and software vulnerabilities, it is a realm best accessed while wielding some trustworthy anti-malware.

For a long time, the Silk Road was the biggest game in darknet commerce. It allowed users to sell a great many illegal things, and inspired a number of similarly designed copycat markets. Transactions there were conducted in bitcoins and other virtual currency, and then goods were shipped through the mail. But a high-profile bust and ensuing court case put several Silk Road admins in jail. The media spotlight has impinged on the Silk Roads relative obscurity, reducing its value as a black marketplace.

From Tumblr.

While Uncle Sam contributes to the development of Tor and similar anonymity resources, the government is also known to take more of a proprietary approach, considering even the dark web to be within American jurisdiction when site hosting is in question. The FBI paid Carnegie Mellon to crack Tor in pursuit of a criminal case. They even waded into the muck and ran a huge sting operation on Playpen, a darknet child porn site by taking over control of the site and running it for weeks as a poisoned well to catch its users.

The dark net is an excellent example of how difficult it is to preventcriminals from using anonymizing services designed to protect honest dissenters. Tors anonymizing functions are critically important to people who rely on it to discuss sensitive topics without fear of reprisal. The debate over how much light should be shone into the dark web is an ongoing topic of discussion. How much illegal activity should be allowed to maintain Tors positive benefits, and is there a way to unmask child molesters and other illicit activity without compromising the security that makes the dark web work?

Now read: 19 ways to stay anonymous and protect your privacy online

Check out our ExtremeTech Explains series for more in-depth coverage of todays hottest tech topics.