1. Home
  2. Deep Dive: The Deep Web

Deep Dive: The Deep Web

Have you ever wondered how a search engine like google comes up with all the information in a matter of a few milli seconds, on a keyword that we input? ‘Bot’ or ‘Robot’, an algorithm, crawls the web similar to a spider, systematically through the WWW and adds new content that it finds, to the search engine’s index.

However, did you know that there is a ‘deep web’, invisible and hidden within the World Wide Web? The content on this is never indexed by standard web search-engines like Google, Bing or Yahoo, which means it is never seen or at least not by people who use these search engines.

‘Deep Web’ provides near anonymity. What we see when we Google the internet, is the surface web, where as little as 1% of all the information is available. Did you know that several websites exist which are not even registered with any search engine? They are all hidden by various network security measures like firewalls and encryption.

Why was a ‘deep web’ crated? The nature of the internet is lack of privacy. So, if one were to spy on enemy nation’s defence communication networks, how does one do it? Use ‘deep web’. Like all defence innovations moving into public domain, this one too did. It is another matter that most defence establishments today use Virtual private networks.

What actually is blocked out from Google like search engines? For a start, databases and intranets. Next, content of Netflix, Amazon Prime Video or HBO Max or any other similar. As a rule, any website that is paywalled, such as online payments, news articles of media houses, or educational content of sites like Coursera or Udemy or Research Journal content or publications requiring subscriptions are blocked. Besides, if google bots were to crawl in our bank accounts or our investment accounts for details, would we accept?

Have you then, ever wondered if most of the information that you may have considered useless or information that you thought was in your exclusive mobile devices or laptops is actually up there on the ‘deep web’, hidden behind HTTP forms?

What kind of information is this? Most of it is useless. Web mails such as Gmail, Yahoo mail, Hotmail or similar, chat messages, private content on social media sites, electronic bank statements, electronic health records (EHR)’s, legal files, online banking details, restricted access social-media pages and profiles, some web forums that require registration for viewing content, and services that users must pay for, protected by paywalls or gateways, video on demand on Netflix, Amazon like portals and online discussion forums, magazines and almost all dailies and newspapers are all there.

Having said that, how would it look like to bypass local restrictions and access TV or movie services that may not be available in certain local areas or download pirated music or steal movies that aren’t yet released in theatres?

That being what it is, if you have deleted a mail or a photograph or any other, for whatever the reason, be sure it can be recovered on the ‘deep web’. This alone must be a compelling reason to be discreet on the information you share on a WhatsApp group or in your personal mails or on your social media platforms.

Any content of the deep web can be located and accessed by a direct URL or IP address, but may require a password or some other security access to get past public-website pages. You will need a different browser. However, if you used ‘Tor’ browser, an onion router, or ‘I2P’ network, be sure, you may be monitored. Still, accessing content on the deep web is relatively safe, and most internet users do it all the time, logging into Gmail or LinkedIn. That ‘Tor’ protects IP when visiting websites and I2P, a proxy network helps journalists reporting from dangerous territories when every other communication device is down are better uses.

Why should anyone be monitored if the ‘deep web’ is searched? It is because the ‘dark web’ is a subset of ‘deep web’. For both, ‘Tor’ the browser is the same. ‘Dark web’ or ‘Freenet’, is a peer-to-peer platform for censorship-resistant, anonymous communication over a ‘darknet’ and hosts a lot of illegal and prohibited content, like personal passwords, black markets for stolen credit cards, false identity documents, drugs, firearms, malware, prostitution, sex trafficking, and child pornography. Even Cyber-attack services, like access to botnets that can conduct DDOS attacks, are available. Many spam and phishing attacks also originate from a dark web marketplace. The ‘dark net’ that allows this is an overlay network within the Internet that can be accessed with specific software, configurations and authorization.

Not everything on the ‘deep web’, is crime related. Therein lies its strength and concern. Its strength is that it helps crime detection both in cyber and physical world, even when people believe they have erased all trails of crime. Its concern is that the information can be misused by unscrupulous operators. User accounts on the deep web contain a lot of personal information that criminals might value, which reason is sufficient to restrict access to much of the deep web. Unfortunately, though deep web’s information isn’t indexed by regular search engines, it still can be accessed through specific algorithms available on GitHub.

The cyber safety experts have a large use for the dark web as they could track yet-unknown vulnerabilities and know where exploits originate before they become widespread threats.

Whatever be the case, the privacy offered by the Tor browser is important. Governments, corporations and governing bodies alike currently participate in unauthorized surveillance of online activity. Some simply don’t want government agencies or even ISPs to know what they’re looking at online, while others have no choice.

Actually, many Tor-based sites are controlled by the police across the globe. Criticising Political ideologies or evading government restrictions, perpetuating scams and crime may not be acceptable. Hence, most governments heavily monitor them. Cyber security agencies look for activity on these sites to decode cybercrimes.

What we know is very little of what’s really on the deep web. Every country researches as much as the cyber enthusiasts and the hackers. So, what must a common man do? Just stick to the basics, and work with a set of morals and ethics. Not ever, delve into unchartered waters.

(Visited 4 times, 1 visits today)