DARPA Is Creating a New Internet, Based Around Search

What you see when you do a basic Web search is only the tip of the iceberg. Most of the information is buried in the "Deep Web." NASA/JPL-Caltech

The Internet you currently know only sees about 4 percent of the actual Internet. The next Internet, being built by the Pentagon’s research and development branch, aims to see the same 4 percent—as well as the other 96 percent.

The program is called Memex, and is being designed with new search functions that could change how we use the Internet. Memex is being developed by the Defense Advanced Research Project Agency (DARPA) with the help of 18 partners, including NASA’s Jet Propulsion Laboratory.

“This is going to have a huge impact,” said Chris Mattmann, the principle investigator for NASA’s side of Memex.

Memex looks to do with raw data what social media did to online chatrooms. It connects everything, shows its relations, and makes it easier for users to find the piece they’re looking for.

DARPA’s key interest in Memex, at least initially, is to help map the Darknet—the Internet beneath the Internet that exists on peer-to-peer networks. They want to use Memex to help stop online human trafficking.

READ MORE

• Fight Against Cybercrime and Terrorism Moves to the Darknet

“It’s a game changing tool for [law enforcement],” Mattmann said, noting, “Memex is already being actively used for law enforcement” including by the New York County District Attorney’s Office.

Searching the Darknet

“Memex to date is focused on finding the bad things that people are doing on the Darknet,” Mattmann said.

Some parts of the Darknet are good, and there are people who use it to bypass government censorship. Mattmann said this area isn’t their focus, noting that they don’t plan on making the Internet “any less anonymous.”

The Darknet is also home to many terrible things. It includes markets for human trafficking, drugs, firearms, pedophilia, hitmen, and hackers. It’s an area of the Internet that law enforcement is still having trouble regulating, partly due to the fact that its networks are not indexed by traditional search engines.

Memex may be able to change the nature of the Darknet. According to DARPA’s website, “The use of forums, chats, advertisements, job postings, hidden services, etc., continues to enable a growing industry of modern slavery.”

By indexing this domain, it states, it would “enable new opportunities to uncover and defeat trafficking enterprises.”

The use of forums, chats, advertisements, job postings, hidden services, etc., continues to enable a growing industry of modern slavery.

— DARPA

Mattmann said Memex is designed to do two key things: it can look at pixels and try to figure out what the small things mean, and it can search metadata—the information behind all images and videos that already exists in abundance, but is invisible to current search engines.

“Metadata is becoming something that’s really important,” Mattmann said, noting “there is metadata associated with images and videos that lets us draw connections with them.”

He said Memex is a technology that can sit on top of the Internet or tools that browse the Darknet, “and it can make relationships between images and people.”

Using this, law enforcement will be able to tell if a picture or video was taken with the same device, and to see which other content was created by the same individual. They can also look for connections between drugs and the people selling them, and images of abducted people and the people who took the pictures.

Some social networking sites already scrub metadata for things like image location, but Mattmann noted “those sites are not what a lot of those bad people are using.”

The New Internet

Memex isn’t just for law enforcement, however, and this is where Mattmann’s real interests are. He wants to ensure that Memex ends up in the hands of the everyday user.

He joined the Memex project “with the full intention that a lot of the technologies the government is investing in at DARPA would go into these technologies that are being downloaded thousands of time every day.”

Parts of Memex are open source, meaning they are publicly available and can be used without licensing fees.

As a member of Apache’s all-volunteer board, Mattman has been interested in bringing Memex to technologies like Apache that are at the foundations of the Internet.

READ MORE

• Hacker War Erupts Over South China Sea Conflict
• Information on Every Federal Employee Allegedly Stolen In Cyberattack

Apache, for example, has already started using Memex. Apache Web servers power close to 53 percent of the Internet. Another core technology, Drupal, has started using it, as has Google.

Mattmann said users will start noticing that their searches return better results, with “more meaningful information, particularly on multimedia content.”

“Searches are going to be more semantically rich,” he said, noting that information will be easier to find, and “the information that comes back is going to be a lot more meaningful.”

The main problem with current search engines, according to the Memex project page at DARPA, is that it uses a “centralized, one-size-fits-all approach that searches the Internet with the same set of tools for all queries.”

The current model is still a heavily “manual process that does not save sessions, requires nearly exact input with one-at-a-time entry, and doesn’t organize or aggregate results beyond a list of links,” it states.

“While that model has been wildly successful commercially,” DARPA states, “it does not work well for many government use cases.”

Of course, the new Internet won’t be released as a new download. It will be integrated gradually into the existing Internet as its development moves along, and this process has already started.

Within a year or two, Mattman said, it’s likely we'll all be using the Memex Internet. Referencing the “ARPANET,” the network created by DARPA that helped create the Internet, Mattman said “the same influence ARPANET had on the Internet, Memex will for search.”