Introduction to the Onion Network Crawler

The Horizon of Events team has long considered the idea of creating a crawler for the popular onion network. Believe it or not, this journey has been quite lengthy. Starting from an older project, there have been several attempts at implementation. As of now, we have reached a certain result, although it is still relatively modest.

Let’s break down what a crawler is for those unfamiliar with the concept.

What Is a Crawler?

A crawler is essentially a component of a search engine, primarily designed to scan web pages and collect information about them. Major search engines like Google and Yandex use such tools. However, as you might know, the onion network is not indexable, making it quite challenging to assign descriptions to specific sites. We are still working towards fully implementing this functionality.

About Our Crawler

We present to you our crawler, which recursively scans links to any depth starting from a specified URL. It is developed in Go and currently compiled only for Windows. To run it, you need to start the Tor browser first.

How to Use the Crawler

• To get information about how to launch the crawler, use the command: ./crawl.exe -h
• Example usage without specifying depth:
  .\crawl.exe --target http://deepweb4wt3m4dhutpxpe7d7wxdftfdf4hhag4sizgon6th5lcefloid.onion/
• Example usage with depth set to 2:
  .\crawl.exe --depth 2 --target http://deepweb4wt3m4dhutpxpe7d7wxdftfdf4hhag4sizgon6th5lcefloid.onion/

Download the Compiled Go Code for Windows

You can download the compiled Go code for Windows systems here:
https://gofile.io/d/f8Y72W