Quick-start
Dependencies
Go 1.13
OpenSearch 7.x
RabbitMQ / AMQP server
NodeJS 9.x
IPFS 0.7
Building
$ go get ./...
$ make
Running
Docker
The most convenient way to run the crawler is through Docker. Simply run:
docker-compose up
This will start the crawler, the sniffer and all its dependencies. Hashes can also be queued for crawling manually by running ipfs-search a <hash>
from within the running container. For example:
docker-compose exec ipfs-crawler ipfs-search add QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv
Ansible deployment
Automated deployment can be done on any (virtual) Ubuntu 16.04 machine. The full production stack is automated and can be found in it’s own repository.