Show HN : I have developed a robots.txt full text search engine.
Show HN : I have developed a robots.txt full text search engine.
As a sub project of my pet project DNSDigger.com i have developed a search engine that index robots.txt (http://en.wikipedia.org/wiki/Robots_exclusion_standard). I actually don't know what poeple could use this for but i suspect poeple that are into SEO could use it and black/white-hat hackers(?). What i need help with is VPS/$$ for spidering. And as i sit on a considereble index of domainnames (though DNSDigger.com 300m hosts) i thought that if you want a TLD indexed sooner than later you could sponsor it with either VPS or Paypal. See it as a strange "kickstarter". And if you think the "design" suck (which it does because there is a lot of things i suck at and design is one of them) you are free to submit anything better. I can put up a banner for your service a limited time. I am adding new hosts/TLD all the time. And if you have to ask what this is you will not have any use for it even if you did ;)
Share cardActual performance
Launch Intel predictions
Analyze your own launch →Incorrect prediction on native model
Similar products
Robots.txt Search Engine
AskHadith – A Hadith search engine based on Mongo text search
AISLER – Search Engine for electronic circuits
Threadbase – A T-Shirt Search Engine
PressPanda, a journalist search engine
A reactive search engine
Vineyard - A search engine for Vine
Sentiment Search Engine
We made a search engine
Healthforum – Drug Side Effect Search Engine