I have a small working search engine called Mohawk Search at [login to view URL] . The crawler is made, written in Perl, but there are some modifications I would like to make.
First of all, I would like the crawler to be able to crawl sites from a seed url on the webserver (a cgi script), so it automatically updates the search engine database. You will need to edit a crawler script already made. You need to make sure it does not overload the web server system.
Second, I want the search engine script to have its own ranking system, nothing complex as Pagerank by Google, but something similar so it does not only display results descending or ascending by date.
I will provide the source code to the search engine and the crawler, and you will need to be modifying it.