i thought our WebCrawler was taking along time... until i found out that yours is taking 8 minutes and 32 seconds...
to succesfully crawl 100 pages on google, then index all of those pages and their links, then write the index to a file takes 1 minute at 40 seconds (give or take 4 seconds) for us, even on one of the bogged down andrew unix servers. 35 seconds of that time is spent indexing (~25 sec) and saving (~10 sec), the rest is spent crawling and parsing (lexing?) pages.
now, i don't know if ours is particularly fast or not, and i'm still concerned about time.... but maybe you should consider looking over your code a little and seeing how you can speed things up.
jason
|