oh wait...
so it is necessary for the index to be able to loaded so that the WebSpider can just continue adding things?
what if we made it so that the index could be loaded and searched, but couldn't be appended to...? would this lose any style points or anything?
it's just that, the index file size gets kind of big. and, although me and my partner haven't finished writing the saveIndex and loadIndex part yet, we had planned on NOT saving the graph structure. just the keywords, the page ID of the pages that each keyword was on, then information for each page (i.e. the page ID, the URL, the relevancy, and possibly the description if the file isn't getting too large).
is this a problem? |