Crawling

There are two parts of a search engine implementation… crawling and searching.

The first part is, obviously, crawling the content and injecting it into the search engine.

For this I wrote a java application that would read the raw PHP content of the mailing list archives, extract the appropriate information, and add it to the search engine.

I started small, with a single mailing list (pctech), and wrote a program to…

1. Read the lists directory to find the year & month subdirectories.
2. For each of those directories, it would read each file that matched the pattern ‘msg99999.html’ (msg\d{5}\.html for those of you who like regex). This represented each individual list message.
3. For each of those message files, it would read the contents, extract the message body, subject, author, and date.
4. This data would be collected in memory and, after processing the last message in the directory, insert the data into thee search engine.

To process the message file content, I used the jsoup library. This would let me easily extract and manipulate the content of the archive message file.

I specifically wanted to avoid redundant content, so I used jsoup’s function to extract the message body and remove any identifiable quoted content. Luckily much of the quoted content is enclosed in blockquote tags.

Since I was reading each archive message individually, I could automatically eliminate messages that shouldn’t be indexed. Things like administrative messages, monthly guidelines, etc.

As each list is processed, I store the ‘high water mark’ for the list consisting of the month/year & message number in a database (H2 embedded database if you’re interested).

Each entry in the Meilisearch database contains metadata …

• List name
• Subject
• Content
• Date (for sorting & display)
• Author
• Year & month of the message (from the directory structure)
• Unique id consisting of the list, year, month, & message number.

After getting the one list working, I started working on the search front end.