Searching

I wrote the search front end in PHP, as it’s one of the easiest languages to implement logic in a web application with. Luckily, there’s a very robust PHP library available for Meilisearch.

The front end is pretty simple, it takes a search term passed in as a request parameter, sends it to the search engine, and then renders the results.

Since nobody wants a huge single page of results, I added paging with a page navigation bar at the bottom. This page navigation is the only place I used AI because my implementation was getting troublesome.

On a lark, I added the ability to do infinite scrolling so, as you scrolled down the page, it loaded new content.

The UI was done with bootstrap, as I’m very familiar with it and I find the components very simple and straight forward.

MVP

(For those unfamiliar, MVP stands for Minimum Viable Product)

Now that I have the crawling & searching functions working, I needed to build an incremental crawl function.

To do this I use the RSS feed (just an XML file) that is automatically built by the archive generation software. This feed contains entries for each new message that is archived.

For each list, I read the appropriate RSS file to find the newest messages, check if the message is higher than the high water mark stored in the H2 database. If it is, I add the new message to the search engine and update the high water mark.

I setup this process to run every 5 minutes on the list server system.