I’ve had a very busy few months (more on that soon). After nearly a year, I’ve finished Michichusa, my search engine. This was easily the most ambitious project I’ve taken on: a fully functional search engine with a crawler, deduplicator, PageRank evaluator, and full-text searcher, plus a graph view for good measure. It’s live now at search.collie.codes, and I’m genuinely proud to finally be at this point.

This is the finale of a run that started with Building a Search Engine and continued through Update 1 and Update 2 .

Search Engine Project Update

I’m sorry for the lack of project updates 🙇 I have been working on a lot of OSS work so this has taken a back seat. However, I have added a lot to the project since my last update . It’s nearing completion.

Overview of the problem

One of the key parts of the design of my search engine is the ability for the spiders to send the pages which they have explored to a central point. Additionally, a more conventional design might use a push/pull system, so the conductor polls each spider in turn and requests the pages which it has seen using pagination. This has several problems.

As I mentioned in my last Building a Search engine , I’ve been busy building my search engine lately, and it’s been a lot of fun diving deep into Kubernetes and learning how to set everything up from scratch.

Overview of the System

I was re-reading “I am feeling lucky” and this inspired me, so I have decided to create a search engine. This is a very ambitious project; however, it will teach me a lot about building a micro service with Kubernetes all from scratch.

My plan is to refactor the spider to use a relational database. I am learning towards each spider having its own SQLite db. Then create a service which can go to the spiders and get their current crawl and add them to a central DB. This will create a single graph of all the visited notes.