Michichusa Is Live
I’ve had a very busy few months (more on that soon). After nearly a year, I’ve finished Michichusa, my search engine. This was easily the most ambitious project I’ve taken on: a fully functional search engine with a crawler, deduplicator, PageRank evaluator, and full-text searcher, plus a graph view for good measure. It’s live now at search.collie.codes, and I’m genuinely proud to finally be at this point.
This is the finale of a run that started with Building a Search Engine and continued through Update 1 and Update 2 .
Try it out at search.collie.codes.
A few things I learned along the way
Hosted Kubernetes is expensive, self-hosting is difficult
Kubernetes is often described in popular writing as “orchestrated Docker across nodes,” and I see why that analogy is appealing, but it hides a lot. I’d compare it more to the difference between a small boat and a container ship. I make the comparison so stark because the container ship has so much going on in areas I wasn’t even aware of, from networking interfaces to the system that schedules nodes based on leader elections.
I started on DigitalOcean, which worked well but was pricey for the compute I actually needed. I ended up self-hosting with k3s on Hetzner instead. A bit of a footgun, but a fun learning experience. Images live in a private container registry, and a cron job in the cluster refreshes the registry tokens; I learned that one the hard way after a token rotation silently broke image pulls.
I also split the cluster into two node pools: one control plane, and worker nodes that scale as needed. I’d previously run the control plane on the same node as the workloads, which caused real problems. Under load the node would lock up, and the control plane couldn’t schedule tasks onto other nodes because it was overloaded itself.
The internet is big
I was finding far too much to explore, far too quickly. The crawl queue was often orders of magnitude larger than the set of pages I’d already visited, so I had to start putting limits in place. It’s now capped at 5 million rows, and new links just get dropped once it’s full instead of growing forever.
Checking that cap on every batch of incoming links would mean running a full COUNT(*) against a table with millions of rows, so instead I cache an approximate count: Postgres already tracks a cheap statistics estimate (pg_class.reltuples) that I refresh every 30 seconds. That’s the database-level cache. It isn’t there for correctness, just so the capacity check doesn’t hammer the database on every insert.
The scale is real: the crawler has made 802,000 fetch attempts so far, discovering 42.7 million links in the process.
Be polite
I kept getting rate-limited when crawling, so I set things up to respect robots.txt, plus a per-domain token-bucket limiter (2 requests/sec, burst of 5) that paces outgoing requests before they ever leave the machine. Most of the politeness happens by never tripping a site’s limit in the first place, which is also why I don’t have a clean “percent rate-limited” stat to show off: the limiter prevents the problem instead of reacting to it after the fact.
What I can measure: of those 802K fetch attempts, 82% succeeded outright. Of the 148K that failed:
- 75.8K were plain 404s
- 25.8K were malformed/invalid URLs (some of what the recent link-extraction hardening was for)
- 15.7K were 403 Forbidden
- 15.4K were crawl-trap loops caught by cycle detection
- 11.3K landed in a catch-all bucket, which is where throttling responses (429/503) show up alongside anything else that didn’t fit a cleaner category
- 3.5K were server errors (5xx)
On top of that, the crawler backed off from robots.txt rules 10,800 times: requests it never made at all, rather than ones that got rejected.
Done is better than perfect
Given the size and complexity of the project, I had to draw some real red lines so it didn’t drag on forever. One weekend I just told myself: no matter how bad the results are, it’s getting finished. I have too many projects I’ve started and never completed, and this was easily the most complex. I was tired of having it half-done and hanging over my head.
Overall I’m really happy with the site. It performs well given the hardware limitations it’s running on: five Go microservices talking over gRPC, a from-scratch PageRank implementation (0.85 damping, 100 random sweeps of up to 100K pages, recomputed every 6 hours), and full-text search blending PostgreSQL relevance (30%) with PageRank (70%), all self-hosted on a k3s cluster provisioned with OpenTofu/Terraform.
You can watch it run live at metrics.collie.codes, on the same Prometheus/Grafana/Alertmanager stack that pages me on Slack when something breaks.
For anyone curious here is link to the Github