I’ve had a very busy few months (more on that soon). After nearly a year, I’ve finished Michichusa, my search engine. This was easily the most ambitious project I’ve taken on: a fully functional search engine with a crawler, deduplicator, PageRank evaluator, and full-text searcher, plus a graph view for good measure. It’s live now at search.collie.codes, and I’m genuinely proud to finally be at this point.

This is the finale of a run that started with Building a Search Engine and continued through Update 1 and Update 2 .

Overview

The usefulness of linters should not be understated, they act as a lightweight form of PR review.
In fact, the Google SRE guide suggests that “nitpick” comments should be left to linters.
Linters help prevent obvious bugs and serve as guardians of code quality.

Background

I’ve been on quite a journey with testing, When I first started they were the part of the codebase constantly breaking and causing endless regeneration. Luckily, that was some time ago and my views have matured, mostly through being burned by not building tests and seeing the difficulty of replicating a bug.

Muzz Social is the largest Muslim social network in the world, we allow for users to post, comment and share their experiences with each other.

Background

At the start of the year we added a simple but effective impressions system. We used the page which the user was requesting to be counted as an impression, we then stored the results of this inside ElastiCache (Valkey). Every time the feed was requested we would then exclude the posts unless the comment count had changed by a significant amount. This change created a very positive change for our users seeing a 22% increase in feed requests over the two months after we rolled out the feature. However there was an inherent issue with this which was that just because the whole page was requested it does not mean it had been seen. So quickly after this was released we started work on its successor named mobile impressions. The system was to be near real time, it also had to be durable as we did not want to drop events as this could cause inconvenience to the users. This was no mean feat as we handle around 150,000 events a minute all of this would need to be processed and handled as quickly as possible. Muzz Social uses an event driven architecture meaning when certain actions happen it creates events which can be tapped into this allows us to extend existing functionality in a scalable and controllable way.

Over the past few weeks I have been making a webcrawler. I wanted to do it as way to get better at Go with useful for learnings for Graph databases as well as being fun. The project made use of cloud native items such as AWS SQS, DynamoDB and optionally Neptune which could be swapped out for Neo4j.

What is a webcrawler?

A webcrawler or web spider is a program which visits a website, and fetches all of the links on that site and then visits them. This is how sites like Google/Bing/DuckDuckGo get the pages to populate when searching.