Posts

I’ve had a very busy few months (more on that soon). After nearly a year, I’ve finished Michichusa, my search engine. This was easily the most ambitious project I’ve taken on: a fully functional search engine with a crawler, deduplicator, PageRank evaluator, and full-text searcher, plus a graph view for good measure. It’s live now at search.collie.codes, and I’m genuinely proud to finally be at this point.

This is the finale of a run that started with Building a Search Engine and continued through Update 1 and Update 2 .

Search Engine Project Update

I’m sorry for the lack of project updates 🙇 I have been working on a lot of OSS work so this has taken a back seat. However, I have added a lot to the project since my last update . It’s nearing completion.

Overview

The usefulness of linters should not be understated, they act as a lightweight form of PR review.
In fact, the Google SRE guide suggests that “nitpick” comments should be left to linters.
Linters help prevent obvious bugs and serve as guardians of code quality.

Background

I’ve been on quite a journey with testing, When I first started they were the part of the codebase constantly breaking and causing endless regeneration. Luckily, that was some time ago and my views have matured, mostly through being burned by not building tests and seeing the difficulty of replicating a bug.

Muzz social is the largest muslim social network in the world, we allow for users to post, comment and share their experiences with each other.

Background

At the start of the year we added a simple but effective impressions system. We used the page which the user was requesting to be counted as an impression, we then stored the results of this inside ElastiCache (Valkey). Every time the feed was requested we would then exclude the posts unless the comment count had changed by a significant amount. This change created a very positive change for our users seeing a 22% increase in feed requests over the two months after we rolled out the feature. However there was an inherent issue with this which was that just because the whole page was requested it does not mean it had been seen. So quickly after this was released we started work on its successor named mobile impressions. The system was to be near real time, it also had to be durable as we did not want to drop events as this could cause inconvenience to the users. This was no mean feat as we handle around 150,000 events a minute all of this would need to be processed and handled as quickly as possible. Muzz social uses an event driven architecture meaning when certain actions happen it creates events which can be tapped into this allows us to extend existing functionality in a scalable and controllable way.

Background

Muzz Social is the world’s leading Muslim-focused social network. We allow people to share and connect with Muslims around the world.

Muzz Social sends push notifications for posts to users who haven’t been on the platform for 48 hours. These notifications are generated by selecting the most engaging and relevant posts, helping to re-engage users and surface content they might have missed. This feature has proven effective in delivering posts that matter to users.

Overview of the problem

One of the key parts of the design of my search engine is the ability for the spiders to send the pages which they have explored to a central point. Additionally, a more conventional design might use a push/pull system, so the conductor polls each spider in turn and requests the pages which it has seen using pagination. This has several problems.

As I mentioned in my last Building a Search engine , I’ve been busy building my search engine lately, and it’s been a lot of fun diving deep into Kubernetes and learning how to set everything up from scratch.

Overview of the System

I was re-reading “I am feeling lucky” and this inspired me, so I have decided to create a search engine. This is a very ambitious project; however, it will teach me a lot about building a micro service with Kubernetes all from scratch.

My plan is to refactor the spider to use a relational database. I am learning towards each spider having its own SQLite db. Then create a service which can go to the spiders and get their current crawl and add them to a central DB. This will create a single graph of all the visited notes.

At first glance, this question might seem trivial. However, the process behind a browser loading a webpage is a complex marvel that we take for granted every day—handling millions of requests and intricate routing to reach servers like Google’s. Let’s take a journey into this process.

We’ll break it down into two main steps: first, discovering the site’s address, and second, retrieving data from that site. For the sake of brevity, some details will be simplified.

As 2024 draws to an end, it’s time for me to reflect on the year’s achievements and challenges.

Professional

Overall, 2024 was a year of growth, and I’m really happy with the improvement in the quality of my work. Here’s what I learned and worked on:

I want to put these out in the open in order to give my self some accountability.

1. More Blog posts

I have started more this year (2024) however I want to continue this trend into 2025. The goal is 1 post a month.

As the year draws to a close, I thought I would reflect on some of the apps I found really useful in 2024.

1. Things3

I have been using Things3 for around six years, and it has been working really well for keeping me focused. In particular, I love having the lock screen widget to pressure me into completing daily tasks.Website

I have been a happy WordPress user for the past eight years. It has served me well, but the time has come to explore other options for hosting my blog.

My Requirements for a New Platform

Before choosing a new platform, I outlined a few key requirements

I recently had to explain why a task was taking longer than expected, and I needed a place to vocalise my thoughts, so here we go. In my experience generally there are two kinds of problems which tasks can be put into(not some tasks can be both). Wide problem being something is not necessarily difficult to implement but it’s changes require changes all over the code bases, an example being a blocking mechanism. Deep problems are problems which will be complex but in a very limited domain. An example might be adding a new feature which is distinct and does not rely on too much other code. Image of wide vs deep problems

I believe in order to understand something you need to be able to explain it to somebody else.

In that light here is me helping you helping me. I will try and do this for other data structures and algorithms.

Binary search is a search method for searching in a ordered list. Order in this sense meaning the list has a hierarchy that might be purely numeric/temporal but it also could be lexicographical like alphabetical ordering.

I have been programming earnestly for about 8 years and professionally for 5 years. I have experience in front-end development, back-end development, and data science. My path is fairly traditional, considering I initially learned by myself and then continued my education as part of my undergraduate studies and during my placement year.

1. Start with Python. Regardless of whether you plan on learning front-end, back-end, or data science, Python will help you grasp the fundamentals without getting bogged down in syntax, which is something I struggled with initially. While syntax is important, it can be a barrier for many people. I started with Java, which wasn’t a horrible idea, but I spent a lot of time learning higher-level concepts like OOP that were only useful much later on.
2. Start creating things as soon as possible. There can be a temptation to watch YouTube videos, follow guides, and copy and paste the results, but this often gives a false sense of learning. When it came to tasks outside the domain of these guides, I struggled. Instead, look up how to do the specific subtasks needed but not the entire project. Learning a language is similar: knowing words and syntax is important, but knowing how they fit together leads to fluency. Example projects to start with include a diary application, a to-do list, and Hangman. All of these are CLI utilities, so you won’t get bogged down in display methods.
3. Learn how to read documentation. It might sound trivial, but being able to read and fully understand documentation without examples is a key skill that will be invaluable in the future.
4. Start reviewing other code and look at how projects are structured, particularly on GitHub.
5. Experiment with other areas of programming. For example, if you are interested in making websites, try learning JavaScript or TypeScript. If you are interested in making applications, try using Tkinter. If you are into making games, consider looking at Godot.

It’s important to have fun while learning. If you don’t enjoy the learning process, you might find programming hard to enjoy. Learning new skills is a massive part of programming, and it’s a constant journey of learning from others.

Over the past few weeks I have been making a webcralwer. I wanted to do it as way to get better at Go with useful for learnings for Graph databases as well as being fun. The project made use of cloud native items such as AWS SQS, DynamoDB and optionally Neptune which could be swapped out for Neo4j.

What is a webcrawler?

A webcrawler or web spider is a program which visits a website, and fetches all of the links on that site and then visits them. This is how sites like Google/Bing/DuckDuckGo get the pages to populate when searching.