Overview of the problem

One of the key parts of the design of my search engine is the ability for the spiders to send the pages which they have explored to a central point. Additionally, a more conventional design might use a push/pull system, so the conductor polls each spider in turn and requests the pages which it has seen using pagination. This has several problems.

As, I mentioned in my last post . I’ve been busy building my search engine lately, and it’s been a lot of fun diving deep into Kubernetes and learning how to set everything up from scratch.

Overview of the System

I was re-reading “I am feeling lucky” and this inspired me, so I have decided to create a search engine. This is a very ambitious project; however, it will teach me a lot about building a micro service with Kubernetes all from scratch.

My plan is to refactor the spider to use a relational database. I am learning towards each spider having its own SQLite db. Then create a service which can go to the spiders and get their current crawl and add them to a central DB. This will create a single graph of all the visited notes.

At first glance, this question might seem trivial. However, the process behind a browser loading a webpage is a complex marvel that we take for granted every day—handling millions of requests and intricate routing to reach servers like Google’s. Let’s take a journey into this process.

We’ll break it down into two main steps: first, discovering the site’s address, and second, retrieving data from that site. For the sake of brevity, some details will be simplified.

As 2024 draws to an end, it’s time for me to reflect on the year’s achievements and challenges.

Professional

Overall, 2024 was a year of growth, and I’m really happy with the improvement in the quality of my work. Here’s what I learned and worked on:

I want to put these out in the open in order to give my self some accountability.

1. More Blog posts

I have started more this year (2024) however I want to continue this trend into 2025. The goal is 1 post a month.

As the year draws to a close, I thought I would reflect on some of the apps I found really useful in 2024.

1. Things3

I have been using Things3 for around six years, and it has been working really well for keeping me focused. In particular, I love having the lock screen widget to pressure me into completing daily tasks.Website

I have been a happy WordPress user for the past eight years. It has served me well, but the time has come to explore other options for hosting my blog.

My Requirements for a New Platform

Before choosing a new platform, I outlined a few key requirements

I recently had to explain why a task was taking longer than expected, and I needed a place to vocalise my thoughts, so here we go. In my experience generally there are two kinds of problems which tasks can be put into(not some tasks can be both). Wide problem being something is not necessarily difficult to implement but it’s changes require changes all over the code bases, an example being a blocking mechanism. Deep problems are problems which will be complex but in a very limited domain. An example might be adding a new feature which is distinct and does not rely on too much other code. Image of wide vs deep problems

I believe in order to understand something you need to be able to explain it to somebody else.

In that light here is me helping you helping me. I will try and do this for other data structures and algorithms.

Binary search is a search method for searching in a ordered list. Order in this sense meaning the list has a hierarchy that might be purely numeric/temporal but it also could be lexicographical like alphabetical ordering.