Project 01/03MC-SYS-02
All projectsSearch engine: scraped pages flow through two Kafka topics and a sanitisation stage into Cassandra.
Build a search engine able to process and index large volumes of data. Two paths must coexist: adding data (scraping more pages into the index) and retrieving it (answering a user's query). The volume requirement calls for a distributed architecture.
Source code is public on GitHub.
Hover or focus a component to trace its connections.
Data paths, in order
Data paths, in order
Source of raw pages. Feeds the back end to index more of the web.
Entry point for both paths. Publishes scraped data to Kafka, forwards queries to the AI stage, and reads and writes users in PostgreSQL.
Raw data topic ("parsing" in the Kafka diagram). The raw-data schema is registered in a schema registry.
Python consumer: quality control on raw data before it is indexed. Republishes clean records.
Clean data topic. Its consumer creates or updates records in Cassandra.
Index store for processed pages.
User interface. Sends queries and account operations to the back end.
Entry point for both paths. Publishes scraped data to Kafka, forwards queries to the AI stage, and reads and writes users in PostgreSQL.
Stage between the back end and Cassandra on the query path, intended to improve result relevance with machine-learning models.
Index store for processed pages.
User interface. Sends queries and account operations to the back end.
Entry point for both paths. Publishes scraped data to Kafka, forwards queries to the AI stage, and reads and writes users in PostgreSQL.
Users and permissions, with role-based access control (RBAC).
Entry point for both paths. Publishes scraped data to Kafka, forwards queries to the AI stage, and reads and writes users in PostgreSQL.
Receives logs; real-time dashboards for performance and error tracking.
Source of raw pages. Feeds the back end to index more of the web.
User interface. Sends queries and account operations to the back end.
Receives logs; real-time dashboards for performance and error tracking.
Entry point for both paths. Publishes scraped data to Kafka, forwards queries to the AI stage, and reads and writes users in PostgreSQL.
Raw data topic ("parsing" in the Kafka diagram). The raw-data schema is registered in a schema registry.
Stage between the back end and Cassandra on the query path, intended to improve result relevance with machine-learning models.
Users and permissions, with role-based access control (RBAC).
Python consumer: quality control on raw data before it is indexed. Republishes clean records.
Clean data topic. Its consumer creates or updates records in Cassandra.
Index store for processed pages.
Scraped pages reach the back end, which publishes them to a Kafka topic. A Python sanitisation stage consumes that topic, checks and cleans the data, and publishes the result to a second topic, treated_data, which is written into Cassandra. On the query side, the front end calls the back end, which passes the query through an AI stage that reads from Cassandra and ranks the results. PostgreSQL stores users and their roles; logs are shipped to Grafana.
Redrawn from the project's data-flow, Kafka and user-management diagrams. The log path to Grafana is drawn from the back end only.
Decision 01 / 07
Decision 02 / 07
Decision 03 / 07
Decision 04 / 07
Decision 05 / 07
Decision 06 / 07
Decision 07 / 07