Modern platforms face a constant challenge: balancing scalability with efficiency, especially as systems grow in size and complexity. At Rightmove, this challenge became evident in our overseas properties sitemap generation process – a critical component for ensuring search engines like Google can discover and rank our pages effectively. However, this process was tied to a legacy monolithic system, creating performance bottlenecks and operational risks.
Recognising the need for a more modern and scalable solution, we leveraged event-driven architecture with Kafka. By rethinking the way we process and generate sitemaps, we not only reduced the generation time from 5 hours to just 30 minutes but also created a foundation for future innovation. This blog post explores the challenges we faced, the solutions we evaluated, and how our approach demonstrates the power of modern architecture to solve real-world problems.
What is a sitemap?
Sitemaps play a critical role in improving website discoverability by guiding search engines like Google to the most relevant and up-to-date pages. For a platform as vast as Rightmove, generating sitemaps efficiently is paramount. While most of our sitemap generation processes have been modernised, the overseas property sitemap remained tied to a legacy monolith system. This inefficiency prompted the team to propose a new approach through an RFC (Request for Comments) process, leveraging Kafka event streaming to transform how sitemaps are created.
Evaluating solutions using RFCs
At Rightmove, the RFC (Request for Comments) process fosters collaboration by allowing teams to propose and refine ideas through feedback from a diverse audience. For the overseas sitemap generation, this process was instrumental in evaluating two potential solutions. While this blog focuses on the technical solution, the details of the RFC process and its impact can be explored in a future article.
Current solution: Replicate the existing batch architecture
Rightmove, as a two-decade-old company, has been actively driving initiatives to modernise and innovate its technology stack. However, even relatively recent systems (5 years old) are sometimes considered legacy due to the pace of innovation. The overseas property sitemap generation process was no exception. Despite efforts to modernise other parts of the sitemap generation, this particular process was still reliant on a monolithic application slated for decommissioning.
The legacy approach to generating sitemaps was riddled with challenges that hindered efficiency and scalability. This process, implemented as a nightly Spring Boot batch job, relied heavily on a single-instance architecture. While leveraging RX Java (reactive java) for parallelisation and caching helped, the scale of the task – generating sitemaps for 266,569 regions – still led to significant limitations:
- Performance Bottlenecks: Completing the process could take up to 5 hours.
- Single Point of Failure: The single-instance design meant that any downtime or unavailability of dependent services during execution could cause the entire process to fail. This lack of resilience made the system highly vulnerable to interruptions, which it did not handle gracefully.
- Increased Load on Services: Hundreds of thousands of REST requests to dependent services added unnecessary stress to the ecosystem.
While this architecture had served its purpose, the system’s fragility and inefficiencies made it clear that a more resilient and scalable solution was essential.
Continuing with the current architecture would have its pros and cons:
- Pros:
- No additional storage requirements.
- Leverages an existing pattern already in use within Rightmove
- Easier debugging of issues, as it runs procedurally within a single process
- Cons:
- Not resilient: A single instance meant any failure disrupted the process.
- Increased load: Added unnecessary stress to other services during nightly execution.
- High coupling: Hundreds of thousands of REST requests increased dependency and complexity.
Proposed solution: Event-driven architecture
This approach leveraged event streams already being published by microservices, such as Regions and Property. The sitemap generator would consume these events, maintain a local state, and construct the sitemap dynamically or periodically.
- Pros:
- Decoupled from other services, reducing dependencies.
- Potential for additional business insights (e.g., live property counts per region).
- Fewer REST requests, reducing load on upstream services.
- Scalable architecture capable of handling dynamic growth.
- Cons:
- Added complexity in maintaining local state and ensuring data consistency.
- Requires additional storage to keep the local state up-to-date.
How it works
The new architecture for sitemap generation is around Kafka event streaming. The sitemap-generator service consumes events from two primary sources:
- Regions Microservice: Events such as region creation, updates, and deletions. Regions can be any region or subregions, i.e. Spain, Andalucia, etc…
- Property Microservice: Events related to property status, such as updates, archiving, or deletions.
The sitemap-generator service maintains a local database to store these events, ensuring that it can dynamically construct the sitemap or generate it periodically. The decision to generate sitemaps dynamically or in batches was delegated to the service team for flexibility.
- Event Consumption:
The sitemap-generator listens to Kafka topics like regions and properties. These events include details such as region IDs, property IDs, and status updates. - State Management:
The service maintains a lightweight database, indexed by region IDs, to store the latest state of properties and regions. For example:- If a property is archived, it is removed from the sitemap.
- If a region is deleted, all associated properties are also removed.
- Scalability:
The service can scale horizontally to handle increased event throughput. By decoupling from REST APIs, it reduces load on other services and ensures real-time updates.
Benefits of the new architecture
This event-driven approach provides several key advantages:
- Improved Efficiency: Eliminates the need for hundreds of thousands of REST requests, reducing the generation time significantly.
- Scalability: The service can scale horizontally to handle more regions or properties.
- Resilience: By consuming events asynchronously, the service is less prone to failures caused by upstream downtime.
- Future Potential: This architecture opens the door for additional features, such as real-time property counts by region or other analytics.
Conclusion
Transitioning the overseas sitemap generation process to an event-driven architecture has improved system efficiency, reduced operational complexity, and laid the groundwork for further improvements. By leveraging the RFC process, the team benefited from a collaborative approach that ensured a well-rounded and future-proof solution. This project serves as a testament to the power of modernising legacy systems and the importance of open communication in driving technical excellence.