8 min read
Modernising Legacy Systems: Step by Step and Low-Risk with Apache Kafka and Debezium
Many companies face the same decision: keep running an outdated core system or replace it entirely in one large project. Both paths carry considerable risk. This article describes a third way, in which the existing system is replaced step by step while it remains in operation.
The cost of standing still
Legacy systems are not just a technical nuisance but a growing business risk. The costs arise in four areas.
Rising maintenance effort
Outdated systems absorb a large share of the IT budget. On top of that come expenses for scarce spare parts, expiring licences and specialised staff. This budget is then missing for innovation.
Rigid structures and data silos
Legacy systems can only be adapted to new requirements with great effort. Important data remains trapped in isolated systems and is not available to other departments, analytics or AI applications.
Security risks
Systems without vendor support no longer receive security updates. Every known vulnerability remains open and increases the risk of data loss, business interruption and regulatory breaches.
Dependency on a few specialists
Knowledge of old technologies and poorly documented systems often rests with a few people. If they leave the company or retire, operational risk rises significantly.
These factors reinforce each other: high maintenance costs crowd out investment, lack of investment weakens competitiveness, and scarce specialists make operations even more expensive. The longer a company waits, the harder modernisation becomes.
Why big-bang projects fail
The obvious way out is often a complete rewrite that replaces the legacy system on a fixed date. This very approach is particularly risky. Systems theorist John Gall offers an explanation, which we discuss in more detail in our article on Systemantics.
Gall’s Law
A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work.
Complex systems contain numerous unknown dependencies that cannot be fully anticipated in a new design. A system that only meets reality at the end of a long project therefore often fails in unexpected ways. A well-known example is the BBC’s Digital Media Initiative, which was cancelled in 2013 after years of development.
Implications for investment decisions
Gall’s Law is not only a rule for development teams but also a principle for risk management:
- Big bang: A large upfront investment rests on assumptions about a future target state that often prove incomplete later.
- Evolutionary approach: The first investment goes into a small, working building block. Every further stage builds on proven results.
The key question is no longer “What does the new system cost in total?” but “What does the next step that demonstrably delivers value cost?”.
The strangler fig pattern
The strangler fig pattern, popularised by Martin Fowler, puts this idea into practice for legacy modernisation.
The principle
The name comes from the strangler fig, which grows around a host tree and gradually replaces it. Applied to software, this means:
- New functionality is built as independent services alongside the legacy system.
- A routing layer in front of both gradually redirects requests to the new services.
- Once all functionality has been taken over, the legacy system is switched off.
Big bang and strangler fig compared
| Criterion | Big-bang rewrite | Strangler fig pattern |
|---|---|---|
| Project risk | High: success or failure is decided at cut-over. | Limited: every change is small and can be rolled back individually. |
| Upfront investment | High: the entire project must be funded before any value is delivered. | Low: investment happens step by step and follows results. |
| First value | After the entire project is complete, often only after years. | With the first service in production, usually within weeks or a few months. |
| Business disruption | Hard cut-over with the risk of extended outages. | Low: the legacy system stays in operation and users are migrated gradually. |
| Feedback | Late, when course corrections are expensive. | Continuous, with every new service. |
The tools: Apache Kafka and Debezium
Apache Kafka: decoupling old and new
Apache Kafka is a distributed event streaming platform. Systems publish events such as “order created” without knowing who will process them. Other systems subscribe to these events independently.
For modernisation, this has three advantages:
- Decoupling: Old and new systems do not need to be connected directly. Maintenance windows in the legacy system do not stop the new services.
- Extensibility: Additional services can use the same data without changing the legacy system again.
- Reliability: Kafka retains events for a configurable period. If a service fails, it can process the missed events afterwards.
Debezium: capturing data changes without touching the application
Debezium implements change data capture, the recording of data changes directly at the source. It reads the database’s transaction log, which the database maintains anyway. The application code and database schema of the legacy system remain unchanged. Depending on the database, only the log configuration needs to be adjusted.
Inserts, updates and deletes are thus transferred to Kafka in near real time, with little additional load and without destabilising the legacy system.
The architecture at a glance
| Component | Role |
|---|---|
| Legacy system and database | Continues to run the core business and remains the leading data source until it is replaced. |
| Debezium | Captures every data change from the transaction log without changing the application. |
| Apache Kafka | Receives the changes and makes them available to all authorised services. Decouples old and new. |
| New services | Take over individual business functions based on data from Kafka and gradually replace the legacy system. |
The migration in six steps
1. Select the first building block
Choose a clearly bounded business function with few dependencies. Domain-driven design methods help identify suitable boundaries. The first building block should deliver noticeable value but should not be the most critical process.
2. Capture data changes with Debezium
A Debezium connector monitors the tables that belong to the selected building block. It first creates a consistent snapshot of the data and then continuously streams all changes.
3. Set up Kafka as the central data layer
Debezium writes the changes to dedicated Kafka topics, for example changes to the customer table to a topic for customer data. These topics are available to all authorised services.
4. Build the first new service
A new, lean service takes over the selected business function. It does not access the legacy database but obtains its data exclusively from Kafka. This ensures decoupling from the start, in line with Gall’s Law: a simple system that works.
5. Run in parallel and validate
The routing layer initially forwards only read requests to the new service, while the legacy system continues to handle all writes. This makes it possible to compare results and performance with real production traffic. After successful validation, the new service also takes over writes. If synchronisation in both directions is temporarily required, the new service publishes its changes as events as well.
6. Replace and repeat
Once the new service has proven itself, the corresponding function in the legacy system is retired and the routing is switched permanently. The process then starts again with the next building block, reusing the infrastructure and experience already in place.
Case studies
Netflix: step by step to microservices
Around 2009, Netflix began gradually moving its monolithic architecture to microservices in the cloud. The migration took several years and was carried out service by service. This allowed the company to solve scaling problems without putting ongoing operations at risk.
Event streaming as the backbone of large platforms
Apache Kafka was originally developed at LinkedIn and today serves as the central backbone for data exchange between systems at many companies with high data volumes. Debezium is widely used as an open-source project to connect existing databases to such platforms.
Challenges and risks
Shifted complexity
Complexity does not disappear, it shifts: from the monolith to a distributed system with clusters, connectors, schema changes and network latency. This complexity is manageable, however. Managed Kafka services or Kubernetes operators such as Strimzi significantly reduce the operational effort.
Data consistency
During the transition, legacy and new systems must hold consistent data. Writing to two systems at the same time is error-prone. Debezium and Kafka guarantee that events are delivered at least once and in order within a partition. For stricter requirements, the outbox pattern is a good fit: a service writes its data and the corresponding event in a single transaction, and Debezium reliably forwards the event.
Organisation and skills
New technologies and ways of working require knowledge that teams often still need to build. The incremental approach helps here too: a small pilot team gains experience, documents results and passes the knowledge on to other teams.
Conclusion
A rewrite in a single step contradicts fundamental experience with complex systems. The evolutionary path with the strangler fig pattern, Apache Kafka and Debezium lowers risk, delivers value earlier and can be steered at every point.
The result is more than a retired legacy system. Data and business logic that were previously locked in a system that was hard to change become available on a modern, event-driven platform. This platform is the foundation for new products, data analytics and AI applications.
Articles from our practice
Open-Source CRM: Eleven Systems Compared
Only the license is free. This article compares SuiteCRM, Twenty, EspoCRM, Odoo, and seven other systems in terms of architecture, licensing terms, and costs over a five-year period. It includes recommendations.
Intelligent Compliance: AI as a Strategic Imperative for the Chemie Industry
In industries where errors can cost millions, AI hallucinations are unacceptable. Discover how a purpose-built RAG system mitigates regulatory risks, ensures compliance with complex regulations like REACH, and turns compliance into a strategic competitive
Transform with Kafka & Debezium, minimize risk.
Outdated IT systems represent a growing liability, hindering agility and blocking innovation. This article presents a proven, incremental approach to modernization. Discover how to renew your core systems step-by-step, minimizing risks and future-proofing
Beyond Command and Control: Leadership by Design
Command and Control emphasizes strict hierarchies and streamlined authority. But in today’s fast-paced world, can this model still deliver – or is it a rigid approach in need of rethinking?
Team Topologies: Structuring Teams for Success
Team Topologies helps organizations design their teams for clarity, speed, and collaboration. Learn how strategic team structures can reduce friction and accelerate delivery.
What complex systems teach us about failure, dysfunction, and unexpected behavior
Systems don’t always do what they’re designed to – they do what they can. In “Systemantics,” John Gall unpacks why systems fail, often in surprising ways. A thought-provoking look at complexity in action.