Over the last decade or so, there has been a strong movement toward a flexible style of building large systems that have lately been called microservices. When you move to microservices, the data distribution becomes the biggest problem. How to move data transactionally in a distributed architecture of hundreds of microservices while keeping the system in a consistent state. how do we provide services with the data they require correctly in minimal time. This talk describes the similar problem we faced while scaling the services at Nexmo, The Vonage API Platform and how are we solving it.
Other ways to solve these problems are either keep a huge shared global database, which is the violation of microservice fundamentals or create an internal Death star in which failure in a single machine you didn’t even know existed can crash your entire system. These solutions are already in place and eventually becomes more and more difficult to modify and scale.
In this talk, I will present a solution which we are using at Nexmo by deconstructing the database and creating an application independent scale and data aware layer. By using this the application can run independently without worrying about the data. By implementing the practices and architectural choices described in this talk audience will be able to decouple the applications from the data layer and other data dependencies to reduce cascading system failures.
This talk will present the following Data handling patterns:
- Replica management for Consistent & Partition Tolerant in data stores
- Command and Query Responsibility Separation to handle high throughout writes and virtual buckets based replication to manage read hot-spotting on a subset of the data corpus
- Data pre-compute and time-travel to create timeline consistent view of data across multiple distributed systems
- Data sourcing from Database replication mechanisms, local transactions and async relaying to an analytics platform
- By-passing network bottlenecks like centralized load-balancers by using data-placement-aware smart local proxies/sidecar