Distributed Architecture for Operational Workloads
There has been a tremendous growth of Big Data in the past few years with Machine Learning being used to draw intelligent insights for competitive advantage. Big Data has been fueled by social media, IoT, and a variety of other data types and sources. As companies have matured in their Machine Learning implementations, there has been an increase in the need to apply models generated by these algorithms to events as they are streaming in, to trigger real-time actions, or for operational dashboards based on those events. To reduce the latency between receiving these events, analyzing them, and deploying the models, these operational workloads need to coexist on the same scale-out data platform that is receiving these events and processing business intelligence or analytic queries.
Additionally, contextual data is needed for both the operational and analytical workloads. Master data, such as customer, product, etc. are needed to provide the context for this event data. Historical data is needed to get a longer-term context for these events. And transactional data is needed to provide the current context. In other words, a distributed data platform is needed to support a hybrid of operational and analytical processing workloads, in order to not only gain intelligence from streaming data, but to make that intelligence operationally actionable in real-time.
Preceding the Big Data revolution, scale-out operational workloads motivated the rise of NoSQL technologies. The early creators and adopters of these technologies faced high volume and velocity of operational events, that databases such as Oracle and Microsoft SQL Server could not keep up with. Scale-out implementations of these databases would have been prohibitively expensive. While document databases such as MongoDB, where a transaction is encapsulated within a single JSON document, eventual consistency models, or row-level immediate consistency solutions, are good for a certain set of operational applications, they do not address a large number of more complex SQL-based ACID transactional requirements. These capabilities are provided now by distributed databases, such as EsgynDB, on scale-out platforms, enabling the hosting of many more OLTP and operational workloads that need elastic linear scalability and high concurrency, without compromising on performance and enterprise class manageability.
The widespread growth in online transactions is coming from the ubiquitous mobile revolution, proliferation of mobile apps, mobile and micro-payments, dramatic increase in the world of users with mobile devices and access to the internet, down to farmers in villages. Current OLTP systems are bursting at the seams, with the inability of vendors like Oracle to keep up with this increase at a reasonable price per transaction without compromising performance. These systems need to be offloaded, migrated, or completely revamped to provide new interfaces, functionality, or to leverage micro-services and container-based distributed application architectures.
Finally, the need for scale-out Operational Data Stores, has also increased commensurate with the increase in OLTP volumes. Call centers, voice recognition systems and Bots to provide support and services, real-time monitoring of transported goods through the supply chain, immediate notification of financial transactions to mobile devices, online or mobile account inquiries, are amongst a myriad of back-end operational applications, increasingly needing to scale-out.
All this points to the fact that if distributed architectures are not part of your current OLTP / operational IT landscape, they soon will be. In this day and age, you want to architect a system that is not siloed, since the lines between operational and analytical workloads is blurring, and latency, introduced by moving / replicating and transforming data between silos, can be a huge competitive disadvantage.