Why success brings its own challenges in IT: Fixing the real-time problems of real-time event streaming with an event portal

Spokesperson: Tom Fairbairn, Distinguished Engineer, Solace

Amidst the recent hype around generative AI, there is a perhaps quieter technology revolution going on: the increasing adoption of event-driven architecture (EDA), an example of which is “event streaming” across the enterprise. Regardless of industry, (manufacturing, retail or even HR, amongst others), adoption of EDA is increasing use cases across the board as organisations realise the value of EDA. From decoupling applications, the ability to be more responsive to internal and external user requests, and reducing operational costs, EDA ultimately allows enterprises the freedom to become more agile and real-time in their day-to-day activities.

A recent IDC Infobrief found that of those who have deployed EDA across multiple use cases, 93% say it either met or exceeded their expectations. Going even further, 82% of those surveyed have stated their plans to apply EDA across their company for a further two/three use cases, within the next 24 months.

Increasing adoption and time-to-market scrambles, along with bigger data flows and more complex deployments lead to difficulties in getting real visibility across a complex, and often rapidly scaling ecosystem of events.

As a result, some are struggling to realise the best return on their EDA investment. The issue can sometimes compound, as an organisation may have reached a stage of maturity regarding multiple streaming use cases, but are left with siloed brokers, clusters, topics and schematics weaved on top of each other. A layer cake effect occurs in the architecture, making it difficult to truly see where they can optimise the information insights chain.

Growing pains are a natural part of scalability, and EDA is no exception.

Three pain points typically arise:

1-Poor visibility requires a clear and clean portal

One of the unsung heroes of EDA is application decoupling, which aids flexibility and agility. However, this can often be detrimental to visibility, making changes to existing applications difficult and risky. This lack of visibility leads to confused ownership, as event producers don’t know who the consumers are, and consumers don’t know who the publishers are. Some event streaming systems, such as Kafka, rely on static or near-static topics which can be abandoned or duplicated.

This directly affects the end-user experience. Say a person is using the application every day and wants to add an extra attribute to the data – they’d have no direct line to contact because they can’t see who produced the data.

An event portal’s single window into the structure of an EDA benefits the event-streaming ecosystem by acting as a native discovery agent to quickly produce visual aids to help locate consumers and producers. Architects and administrators on the business side are able to view the relationship between producers and consumers, as well as their event interface and KPIs, including their most and least-used topics.

Another issue that occurs when scaling infrastructure and information flows is the inability to fully grasp the downstream impact of changes made to the architecture. Evolving and architecture will require visibility on whether microservices are affected by a given event before even deploying a new feature or function, to ensure it doesn’t bring down that system. This must be done sometimes in seconds. While real-time is part of the solution, an event portal can automate the scanning of an internal system, helping visualise a complete view of endpoints and event streams, rather than making the business manually draw them between microservices. For those looking to better understand their infrastructure, particularly as they scale up, an event portal helps deploy new services faster.

2-Event Streaming is not a one-and-done – the issue with limited sharing and reuse

Data has been the new oil for quite some time, and much like oil, when one needs to see data, they often need it right there and then. Real-time data is the most valuable data that can exist within your organisation, but if it is siloed in one specific department, overall business decision-makers don’t know about it, and that means it is not providing its full potential.

Developers must understand that their hard work won’t be realised until they can catalogue all that data and the benefits that come with it. Otherwise, it may get stale and quickly become out of date.

An event portal can create a real-time catalogue of event data, able to list all topics, event streams and schemas for each application, as well as the owner and best point of contact. Not only does this speed up internal developments by letting developers share, discover, and re-use existing event-streaming assets, but they can do so outside of the organisation so that customers and partners can benefit from it too – an often neglected consideration when technologies such as Kafka focus on operational management.

3-Security and governance in event streams

The fact that event-driven systems are dynamic and decentralised is a major selling point. However, that can run the risk of unique security challenges.

A trade-off that commonly occurs when implementing event streaming is that, as they include access control rules, developers tend to be too permissive to ensure agility, creating a lack of visibility or proper cataloguing of the data. With time, that runs the risk of interfering with business data security, governance and compliance. The issue only gets worse over time as the use of data evolves across the applications, with each new human addition.

With a thinning attack surface and an increase in never-before-seen malware variants, security in business must pivot from reactive to proactive.

An event portal helps users organise systems into application domains, create and import payload schema definitions in a variety of formats and better define event interactions. This improves IT governance by more easily allowing control for who can access which resources while giving those the ability to create and track every version of EDA objects as they evolve.

By putting visibility at the forefront, administrators can spend less time worrying about ensuring their security is in line with internal and regulatory policies. Outside of real-time visibility, the true value of event-driven architecture lies in the ability to easily spot and re-use all of these data assets. However, that can only be done if they are documented, managed and governed properly. The event portal is necessary to solve this, acting as a discovery tool, de facto guide, and being able to dictate the more effective way to manage event streams.

Real-time event streams and the success of powerful architectures have brought with them the need to address the rapidly growing complexity of event streaming estates. Single, multi-broker event portal technology is the solution to helping organisations discover, document, govern and manage the lifecycle of their real-time event streams across the enterprise, and it will become increasingly important as more organisations embrace EDA as a foundational platform.