Introduction
Foundation of Arcology Network
Streamer is an advanced, flexible, loose coupling, highly scalable, and in-house developed event-driven architecture (EDA) development framework by Arcology team. Streamer serves as a unified solution for inter-thread and inter-process communication, allowing seamless data exchange between modules regardless of deployment method or location.
Streamer unifies inter-thread and inter-process communication under the same hood. It is designed to ensure that modules always interact with a unified event broker, regardless of the underlying communication mechanisms.
Communication
The event broker receives events from publisher actors and distributes them to subscriber actors based on their interests. In Streamer, an actor usually communicates asynchronously with others. Actors can publish and subscribe to events through the event broker.
For inter-thread use, Streamer permits lockless inter-thread communication by replacing the basic concurrency tools such as channels, lock, and queues, with a unified and pub/sub-based event architecture.
For inter-process communication, Streamer hides all the intricacies from the developer. Interacting with another thread residing in a different thread is just as seamless as communicating with threads in the same process.
Features
Compared to other similar systems, Streamer:
Abstracts underlying event broker logic into a single, unified framework.
Provides high performance lockless communication.
Comes with some built-in event filters to manage complex event flows effectively.
Seamlessly integrates with Apache Kafka for high-throughput inter-process communication.
Separates implementation from deployment, allowing developers to focus on logic without worrying about how deployment may affect throughout.
Streamer & Kafka
Abstraction of Event Broker Logic
Abstracts complexities into a unified system, minimizing platform-specific code.
Specialized as a distributed streaming platform.
Unified Event Management
Provides a unified approach for inter-thread and inter-process communication.
Optimized for distributed streaming scenarios.
Built-in Event Filtering Mechanism
Offers a built-in mechanism for managing complex event flows efficiently.
Primarily used for message queueing with streaming capabilities.
Integration with 3rd-party Solutions
Seamlessly integrates with Kafka for enhanced interoperability.
Positioned as a standalone distributed streaming platform.
Scalability and Durability
Emphasizes abstraction and flexibility for deployment scenarios.
Known for scalability, durability, and fault-tolerant data streaming.
Use Cases
Suited for event-driven architecture with high scalability requirement.
Ideal for distributed streaming with scalability, durability, and fault tolerance.
Streamer incorporates Apache for high-throughput inter-process communication.
Components
There are two major entities in the whole design, event Broker and Actors.
Broker: The event broker is responsible for managing the event flow to and from the actors.
Actors: An actor is an entity that react to the events subscribed from the broker.
Workflow
In Streamer, an Event Broker serves as the central hub for managing and distributing events within the system. Each actor comprises a State Machine that controls overall actor state transitions, an Event Filter governing event relationships and triggering the associated Worker Thread. Developers embed custom logic in the Worker Thread for event and data processing. The Message Wrapper encapsulates the Worker Thread's return values into an event format acceptable to the Event Broker.
Last updated