Introduction

Foundation of Arcology Network

Streamer is an advanced, flexible, loose coupling, highly scalable, and in-house developed event-driven architecture (EDA) development framework by Arcology team. Streamer serves as a unified solution for inter-thread and inter-process communication, allowing seamless data exchange between modules regardless of deployment method or location.

Streamer unifies inter-thread and inter-process communication under the same hood. It is designed to ensure that modules always interact with a unified event broker, regardless of the underlying communication mechanisms.

Communication

The event broker receives events from publisher actors and distributes them to subscriber actors based on their interests. In Streamer, an actor usually communicates asynchronously with others. Actors can publish and subscribe to events through the event broker.

For inter-thread use, Streamer permits lockless inter-thread communication by replacing the basic concurrency tools such as channels, lock, and queues, with a unified and pub/sub-based event architecture.

For inter-process communication, Streamer hides all the intricacies from the developer. Interacting with another thread residing in a different thread is just as seamless as communicating with threads in the same process.

Features

Compared to other similar systems, Streamer:

• Abstracts underlying event broker logic into a single, unified framework.

• Provides high performance lockless communication.

• Comes with some built-in event filters to manage complex event flows effectively.

• Seamlessly integrates with Apache Kafka for high-throughput inter-process communication.

• Separates implementation from deployment, allowing developers to focus on logic without worrying about how deployment may affect throughout.

Streamer & Kafka

	Streamer	Kafka
Abstraction of Event Broker Logic	Abstracts complexities into a unified system, minimizing platform-specific code.	Specialized as a distributed streaming platform.
Unified Event Management	Provides a unified approach for inter-thread and inter-process communication.	Optimized for distributed streaming scenarios.
Built-in Event Filtering Mechanism	Offers a built-in mechanism for managing complex event flows efficiently.	Primarily used for message queueing with streaming capabilities.
Integration with 3rd-party Solutions	Seamlessly integrates with Kafka for enhanced interoperability.	Positioned as a standalone distributed streaming platform.
Scalability and Durability	Emphasizes abstraction and flexibility for deployment scenarios.	Known for scalability, durability, and fault-tolerant data streaming.
Use Cases	Suited for event-driven architecture with high scalability requirement.	Ideal for distributed streaming with scalability, durability, and fault tolerance.

Streamer incorporates Apache for high-throughput inter-process communication.

Components

There are two major entities in the whole design, event Broker and Actors.

• Broker: The event broker is responsible for managing the event flow to and from the actors.

• Actors: An actor is an entity that react to the events subscribed from the broker.

Workflow

In Streamer, an Event Broker serves as the central hub for managing and distributing events within the system. Each actor comprises a State Machine that controls overall actor state transitions, an Event Filter governing event relationships and triggering the associated Worker Thread. Developers embed custom logic in the Worker Thread for event and data processing. The Message Wrapper encapsulates the Worker Thread's return values into an event format acceptable to the Event Broker.