Following logs

Info

The information in this chapter is relevant since Logdy v0.6

The above drawing presents a possible setup of Logdy with multiple writing processes and multiple reading clients at the same time.

• There could be multiple processes writing simultaneously to Logdy
• There could be multiple clients, reading/following messages from a buffer. Each client keeps it's a state of their own cursor

Message buffer

There are two mechanisms in Logdy that collect and buffer logs. In this chapter, you will learn how they work and what is an effective usage.

Backend

The buffer on the backend (in Logdy process) is implemented as an effective ring buffer. That buffer is able to keep hundreds of thousands messages in memory (even millions, the limit is only the operating memory available to the Logdy process). The capacity of this buffer is fixed and defined on a process startup (with a CLI option). All messages that has been delivered to Logdy process are put into buffer and kept there. Since the buffer is fixed size, once the buffer reaches it's limits, oldest messages will be removed (FIFO).

Since this buffer keeps all of the messages, each client can read messages at a different offset.

Browser

Each client, who is effectively a browser connects to a backend and starts reading messages from a ring buffer through a websocket connections. The client keeps a buffer of the messages on it's own. This is due to the fact that each message is parsed with custom code once it reaches the client. The buffer is persisted to a localStorage on the browser, this lets us save some work the browser needs to do after a page reload.

This buffer is configurable in size (see CLI reference) dynamically and a client can change that setting without restarting either client or the backend process.

Since browsers are not designed to handle big amounts of data, depending on a machine and complexity of parsed logs, 2,000 logs in a buffer provides a balance between performance and enough big window for browsing logs.

Flow control

The above screenshot presents controls that let the client control an ingestion of logs messages.

Pause logs

1st button - Since the client can receive newly added messags, it can also command the backend to pause. This means, a user can pause streaming and inspect a current batch.

Follow new logs

2nd button - As previously mentioned Logdy process streams new messages it receives to a client. This button will trigger following new log messages that has been added to a buffer.

Resume following logs

3rd button - Connecting all of the concepts together, after stream is paused, you can resume it from a place (offset) where it was paused.