Every application has a narrator commenting its execution, be it a humble println or a more structured log. But this narrator
is unreliable: It decides what's important and what's not, forgets to mention the juiciest
parts of the plot, and usually rambles for gigabytes.
Obviously, though, there are more things coming out of your application: Metrics, tracing events, all the system chatter
that surrounds a running process. Every single sub-system supporting your application, be it
databases, message queues, git dags, which deep down are logs.
Meanwhile, architects have been promoting Event Sourcing as a full and faithful representation of the application's business.
From the "domain" emerges a well-defined language, exhaustively describing all event affecting
the state of the application and focusing on its dynamics.
But what if there was
One Log? What if we used well structured messages, integrating in a single stream
dtrace application logs, iostat metrics, prometheus signals, and domain events. What if we
relinquished up-front filtering and throttling and let serendipity do its job? What if the
separate realms of information (business events, kibana views, grafana boards) we work with
were just views of a big stream of log events?
This session will be a live-running experiment pushing knobs to 11 and explore what information we can harvest from this
hoard of data. Building upon a simple event source application, we'll aggregate more events,
implement traffic replay as a reverse event log, embrace system logs, and see what happens
when we don't silence the narrator but let it loose.
Arnaud Bailly & Yann Schwartz