Every application has a narrator commenting its execution, be it a humble println or a more structured log. But this narrator is unreliable: It decides what's important and what's not, forgets to mention the juiciest parts of the plot, and usually rambles for gigabytes.
Obviously, though, there are more things coming out of your application: Metrics, tracing events, all the system chatter that surrounds a running process. Every single sub-system supporting your application, be it databases, message queues, git dags, which deep down are logs.
Meanwhile, architects have been promoting Event Sourcing as a full and faithful representation of the application's business. From the "domain" emerges a well-defined language, exhaustively describing all event affecting the state of the application and focusing on its dynamics.
But what if there was
One Log? What if we used well structured messages, integrating in a single stream dtrace application logs, iostat metrics, prometheus signals, and domain events. What if we relinquished up-front filtering and throttling and let serendipity do its job? What if the separate realms of information (business events, kibana views, grafana boards) we work with were just views of a big stream of log events?
This session will be a live-running experiment pushing knobs to 11 and explore what information we can harvest from this hoard of data. Building upon a simple event source application, we'll aggregate more events, implement traffic replay as a reverse event log, embrace system logs, and see what happens when we don't silence the narrator but let it loose.
Arnaud Bailly & Yann Schwartz