SYS-CON MEDIA Authors: Maria C. Horton, Jason Bloomberg, Elizabeth White, Zakia Bouachraoui, Andy Thurai

Blog Feed Post

From Scala Unified Logging to Full System ObservabilityPart 2 of 3: How We Made Logging Great Again

Jonathan is a platform engineer at VictorOps, responsible for system scalability and performance. This is the second part in a series on system visibility, the Detection and Analysis part of the Incident Management Lifecycle. If you missed it, read Part 1 first.

Unified Interface

Like any good engineering approach, we wanted a simple, thin facade that would codify our use-cases, provide a single extension point, and be low-overhead to maintain. With the exception of appender performance, we needed unification of the interfaces in order to address all the concerns mentioned in Part 1. We did just that.

We created a top level Logger that would house all supplemental functionality and become our primary interface regardless of whether we’re logging from a “typical” class, an actor, or a controller. There are two implementations: ClassLogger and ActorLogger which behind the scenes make use of SLF4J and Akka logging, respectively. So far, we haven’t had a use case for a separate controller-oriented logger and simply used the ClassLogger in our controllers.


Next, we created a Logging trait in order to further standardize our logger names and break the stigma of a separate Logger for each class in a class hierarchy, in favor of a single Logger per instance. In other words, we lose the context of which hierarchical class in favor of the gains it brings. This has the benefit of providing better context to any log statements called from a parent class or trait mixin. This also removed further developer decisions by mixing in a logger with the same field name, logger name, and identity* which we’ll get into next, regarding log variables.

*identity–in this case I’m referring to the information you want printed in every log statement from that class. For example, if the class is associated to a specific user, the username may be a part of its identity.

Our supplemental functionality

Support for Log Variables

The goal of codifying use-cases was achieved through adding first class support for log variables — which we define as any key-value pair you want easily parsed from your log statement. This meant the developer was now able to simply pass a tuple (or with macros, just the aptly named variable) to the log method and it was formatted by the facade. No more formatting errors, like this:

Notice the missing closing brace and, in this case, the inconsistency of this statement using colon+space instead of equals. That simple mistake would lead to a failure to extract the variable in your log parsing (in Sumo/Elastic/Splunk/etc) making it somewhat hidden and potentially yield the log statement useless.

Once we had basic support for log variables we got some ideas. For example, how about allowing a class to set class-level log variables that are printed in every log statement from a class? We did just that. Then, we added the ability to simply create a formatted string using the same functionality – useful for strings given to other objects, like a failed Future. Below, see how we were able to expand the use of class-level log variables as well as string creation into failure messages:

Implicit loggers for utility classes


Another perspective on log variables is that they allow you to capture the context of a log statement which helps to describe the data being acted upon. Implicit loggers allow utility objects to assume the identity of the caller when logging relevant information about their job. So, instead of telling you about a needle in a haystack:

We now specify who wanted it (since the class represents the caller) as well as any context they set up in their logger (like the org and user in this example):

Sanitizing PII

A feature worth considering is sanitizing PII (personally identifiable information) in log statements. There are some references to functionality like this in the GoDaddy logger and others. So, focusing on maintaining a thin facade, you could completely replace Logback with the GoDaddy Logger and add this functionality without modifying many, if any, log statements directly.

Is Logging Great Again?

This simple solution totaled 452 lines of code, including tests and nearly 100 lines of comments to aid in migrating files — the developer responses were surprising to say the least.

Remember the last time you got a splinter in your foot? Such a simple task of removing the splinter relieves that constant annoyance that can easily build up if not addressed. Ha ha! Our state of logging was just like that splinter. Relieved comments began to trickle in, one by one, on how devs have been pleasantly surprised with the new experience of logging on our platform.

Most of those comments centered on how the unified logging was implemented at the right level of abstraction (flexibility versus structure) and that it made logging simple and, because of that, enjoyable to use. These days at VictorOps, developers are progressively migrating the files they touch in order to reap the benefits and the simplicity they’ve become accustomed to.

Here’s a before/after displaying the reuse, codified formatting, and maintainability improvements:

With this change, we progressed in two very important ways as a team: We addressed some tech debt that had become a larger nuisance than we had realized; and, we improved our log searchability by providing simplicity and just enough structure to get the job done.

So, with our logging in a much happier place, we were prepared to move to the next step of improving our monitoring and instrumentation. In Part 3, we’ll talk about how metrics have become our next focal point.

The post From Scala Unified Logging to Full System Observability
Part 2 of 3: How We Made Logging Great Again
appeared first on VictorOps.

Read the original blog entry...

More Stories By VictorOps Blog

VictorOps is making on-call suck less with the only collaborative alert management platform on the market.

With easy on-call scheduling management, a real-time incident timeline that gives you contextual relevance around your alerts and powerful reporting features that make post-mortems more effective, VictorOps helps your IT/DevOps team solve problems faster.

Latest Stories
In today's always-on world, customer expectations have changed. Competitive differentiation is delivered through rapid software innovations, the ability to respond to issues quickly and by releasing high-quality code with minimal interruptions. DevOps isn't some far off goal; it's methodologies and practices are a response to this demand. The demand to go faster. The demand for more uptime. The demand to innovate. In this keynote, we will cover the Nutanix Developer Stack. Built from the foundat...
Two weeks ago (November 3-5), I attended the Cloud Expo Silicon Valley as a speaker, where I presented on the security and privacy due diligence requirements for cloud solutions. Cloud security is a topical issue for every CIO, CISO, and technology buyer. Decision-makers are always looking for insights on how to mitigate the security risks of implementing and using cloud solutions. Based on the presentation topics covered at the conference, as well as the general discussions heard between sessio...
"NetApp's vision is how we help organizations manage data - delivering the right data in the right place, in the right time, to the people who need it, and doing it agnostic to what the platform is," explained Josh Atwell, Developer Advocate for NetApp, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Sold by Nutanix, Nutanix Mine with Veeam can be deployed in minutes and simplifies the full lifecycle of data backup operations, including on-going management, scaling and troubleshooting. The offering combines highly-efficient storage working in concert with Veeam Backup and Replication, helping customers achieve comprehensive data protection for all their workloads — virtual, physical and private cloud —to meet increasing business demands for uptime and productivity.
The Software Defined Data Center (SDDC), which enables organizations to seamlessly run in a hybrid cloud model (public + private cloud), is here to stay. IDC estimates that the software-defined networking market will be valued at $3.7 billion by 2016. Security is a key component and benefit of the SDDC, and offers an opportunity to build security 'from the ground up' and weave it into the environment from day one. In his session at 16th Cloud Expo, Reuven Harrison, CTO and Co-Founder of Tufin, ...
While the focus and objectives of IoT initiatives are many and diverse, they all share a few common attributes, and one of those is the network. Commonly, that network includes the Internet, over which there isn't any real control for performance and availability. Or is there? The current state of the art for Big Data analytics, as applied to network telemetry, offers new opportunities for improving and assuring operational integrity. In his session at @ThingsExpo, Jim Frey, Vice President of S...
"We were founded in 2003 and the way we were founded was about good backup and good disaster recovery for our clients, and for the last 20 years we've been pretty consistent with that," noted Marc Malafronte, Territory Manager at StorageCraft, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Historically, some banking activities such as trading have been relying heavily on analytics and cutting edge algorithmic tools. The coming of age of powerful data analytics solutions combined with the development of intelligent algorithms have created new opportunities for financial institutions. In his session at 20th Cloud Expo, Sebastien Meunier, Head of Digital for North America at Chappuis Halder & Co., discussed how these tools can be leveraged to develop a lasting competitive advantage ...
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settl...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm. In their Day 3 Keynote at 20th Cloud Expo, Chris Brown, a Solutions Marketing Manager at Nutanix, and Mark Lav...
@CloudEXPO and @ExpoDX, two of the most influential technology events in the world, have hosted hundreds of sponsors and exhibitors since our launch 10 years ago. @CloudEXPO and @ExpoDX New York and Silicon Valley provide a full year of face-to-face marketing opportunities for your company. Each sponsorship and exhibit package comes with pre and post-show marketing programs. By sponsoring and exhibiting in New York and Silicon Valley, you reach a full complement of decision makers and buyers in ...
According to the IDC InfoBrief, Sponsored by Nutanix, “Surviving and Thriving in a Multi-cloud World,” multicloud deployments are now the norm for enterprise organizations – less than 30% of customers report using single cloud environments. Most customers leverage different cloud platforms across multiple service providers. The interoperability of data and applications between these varied cloud environments is growing in importance and yet access to hybrid cloud capabilities where a single appl...
"At the keynote this morning we spoke about the value proposition of Nutanix, of having a DevOps culture and a mindset, and the business outcomes of achieving agility and scale, which everybody here is trying to accomplish," noted Mark Lavi, DevOps Solution Architect at Nutanix, in this SYS-CON.tv interview at @DevOpsSummit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"Cloud computing is certainly changing how people consume storage, how they use it, and what they use it for. It's also making people rethink how they architect their environment," stated Brad Winett, Senior Technologist for DDN Storage, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
A look across the tech landscape at the disruptive technologies that are increasing in prominence and speculate as to which will be most impactful for communications – namely, AI and Cloud Computing. In his session at 20th Cloud Expo, Curtis Peterson, VP of Operations at RingCentral, highlighted the current challenges of these transformative technologies and shared strategies for preparing your organization for these changes. This “view from the top” outlined the latest trends and developments i...