SYS-CON MEDIA Authors: Pat Romanski, Liz McMillan, Yeshim Deniz, Elizabeth White, Courtney Abud

Blog Feed Post

Tales from the Field: Debugging Applications in Production with Information Points

Production debugging is one of the most difficult parts of the job for any software engineer and one of the most underrated problems faced by IT. Developers usually rely on logs to troubleshoot production issues. They go through hundreds of lines of logs, sorting through complex logic as their stress levels rise, acutely aware that the bug they are looking for could be crippling the business. It’s painstaking, laborious work at best, and all too often the relevant logs are not available.

Developers need better tools to debug production issues faster. At stake is not only lost revenue but often something even more valuable: The trust of their customers.

I’ve spent many late nights staring at my computer trying to find the root cause of an issue in production environments. The general lack of insights into what is happening in the production environment makes this hard enough, but if the code has been inherited (legacy code) or involves timed elements such as background processes or cron jobs, things get even more difficult to track down. Engineers (myself included) often don’t have a clear understanding of the entire application, so it’s hard to know if the issues we’re tracking down are related to a change we made, or something that changed in another part of the codebase. We look for answers in the logs, but it feels like we are hunting for a needle in a haystack. Adding additional logs in search of more relevant data only increases the amount of proverbial straw and is probably not an option if the issue we are dealing with is time-sensitive.

During these late nights, I used to wish that production environments were more like a local developer environment where debugging is relatively easy, thanks to tools like the debugger in my local IDE or browser. Wouldn’t it be nice to have a debugger or a dynamic logging device in production? Developers need a magic wand (or magnet) to get all the needles from the haystack of production logs! Well, it turns out that the Information Points feature of AppDynamics was the magic wand I was looking for.

What are information points?

If you are not familiar with AppDynamics, consider an information point as a tool that allows you to inspect the input parameters or return value of any invocation of a method along with additional metrics about the execution time of the method for each invocation. If you are familiar with AppDynamics, information points are similar to data collectors in business transactions. However, while data collectors show application data in the context of a business transaction, information points reflect data state across all invocations of a method, independently of business transactions. They also let you apply computations to the values, for example, representing the sum or average for a method return value or input parameter.

Below is an example of how I have used information points as a production debugging tool. You will notice the debug flow is very similar to the way developers find and fix issues using an IDE.

One of our customers was reaching the data limits for one of the metadata items we collect, and the customer was adamant that the stale/old data was not getting purged. Our operations engineers increased the limits a couple of times, and the issue got escalated to my engineering team. The system was designed in a way where the stale/old data was deleted by a background cron task. The same background task was used for all similar data, so it was hard to diagnose what was going on and whether the background task to clean old data was being invoked for that particular account and metadata records. There were no relevant logs available to debug this issue further.

To resolve the issue I created a new information point on the Information Point Page with a few clicks as shown in Figure 1 (below). The information point was created on the delete method of the background task for that particular account and metadata record. I also created a custom metric for the return value which returned the number of deleted entities.

Code block for the delete background task:

DeleteBackgroundTask {

public int deleteStaleEntries(int accountId, String entityType) {

Delete code…

}

}

Screen Shot 2017-10-26 at 8.38.56 PM.png

Figure 1: Information points created with custom metric.

Within the next few minutes, I was able to confirm that the background job to delete stale entries was triggered every 10 minutes and was working as per design. I also was able to see how many records were deleted.

Screen Shot 2017-10-26 at 8.39.32 PM.png

Next, I created another information point for this particular account on the method that was creating the stale entries. This information point had a custom metric that collected the number of records being created. From this information, I was able to determine that the customer was creating these records at a rate higher than the documented limits, and our delete task could not keep up. This information was then conveyed to the customer, they were able to adjust their usage, and the entire issue was solved within less than hour.

The Information Points feature in AppDynamics has truly changed my life. Information points help me understand what is going on in real time, and I use them regularly to debug production issues. They are also used by our quality engineers to test complex background tasks. With information points, problems can be easily isolated to a particular method or segment of code.

I have just one word of caution: There is a limit on the number of information points that can be added to the system as collecting too many can impact your own application performance. Please make sure you delete the information points that you create during your debugging session so you are prepared for the next one. Happy debugging!

For more details on information points, check out the docs here. You can also learn more about AppDynamics with our guided tour or by scheduling a demo today.

The post Tales from the Field: Debugging Applications in Production with Information Points appeared first on Application Performance Monitoring Blog | AppDynamics.

Read the original blog entry...

More Stories By AppDynamics Blog

In high-production environments where release cycles are measured in hours or minutes — not days or weeks — there's little room for mistakes and no room for confusion. Everyone has to understand what's happening, in real time, and have the means to do whatever is necessary to keep applications up and running optimally.

DevOps is a high-stakes world, but done well, it delivers the agility and performance to significantly impact business competitiveness.

Latest Stories
Today most companies are adopting or evaluating container technology - Docker in particular - to speed up application deployment, drive down cost, ease management and make application delivery more flexible overall. As with most new architectures, this dream takes significant work to become a reality. Even when you do get your application componentized enough and packaged properly, there are still challenges for DevOps teams to making the shift to continuous delivery and achieving that reducti...
Here to help unpack insights into the new era of using containers to gain ease with multi-cloud deployments are our panelists: Matt Baldwin, Founder and CEO at StackPointCloud, based in Seattle; Nic Jackson, Developer Advocate at HashiCorp, based in San Francisco, and Reynold Harbin, Director of Product Marketing at DigitalOcean, based in New York. The discussion is moderated by Dana Gardner, principal analyst at Interarbor Solutions.
Skeuomorphism usually means retaining existing design cues in something new that doesn’t actually need them. However, the concept of skeuomorphism can be thought of as relating more broadly to applying existing patterns to new technologies that, in fact, cry out for new approaches. In his session at DevOps Summit, Gordon Haff, Senior Cloud Strategy Marketing and Evangelism Manager at Red Hat, discussed why containers should be paired with new architectural practices such as microservices rathe...
In 2014, Amazon announced a new form of compute called Lambda. We didn't know it at the time, but this represented a fundamental shift in what we expect from cloud computing. Now, all of the major cloud computing vendors want to take part in this disruptive technology. In his session at 20th Cloud Expo, John Jelinek IV, a web developer at Linux Academy, will discuss why major players like AWS, Microsoft Azure, IBM Bluemix, and Google Cloud Platform are all trying to sidestep VMs and containers...
Using serverless computing has a number of obvious benefits over traditional application infrastructure - you pay only for what you use, scale up or down immediately to match supply with demand, and avoid operating any server infrastructure at all. However, implementing maintainable and scalable applications using serverless computing services like AWS Lambda poses a number of challenges. The absence of long-lived, user-managed servers means that states cannot be maintained by the service. Lo...
With the new Kubernetes offering, ClearDATA solves one of the largest challenges in healthcare IT around time-to-deployment. Using ClearDATA's Automated Safeguards for Kubernetes, healthcare organizations have access to the container orchestration to dynamically deploy new containers on demand, monitor the health of each container for threats and seamlessly roll back faulty application updates to a previous version, avoid system-wide downtime and ensure secure continuous access to patient data.
Isomorphic Software is the global leader in high-end, web-based business applications. We develop, market, and support the SmartClient & Smart GWT HTML5/Ajax platform, combining the productivity and performance of traditional desktop software with the simplicity and reach of the open web. With staff in 10 timezones, Isomorphic provides a global network of services related to our technology, with offerings ranging from turnkey application development to SLA-backed enterprise support. Leadin...
With the rise of Docker, Kubernetes, and other container technologies, the growth of microservices has skyrocketed among dev teams looking to innovate on a faster release cycle. This has enabled teams to finally realize their DevOps goals to ship and iterate quickly in a continuous delivery model. Why containers are growing in popularity is no surprise — they’re extremely easy to spin up or down, but come with an unforeseen issue. However, without the right foresight, DevOps and IT teams may lo...
Platform9, the open-source-as-a-service company making cloud infrastructure easy, today announced the general availability of its Managed Kubernetes service, the industry's first infrastructure-agnostic, SaaS-managed offering. Unlike legacy software distribution models, Managed Kubernetes is deployed and managed entirely as a SaaS solution, across on-premises and public cloud infrastructure. The company also introduced Fission, a new, open source, serverless framework built on Kubernetes. These ...
Emil Sayegh is an early pioneer of cloud computing and is recognized as one of the industry's true veterans. A cloud visionary, he is credited with launching and leading the cloud computing and hosting businesses for HP, Rackspace, and Codero. Emil built the Rackspace cloud business while serving as the company's GM of the Cloud Computing Division. Earlier at Rackspace he served as VP of the Product Group and launched the company's private cloud and hosted exchange services. He later moved o...
As you know, enterprise IT conversation over the past year have often centered upon the open-source Kubernetes container orchestration system. In fact, Kubernetes has emerged as the key technology -- and even primary platform -- of cloud migrations for a wide variety of organizations. Kubernetes is critical to forward-looking enterprises that continue to push their IT infrastructures toward maximum functionality, scalability, and flexibility. As they do so, IT professionals are also embr...
Kubernetes is a new and revolutionary open-sourced system for managing containers across multiple hosts in a cluster. Ansible is a simple IT automation tool for just about any requirement for reproducible environments. In his session at @DevOpsSummit at 18th Cloud Expo, Patrick Galbraith, a principal engineer at HPE, will discuss how to build a fully functional Kubernetes cluster on a number of virtual machines or bare-metal hosts. Also included will be a brief demonstration of running a Galer...
DevOps is under attack because developers don’t want to mess with infrastructure. They will happily own their code into production, but want to use platforms instead of raw automation. That’s changing the landscape that we understand as DevOps with both architecture concepts (CloudNative) and process redefinition (SRE). Rob Hirschfeld’s recent work in Kubernetes operations has led to the conclusion that containers and related platforms have changed the way we should be thinking about DevOps and...
Cloud-Native thinking and Serverless Computing are now the norm in financial services, manufacturing, telco, healthcare, transportation, energy, media, entertainment, retail and other consumer industries, as well as the public sector. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that pro...
Docker is sweeping across startups and enterprises alike, changing the way we build and ship applications. It's the most prominent and widely known software container platform, and it's particularly useful for eliminating common challenges when collaborating on code (like the "it works on my machine" phenomenon that most devs know all too well). With Docker, you can run and manage apps side-by-side - in isolated containers - resulting in better compute density. It's something that many developer...