SYS-CON MEDIA Authors: Yeshim Deniz, Elizabeth White, Greg Ness, Sean Houghton, Glenn Rossman

Article

Cassandra Nodetool Internals

Let's go inside the cassandra nodetool utility.

Relational database management systems are the most commonly used system to store and use data, but for extremely large amounts of data, this kind of system doesn’t scale up properly.

The concept of "NoSQL"(Not Only SQL) has been spreading due to the growing demand for relational database alternatives. The biggest motivation behind NoSQL is scalability. NoSQL solutions can offer a way to store and use extremely large amounts of data, but with less overhead, less work, better performance, and less downtime.

Apache Cassandra implements the “NoSQL” concept. It was developed at Facebook to power their Inbox Search feature, and it became an Apache open source project. Twitter, Digg, Reddit and quite a few others started using it invo

Cassandra exposes a number of management operations via Java Management Extensions (JMX). Java Management Extensions (JMX) is a Java technology that supplies tools for managing and monitoring Java applications and services. Any statistic or operation that a Java application has exposed as an MBean can then be monitored or manipulated using JMX.

In this article the goal is to go inside Cassandra nodetool, and see how easy it would be to extend it or build a Cassandra monitoring UI. For that we use JArchitect to understand how the nodetool works internally.

The nodetool utility is a command line interface for Cassandra. You can use it to help manage a cluster. It’s used like this nodetool -h HOSTNAME [-p JMX_PORT] COMMAND
Here are some available commands:

  ring                   - Print informations on the token ring
  join                   - Join the ring
  info                   - Print node informations (uptime, load, ...)
  cfstats                - Print statistics on column families
  clearsnapshot          - Remove all existing snapshots
  version                - Print cassandra version
  tpstats                - Print usage statistics of thread pools
  drain                  - Drain the node (stop accepting writes and flush all column families)

The nodetool classes exist in the org.apache.cassandra.tools package, and the entry class is NodeCmd.

NodeCmd

Let’s search for methods invoked from the main method from the NodeCmd class by executing the following CQLinq query:

from m in Methods where m.IsUsedBy ("org.apache.cassandra.tools.NodeCmd.main(String[])")
select new { m, m.NbBCInstructions }

cassandra1

To treat commands, the main method uses the Apache Commons CLI library which provides an API for parsing command line options. It's also able to print help messages of all the options available.

For each command the NodeCmd switch to the appropriate method to do the job. And the NodeCmd collaborates with the NodeProbe class.

NodeProbe

Let’s discover how NodeProbe achieve its task, for that we can search for all types used by it.

from t in Types where t.IsUsedBy ("org.apache.cassandra.tools.NodeProbe")
select new { t }
 

cassandra2

The NodeProbe class uses mainly the management beans; it acts as a facade and redirects each command to the appropriate JMX bean.

Here is the list of all Cassandra JMX beans:

from t in Types
where t.NameLike (@"mbean\i") && t.IsInterface 
select  t

cassandra3

Many managed beans are available which gives the possibility to create tools exploiting their capabilities, to help administrators monitor and manage the Cassandra cluster.

The idea of such tools is to interact with the JMX beans and invoke some of their methods, for that we need to create a JMX proxy by invoking JMX.newMBeanProxy.

Let’s search for methods which create a JMX proxy.

from m in Methods where m.IsUsing ("javax.management.JMX.newMBeanProxy(MBeanServerConnection,ObjectName,Class)")
select new { m, m.NbBCInstructions }

cassandra4

The NodeProbe which acts as a facade create these proxies, we can take as example the connect method which is invoked to create these proxies, and discover some methods invoked by it.

cassandra5

After the creation of proxies, all the commands will be just a redirection, for example let’s search for methods used by NodeProbe.forceRemoveCompletion.

from m in Methods where m.IsUsedBy ("org.apache.cassandra.tools.NodeProbe.forceRemoveCompletion()")
select new { m, m.NbBCInstructions }
 

cassandra6

Only the StorageServiceMBean.forceRemoveCompletion is used, all the logic of the treatment is in the server side. TheStorageServiceMBean is implemented by the StorageService class and here are all methods used by the StoageService. forceRemoveCompletion method:

cassandra7

Possible Design improvement

We discovered that NodeProbe is just a facade to the JMX beans, but what about NodeCmd class, it uses any JMX beans directly?

As shown before NodeCmd not create any JMX proxy, the creation of all the proxies is achieved by the NodeProbe. So we can conclude that all JMX beans invoking are from the NodeProbe class, and the responsibility of the NodeCmd class is just to treat the command line and ask the NodeProbe class to do the job. And to check that let’s search for JMX beans used directly by the NodeCmd class.

from t in Types where t.IsUsedBy ("org.apache.cassandra.tools.NodeCmd") && t.NameLike (@"mbean\i") 
select  t

cassandra8

The NodeCmd uses also some JMX beans like EndpointSnitchInfoMBean, and here are all the methods using this bean.

from m in Methods where m.IsUsing ("org.apache.cassandra.locator.EndpointSnitchInfoMBean")
select new { m, m.NbBCInstructions }

cassandra9

NodeCmd and NodeProbe use it, the NodeCmd doesn’t have the proxy but ask it from the NodeProbe class as shown in this dependency graph:

cassandra10

Maybe it will be better to refactor the nodetool and let only NodeProbe redirect commands to the JMX proxies, and acts as the only facade to the management capabilities, and the responsibility of the NodeCmd will be only the parsing of the command line and redirect to the NodeProbe class. Only a few commands are treated directly from the NodeCmd class and delegate this task to the NodeProbe is not a difficult task.

Conclusion
Cassandra expose many management capabilities thought JMX beans, and not all of them are treated by the nodetool utility, it’s very easy to add other commands to it, we can just take as example the already existing commands and add your own. Building administrator tools is also easy, because all the logic is in the Cassandra server side, you have just to develop a nice GUI and interact with the JMX beans. If you want to develop your own administration tool, the nodetool is a good beginning; understanding how it works will facilitate a lot your task.

More Stories By Lahlali Issam

Lahlali Issam Lead Developer at JavaDepend, a tool to manage and understand complex Java code. With JavaDepend, software quality can be measured using Code Metrics, visualized using Graphs and Treemaps, and queried using CQL language, a SQL like to query the code base.

Latest Stories
How do APIs and IoT relate? The answer is not as simple as merely adding an API on top of a dumb device, but rather about understanding the architectural patterns for implementing an IoT fabric. There are typically two or three trends: Exposing the device to a management framework Exposing that management framework to a business centric logic Exposing that business layer and data to end users. This last trend is the IoT stack, which involves a new shift in the separation of what stuff happe...
The Internet of Things will put IT to its ultimate test by creating infinite new opportunities to digitize products and services, generate and analyze new data to improve customer satisfaction, and discover new ways to gain a competitive advantage across nearly every industry. In order to help corporate business units to capitalize on the rapidly evolving IoT opportunities, IT must stand up to a new set of challenges. In his session at @ThingsExpo, Jeff Kaplan, Managing Director of THINKstrateg...
The Internet of Things promises to transform businesses (and lives), but navigating the business and technical path to success can be difficult to understand. In his session at @ThingsExpo, Sean Lorenz, Technical Product Manager for Xively at LogMeIn, demonstrated how to approach creating broadly successful connected customer solutions using real world business transformation studies including New England BioLabs and more.
An entirely new security model is needed for the Internet of Things, or is it? Can we save some old and tested controls for this new and different environment? In his session at @ThingsExpo, New York's at the Javits Center, Davi Ottenheimer, EMC Senior Director of Trust, reviewed hands-on lessons with IoT devices and reveal a new risk balance you might not expect. Davi Ottenheimer, EMC Senior Director of Trust, has more than nineteen years' experience managing global security operations and asse...
DevOps is all about agility. However, you don't want to be on a high-speed bus to nowhere. The right DevOps approach controls velocity with a tight feedback loop that not only consists of operational data but also incorporates business context. With a business context in the decision making, the right business priorities are incorporated, which results in a higher value creation. In his session at DevOps Summit, Todd Rader, Solutions Architect at AppDynamics, discussed key monitoring techniques...
P2P RTC will impact the landscape of communications, shifting from traditional telephony style communications models to OTT (Over-The-Top) cloud assisted & PaaS (Platform as a Service) communication services. The P2P shift will impact many areas of our lives, from mobile communication, human interactive web services, RTC and telephony infrastructure, user federation, security and privacy implications, business costs, and scalability. In his session at @ThingsExpo, Robin Raymond, Chief Architect...
What do a firewall and a fortress have in common? They are no longer strong enough to protect the valuables housed inside. Like the walls of an old fortress, the cracks in the firewall are allowing the bad guys to slip in - unannounced and unnoticed. By the time these thieves get in, the damage is already done and the network is already compromised. Intellectual property is easily slipped out the back door leaving no trace of forced entry. If we want to reign in on these cybercriminals, it's hig...
The definition of IoT is not new, in fact it’s been around for over a decade. What has changed is the public's awareness that the technology we use on a daily basis has caught up on the vision of an always on, always connected world. If you look into the details of what comprises the IoT, you’ll see that it includes everything from cloud computing, Big Data analytics, “Things,” Web communication, applications, network, storage, etc. It is essentially including everything connected online from ha...
The 3rd International @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to th...
We are reaching the end of the beginning with WebRTC, and real systems using this technology have begun to appear. One challenge that faces every WebRTC deployment (in some form or another) is identity management. For example, if you have an existing service – possibly built on a variety of different PaaS/SaaS offerings – and you want to add real-time communications you are faced with a challenge relating to user management, authentication, authorization, and validation. Service providers will w...
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
SYS-CON Media announced that Centrify, a provider of unified identity management across cloud, mobile and data center environments that delivers single sign-on (SSO) for users and a simplified identity infrastructure for IT, has launched an ad campaign on Cloud Computing Journal. The ads focus on security: how an organization can successfully control privilege for all of the organization’s identities to mitigate identity-related risk without slowing down the business, and how Centrify provides ...
The Internet of Things will greatly expand the opportunities for data collection and new business models driven off of that data. In her session at @ThingsExpo, Esmeralda Swartz, CMO of MetraTech, discussed how for this to be effective you not only need to have infrastructure and operational models capable of utilizing this new phenomenon, but increasingly service providers will need to convince a skeptical public to participate. Get ready to show them the money!
The Internet of Things is tied together with a thin strand that is known as time. Coincidentally, at the core of nearly all data analytics is a timestamp. When working with time series data there are a few core principles that everyone should consider, especially across datasets where time is the common boundary. In his session at Internet of @ThingsExpo, Jim Scott, Director of Enterprise Strategy & Architecture at MapR Technologies, discussed single-value, geo-spatial, and log time series dat...
In her General Session at 15th Cloud Expo, Anne Plese, Senior Consultant, Cloud Product Marketing, at Verizon Enterprise, focused on finding the right mix of renting vs. buying Oracle capacity to scale to meet business demands, and offer validated Oracle database TCO models for Oracle development and testing environments. Anne Plese is a marketing and technology enthusiast/realist with over 19+ years in high tech. At Verizon Enterprise, she focuses on driving growth for the Verizon Cloud platfo...