Praxis - Hadoop

In this section, we discuss some of the common issues users run into when running an HBase cluster under load.

Versions

Up until HBase 0.20, HBase aligned its versioning with that of Hadoop. A particular HBase version would run on any Hadoop that had a matching minor version, where minor version in this context is considered the number between the periods (e.g., 20 is Praxisthe minor version of an HBase 0.20.5). HBase 0.20.5 would run on an Hadoop 0.20.2, but HBase 0.19.5 would not run on Hadoop 0.20.0. With HBase 0.90,‖ the version relationship was broken. The Hadoop release cycle has slowed and no longer aligns with that of HBase developments. Also, the intent is that now a particular HBase version can run on multiple versions of Hadoop. For example, HBase 0.90.x will work with both Hadoop 0.20.x and 0.21.x.

This said, ensure you are running compatible versions of Hadoop and HBase. Check the requirements section of your download. Incompatible versions will throw an exception complaining about the version mismatch, if you are lucky. If they cannot talk to each sufficiently to pass versions, you may see your HBase cluster hang indefinitely, soon after startup. The mismatch exception or HBase hang can also happen on upgrade
if older versions of either HBase or Hadoop can still be found on the classpath because of imperfect cleanup of the old software.

HDFS

HBase’s use of HDFS is very different from how it’s used by MapReduce. In MapReduce, generally, HDFS files are opened, with their content streamed through a map task and then closed. In HBase, data files are opened on cluster startup and kept open so that we avoid paying the file open costs on each access. Because of this, HBase tends to see issues not normally encountered by MapReduce clients:

Running out of file descriptors

Because we keep files open, on a loaded cluster, it doesn’t take long before we run into system- and Hadoop-imposed limits. For instance, say we have a cluster that has three nodes each running an instance of a datanode and a regionserver and we’re running an upload into a table that is currently at 100 regions and 10 column families. Allow that each column family has on average two flush files. Doing the math, we can have 100 × 10 × 2, or 2,000, files open at any one time. Add to this total miscellaneous other descriptors consumed by outstanding scanners and Java libraries. Each open file consumes at least one descriptor over on the remote datanode.

The default limit on the number of file descriptors per process is 1,024. When we exceed the filesystem ulimit, we’ll see the complaint about Too many open files in logs, but often you’ll first see indeterminate behavior in HBase. The fix requires increasing the file descriptor ulimit count.# You can verify that the HBase process is running with sufficient file descriptors by looking at the first few lines of a regionservers log. It emits vitals such as the JVM being used and environment settings such as the file descriptor ulimit.

Why 0.90? We wanted there to be no confusion that a break had been made, so we put a large gap between our new versioning and that of Hadoop’s.

See the HBase FAQ for how to up the ulimit on your cluster.

Running out of datanode threads

Similarly, the Hadoop datanode has an upper bound of 256 on the number of threads it can run at any one time. Given the same table statistics quoted in the preceding bullet, it’s easy to see how we can exceed this upper bound relatively early, given that in the datanode as of this writing each open connection to a file block consumes a thread. If you look in the datanode log, you’ll see a complaint like xceiverCount 258 exceeds the limit of concurrent xcievers 256 but again, you’ll likely see HBase act erratically before you encounter this log entry. Increase the dfs.datanode.max.xcievers (note that the property name is misspelled) count inHDFS and restart your cluster.

Sync

You must run HBase on an HDFS that has a working sync. Otherwise, you will lose data. This means running HBase on Hadoop 0.21.x or on a Hadoop that has been built from the branch-0.20-append† branch, which adds a working sync/append to Hadoop 0.20.

UI

HBase runs a web server on the master to present a view on the state of your running cluster. By default, it listens on port 60010. The master UI displays a list of basic attributes such as software versions, cluster load, request rates, lists of cluster tables, and participating regionservers. Click on a regionserver in the master UI and you are taken to the web server running on the individual regionserver. It lists the regions this serveris carrying and basic metrics such as resources consumed and request rates.

Metrics

Hadoop has a metrics system that can be used to emit vitals over a period to a context (this is covered in “Metrics” ). Enabling Hadoop metrics, and in particular tying them to Ganglia or emitting them via JMX, will give you views on what is happening on your cluster currently and in the recent past. HBase also adds metrics of its own request rates, counts of vitals, resources used that can be caught by a Hadoop context. See the file hadoop-metrics.properties under the HBase conf directory.

See the HBase troubleshooting guide for more detail on this issue.

You can find the branch-0.20-append branch at http://svn.apache.org/repos/asf/hadoop/common/branches/ branch-0.20-append/

On regionserver crash, before Hadoop 0.21 or 0.20-append, edits written to the commit log kept in HDFS were not recoverable as files that had not been properly closed lost all edits no matter how much had been
written them at crash time.

Yes, this file is named for Hadoop, though it’s for setting up HBase metrics.

Schema Design

HBase tables are like those in an RDBMS, except that cells are versioned, rows are sorted, and columns can be added on the fly by the client as long as the column family they belong to preexists. These factors should be considered when designing schemas for HBase, but far and away the most important concern designing schemas is consideration of how the data will be accessed. All access is via primary key so the key designshould lend itself to how the data is going to be queried. The other property to keep in mind when designing schemas is that a defining attribute of column(-family)-oriented stores, like HBase, is that it can host wide and sparsely populated tables at no incurred cost.

Joins

There is no native database join facility in HBase, but wide tables can make it so that there is no need for database joins pulling from secondary or tertiary tables. A wide row can sometimes be made to hold all data that pertains to a particular primary key.

Row keys

Take time designing your row key. In the weather data example in this chapter, the compound row key has a station prefix that served to group temperatures by station. The reversed timestamp suffix made it so temperatures could be scanned ordered from most recent to oldest. A smart compound key can be used to cluster data in ways amenable to how it will be accessed.

Designing compound keys, you may have to zero-pad number components so row keys sort properly. Otherwise, you will run into the issue where 10 sorts before 2 when only byte-order is considered (02 sorts before 10).If your keys are integers, use a binary representation rather than persist the string version of a number it consumes less space.

Counters

At StumbleUpon, the first production feature deployed on HBase was keeping counters for stumbleupon.com frontend. Counters used to be kept in MySQL, but the rate of change was such that drops were frequent and the load imposed by the counter writes was such that web designers self-imposed limits on what was counted. Using the incre mentColumnValue() method on org.apache.hadoop.hbase.HTable, counters can be incremented many thousands of times a second.

“Column-Stores for Wide and Sparse Data” by Daniel J. Abadi.

Bulk Load

HBase has an efficient facility for bulk loading HBase by writing its internal data format directly into the filesystem from MapReduce. Going this route, it’s possible to load an HBase instance at rates that are an order of magnitude or more beyond those attainable by writing via the HBase client API. The facility is described at http://hbase.apache.org/ docs/current/bulk-loads.html. It’s also possible to bulk load into a live table.


All rights reserved © 2018 Wisdom IT Services India Pvt. Ltd DMCA.com Protection Status

Hadoop Topics