org.apache.hadoop.hdfs.server.datanode

Class DataNode

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.hdfs.server.datanode.DataNode

All Implemented Interfaces:

Runnable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol, org.apache.hadoop.hdfs.protocol.FSConstants, org.apache.hadoop.hdfs.server.datanode.DataNodeMXBean, org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol, org.apache.hadoop.ipc.VersionedProtocol

public class DataNode
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol, org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol, org.apache.hadoop.hdfs.protocol.FSConstants, Runnable, org.apache.hadoop.hdfs.server.datanode.DataNodeMXBean

DataNode is a class (and program) that stores a set of blocks for a DFS deployment. A single deployment can have one or many DataNodes. Each DataNode communicates regularly with a single NameNode. It also communicates with client code and other DataNodes from time to time. DataNodes store a series of named blocks. The DataNode allows client code to read these blocks, or to write new block data. The DataNode may also, in response to instructions from its NameNode, delete blocks or copy blocks to/from other DataNodes. The DataNode maintains just one critical table: block-> stream of bytes (of BLOCK_SIZE or less) This info is stored on a local disk. The DataNode reports the table's contents to the NameNode upon startup and every so often afterwards. DataNodes spend their lives in an endless loop of asking the NameNode for something to do. A NameNode cannot connect to a DataNode directly; a NameNode simply returns values from functions invoked by a DataNode. DataNodes maintain an open server socket so that client code or other DataNodes can read/write data. The host/port for this server is reported to the NameNode, which then sends that information to clients or other DataNodes that might be interested.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hdfs.protocol.FSConstants

org.apache.hadoop.hdfs.protocol.FSConstants.DatanodeReportType, org.apache.hadoop.hdfs.protocol.FSConstants.SafeModeAction, org.apache.hadoop.hdfs.protocol.FSConstants.UpgradeAction

Field Summary

Fields

Modifier and Type	Field and Description
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner	blockScanner
org.apache.hadoop.util.Daemon	blockScannerThread
org.apache.hadoop.hdfs.server.datanode.FSDatasetInterface	data
static String	DATA_DIR_KEY
static String	DATA_DIR_PERMISSION_KEY
static String	DN_CLIENTTRACE_FORMAT
org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration	dnRegistration
static String	EMPTY_DEL_HINT
org.apache.hadoop.ipc.Server	ipcServer
static org.apache.commons.logging.Log	LOG
org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol	namenode
static int	PKT_HEADER_LEN Header size for a packet

Fields inherited from interface org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol

versionID

Fields inherited from interface org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol

versionID

Fields inherited from interface org.apache.hadoop.hdfs.protocol.FSConstants

BLOCKREPORT_INITIAL_DELAY, BLOCKREPORT_INTERVAL, BUFFER_SIZE, DEFAULT_BLOCK_SIZE, DEFAULT_DATA_SOCKET_SIZE, HEARTBEAT_INTERVAL, LAYOUT_VERSION, LEASE_HARDLIMIT_PERIOD, LEASE_RECOVER_PERIOD, LEASE_SOFTLIMIT_PERIOD, MAX_PATH_DEPTH, MAX_PATH_LENGTH, MIN_BLOCKS_FOR_WRITE, QUOTA_DONT_SET, QUOTA_RESET, SIZE_OF_INTEGER, SMALL_BUFFER_SIZE

Method Summary

Methods

Modifier and Type	Method and Description
protected void	checkDiskError() Check if there is a disk failure and if so, handle the error
protected void	checkDiskError(Exception e) Check if there is no space in disk
static DataNode	createDataNode(String[] args, org.apache.hadoop.conf.Configuration conf) Instantiate & Start a single datanode daemon and wait for it to finish.
static DataNode	createDataNode(String[] args, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.SecureResources resources) Instantiate & Start a single datanode daemon and wait for it to finish.
static org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol	createInterDataNodeProtocolProxy(org.apache.hadoop.hdfs.protocol.DatanodeInfo info, org.apache.hadoop.conf.Configuration conf, int socketTimeout, boolean connectToDnViaHostname)
static InetSocketAddress	createSocketAddr(String target) Deprecated.
Long	getBalancerBandwidth() Get current value of the max balancer bandwidth in bytes per second.
org.apache.hadoop.hdfs.protocol.Block	getBlockInfo(org.apache.hadoop.hdfs.protocol.Block block)
org.apache.hadoop.hdfs.protocol.BlockLocalPathInfo	getBlockLocalPathInfo(org.apache.hadoop.hdfs.protocol.Block block, org.apache.hadoop.security.token.Token<org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier> token)
org.apache.hadoop.hdfs.server.protocol.BlockMetaDataInfo	getBlockMetaDataInfo(org.apache.hadoop.hdfs.protocol.Block block)
static DataNode	getDataNode() Return the DataNode object
org.apache.hadoop.hdfs.server.datanode.FSDatasetInterface	getFSDataset() This method is used for testing.
String	getHostName()
String	getHttpPort()
static InetSocketAddress	getInfoAddr(org.apache.hadoop.conf.Configuration conf) Determine the http server's effective addr
String	getNamenode() Return the namenode's identifier
InetSocketAddress	getNameNodeAddr()
String	getNamenodeAddress()
long	getProtocolVersion(String protocol, long clientVersion)
String	getRpcPort()
InetSocketAddress	getSelfAddr()
static InetSocketAddress	getStreamingAddr(org.apache.hadoop.conf.Configuration conf)
String	getVersion()
String	getVolumeInfo() Returned information is a JSON representation of a map with volume name as the key and value is a map of volume attribute keys to its values
int	getXceiverCount() Number of concurrent xceivers per node.
static DataNode	instantiateDataNode(String[] args, org.apache.hadoop.conf.Configuration conf) Instantiate a single datanode object.
static DataNode	instantiateDataNode(String[] args, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.SecureResources resources) Instantiate a single datanode object.
static void	main(String[] args)
static DataNode	makeInstance(String[] dataDirs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.SecureResources resources) Make an instance of DataNode after ensuring that at least one of the given data directories (and their parent directories, if necessary) can be created.
protected Socket	newSocket() Creates either NIO or regular depending on socketWriteTimeout.
protected void	notifyNamenodeReceivedBlock(org.apache.hadoop.hdfs.protocol.Block block, String delHint)
void	offerService() Main loop for the DataNode.
org.apache.hadoop.hdfs.protocol.LocatedBlock	recoverBlock(org.apache.hadoop.hdfs.protocol.Block block, boolean keepLength, org.apache.hadoop.hdfs.protocol.DatanodeInfo[] targets)
org.apache.hadoop.util.Daemon	recoverBlocks(org.apache.hadoop.hdfs.protocol.Block[] blocks, org.apache.hadoop.hdfs.protocol.DatanodeInfo[][] targets)
void	run() No matter what kind of exception we get, keep retrying to offerService().
static void	runDatanodeDaemon(DataNode dn) Start a single datanode daemon and wait for it to finish.
void	scheduleBlockReport(long delay) This methods arranges for the data node to send the block report at the next heartbeat.
static void	secureMain(String[] args, org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.SecureResources resources)
static void	setNewStorageID(org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration dnReg)
void	shutdown() Shut down this instance of the datanode.
org.apache.hadoop.hdfs.server.protocol.BlockRecoveryInfo	startBlockRecovery(org.apache.hadoop.hdfs.protocol.Block block)
String	toString()
void	unRegisterMXBean()
void	updateBlock(org.apache.hadoop.hdfs.protocol.Block oldblock, org.apache.hadoop.hdfs.protocol.Block newblock, boolean finalize)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

DN_CLIENTTRACE_FORMAT

public static final String DN_CLIENTTRACE_FORMAT

See Also:

Constant Field Values

namenode

public org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol namenode

data

public org.apache.hadoop.hdfs.server.datanode.FSDatasetInterface data

dnRegistration

public org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration dnRegistration

EMPTY_DEL_HINT

public static final String EMPTY_DEL_HINT

See Also:

Constant Field Values

blockScanner

public org.apache.hadoop.hdfs.server.datanode.DataBlockScanner blockScanner

blockScannerThread

public org.apache.hadoop.util.Daemon blockScannerThread

DATA_DIR_KEY

public static final String DATA_DIR_KEY

See Also:

Constant Field Values

DATA_DIR_PERMISSION_KEY

public static final String DATA_DIR_PERMISSION_KEY

See Also:

Constant Field Values

ipcServer

public org.apache.hadoop.ipc.Server ipcServer

PKT_HEADER_LEN

public static final int PKT_HEADER_LEN

Header size for a packet

See Also:

Constant Field Values

Method Detail

createSocketAddr

@Deprecated
public static InetSocketAddress createSocketAddr(String target)
                                          throws IOException

Deprecated.

Use NetUtils.createSocketAddr(String) instead.

Throws:

IOException

unRegisterMXBean

public void unRegisterMXBean()

getInfoAddr

public static InetSocketAddress getInfoAddr(org.apache.hadoop.conf.Configuration conf)

Determine the http server's effective addr

newSocket

protected Socket newSocket()
                    throws IOException

Creates either NIO or regular depending on socketWriteTimeout.

Throws:

IOException

getDataNode

public static DataNode getDataNode()

Return the DataNode object

createInterDataNodeProtocolProxy

public static org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol createInterDataNodeProtocolProxy(org.apache.hadoop.hdfs.protocol.DatanodeInfo info,
                                                                                            org.apache.hadoop.conf.Configuration conf,
                                                                                            int socketTimeout,
                                                                                            boolean connectToDnViaHostname)
                                                                                                     throws IOException

Throws:

IOException

getNameNodeAddr

public InetSocketAddress getNameNodeAddr()

getSelfAddr

public InetSocketAddress getSelfAddr()

getNamenode

public String getNamenode()

Return the namenode's identifier

setNewStorageID

public static void setNewStorageID(org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration dnReg)

shutdown

public void shutdown()

Shut down this instance of the datanode. Returns only after shutdown is complete. This method can only be called by the offerService thread. Otherwise, deadlock might occur.

checkDiskError

protected void checkDiskError(Exception e)
                       throws IOException

Check if there is no space in disk

Parameters:

e - that caused this checkDiskError call

Throws:

IOException

checkDiskError

protected void checkDiskError()

Check if there is a disk failure and if so, handle the error

getXceiverCount

public int getXceiverCount()

Number of concurrent xceivers per node.

Specified by:

getXceiverCount in interface org.apache.hadoop.hdfs.server.datanode.DataNodeMXBean

offerService

public void offerService()
                  throws Exception

Main loop for the DataNode. Runs until shutdown, forever calling remote NameNode functions.

Throws:

Exception

notifyNamenodeReceivedBlock

protected void notifyNamenodeReceivedBlock(org.apache.hadoop.hdfs.protocol.Block block,
                               String delHint)

run

public void run()

No matter what kind of exception we get, keep retrying to offerService(). That's the loop that connects to the NameNode and provides basic DataNode functionality. Only stop when "shouldRun" is turned off (which can only happen at shutdown).

Specified by:

run in interface Runnable

runDatanodeDaemon

public static void runDatanodeDaemon(DataNode dn)
                              throws IOException

Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting.

Throws:

IOException

instantiateDataNode

public static DataNode instantiateDataNode(String[] args,
                           org.apache.hadoop.conf.Configuration conf)
                                    throws IOException

Instantiate a single datanode object. This must be run by invoking runDatanodeDaemon(DataNode) subsequently.

Throws:

IOException

instantiateDataNode

public static DataNode instantiateDataNode(String[] args,
                           org.apache.hadoop.conf.Configuration conf,
                           org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.SecureResources resources)
                                    throws IOException

Instantiate a single datanode object. This must be run by invoking runDatanodeDaemon(DataNode) subsequently.

Parameters:

resources - Secure resources needed to run under Kerberos

Throws:

IOException

createDataNode

public static DataNode createDataNode(String[] args,
                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Instantiate & Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting.

Throws:

IOException

createDataNode

public static DataNode createDataNode(String[] args,
                      org.apache.hadoop.conf.Configuration conf,
                      org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.SecureResources resources)
                               throws IOException

Instantiate & Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting. LimitedPrivate for creating secure datanodes

Throws:

IOException

makeInstance

public static DataNode makeInstance(String[] dataDirs,
                    org.apache.hadoop.conf.Configuration conf,
                    org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.SecureResources resources)
                             throws IOException

Make an instance of DataNode after ensuring that at least one of the given data directories (and their parent directories, if necessary) can be created.

Parameters:

dataDirs - List of directories, where the new DataNode instance should keep its files.

conf - Configuration instance to use.

resources - Secure resources needed to run under Kerberos

Returns:

DataNode instance for given list of data dirs and conf, or null if no directory from this directory list can be created.

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object

scheduleBlockReport

public void scheduleBlockReport(long delay)

This methods arranges for the data node to send the block report at the next heartbeat.

getFSDataset

public org.apache.hadoop.hdfs.server.datanode.FSDatasetInterface getFSDataset()

This method is used for testing. Examples are adding and deleting blocks directly. The most common usage will be when the data node's storage is similated.

Returns:

the fsdataset that stores the blocks

secureMain

public static void secureMain(String[] args,
              org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.SecureResources resources)

main

public static void main(String[] args)

getBlockMetaDataInfo

public org.apache.hadoop.hdfs.server.protocol.BlockMetaDataInfo getBlockMetaDataInfo(org.apache.hadoop.hdfs.protocol.Block block)
                                                                              throws IOException

Specified by:

getBlockMetaDataInfo in interface org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol

Throws:

IOException

startBlockRecovery

public org.apache.hadoop.hdfs.server.protocol.BlockRecoveryInfo startBlockRecovery(org.apache.hadoop.hdfs.protocol.Block block)
                                                                            throws IOException

Specified by:

startBlockRecovery in interface org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol

Throws:

IOException

recoverBlocks

public org.apache.hadoop.util.Daemon recoverBlocks(org.apache.hadoop.hdfs.protocol.Block[] blocks,
                                          org.apache.hadoop.hdfs.protocol.DatanodeInfo[][] targets)

updateBlock

public void updateBlock(org.apache.hadoop.hdfs.protocol.Block oldblock,
               org.apache.hadoop.hdfs.protocol.Block newblock,
               boolean finalize)
                 throws IOException

Specified by:

updateBlock in interface org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol

Throws:

IOException

getProtocolVersion

public long getProtocolVersion(String protocol,
                      long clientVersion)
                        throws IOException

Specified by:

getProtocolVersion in interface org.apache.hadoop.ipc.VersionedProtocol

Throws:

IOException

getBlockLocalPathInfo

public org.apache.hadoop.hdfs.protocol.BlockLocalPathInfo getBlockLocalPathInfo(org.apache.hadoop.hdfs.protocol.Block block,
                                                                       org.apache.hadoop.security.token.Token<org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier> token)
                                                                         throws IOException

Specified by:

getBlockLocalPathInfo in interface org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol

Throws:

IOException

recoverBlock

public org.apache.hadoop.hdfs.protocol.LocatedBlock recoverBlock(org.apache.hadoop.hdfs.protocol.Block block,
                                                        boolean keepLength,
                                                        org.apache.hadoop.hdfs.protocol.DatanodeInfo[] targets)
                                                          throws IOException

Specified by:

recoverBlock in interface org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol

Throws:

IOException

getBlockInfo

public org.apache.hadoop.hdfs.protocol.Block getBlockInfo(org.apache.hadoop.hdfs.protocol.Block block)
                                                   throws IOException

Specified by:

getBlockInfo in interface org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol

Throws:

IOException

getStreamingAddr

public static InetSocketAddress getStreamingAddr(org.apache.hadoop.conf.Configuration conf)

getHostName

public String getHostName()

Specified by:

getHostName in interface org.apache.hadoop.hdfs.server.datanode.DataNodeMXBean

getVersion

public String getVersion()

Specified by:

getVersion in interface org.apache.hadoop.hdfs.server.datanode.DataNodeMXBean

getRpcPort

public String getRpcPort()

Specified by:

getRpcPort in interface org.apache.hadoop.hdfs.server.datanode.DataNodeMXBean

getHttpPort

public String getHttpPort()

Specified by:

getHttpPort in interface org.apache.hadoop.hdfs.server.datanode.DataNodeMXBean

getNamenodeAddress

public String getNamenodeAddress()

Specified by:

getNamenodeAddress in interface org.apache.hadoop.hdfs.server.datanode.DataNodeMXBean

getVolumeInfo

public String getVolumeInfo()

Returned information is a JSON representation of a map with volume name as the key and value is a map of volume attribute keys to its values

Specified by:

getVolumeInfo in interface org.apache.hadoop.hdfs.server.datanode.DataNodeMXBean

getBalancerBandwidth

public Long getBalancerBandwidth()

Get current value of the max balancer bandwidth in bytes per second.

Returns:

bandwidth Blanacer bandwidth in bytes per second for this datanode.