Stack Overflow
Stack Overflow

Questions tagged [namenode]

The Hadoop NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept.

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself.

Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The NameNode responds to the successful requests by returning a list of relevant DataNodes where the data lives.

190 questions
2
votes
1 answer

Do we need to put namenode in safe mode before restarting the job tracker?

I have a Hadoop cluster running Cloudera's CDH3, Apache Hadoop's 0.20.2 equivalent. I want to restart the job-tracker as there are some jobs which are not getting killed. I tried killing them from the command line, the command executes successfully,…
Varun Risbud
  • 145
  • 9
2
votes
1 answer

Hadoop: Why I got "Max Non Heap Memory is -1 B." message in Namenode Information Web? What it does mean?

I have a cluster to use Hadoop 2.6.5 (one master that works as namenode and datanode, and two slaves) that I made using VirtualBox (easch node has Xubuntu 16.04 installed). A priori, the installation is right because I ran a wordcount example and…
mjbsgll
  • 666
  • 5
  • 20
1
vote
1 answer

How often are blocks on HDFS replicated?

I have a question regarding hadoop hdfs blocks replication. Suppose a block is written on a datanode and the DFS has a replication factor 3, how long does it take for the namenode to replicate this block on other datanodes? Is it instantaneuos? If…
wayne
  • 11
  • 2
1
vote
0 answers

What is the prefered solution for corrupted namenode metadata

we have HDP cluster , version 2.6.5 cluster include management of two name-node ( one is active and the secondary is standby ) and 65 datanode machines we have problem with the standby name-node that not started and from the namenode logs we can see…
jessica
  • 1,804
  • 4
  • 24
1
vote
0 answers

Add datanode and backup existed data to a standalone Hadooop on windows machine

I have installed a standalone mode Hadoop on windows machine locally, with one datanode and the replication factor set as 1. I have already uploaded some data onto the datanode. Let us call this existing datanode as datanode1. I would like to add…
Yu Xiang
  • 115
  • 2
  • 13
1
vote
0 answers

ambari + namenode HA and hdfs zkfc -formatZK

we have production HDP cluster , HDP version - 2.6.5 Ambari version - 2.6.1 We are performing now the change hostname configuration according to the document -…
Judy
  • 1,017
  • 1
  • 10
  • 28
1
vote
0 answers

Airflow conn_id with multiple server

I am using WebHDFSSensor and for that we need to provide namenode. However, active namenode and standBy namenode change. I can't just provide current namenode host to webhdfs_conn_id. I have to create connection from both host. I tried to provide…
Ayush Goyal
  • 365
  • 1
  • 12
1
vote
6 answers

Hadoop namenode metadata

I am a bit confused by the Hadoop architecture. What kind of file metadata is stored in Hadoop Namenode? From Hadoop wiki, it says Namenode stores the entire system namespace. Does information like last modified time, created time, file size,…
leon
  • 8,725
  • 18
  • 57
  • 73
1
vote
1 answer

No resources in GCP Dataproc node to start new SparkSession

I am working on a use case where I have to process a huge amount of data (multiple tables) and I am trying to submit this as a batch job to the Dataproc cluster(PySpark). My code looks something like this from pyspark import SparkContext from…
Nightwing
  • 37
  • 6
1
vote
1 answer

hdfs + namenode + edit files increasing with huge size and how to limit the size of edit files

we have HDP cluster with 7 datanodes machines under /hadoop/hdfs/namenode/current/ we can see more then 1500 edit files each file is around 7M to 20M as the following 7.8M …
jessica
  • 1,804
  • 4
  • 24
1
vote
0 answers

Hadoop namenode and secondary nemenode concept

I want to share all about our case. We have Hadoop cluster with 2 name nodes, one active name node, and one standby name node. After some time we notice that the active name node and secondary name node are down for 3 days. After reviewing the name…
jessica
  • 1,804
  • 4
  • 24
1
vote
0 answers

Hadoop Namenode and Secondary Namenode not starting with PDSH exit code 1

aqib@aqib-Inspiron-5521:~$ start-dfs.sh Starting namenodes on [aqib-Inspiron-5521] pdsh@aqib-Inspiron-5521: aqib-Inspiron-5521: ssh exited with exit code 1 Starting datanodes Starting secondary namenodes [aqib-Inspiron-5521] pdsh@aqib-Inspiron-5521:…
Aqib jallal
  • 11
  • 1
1
vote
0 answers

How to recover bad namenode from good namenode

I want to share all , about how to recover second bad namenode when using the good namenode example of one bad namenode and one good namenode So for this scenario lets say the following The bad namenode is on machine hadoop1 The good namenode is…
jessica
  • 1,804
  • 4
  • 24
1
vote
1 answer

Memory for Namenode(s) in Hadoop

Environment: The production cluster has 2 name-nodes (active and standby namely) and the nodes are SAS drives in Raid-1 configuration. These nodes have nothing but the master services (NN and Standby NN) running on each. They have a Ram of 256GB…
Santosh Bitra
  • 41
  • 3
1
vote
1 answer

Hodoop namenode not starting

When I use start-all.cmd, then datanode, resourcemanager, nodemanager are working properly but namenode is not working! 19/11/04 22:09:14 WARN namenode.FSNamesystem: Encountered exception loading fsimage java.io.IOException: NameNode is not…
Mihir Mistry
  • 11
  • 1
1 2
3
…
12 13