Stack Overflow
Stack Overflow

Questions tagged [namenode]

The Hadoop NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept.

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself.

Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The NameNode responds to the successful requests by returning a list of relevant DataNodes where the data lives.

190 questions
19
votes
1 answer

How does Hadoop Namenode failover process works?

Hadoop defintive guide says - Each Namenode runs a lightweight failover controller process whose job it is to monitor its Namenode for failures (using a simple heartbeat mechanism) and trigger a failover should a namenode fail. How come a…
K246
  • 951
  • 1
  • 7
  • 13
16
votes
1 answer

Connection reset by peer while running Apache Spark Job

We have two HDP cluster's setup let's call them A and B. CLUSTER A NODES : It contains a total of 20 commodity machines. There are 20 data nodes. As namenode HA is configured, there is one active and one standby namenode. CLUSTER B NODES : It…
Aniketh Jain
  • 543
  • 5
  • 21
13
votes
4 answers

error in namenode starting

When i try to start the hadoop on master node i am getting the following output.and the namenode is not starting. [hduser@dellnode1 ~]$ start-dfs.sh starting namenode, logging to…
Tejas
  • 238
  • 1
  • 2
  • 12
10
votes
3 answers

Hadoop Installation Issue on Windows

I have been trying to install Hadoop on Windows 7 for quite sometime now. I am following this blog for instructions. But unfortunately I have not been be able to run the Namenode. There seems to be issue with hdfs-site.xml file but I dont see…
Ahmed S. Durrani
  • 1,397
  • 1
  • 15
  • 36
9
votes
2 answers

Difference between fs.defaultFS and fs.default.name

I know both these properties are related to namenode. But what is the actual difference between them?
pramesh
  • 1,694
  • 1
  • 13
  • 23
6
votes
3 answers

Namenode file quantity limit

Any one know how many bytes occupy per file in namenode of Hdfs? I want to estimate how many files can store in single namenode of 32G memory.
billow
  • 127
  • 1
  • 1
  • 9
5
votes
0 answers

HDFS No valid image files found

we have a old hadoop cluster machine hadoop - version 2.6 all machines in the cluster are redhat version - 7.3 we have a problem to start the standby name-node on the last master machine from the logs ( under /var/log/hadoop/hdf ) , we can see…
enodmilvado
  • 383
  • 5
  • 18
4
votes
2 answers

When do YARN and NameNode interact

When a job is submitted, when do YARN and NameNode interact? When a job is submitted, who does it get sent to? Could someone explain the end-to-end flow - how hadoop ecosystem works? Thanks!
user2498121
  • 43
  • 4
4
votes
1 answer

"start-all.sh" and "start-dfs.sh" from master node do not start the slave node services?

I have updated the /conf/slaves file on the Hadoop master node with the hostnames of my slave nodes, but I'm not able to start the slaves from the master. I have to individually start the slaves, and then my 5-node cluster is up and running. How can…
ingmid
  • 51
  • 3
4
votes
1 answer

Is Namenode still necessary if I use S3 instead of HDFS?

Recently I am setting up my Hadoop cluster over Object Store with S3, all data file are store in S3 instead of HDFS, and I successfully run spark and MP over S3, so I wonder if my namenode is still necessary, if so, what does my namenode do while I…
TaoCHEN92
  • 383
  • 3
  • 6
4
votes
1 answer

Query on hadoop namenode -format command

While doing "hadoop namenode -format", below message comes. Re-format filesystem in Storage Directory /opt/data/temp/dfs/name ? (Y or N) What should one give here? "Y" or "N". If given Y, will it lose data from HDFS?
earl
  • 636
  • 9
  • 28
4
votes
2 answers

what's the actual ideal NameNode memory size when meet a lot files in HDFS

I will have 200 million files in my HDFS cluster, we know each file will occupy 150 bytes in NameNode memory, plus 3 blocks so there are total 600 bytes in NN. So I set my NN memory having 250GB to well handle 200 Million files. My question is that…
Jack
  • 4,626
  • 8
  • 46
  • 98
3
votes
1 answer

hadoop cluster with active standby namenode + gap in the edit log

we have ambari cluster , HDP version 2.6.5 cluster include management of two name-node ( one is active and the secondary is standby ) and 65 datanode machines we have problem with the standby name-node that not started and from the namenode logs we…
jessica
  • 1,804
  • 4
  • 24
3
votes
0 answers

Namenode in hadoop cluster and fsimage and Edit_logs consept

I want to give short background about the namenodes and fsimage/edit_logs , and how namenode works in hadoop clusters, The NameNode stores modifications to the file system as a log appended to a native file system file, edits. When a NameNode…
jessica
  • 1,804
  • 4
  • 24
3
votes
0 answers

NameNode only has to wait for blocks to be minimally replicated before returning successfully

I have a question about the block reports sent to the NameNode from the DataNodes when a client writes in HDFS, as well as the ack from the NameNode to the client about closing the file. Could someone kindly elaborate this paragraph of the Hadoop…
triples
  • 31
  • 2
1
2 3
…
12 13