Stack Overflow
Stack Overflow

Questions tagged [distributed-computing]

utilizing more than one computer, connected to each other with a communication link to accomplish a common task.

Distributed computing is where multiple computing units are connected to achieve a common task. The larger computing power enables a lot more tasks to be performed than in a single unit, and searches can be coordinated for efficiency. Successes usually give the finder credit.

Distributed computing projects include hunting large prime number, and analysing DNA codes.

Projects

References

2586 questions
387
votes
6 answers

Explaining Apache ZooKeeper

I am trying to understand ZooKeeper, how it works and what it does. Is there any application which is comparable to ZooKeeper? If you know, then how would you describe ZooKeeper to a layman? I have tried apache wiki, zookeeper sourceforge...but I…
topgun_ivard
  • 7,648
  • 8
  • 34
  • 44
317
votes
17 answers

Spark - repartition() vs coalesce()

According to Learning Spark Keep in mind that repartitioning your data is a fairly expensive operation. Spark also has an optimized version of repartition() called coalesce() that allows avoiding data movement, but only if you are decreasing the…
Praveen Sripati
  • 29,779
  • 15
  • 74
  • 108
244
votes
2 answers

What are workers, executors, cores in Spark Standalone cluster?

I read Cluster Mode Overview and I still can't understand the different processes in the Spark Standalone cluster and the parallelism. Is the worker a JVM process or not? I ran the bin\start-slave.sh and found that it spawned the worker, which is…
Manikandan Kannan
  • 7,246
  • 12
  • 35
  • 60
217
votes
6 answers

What is the difference between cache and persist?

In terms of RDD persistence, what are the differences between cache() and persist() in spark ?
Ramana
  • 6,443
  • 7
  • 25
  • 30
200
votes
3 answers

What determines Kafka consumer offset?

I am relatively new to Kafka. I have done a bit of experimenting with it, but a few things are unclear to me regarding consumer offset. From what I have understood so far, when a consumer starts, the offset it will start reading from is determined…
Asif Iqbal
  • 3,450
  • 4
  • 20
  • 29
127
votes
25 answers

Calculate the median of a billion numbers

If you have one billion numbers and one hundred computers, what is the best way to locate the median of these numbers? One solution which I have is: Split the set equally among the computers. Sort them. Find the medians for each set. Sort the sets…
anony
  • 1,433
  • 3
  • 13
  • 10
89
votes
5 answers

Is it possible to add partitions to an existing topic in Kafka 0.8.2

I have a Kafka cluster running with 2 partitions. I was looking for a way to increase the partition count to 3. However, I don't want to lose existing messages on the topic. I tried stopping Kafka, modifying the server.properties file to increase…
Asif Iqbal
  • 3,450
  • 4
  • 20
  • 29
74
votes
3 answers

Meaning of inter_op_parallelism_threads and intra_op_parallelism_threads

Can somebody please explain the following TensorFlow terms inter_op_parallelism_threads intra_op_parallelism_threads or, please, provide links to the right source of explanation. I have conducted a few tests by changing the parameters, but the…
itsamineral
  • 1,119
  • 3
  • 12
  • 18
60
votes
4 answers

Apache Spark vs Akka

Could you please tell me the difference between Apache Spark and AKKA, I know that both frameworks meant to programme distributed and parallel computations, yet i don't see the link or the difference between them. Moreover, I would like to get the…
user4658980
59
votes
5 answers

Difference between cloud computing and distributed computing?

I wanted to know about the difference about cloud computing and distributed computing. I read an article about cloud computing and got a feeling that somewhere there is a relation between cloud computing and distributed computing and so wanted to…
Rachel
  • 91,207
  • 112
  • 255
  • 361
56
votes
1 answer

"Eventual Consistency" vs "Strong Eventual Consistency" vs "Strong Consistency"?

I came across the concept of "Strong Eventual Consistency" . Is it supposed to be stronger than "Eventual Consistency" but weaker than "Strong Consistency"? Could someone explain the differences among these three concepts with applicable…
njzhxf
  • 797
  • 1
  • 6
  • 9
49
votes
4 answers

Service discovery vs load balancing

I am trying to understand in which scenario I should pick a service registry over a load balancer. From my understanding both solutions are covering the same functionality. For instance if we consider consul.io as a feature list we have: Service…
Lucian Enache
  • 2,210
  • 5
  • 31
  • 55
48
votes
1 answer

Flattening Rows in Spark

I am doing some testing for spark using scala. We usually read json files which needs to be manipulated like the following example: test.json: {"a":1,"b":[2,3]} val test = sqlContext.read.json("test.json") How can I convert it to the following…
Nir Ben Yaacov
  • 1,006
  • 2
  • 15
  • 29
47
votes
2 answers

What is a task in Spark? How does the Spark worker execute the jar file?

After reading some document on http://spark.apache.org/docs/0.8.0/cluster-overview.html, I got some question that I want to clarify. Take this example from Spark: JavaSparkContext spark = new JavaSparkContext( new…
EdwinGuo
  • 1,595
  • 2
  • 16
  • 24
45
votes
1 answer

How does pytorch's parallel method and distributed method work?

I'm not an expert in distributed system and CUDA. But there is one really interesting feature that PyTorch support which is nn.DataParallel and nn.DistributedDataParallel. How are they actually implemented? How do they separate common embeddings and…
erickg
  • 841
  • 6
  • 13
1
2 3
…
99 100