Stack Overflow
Stack Overflow

Questions tagged [dataflow]

Dataflow programming is a programming paradigm in which computations are modeled through directed graphs: nodes are instructions and data flows through the connections between them.

Dataflow programming is a programming paradigm which models programs as directed graphs and calculation proceeds in a way similar to electrical circuits. More precisely:

  • nodes are instructions that takes one or more inputs, perform calculation on them and present the result(s) as output;
  • edges connects inputs and outputs of the instructions -- this way the output of an instruction can be fed directly to the input on another node to trigger another calculation;
  • data "travels" using the directed edges and triggers the instructions as they pass through the nodes.

Often dataflow programming languages are visual, the most prominent example being LabView.

Resources

964 questions
-1
votes
1 answer

Dependent data processing pipelines where files arrive asynchronously

I have several data dependent tasks/pipelines of which some depend on the completion of another. What makes it even harder is that the data can arrive asynchronously, meaning that certain tasks need to wait until all the files or tasks in the…
JohnAndrews
  • 5,088
  • 9
  • 57
  • 121
-1
votes
1 answer

How to create gcloud dataflow from linux console

I have already created gcloud dataflow on the Google cloud platform and now I need to do it from the Linux console. The main questions: How should it look? How can I work with --parameters of gcloud?
Alexander Antonenko
  • 1
  • 3
-1
votes
2 answers

Adding support for JS UDF in Google Dataflow template

I have this code from Google Cloud Platform Dataflow Templates. I wish to add more functionalities to it, namely, I wish to add support for JavaScript UDF. When I try to compile the file, using this: mvn compile exec:java…
tHappy
  • 21
  • 4
-1
votes
1 answer

window Data hourly(clockwise) basis in Apache beam

I am trying to aggregate streaming data for each hour(like 12:00 to 12:59 and 01:00 to 01:59) in DataFlow/Apache Beam Job. Following is my use case Data is streaming from pubsub, It has a timestamp(order date). I want to count no of orders in each…
Kamal Sharma
  • 1
  • 1
-1
votes
1 answer

Continus assignment to verilog module instance

Is it possible to use assign keyword with module instances? Lets assume I have a module logarithm which works as it's intended. In some other module, I want to have: A = log(B) + log(C). Is there any efficient way other than the following to do…
Reza Abdolhakim
  • 13
  • 5
-1
votes
1 answer

SSIS Package Writes 0 rows but run DataFlow only and get 400k rows

I use my own login. I had to add a new join to OLEDB source query and tested with execute of Dataflow task and works 400k rows. Then run within package return is 0 rows and no error or warning
MacAries
  • 1
  • 3
-1
votes
2 answers

Dataflow Task.WhenAll causes A task was canceled Exception

I am new to Dataflow, and I follow this walkthrough How to: Cancel a Dataflow Block. I click add button first, and then click cancel, but I got exception about "A task was canceled Exception" after clicking cancel button. I fail to find any way to…
Edward
  • 22,080
  • 7
  • 44
  • 80
-1
votes
1 answer

Writing a protobuf object to GCS

I am making a dataflow (java api) which will write protobuf object to gcs. Can anyone please help me with how to do this. A sample of code or something would be helpful.
Raman
  • 172
  • 14
-1
votes
1 answer

Trigger draw(rect) function

In my app I have a chart into an UIView. This chart is drawed using override func draw(_ rect: CGRect) {}. Trouble is that datas to populate the chart comes from the ViewController which contains the UIView and they need some time to be calculated…
Matte.Car
  • 1,915
  • 4
  • 21
  • 39
-1
votes
1 answer

How can I transfer data on play framework

I want to create single page application and I am beginner at play framework. I install the play activator and create play-java project then I import the project in eclipse. I put my login.html and necessary js files (index.js, controller.js etc)…
artless7
  • 13
  • 7
-1
votes
1 answer

BufferBlock -> ActionBlock Not Firing

In Short I create an BatchBlock which links to an ActionBlock, which calls a async method, but this method never fires. Details Here is how I create the blocks var instance = new ConnectionSaveStep(repo, progress, total); var batch =…
Boas Enkler
  • 11,465
  • 12
  • 61
  • 128
-1
votes
1 answer

failed to compile dataflow sample

I tried the Google sample dataflow but I get the following error when I run maven install as described in the following link: [ERROR] Failed to execute goal on project google-cloud-dataflow-java-examples-all: Could not resolve dependencies for…
Mohamed Sylla
  • 1
  • 1
-1
votes
1 answer

Unspecific Dataflow Import error in Magento

I'm trying to import data into Magento through Dataflow. I've tried both xml and csv files but huge error messages starting with the following keep coming up:
ReadySteadyFlow
  • 41
  • 4
-2
votes
1 answer

How to create multi zone(regional) dataflow using terraform?

dataflow list I want to change instance group location from asia-northeast1-a(zonal) to asia-norhteast1(regional) but dataflow job automatically select single zone. I wrote terraform code below as reference but maybe this is not a terraform issue…
hiko1129
  • 3
  • 2
1 2 3
…
64
65