Stack Overflow
Stack Overflow

Questions tagged [large-scale]

42 questions
563
votes
8 answers

Large-scale design in Haskell?

What is a good way to design/structure large functional programs, especially in Haskell? I've been through a bunch of the tutorials (Write Yourself a Scheme being my favorite, with Real World Haskell a close second) - but most of the programs are…
Dan
  • 5,773
  • 3
  • 15
  • 13
17
votes
6 answers

With Haskell, how do I process large volumes of XML?

I've been exploring the Stack Overflow data dumps and thus far taking advantage of the friendly XML and “parsing” with regular expressions. My attempts with various Haskell XML libraries to find the first post in document-order by a particular user…
Greg Bacon
  • 121,231
  • 29
  • 179
  • 236
12
votes
2 answers

Applying a function in each row of a big PySpark dataframe?

I have a big dataframe (~30M rows). I have a function f. The business of f is to run through each row, check some logics and feed the outputs into a dictionary. The function needs to be performed row by row. I tried: dic = dict() for row in…
mommomonthewind
  • 3,057
  • 6
  • 31
  • 58
10
votes
3 answers

Dealing with large files in Haskell

I have a large file (4+ gigs) of, lets just say, 4 byte floats. I would like to treat it as List, in the sense that I would like to be able to use map, filter, foldl, etc. However, instead of producing a new list with the output, I would like to…
Jonathan Fischoff
  • 1,397
  • 11
  • 22
8
votes
3 answers

How to solve large-scale nonlinear optimization problems with Ceres?

I need to optimize a surface represented by a 2D grid of points to produce normal vectors of the surface that align with provided target normal vectors. The grid size is likely to be between 201x201 and 1001x1001. That means that the number of…
Eric
  • 234
  • 3
  • 10
7
votes
8 answers

Brain modelling

Just wondering, since we've reached 1 teraflop per PC, yet we are still not able to model an insect's brain. Has anyone seen a decent implementation of a self-learning, self-developing neural network?
Andy
  • 2,476
  • 1
  • 27
  • 47
4
votes
1 answer

Whats the best structure for a large scale python tornado project?

I'm using a mongodb database backend with tornado as my core. I currently just have my main file with a bunch of handlers in it. It's a multiuser webapp with links between users, aka a "friend" system. My current structure is: templates/ …
carboncomputed
  • 1,518
  • 3
  • 19
  • 36
3
votes
0 answers

How to decouple application architecture from React library?

We're planning to implement our front-end application based on Large-Scale JavaScript Application. There's a concern in our team to not be coupled to any JavaScript based frameworks like React or Angular. How could it be designed to be decoupled…
Ahmad Payan
  • 1,933
  • 1
  • 19
  • 31
3
votes
0 answers

How to fan-out/distribute 500,000 records with Firebase?

Requirements: Users can register to company & unsubscribe freely. Users should get only the posts of companies they are registered to. Users shouldn't see posts they already liked/disliked Currently, what we are doing is, when a user joins a…
Uri Brodsky
  • 59
  • 4
3
votes
1 answer

Optimizing a query for optional fields from another table

I have a innodb table called items that powers one ecommerce site. The search system allows you to search for optional/additional fields, so that you can e.g. search for only repaired computers or cars only older than 2000. This is done via…
aez1
  • 31
  • 3
2
votes
3 answers

Is there any must-have kind of book about Symfony-2.0 and large scale projects?

I'm looking for a guide book for Symfony framework, specially version-2.0. I'm interested in more advanced guides, specially about how to "squeeze" the maximum performance from symfony, optimizations for medium to large scale projects where even…
marcio
  • 8,631
  • 9
  • 45
  • 81
2
votes
1 answer

Large-scale models and FMU for model exchange

I want to export large-scale models from Openmodelica to FMU for model exchange. As first step, I'm using this model for testing purposes: model BigModel "Tank-Pipe-Tank cloned n times" // constant Real g=9.8; // parameter Integer n = 1; …
Neotake
  • 79
  • 6
2
votes
0 answers

How to independently scale M web app servers and N cache servers?

I have M web app servers (with the same app on them, shared-nothing), behind a load balancer. The app running requires heavy caching, so I would like to have N cache servers behind (using Memcached). I would like to use consistent hashing, so I…
v_i_m
  • 171
  • 1
  • 2
  • 6
2
votes
4 answers

How to avoid large matrix multiplication in Matlab

There are two large matrices in my code, which have the same number of columns and different number of rows. Like A(20000X4000) and B(30000X4000). Both are 0-1 sparse. I should check each row of A with all rows of B and count the number of common…
Alef
  • 41
  • 1
  • 7
2
votes
1 answer

Large shared code base, extend functionality

Our application is a large scale Javascript application, using the Javascript MVC framework. We are using the MVC app folder across all of our sites via an SVN:external, and each site also has their own files. The settings files are specific to the…
Scott Warren
  • 233
  • 2
  • 11
1
2 3