Fastest way to search in millions of records which is the combination of two js objects

Question

I have millions of objects of messages

messages: [
    {
      id: Int,
      text: String,
      userId: Int,
      receiverId: Int,
    },

and thousands of users

  users: [
    {
      id: Int,
      name: String,
    },

I need to process two objects and return an object in format

[{ message, userFromName, userToName }]

I read about array methods like find, filter, some and all of these are slower than native for and foreach.

I also wrote a function which two foreach loops

msgData.forEach(function(msg,i) {
    ...iterating every msg 
    userData.forEach(function(user) {
       ...iterating every user id over message sender and receiver id
    });
});

The complexity of the code O(n)square

How to get the required format in less amount of time?

Maybe it would be more efficient if you made the server process this data. — Nicolas, Nov 20 '19 at 17:20
If your data can be converted to object with the id being the key, like `messages = {"id1": {...}, "id2": {...}}` and `users = {"id1": {...}, "id2": {...}}` then you can drastically improvement performance since look ups are now done by the Object key, not iterating through an array. — Kousha, Nov 20 '19 at 17:20
"*I need to process two objects*" - which ones? "*and return an object in format*" - do you mean returning an array with multiple objects? — Bergi, Nov 20 '19 at 17:21
What makes you think that `find` or `some` are slower than `forEach`? — Bergi, Nov 20 '19 at 17:22
Put them in a few `Map`s: messages keyed by sender, messages keyed by receiver, users keyed by id. That should allow you to do most searches efficiently enough. — Bergi, Nov 20 '19 at 17:24
@Bergi - Please read https://codeburst.io/javascript-performance-test-for-vs-for-each-vs-map-reduce-filter-find-32c1113f19d7 — Puneet Kushwah, Nov 20 '19 at 17:30
@PuneetKushwah That article makes no sense. The `forEach` method has exactly the same disadvantages as a the other array iteration methods. — Bergi, Nov 20 '19 at 17:34
@Bergi - I have also read the some thing on many other stackoverflow questions... if you are dealing in millions go for FOR loop or FOREACH loop... however I am not sure how correct it is. — Puneet Kushwah, Nov 20 '19 at 17:37
@PuneetKushwah The only reasonable advise is to do your own benchmarks on your actual use case with your real data. That said, modern engines try to optimise all looping methods the same. And as always: if you are looping over millions of records, your data structure was chosen wrong. — Bergi, Nov 20 '19 at 17:41

Klaycon · Accepted Answer · 2019-11-20T17:36:30.820

Convert users to a dictionary for fast, non-iterative lookup:

Native Object:

let userDict = users.reduce((o,u)=> {
  o[u.id]=u.name;
  return o;
}, {});

Map:

let userDict = new Map();
userDict.forEach(u => userDict.set(u.id,u.name));

This is O(n). With this dictionary you can simplify the msgData forEach to this:

let result = msgData.map(msg => {
  return { message: msg.text, userFromName: userDict[msg.userId], userToName: userDict[msg.receiverId]};
 });

This is also O(n) as both object and Map lookups are O(1). See this question for performance details on those. Either way you go will be a significant performance improvement over your current O(n^2) solution.

score 0 · Answer 2 · answered Nov 21 '19 at 05:56

Assuming that you will have a key in common between the two arrays, you could use a Hash Map Data Structure, in Javascript can be represented as a simple plain object, for example:

const users = [{id: 1, name: 'Marcela'}, {id:2, name: 'Ana'}];
const messages = [{id: 1, message: 'She is a great Graphic Designer'}, {id: 2, message: 'She likes watch movies'}];

// #1 convert one array to a hashmap
const userHashMap = {};

// O(N)
for (let user of users) {
  userHashMap[user.id] = user;
}

// above code will store a DS like 
// {
//     1: 
//         {
//             id: 1,
//             name: 'Marcela'
//         }, 
//     2:
//         {
//             id: 2,
//             name: 'Ana'
//         }
// }

// #2 Now you can iterate over messages in O(N)
// and we can get the values of users 
// in constant time O(1) (That's the most important use of HashMaps):

messages.forEach(message => {
  console.log(`This id message ${message.id} belongs to ${userHashMap[message.id].name}`);
  console.log(`Message for ${userHashMap[message.id].name}: ${message.message}`);
});

Remember, we reduced time complexity using a technique that has an impact in memory (object HashMap is storing data for every user so space complexity is O(N)).

Fastest way to search in millions of records which is the combination of two js objects

2 Answers2