Nodejs: How To Use Async.js To Process A List Of Items In Database

I have this problem: I get a list of items from an API and I have to save them in database. For each of them I have to display some feedback, like ' - item name. Saving in DB..' an

Solution 1:

Based on the async documentation using async.each

Applies the function iteratee to each item in arr, in parallel.

And look they even warned that this does not preserve the order since its parallel:

Note, that since this function applies iteratee to each item in parallel, there is no guarantee that the iteratee functions will complete in order.

If I understand correctly, you wish your items (assuming you have A,B,C) to be executed in the following order:

//This uses async.eachSeries
For each - A
SaveInDB called.
Timeout set. Waiting to timeout ...
Processing DB transaction for > A ....... Done.
Success! Next....
For each - B
SaveInDB called.
Timeout set. Waiting to timeout ...
Processing DB transaction for > B ....... Done.
Success! Next....
For each - C
SaveInDB called.
Timeout set. Waiting to timeout ...
Processing DB transaction for > C ....... Done.
Success! Next....

And not in the following order (or maybe different, its parallel, no order guaranteed):

//uses async.each 
For each - A
SaveInDB called.
Timeout set. Waiting to timeout ...
For each - B
SaveInDB called.
Timeout set. Waiting to timeout ...
For each - C
SaveInDB called.
Timeout set. Waiting to timeout ...
Processing DB transaction for > A ....... Done.
Success! Next....
Processing DB transaction for > B ....... Done.
Success! Next....
Processing DB transaction for > C ....... Done.
Success! Next....

So change your method from async.each to async.eachSeries as well as to make sure that you don't get confused see below saveInDB function.

functionsaveInDB(item, callback) {
   //this will not wait for timeout. console.log('SaveInDB called.');

   // Simulating some delaysetTimeout(function () {
     //everything in this block waits for timeout. console.log('Processing DB transaction for > ' + item + ' ....... Done.');
     console.log('Success! Next....');
     callback(null)
   }, 5000);

   //this will not wait for timeoutconsole.log('Timeout set. Waiting to timeout ...');
 }

I would give it a go with real database, rather than simulation with timeout. Be careful with timeout, see below from documentation:

It is important to note that your callback will probably not be called in exactly delay milliseconds - Node.js makes no guarantees about the exact timing of when the callback will fire, nor of the ordering things will fire in. The callback will be called as close as possible to the time specified.

Solution 2:

async.each(data.results, function (result, cb_results) {
       //try your here cb_results();
    }, function (err) {
        if (err) { throw err; }

    });

Example:

var returnData = [];
    async.each(data.results, function (result, cb_results) {
       var myJson = {};
       myJson.id = result.id;
       ................
       returnData.push(myJson);
       cb_results();
    }, function (err) {
        if (err) { throw err; }
        callback(returnData);
    });