Execute Many Promises Sequentially (concept)

(My target is clarify my concept about the problem, not code) I want execute a array of promises sequentially, but nodeJS throw a strange error about many promises executed in para

Solution 1:

I would have used something like a worker pool instead of executing things in a batch of 20 each time, you will always end up waiting for the last one to finish before you start next 20 batch, instead you should set a limit of how many continious download you want to do so you have no more then 20 promises and not a long chain of 9000

The same thing can be accomplish with iterators also. (a same iterator can be passed to different workers and while someone calls the first item the next worker will always get the next one)

So with zero dependencies i would do something like this:

constsleep = n => newPromise(rs =>setTimeout(rs, 1000))

asyncfunctionsequentialDownload(iterator) {
  for (let [index, url] of iterator) {
    // figure out where to save the fileconst path = path.resolve(__dirname, 'images', index + '.jpg')
    // download all images as a streamconst res = await axios.get(index, { responseType: 'stream' })

    // pipe the stream to discconst writer = fs.createWriteStream(path)
    res.data.pipe(writer)

    // wait for the download to completeawaitnewPromise(resolve => writer.on('finish', resolve))
    // wait a extra 5 secawaitsleep(5000)
  }
}

const arr = [url1, url2, url3] // to be downloadedconst workers = newArray(20) // create 20 "workers"
  .fill(arr.entries()) // fill it with same iterator
  .map(sequentialDownload) // start workingPromise.all(workers).then(() => {
  console.log('done downloading everything')
})