Usually, the machine that executes our code has limited resources. Doing everything at once might not only hurt but can also hang our process and make it stop responding altogether.
When we want to crawl 100 websites, we should crawl, for example, 5 at once, to not take up all the available bandwidth. As soon as one website is crawled, the next one is ready to go.
Generally speaking, all “heavy” operations should be laid out in time and not executed all-at-once for better performance and to save resources.
Implementation
If you are familiar with my previous post about implementing promises, then you are going to notice a lot of similarities.
We register a given task by adding it to our tasksQueue and then we call executeTasks.
Now we execute as many tasks as our limit allows us - one by one, each time adding 1 to our counter called tasksActiveCount.
When the executed task finishes we remove 1 from tasksActiveCount and again call executeTasks.
Below we can see an example of how it works.
The limit is set to 3, so as the first two tasks are taking very long to process, we can see the third “slot” getting opened from time to time allowing the next task in the queue to be executed.
Always three there are, no more, no less.
Executing heavy and light tasks with the limit of 3.
You can see the code in the repository.
Thank you very much for reading! Can you think of any other way of achieving the same effect?
Also available on FreeCodeCamp.
