Usually, the machine that executes our code has limited resources. Doing everything at once might not only hurt but can also hang our process and make it stop responding altogether.
When we want to crawl 100 websites, we should crawl, for example, 5 at once, to not take up all the available bandwidth. As soon as one website is crawled, the next one is ready to go.
Generally speaking, all “heavy” operations should be laid out in time and not executed all-at-once for better performance and to save resources.
If you are familiar with my previous post about implementing promises, then you are going to notice a lot of similarities.
We register a given task by adding it to our tasksQueue and then we call executeTasks.
Now we execute as many tasks as our limit allows us – one by one, each time adding 1 to our counter called tasksActiveCount.
When the executed task finishes we remove 1 from tasksActiveCount and again call executeTasks.
Below we can see an example of how it works.
The limit is set to 3, so as the first two tasks are taking very long to process, we can see the third “slot” getting opened from time to time allowing the next task in the queue to be executed.
Always three there are, no more, no less.
You can see the code in the repository.
Thank you very much for reading! Can you think of any other way of achieving the same effect? Share them down below.
If you have any questions or comments feel free to put them in the comment section below or send me a message.
Check out my social media!