Using parfor to Make Many Web Requests - MATLAB
Video Player is loading.
Current Time 0:00
Duration 47:27
Loaded: 0.35%
Stream Type LIVE
Remaining Time 47:27
 
1x
  • Chapters
  • descriptions off, selected
  • captions off, selected
  • en (Main), selected
    Video length is 47:27

    Using parfor to Make Many Web Requests

    My colleague asked me to access all the pages on a web server in order to populate its cache. I plan to use parfor to get through the more than 250k pages in a timely manner. I will also need to not go too fast and overload the server.

    I’ve used parfor before for web page access and found that it is one of the rare situations where you can use more MATLAB® workers than available physical or even logical processors, which is not normally recommended. It works because web requests require a lot of waiting, and often the processing I need to do in MATLAB is little.

    I’ll admit this video gets a little boring near the end as I try out different numbers of workers. Remember that you can increase the playback speed of the video in the lower right corner of the player.

    Originally posted on Stuart’s MATLAB Videos blog.

    Recorded: 26 May 2021

    Related Products