"webread" does not work for Yahoo Option Chains page

Going to any Yahoo Option Chain page, copy and paste the URL to webread returns an error. For example:

webread('https://finance.yahoo.com/quote/AAPL/options/')

returns

The server returned the status 404 with message "Not Found" 

It works for stock prices, but not for option chains. Is this a Yahoo protection issue or a limitation of webread?

[Walter Roberson]

Note that

webread('https://finance.yahoo.com/quote/AAPL/')

works.

[Cengiz Esmersoy]

Yes, it works for stocks, but not for options. This makes me think there may be an access block by Yahoo, but this would typically be status 401.

And, I understand from blog traffic Pyton can access it, although I have not tried myself.

[Walter Roberson]

Access blocks would typically be error 403, Access Prohibitted.

Answer 1

[Kanishk]

Hi Cengiz,

It seems that when trying to use ‘webread’ to access the Yahoo Finance options page ‘404 Not Found’ error is encountered.

The issue appears to be related to the "User-Agent" field in the HTTP request. Some servers, including Yahoo Finance, selectively block requests from certain clients if the "User-Agent" field is not set or is set to an unrecognized value.

Although the field is not mandatory in HTTP requests, to resolve this issue, you can specify the "User-Agent" in the ‘webread’ request to mimic a web browser using ‘weboptions’.

Please follow the below code snippet to resolve the error:

url = 'https://finance.yahoo.com/quote/AAPL/options';

% Set the web options with a User-Agent to mimic a web browser 
options = weboptions('UserAgent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'); 

try 
    data = webread(url, options); 
    % Save retrieved page in HTML file 
    fid = fopen('AAPLOptions.html','w'); 
    fprintf(fid, '%s\n',data); 
    fclose(fid); 
catch err 
    disp(['Error: ', err.message]); 
end 

I was able to resolve the above error following the above steps.

The ‘User-Agent’ property of your browser can be found here

https://www.whatismybrowser.com/detect/what-is-my-user-agent/

Please go through the following documentation pages to learn more about ‘webread’ and ‘weboptions’:

• https://www.mathworks.com/help/releases/R2020b/matlab/ref/weboptions.html
• https://www.mathworks.com/help/releases/R2020b/matlab/ref/webread.html

Hope this helps!

Thanks

Kanishk