HTTP Request returns partial data

I get the following error after contacting a REST service:

Warning: The server returned "15784" bytes, but "-1" were expected. The reason is "transfer closed with outstanding read data remaining"; 

The relevant code is simply:

msg = matlab.net.http.RequestMessage;
msg.Method = matlab.net.http.RequestMethod.GET;
accept = matlab.net.http.field.AcceptField(acceptType);
msg.Header = [accept];
[rsp, ~, ~] = msg.send(url);

This happens when the query in question takes a relatively long time, say 3-5 minutes. The options are set to have Inf timeout:

MaxRedirects: 20
ConnectTimeout: 10
UseProxy: 1
ProxyURI: []
Authenticate: 1
Credentials: [1×1 matlab.net.http.Credentials]
UseProgressMonitor: 0
SavePayload: 0
ConvertResponse: 1
DecodeResponse: 1
ProgressMonitorFcn: []
CertificateFilename: "default"
VerifyServerName: 1
DataTimeout: Inf
ResponseTimeout: Inf
KeepAliveTimeout: Inf

and so can't be to blame.

The specific warning message comes from:

function copyContentToFile(obj, filename)
% Copy the content from the Web service to a file.  This is used for content
% types which can only be converted using MATLAB functions that read data
% from files.  This may throw an exception if copy fails.

assert(obj.ConnectionIsOpen)
copyContentToFile(obj.Connection, filename);
end

when copyContentToFile is called.

My suspicion is that the problem is that matlab cannot handle transfer-encoding: chunked. This is a header that I see on the response. I am guessing that is why the warning appears, and then I only get back the first chunk of data, and not all data.

Does anyone know why this error occurs?

Answer 1

[Hassaan]

1. Server Closes Connection Prematurely

The server might be closing the connection before all the intended data has been sent, perhaps due to a timeout or misconfiguration on the server side. This would lead to the client expecting more data than it actually receives.

Suggestion: If possible, investigate the server configuration or contact the server administrator to ensure it's configured to handle long-running requests appropriately.

2. Network Interruptions or Proxy Issues

Network issues or proxies between your client and the server might be interfering with the connection, especially for long-running requests.

Suggestion: Test the request from a different network or machine, if possible, to rule out local network or proxy issues.

3. MATLAB HTTP Options Configuration

Although you've set a lot of the relevant timeouts to Inf, ensuring no timeouts on MATLAB's side, there might be other configurations or limits in the MATLAB HTTP engine or the underlying system that are causing the issue.

4. Workaround: Retry Mechanism

If the data you're expecting can be requested in smaller parts or if the service supports range requests, implementing a retry mechanism or pagination in your request logic could be a workaround. This would involve breaking down the long-running request into shorter, more manageable requests.

5. Increase Server Send Buffer

On some occasions, the server's send buffer size might be insufficient for large responses or slow connections, causing the server to close the connection if the buffer overflows.

Suggestion: Check if the server allows configuration of the send buffer size and increase it if possible.6. Debugging with External Tools

To further diagnose the issue, you could use external tools like Wireshark or Postman to mimic the requests and observe the behavior. These tools might give you additional insights into whether the issue is with the specific way MATLAB handles the requests or a more general server-side issue.

Suggestion: Mimic the request using Postman or cURL and observe if the issue persists. If it doesn't, the problem might be specific to how MATLAB handles the request or interprets the response.

If none of these suggestions resolves the issue, the best course of action would be to reach out to MATLAB support for assistance. They might be able to provide more specific advice or identify if there's a bug in the MATLAB HTTP client handling.

-----------------------------------------------------------------------------------------------------------------------------------------------------

If you find the solution helpful and it resolves your issue, it would be greatly appreciated if you could accept the answer. Also, leaving an upvote and a comment are also wonderful ways to provide feedback.

It's important to note that the advice and code are based on limited information and meant for educational purposes. Users should verify and adapt the code to their specific needs, ensuring compatibility and adherence to ethical standards.

Professional Interests

• Technical Services and Consulting
• Embedded Systems | Firmware Developement | Simulations
• Electrical and Electronics Engineering

Feel free to contact me.

Answer 2

[Mikael Öhman]

I already used chat gpt and got that answer. Posting here to get human answers.

[Hassaan]

Good to know. MATLAB community will definetly guide you further.