Is Thingspeak MQTT reliable?

I have a couple of NodeMCUs that are sending data to TS every 30 minutes since 2018, with no issues, using ThingSpeak API. These days I tried to give MQTT a chance, so I changed my code in order to use ThingSpeak MQTT broker to publish my data. I followed the code example from this page.

The result is really disappointing: while the devices are still sending the data every 30 minute (with no error codes) the data is not published every time, so I have my chart full of "holes". Sometimes they are published once an hour, some other every 2 hours, randomly.

I have checked the code many times, but it is quite straight-forward, so I have the suspect that the QoS=0 is not giving the best results.

Answer 1

[Vinod]

I just tried this example for a few hours, admittedly it is on a Raspberry Pi, and had reliable results with no "holes" in the data. If your code on the NodeMCU is substantially different, can you share that so we may use that to help you troubleshoot? (Please mask/delete any API keys or credentials in your code you share)

[Fabio Marzocca]

Hi Vinod,

you can find the code here.

[Vinod]

A few things I see straight away without trying to put this code on a device:

• The loop() seems to have nothing. Most examples of the library call client.loop() inside of the Arduino loop()

• There seems to be no delay() whatsoever when publishing to ThingSpeak in the sendDataToTS() function. With a free account, one can update a channel every 15 seconds at most. With a paid license, one can increase that rate to up to once per second. Knowingly exceeding rates allowed by your license type borders on abuse of service and might result in the channel ingest getting throttled.

• There's a lot of libraries used. It is not clear to me that one of them is not using up all free memory and causing runtime errors that result in unpredictable behavior.

My recommendation is to simplify the sketch -- perhaps start with one of the examples with the PubSubClient library - and add on the necessary parts from there. Remember to ensure that your sendDataToTS() has adequate delays so you don't end up hammering the service.

[Fabio Marzocca]

As I said, I am sending data to TS every 30 minutes. The loop does nothing because the code never reaches it. It executes the setup and then a pin changes state, which drives a TPL5110 to shutdown power at the NodeMCU and start a cycle count of 30 minutes.

So, the sendDataToTS() is executed once, then device powers off and restart after 30 minutes.

The same sketch is working since 2018 by using the ThingSpeak.h library.

[Fabio Marzocca]

This is the flowchart: https://www.acronico.it/wp-content/uploads/2018/09/Meteorino-flowchart-v.2.jpg

[Fabio Marzocca]

There is no memeory issue: I tested with the serial console and the debugging messages. It works but onve every now and then it does not send the data (or, as no error message is sent through Telegram, I suppose it sends the data but the data is not displayed).

Just watch at the last 24hours of data on channel 563760 (previous data was wrong as I was doing tests).

[Vinod]

Thanks for clarifying it - I missed the fact that you have circuitry that ensures the request only gets sent every 30 minutes.

When I look at the server logs, I definitely see that not all requests ~30 minutes apart are coming in. There are gaps in the timestamps of the received message; Serveral of them didn't even make it to our HiveMQ server. Below is a screenshot of the timestamps in US EDT of requests to update your channel from the latter half of 12/30. I've marked some of the missing slots.

This could be on account of the QoS 0, the PubSubClient's robustness in the network conditions, etc.

I'm not sure I have any more suggestions, other than to try connecting your device over wired connection rather than WiFi. Then again, that might not be possible with the NodeMCUs you're using.

One other suggesiton is to possibly try a C program on your PC or a (wired) Raspberry Pi that is programmed to do something similar to your code on the NodeMCU to confirm that is not seeing dropped points.

[Fabio Marzocca]

Thank you Vinod, I will make some tests.

[Fabio Marzocca]

I have updated all the libraries in the sketch and re-uploaded it at 7:00 am EDT. Let's see if it is better now...

[Fabio Marzocca]

As you can see... it was not a solution. Still "holes".

[Fabio Marzocca]

Tomorrow I will reload my previous sketch, the one using Thingspeak.h library and REST calls, that was working for 2 years. Definitely MQTT is not an option to upload data to thingspeak, I am sorry.

[Vinod]

Thank you for testing it, Fabio.

We'll look into adding some examples in the ThingSpeak library that shows how to use MQTT to ThingSpeak reliably from devices like the ESP32 or NodeMCU. That should provide a good starting point.

[Fabio Marzocca]

Hi Vinod,

I have replaced yesterday the sketch with MQTT code with the one using the web API calls. You can see the results yourself on the channel.

At this point we can exclude any network-related problem, restrisrting the troubleshooting to just 2 items: the PubSub library or Thingspeak's broker.