Thingspeak now unable to parse website

I have a ThingHTTP request that parses my local council website and extracts the bins which are collected each week. I then display this on an arduino with colour LEDs.

This worked fine for years but now for some reason the Thingspeak request cannot parse the website any more???

The conucil website is:

https://www.salford.gov.uk/bins-and-recycling/bin-collection-days/your-bin-collections/?UPRN=100012691624

This isn't my address but a random house nearby ;)

I've copied the XPath for the string about this week (//*[@id="standard-page"]/div/div/div[2]/p[1]/text()[2])

Then used the GET request on thingspeak - This used to work fine.

Now it gives me :

Error parsing document, try a different parse string.

Any ideas?

[Steven Lord]

You have confirmed that the layout of the page hasn't changed, and that the council hasn't changed who/what is allowed to access the website (choosing to use a robots.txt to block the scraping mechanism that ThingSpeak uses to read the data, for example?)

[Mark Quinn]

The layout hasn't changed. I checked the XPath and I'm sure that's correct.

I don't know how to check if the website has a scraper blocker though - how would I go about checking for that?

Answer 1

[Vinod]

I can't access the URL from the public internet in the US. I believe this to be the reason the ThingSpeak servers can't get to the page and parse the content. I also tried to use Opera browser's VPN to see if this page is accessible in other locations, and it is not accessible.

Have there been changes to the page's availability outside the UK?

[Mark Quinn]

Ah! That could be an issue... I don't know why they would have done that but if they have then I assume that I am somewhat scuppered ... unless ThinkSpeak can run through a VPN to appear in the UK.

Do you know if there is a way to do that or if there are any other alternatives?

[Vinod]

The one thought I had was to see if we could run it through Google Translate. However, it appears even Google Translate cannot get to the site. See:

https://translate.google.com/translate?sl=es&tl=en&hl=en&u=https://www.salford.gov.uk/bins-and-recycling/bin-collection-days/your-bin-collections/?UPRN%3D100012691624&client=webapp

The alternative I can think of is to find a free online proxy server based in the UK that can allow the page to be accessed outside the UK.

[Mark Quinn]

That's a clever approach- pity it didn't work 😞

I think I might have to look at using the Arduino to parse the website directly rather than using an API... I can't imagine this being fun 😅