E-07 and web lock up

[MrEd]

Hi Ray, and everyone else. 

We've got a couple of Afterburner units, and generic Chinese heaters. We're using them for keeping our polytunnels full of plants frost free. After a long summer sleep, they've been revived as it's getting cold in the UK now. I updated the firmware on one unit and have had some issues since. (Of course because everything was off for quite a while, it may not have been the update.)

Afterburner: V3.5.2
Date: 2 Jul 2024
MQTT Webburner: 3.5.1.18
Date: 07 Jul 2024

I've twice had e-07 showing up, which as I understand it suggests a fault on the blue wire. I'll check in a bit more depth when I can see it and maybe give the connectors a little clean. I could potentially swap the units over too if that is worth a try. 

At the same time as this E-07 was showing, the web UI was behaving a bit weirdly, the web page loaded, but was not connecting to the websocket. On both occasions I resorted to removing the power and afterwards it was back to working fine again. During this, the Virtual Console (nice feature BTW) was still responsive, but I don't believe that the MQTT broker got any messages at all, the last thing showing in HomeAssistant was E-00 OK.

It ran for a couple of days between E-07 / unresponsive web front end.

As an aside, the logging feature seems like it could be useful, but is there any way the logs could be sent to a syslog server? I believe the protocol is really simple so it might be a handy to monitor how it's all going.

Thanks,
Ed.

[afterburner]

Whenever I hear of "No temperature sensor" or comms issues, my immediate suspicion is EMI from some other device.

Cheap Chinese battery chargers are notorious for having their mains input input EMI suppression components deliberately left out, replaced with wire links.
These components are meant to stop switching hash from the switch-mode supply's action re-emerging on the mains wiring, from where it can radiate widespread.

The EMI permeates the digital communications to the DS18B20, or ECU, leading to corrupted data and CRC errors.

So, it's "been off for a while", but has anything new been added otherwise, unrelated to the Afterburner is my first question...?

The issue of syslog has been brought up in the past, but sadly such things rely on lots of things working correctly in the first instance.
Even the web logging is "mickey mouse" as to record to a file relies upon the browser being enabled with an experimental flag to allow local file system writing.
Web pages normally do run in a sandbox, with no access to the host's file system for write access (outside of usual HTTP file GET).

Telnet can be used to gather logs, but things like system errors from IDF kernel code may not appear.
Arduino libraries can, via a Serial redirect I devised to allow that to happen.

The most reliable method is a direct serial debug connection to the debug port. All possible debug information appears that way.

The only reason why the web page logging was added was a side effect of providing debug menu access via the web page.
But as mentioned the available logging is not that fantastic, and telnet is still better for achieving that.

The other gremlin with time is connections oxidising, and causing higher resistance connections than normal.
This normally manifests as a constant boot loop, as voltage drops when wifi is started trigger a brownout reboot.
Usually the act of unplugging/re-plugging every connection a couple of time on the heater's controller leg is sufficient to scrape clean metal back.
ie JST plug at AB, 3 pin OEM controller plug, main ECU 9 pin plug.

[MrEd]

Thanks very much for replying.

I shall give it all a good clean, as you can imagine it's a fairly damp environment in the polytunnels. Less so now we've got rid of the horrid gas heaters and can use the diesel heater instead! 

In terms of EMI, the only thing plugged in nearby is the charger. Perhaps not the highest quality one, but bought from Halfords in the UK, so at least has some likelihood if a real CE mark/testing. It was the same one as last year, but it's certainly worth a try to rule that out.

I fully understand the pain of trying to retrofit something into existing software. Now you've got me thinking about a serial to syslog bridge... I've got too many projects!

Thanks,
Ed.

[MrEd]

I'm afraid I've not been able to do any physical maintenance to the wires yet, but something strange is happening.

I've been noticing that the web UI will hang, waiting on connection to the websocket. I've also tried using the mqtt broker for the web UI and that's never managed to connect successfully. 

So last night, I rolled back the firmware and web files to the previous stable version. It connected straight away, and the heater has been in E-00 ever since. It's been running on the thermostat over night, keeping the polytunnels toasty.

I've also noticed that the E-07 has never made it to the HomeAssistant logging via mqtt.

So, I'm not sure it changes anything, and I'm still going to do the cleaning of the terminals, but it does seem a lot more stable on the previous firmware. Time will tell though!

[MrEd]

It turns out that my father may have been looking at the device physically at the same time I was looking at it virtually. I hate it when people change two variables at the same time!
He noticed that the main power connection was really quite poor and we think that might have been causing a brownout whenever it started to draw current.
I'll leave it alone this week, as it's likely to be cold, and upgrade again if it's proved itself to be stable now.
Thanks!

[MrEd]

Unfortunately, no such luck. We're still getting intermittent behaviour on both units. Probably still some cleaning on the connectors, but at the moment I'm most concerned about being able to detect that it's gone wrong.

I've been watching the unit happily clicking in and out, and keeping the temperature as it should be after giving it a power off and power on at 8AM today. 

Now 15:30ish we're back to the state where I don't seem to be able to recover remotely:

Connecting to ws:<ipaddress>:80 and nothing else on the web front end. 

Telnet seems alive however. Is there a way to do a full reset this way? I've tried using debug -> ! to test the software watchdog, but the websocket never seems to recover. 

Nothing in the logging in telnet screams "I'm broken" to me, and it appears to be publishing to mqtt fine. According to MQTT it's E-00 OK but according to the virtual console (which works it seems), it's E-07 Comms fault. 

Rebooting via the virtual console seems to reboot according to the log via telnet, but doesn't cause the error state to be published by MQTT or reset the comms fault, the only thing that seems to bring it back to life is removing the three pin plug for a while.

This is all on v 3.5.2

After the reboot I did find:
Start Request!
Start denied - no heater data

in the telnet log. But according to the MQTT broker, it's all E-00 OK still.

[MrEd]

Right, we've given everything a good clean, shortened the 12 volt connections as we were seeing quite a voltage drop, and I'm seeing the situation where all seems to be going fine, then it just seems to stop responding and then goes to "Shutting down" then once that's finished, starts up again. The log seems pretty unhelpful:

JSON send: {"FanRPM":4290}

JSON send: {"FanVoltage":7.2}
JSON send: {"IP_STARSSI":-47}
JSON send: {"FanRPM":4270}
JSON send: {"PumpCount":3834.056,"SysTotalFuel":8128.319}
JSON send: {"TempBody":215,"FanVoltage":7.4}
JSON send: {"FanRPM":4290}
JSON send: {"FanVoltage":7.3}
JSON send: {"FanRPM":4300,"FanVoltage":7.4}
JSON send: {"IP_STARSSI":-46}
JSON send: {"PumpCount":3891.23,"SysTotalFuel":8129.577}
JSON send: {"FanRPM":4290,"FanVoltage":7.3}
JSON send: {"FanRPM":4300,"FanVoltage":7.4}
JSON send: {"TempCurrent":9.6,"Temp1Current":9.6}
JSON send: {"FanRPM":4290}
JSON send: {"PumpCount":3948.425,"SysTotalFuel":8130.835}
JSON send: {"FanRPM":4300}

And that was all she wrote.