Extremely high load when recording

Jeffrey_Weitz · October 9, 2021, 2:03am

Hi.

Not sure what happed, but in the last couple of weeks my DVR keeps freezing up I think when there are two recordings or maybe when I get to 3.

It almost comes to a halt, and when I look at the status, it’s running at 20 to 30 load average. I can barely load the web UI and everything stutters if we are watching something.

I’ve tried disabling commercial detection since it would eventually get a lineup of recordings waiting to be detected and this really hasn’t fixed anything.

I even tried software transcoding and deleting a bunch of recordings and it’s still not working correctly.

We’ve never noticed this until maybe a week or two ago.

I’m not really sure what to do here, but does anyone have any ideas?

tmm1 · October 9, 2021, 2:07am

Can you screenshot htop showing cpu usage and processes?

Also click Support > Troubleshooting > Submit Diagnostics on your dvr web UI.

tmm1 · October 9, 2021, 2:09am

Have you rebooted?

Using the Pi image right? What kind of hard drive?

Possible it is something similar to what @adamandaj has experienced with his Pi setup.

Jeffrey_Weitz · October 9, 2021, 2:23am

Hi, thanks of the reply.

Yes, it's using the Pi image. I have tried rebooting a couple of times, as well as updating to a couple of different pre-release versions.

Everything works for great for a while after reboot or update, but after a couple of hours I'm back at high load once a couple of recordings get started.

Hard drive is:

Bus 002 Device 002: ID 1058:2626 Western Digital My Passport 2626

It's not terribly bad right now in the screenshot, only running at ~ 20 load avg.

Logs have been submitted as 8409cb82-67c3-4215-8b6f-4d6b4ecf1132 .

tmm1 · October 9, 2021, 2:41am

Hmmm very strange.

If you run iostat 1 and watch the %idle and %iowait, what kind of numbers do you see?

Jeffrey_Weitz · October 9, 2021, 2:59am

Mostly just a lot of disk writes.

https://pastebin.pl/view/5282e0a6

I see a lot of posts about using a powered USB hub for the hard drive. I don't have one but would get one to try that if that might be a help.

tmm1 · October 9, 2021, 3:05am

The high load averages are very confusing.

You could try updating to the new image which comes with v5.10 kernel (vs 4.9). Might help or not. To update you would click and hold the first Check for Update button in the OS section.

Jeffrey_Weitz · October 9, 2021, 3:06am

I'll give that a try and report back. thanks.

adamandaj · October 9, 2021, 3:24pm

100% the same issue, and it's happened now three times with various WD drives, about three months after being put in service. I'm not sure if the drives are just not designed for the load (though a few months shouldn't cause three different drives to fail) or some kind of trouble with exFAT.

I have another instance running with an SSD with much lower load, going strong for a few months with no problem.

I suspect it's exFAT and not the drive but my next try will just be with an SSD and then just copy stuff off every once in awhile.

One last thing: the powered hub did not help me.

ajp

tmm1 · October 9, 2021, 4:39pm

@adamandaj do you also see high load numbers when it happens?

I'm going to add some profiling tools to the next OS image so we can capture more data about what's being slow.

tmm1 · October 9, 2021, 4:42pm

Okay please try:

distro-upgrade 2021.1009.1631
systemctl reboot
perf top

adamandaj · October 9, 2021, 5:06pm

Yes, high load.

Will do it tomorrow -- not at home and Tailscale isn't persistent after a reboot. Hope we can track it down!

tmm1 · October 9, 2021, 5:20pm

How do you setup tailscale on the Pi? I'd like to make that easier via web UI

adamandaj · October 10, 2021, 3:16am

Docker package fastandfearless/tailscale seems to work. But it's not persistent with tailscale up at reboot and I haven't dug in to fix it.

The best use for it is downloading stuff remotely to iOS when traveling.

adamandaj · October 10, 2021, 3:18am

docker run -d --restart unless-stopped --name=tailscaled -v /var/lib:/var/lib -v /dev/net/tun:/dev/net/tun --network=host --privileged fastandfearless/tailscale tailscaled

docker exec tailscaled tailscale up

Jeffrey_Weitz · October 11, 2021, 11:04pm

Since I upgraded, I have not had any further issues with load, even while recording multiple shows. I'll keep monitoring and let you know if I see any more issues.

Jeffrey_Weitz · October 16, 2021, 2:14am

So, looks like the high load is back now. Seems to be after I had 3 recordings at once, is there an additional upgrade to grab additional info?