DVR server degradation during guide download

Not sure when this issue started, but I first noticed it yesterday. During my daily guide download/update process, my DVR server is being pushed to the limit memory wise (as far as I can tell), which is causing the whole DVR system to grind almost to a halt. DVR is running on a DS218+, and has been for at least 2 years, and I’ve never seen this issue before.

I still have the default 2GB memory installed that came with the box, and that has always been plenty. This morning again I noticed during the guide data download/update, my available server memory plummeted, and if I closed out of my iOS client (running on iPhone 12 Pro), and tried to reconnect, the app told me it couldn’t find my DVR server. If I try to access the DVR server settings web admin page directly during this process, it does load, eventually, but at a magnitude of several orders slower than when the guide update is not running.

I have attached some photos showing server memory and CPU usage. Any suggestions would be welcomed. Let me know if more info is needed. Thanks!

Can you sort by memory? The DVR only says 600mb so what is using the other 1GB of RAM?

I did try sorting it, and it doesn’t look like it is an option to do that. When the guide update process finally finished, I noticed my overall memory usage was still very high compared to what it normally is when the system is idle.

I did a full reboot of the DS218+, and now my memory usage is back to normal, only about 22-25% at idle, and the DVR system as a whole is back to being very responsive again.

I’m going to guess there was something else not releasing memory perhaps? I’ll keep an eye on it and see what happens over the next few days, and will post back if it continues to be an issue.

Thanks for the quick response!

I had a similar issue recently where my ds218+ got very bogged down. Load averages were in the double digits, if I remember right. I have never had that happen before. I think Channels had two recordings going, a comskip, and one client playing an original quality recording at the time. So, a decent load. I looked at the resource monitor and found that the disk(s) were struggling to keep up. They are mirrored WD Reds. There was also just a bit of memory paging happening. The recordings made at this time were messed up, with portions missing and pixelation, etc. Anyway, I rebooted the NAS the next morning and it’s been fine since. Possibly not related at all but I thought I would mention it as that’s the first time I’ve ever seen it struggle with the load.

I normally see 60-70MB memory use when Channels DVR is idle on my two Synology NAS's (DS1513+ & DS1019+). It will peak over 200MB during guide updates along with CPU usage peaking 50-80%.
Once the guide update is complete, the CPU goes back to normal and the memory use slowly drops.

I can sort by memory use in Task Manager

Same as @Macnbaish, on the occasions I have seen degraded DVR services, it is almost always due to io wait. Which can be most easily tracked using top:

When that number starts climbing, there are problems. It doesn't have to be high-bandwidth transfers either. If you have an application running that does constant small database updates running concurrently, it going to increase this number.

I've got surveillance station on this box, but don't run the live view constantly in high bitrate streams. Only if I am watching for something (like a car to leave the space I want to park in).

Also, UniFi, DNS, HomeBridge, Time Machine, and similar small services that could be run on the Synology are offloaded to run on a Raspberry Pi instead. Synology indexing service is disabled. And, since it is a btrfs volume, which does copy-on-write and leads to fragmented files, I run a scheduled defragmentation on some parts of the drive on Saturday mornings.
My task scheduler script for that looks like this: "/sbin/btrfs filesystem defragment -r /volume1 /volume1/DVR". Don't use snapshot replication on your DVR directory.

I was using the iOS Synology app at the time to get those screen captures, but it would make sense that the actual web interface allows sorting of the columns, thanks for that info!

The only other item of consequence I run on the Synology box is a Plex server, but I am the only user of it, and a light user at that. When I first set it up, I turned off every other application that I would not be using, just to minimize the impact on DVR and Plex operations.

I probably should have just started with a reboot as the first troubleshooting step, derp. I have been spoiled by so few issues with both the Synology box itself and Channels that I didn't think of the easiest solution first.

Real quick, can someone explain what the numbers mean for "load averages" on the DVR web admin page? I never understood how to read those.

Thanks for the help from everyone! :slight_smile:

Those are the system's load averages over the last 1, 5, and 15 minute time periods.

Thanks. What units are those numbers in? CPU GHz, or something else?

its the number of tasks in queue for a processor core. So if you have 4 cores, a number below 4 is a good load average. If above 4, you have stuff that is waiting to be processed. Similarly, with a 2-core processor, your ideal numbers would be below 2.

0–N, where N is the number of processors/cores your system has. So if your system has 4 cores, then those averages can be anything from 0 (no load) to 4 (100% load on all 4 cores). So a value of 0.3 for a 4–core system means your system average load was 7.5% (0.3 / 4 = 7.5).

1 Like

Load average includes the io wait that I mentioned above. So your CPU might not actually be working very hard, except that it is waiting for the disk (which will slow down everything).

Ahh, makes more sense now, thanks everyone!

http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html

3 Likes

Wow, that's extraordinary perseverance!
Thanks for sharing that great detective story.

1 Like