Channels DVR Buffering Has Returned

These messages are bad news and indicate either a network issue or that the disk you're using to record is failing.

You said the speedtest shows 100MB/s but I would expect much more for a gigabit wired network.

I think, if he's using the FireTV 4k with the amazon adapter, the max ethernet speed he'll get is 100MB/s. I actually removed the adapter on mine and was able to get faster speeds using my wifi.

thanks for the quick response..the buffer msg is from 1 of the tuners in the HDHR.

The drive is a brand new Seagate 2 TB USB 3.0 device. I have not detected any drive errors.I rechecked my network cabling. The cabling from the Cox modem, to the TP link router to the PC all is connected to the router itself. The only device out of that connectivity is the Fire TV and that e-net dongle connection. That connection is thru a gigabit Ethernet switch to the router..

I rewatched the program that we were watching last night when the buffering started. It played today without issue. Could the problem have something to do with my processor and a possible lack of CPU power?

We typically record and watch previously recorded programs at the same time.At times we may be recording more than 1 program at a time between the HDHR ans Playstation Vue. We also have Commercial Dedtection on, which I understand puts a significant amount of load on the cpu when it is doing it's work.

I have turned off the commercial detection in the server and will try to re create the problem. Any thoughts on the CPU horsepower?? Could we be over driving it with all the co incident record/playback activity?

Thanks for your feedback..

Bob

It's possible that it might be a CPU issue. You could check the Task Manager while recordings are happening to see how much the system is loaded.

Boogmeister1 is correct. The Amazon dongle for Firetv 4K is limited to 10/100 speeds (BUMMER). Because I have a 2nd Firetv upstairs in BR, my wifi is limited to N speeds instead of AC because of signal strength aka walls etc. May needd to upgrade to MESH Router in the future.

I will be testing and switching to Task Manager monitoring to see if CPU/GPU performance is the culprit.

Thanks

Bob

If you're having bandwidth issues with 100Mbps connections, I'm not sure how much of an improvement mesh wireless is going to be. Bandwidth and throughput of mesh networking is much reduced over straight wireless because of the need to maintain a connection to all of the nodes in a bidirectional manner.

I have no doubt they've improved over the last couple of years, they still aren't as good as marketing materials will lead you to believe. Mesh is best for expanding coverage to hard to reach places in an easy manner, not for achieving highspeed everywhere.

(Also, the routers used by mesh systems can introduce another layer of complexity that makes network troubleshooting much more difficult at times.)

racameron..thanks for the feedback on Mesh..I don't think I have a bandwidth problem in this instance. I've seen no frame transmit/receive errors or network errors of any kind in the log. I'm thinking this is more of an issue with the speed/capacity of the CPU/APU. Spec wise (2 cores) it appears to be enough, but, with the amount of co-incident recording and playback we do, it could be the issue.
We typically record at least 1 pgm from 1 of the 4 HDHR tuners and 1 pgm from Playstation Vue per hour in the evening from 7 to 10PM each night. Additionally, we could be watching a program previously recorded during that same period of time.
We have the Commercial Detection turned on and I understand that takes up a lot of CPU while it's doing it's thing at the end of a given recording and I have hardware transcoding selected so that could be a contributor too.
I run the playback from the PC to a HT receiver via HDMI to the TV. I now have a window open on the PC input of the receiver with Task Mgr running to see if the CPU is pegged during 1 of the buffering episodes if I can get the input switched fast enough to catch it.
We'll have to see what happens..

Bob

OK, so let's analyze possible bottlenecks here. The one that stands out to me immediately is the Seagate drive, USB 3.x or not and the fact that you're recording multiple streams while playing some. So you are accessing a minimum of 3 files simultaneously on that drive. While single continuous file transfers can hit 130MB/s sustained on modern drives, as soon as you drop a second transfer in there, it drops to half or less. A third makes it drop again. And that's for the most optimistic performance numbers for a spinning drive, in reality you're generally looking at least 10-20% lower max numbers on a populated drive, worse if it's been heavily used and the file system is fragmented. You're also not running the latest greatest hardware there, so my guess without further info is that those numbers I'm quoting are likely quite a bit above the peaks you're looking at in real life.

Finally, let's look at comm skip. That reads the file as fast as possible as it's a simple scan. That can actually cause write performance to drop significantly as the write requests get queued up behind a large block read request.

So that gives you the down and dirty on why you're suffering problems. Your post helped me figure out why I started having issues when we hooked up TV Everywhere just recently, so thanks for that.

As soon as I look into the configuration and figure out how I can reduce the hit, I'll post a followup, although I'm currently temporarily on a WD 4TB USB, I was actually looking to move that to a 2TB external SSD, which would solve those types of problems. (Random seeks are cheap on SSDs)

2lamas
Read your evaluation of my issue. Lots of good feedback. Thanks so much.

Last night while watching a previous recording, playback locked up..screen went black! jumped over to receiver and changed to PC to look at Task Mgr performance..."I" drive, where recordings are stored and played back from was 100% busy.This is the USB 3.0 drive discussed earlier.

I ordered a 1TB SSD drive this morning. Should be here tomorrow afternoon and will replace the "I" drive ASAP. Will cc everything off "I"and move to SSD.

I'll update as soon as things are completed and tested.

Thanks for the feedback..

Bob.

1 Like

1 TB SSD installed. Buffering has been significantly reduced to perhaps once an evening and that lasts for 1-2 seconds max. Task Mgr shows no more 100% activity on the new SSD when it happens. I looked at the processor and during 1 instance there was a spike in activity for a very short period on time. I think at this point I'm at the mercy of a 1 core APU/mobo that is 6 years old.

Thanks for the assistance and recommendations on how to fix this issue.
Now I have to figure out what is going to replace PS Vue since it is going away in Jan 2020.

Thanks
Bob

1 Like

I purchased a 4 TB WD for testing DVR. This is my first experience with Channels DVR. I installed on my Windows workstation, which is a beefy box.

[OS Windows Microsoft Windows 10 Pro 10.0.18363 Build 18363, CPU 12 cores / Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz, RAM 31.91 GB 53.0% free ]

We have the HDHR Quad, everything is wired 1G or higher (the workstation is 10G). We have a 1.2G Internet connection. Also recording from Vue which is a killer feature BTW.

The family jumped in and really loaded things up - right now there are 12 jobs running. Or at least trying to run. As the other members posted, I'm seeing 100% disk usage, I think this means my drive is also too slow. I'll move the DVR to another drive I have fairly soon. During operation my workstation is noticeably bogged. Channels DVR is using about 40% of the CPU. (That's a lot!)

The log is loaded with:
2019/11/30 17:10:57 [WRN] Buffer for 1078E069 ch4.1 is more than 50% full (clients=1, len=16796760)
2019/11/30 17:10:59 [WRN] Buffer for 1078E069 ch7.1 is more than 75% full (clients=1, len=25166424)
2019/11/30 17:11:01 [WRN] Buffer for 1078E069 ch4.1 is more than 75% full (clients=1, len=25169056)
2019/11/30 17:11:03 [WRN] Buffer for 1078E069 ch7.1 is more than 95% full (clients=1, len=31879340)

Ok so next steps:

  • Is there a diff btw Windows and Linux? I can boot into Linux and try again.
  • I'll move to a faster disk but I think that an array might work better. Any more feedback on what it takes to run 20 jobs and also stream out to 2 - 3 devices?

Looks like your problem is probably disk throughput related, not network related. Just from that brief snippet, it looks like you've got 4 concurrent operations going on your drive:

  1. Writing the stream from channel 4.1
  2. Reading the stream (to deliver to clients) from channel 4.1
  3. Writing the stream from channel 7.1
  4. Reading the stream (to deliver to clients) from channel 7.1

Also, since those are OTA channels, they are probably high bitrate MPEG-2 streams. If you have additional streams in progress, they are probably adding to your disk bottlenecks, too.

(As far as the difference between Windows and Linux, I doubt it. The DVR server is written in Go, which is pretty similar across platforms.)

Thanks for the feedback, after some more research and testing things are much improved.

First, the drive I picked up at Best Buy is:
Seagate - Barracuda 4TB Internal SATA Model: ST4000DMA05 (st4000dm004-2cv104) [5400 RPM]

I moved the Channels workload to a:
Seagate 3TB 7200RPM SATA 6.0Gb/s ST3000DM001 64MB

The replacement drive is working much better so far. Disk activity is hovering around 5-10% most of the time, although I have seen one instance of the buffer WRN messages.

The other drive hits 100% disk activity when reading and writing at the same time. It had some good reviews at Best Buy, but other sites like NewEgg have 100's of bad reviews for the ST4000DM004 - so far the drive is claimed to be SMR - I haven't been in the drive game since I went mostly SSD, but it seems like SMR would choke and in practice I did experience extremely bad performance from the drive.

Thanks everyone, this is a great community. The DVR was down at our house for a bit and I heard about it right away from my wife and kids. We love this product!

3 Likes

3.2 gig ram available is the problem...presumably 4 gig with some being used for integrated graphics....half is already being used just at idle. Add at least another 4 and you will be fine

So @proline, could this be my problem and why I am getting the buffering issues as well? I am showing 2.9GB RAM on my Shield.

OS
NVIDIA SHIELD Android TV
Android
(kernel: 4.9.140-tegra-g8e6798163e77)

CPU
4 cores / ARMv8 Processor rev 1 (v8l)

RAM
2.89 GB
17.1% free

Here are my logs from last night. These happened when I tried to record three shows at once that all started at the same time. (Last Man Standing on FOX, Super Store on NBC and Young Sheldon on CBS):

2020/01/16 14:59:00.023052 [DVR] Starting job 1579222740-51 Superstore on ch=[6000 1008]
2020/01/16 14:59:00.474570 [TNR] Opened connection to 131AEEA1/0 for ch1008 WGALDT
2020/01/16 14:59:00.474683 [DVR] Starting job 1579222740-52 Young Sheldon on ch=[1021]
2020/01/16 14:59:00.751892 [DVR] Recording for job 1579222740-51 from 131AEEA1 ch1008 into "TV/Superstore/Superstore S05E12 Myrtle 2020-01-16-1459.mpg" for 31m59.974182616s
2020/01/16 14:59:00.771859 [IDX] Generating video index for job 1579222740-51
2020/01/16 14:59:00.976673 [TNR] Opened connection to 131AEEA1/1 for ch1021 WHPDT
2020/01/16 14:59:00.976852 [DVR] Starting job 1579222740-31 Last Man Standing on ch=[1043]
2020/01/16 14:59:00.976920 [DVR] Waiting 59m59.02309397s until next job 1579226340-79 Dr. Pimple Popper
2020/01/16 14:59:01.018450 [DVR] Recording for job 1579222740-52 from 131AEEA1 ch1021 into "TV/Young Sheldon/Young Sheldon S03E12 Body Glitter and a Mall Safety Kit 2020-01-16-1459.mpg" for 32m59.525258397s
2020/01/16 14:59:01.040856 [IDX] Generating video index for job 1579222740-52
2020/01/16 14:59:01.474264 [TNR] Opened connection to 131AEEA1/2 for ch1043 WPMTDT
2020/01/16 14:59:01.509913 [DVR] Recording for job 1579222740-31 from 131AEEA1 ch1043 into "TV/Last Man Standing/Last Man Standing S08E05 The Office Mysterious Ways 2020-01-16-1459.mpg" for 1h1m59.023036002s
2020/01/16 14:59:01.533961 [IDX] Generating video index for job 1579222740-31
2020/01/16 14:59:14.498427 [WRN] Buffer for 131AEEA1 ch1008 is more than 50% full (clients=1, len=16777684)
2020/01/16 14:59:17.710220 [WRN] Buffer for 131AEEA1 ch1043 is more than 50% full (clients=1, len=16777684)
2020/01/16 14:59:18.792586 [WRN] Buffer for 131AEEA1 ch1008 is more than 75% full (clients=1, len=25165868)
2020/01/16 14:59:22.220248 [WRN] Buffer for 131AEEA1 ch1008 is more than 95% full (clients=1, len=31877468)
2020/01/16 14:59:22.879393 [WRN] Buffer for 131AEEA1 ch1008 is more than 99% full (clients=1, len=33219788)
2020/01/16 14:59:23.093154 [WRN] Buffer for 131AEEA1 ch1043 is more than 75% full (clients=1, len=25165868)
2020/01/16 14:59:27.379464 [WRN] Buffer for 131AEEA1 ch1043 is more than 95% full (clients=1, len=31877468)
2020/01/16 14:59:28.233618 [WRN] Buffer for 131AEEA1 ch1043 is more than 99% full (clients=1, len=33219788)
2020/01/16 14:59:28.882229 [WRN] Buffer for 131AEEA1 ch1021 is more than 50% full (clients=1, len=16777684)
2020/01/16 14:59:37.772992 [WRN] Buffer for 131AEEA1 ch1021 is more than 75% full (clients=1, len=25165868)
2020/01/16 14:59:39.558113 [WRN] Buffer for 131AEEA1 ch1021 is more than 50% full (clients=1, len=16777684)
2020/01/16 14:59:46.973929 [WRN] Buffer for 131AEEA1 ch1021 is more than 75% full (clients=1, len=25165868)
2020/01/16 14:59:52.657593 [WRN] Buffer for 131AEEA1 ch1021 is more than 95% full (clients=1, len=31877468)
2020/01/16 14:59:53.550155 [WRN] Buffer for 131AEEA1 ch1021 is more than 99% full (clients=1, len=33219788)....................................................................................................

(keeps repeating the above cycle until it hits the below error, then continues same as above constantly):
2020/01/16 15:12:39.754957 [M3U8] Generator: err=cannot create temp file: open /storage/A8E26A9EE26A710C/NVIDIA_SHIELD/Streaming/m3u8/2201/stream.m3u8001996029: no such file or directory

So would this be a Hard Drive issue, or a Host DVR Server (Shield) RAM issue? The devs here mentioned maybe HDD before, and I am thinking of replacing this setup with an NAS, but I don't want to spend the money if I don't have to. They mentioned it could possibly be because the external hard drive connected to my Shield being used for the DVR is using the Shield's internal 5V power through the USB data cable and not a separate power supply plugged into the external hard drive. That would be a MUCH cheaper fix than a new NAS and hard drive for it, plus much less hassle trying to walk my daughter through moving the DVR Server over to the NAS, since it is remote at here place, thousands of miles away!

That is HDD either full or not available. Check the HDD.

1 Like

OK thanks. So since I know it isn't full based on the available space % shown, then it sounds like it is indeed the HDD not available, as mentioned by @tmm1 and @eric previously, which is most likely because it needs its own power supply as they've suggested. This is a good thing since the solution appears much cheaper than replacing the whole system.

So you don't think the RAM amount has anything to do with this, as @proline mentioned above, in the quote below? I have even less RAM at 2.9GB, of which only 17.1% showed as available.

From looking at that log it looks to be filling ram that is your buffer % full warnings because it can not write the buffer to the HDD to free up ram that would be your can not create file error. I would definitely be looking to power that drive with an external power brick.

1 Like

Excellent, thanks!

There is a fixed size buffer (maximum 64MB) per stream. The buffering warnings you're seeing is because the drive cannot keep up with how much is being written to it.

1 Like