Ubuntu Server poor performance ...fragmented drive

[Tex_B]

I'm just curious if anyone else running Ubuntu Server has experienced new performance issues in the last week or two?

I noticed the issue the day after my unattended upgrades, so it's high on my list of suspicion but hard for me to confirm. I'm running out of ideas, and probably going rebuild the server or try to restore from a 4 month old tar backup. This week we've had to rely on Hulu to watch recordings.

Issue:-
If I record from multiple sources at once, playback while recording freezes completely, and comskip takes 20 minutes or more per show. When I can get playback to work (usually after it is no longer recording anything), recordings during that time are scrambled and it freezes the app if I try to play through the bad quality. CPU Usage is hovering around 50% when only recording 1 show, detecting commercials on another show, and processing newly recorded files.

Some troubleshooting-
Speedtest shows the server still connecting to the network at gigabit speeds, and the 2 HDHomerun tuners are plugged into the same switch as the server. Antenna signal quality is 100%. I've force closed the client app and updated it, and disabled commercial skip. I rebooted the HDHomeRuns, router, switch, and server. Yesterday I updated the HDHomeRuns to the latest firmware, and I even switched to Channels DVR pre-release to see if any new updates addressed the issue. I looked at the server logs and see some errors around com skip when it takes a long time, and around when I reboot the server or HDHomeRuns, but thats about it.

Hardware background-
I've been running Channels DVR on my Ubuntu Mini PC for about 7 months now, and it has been rock solid. It has a 10th gen i5 8 core CPU with 8 GB of Ram, and a 4 TB external drive for recordings. When it was new I did some stress tests and it was obvious the hardware was overkill, able rip through anything I could throw at it- I recorded 10 streams (2 streaming, 8 OTA), while simultaneously playing back to my phone and transcoding playback to my web browser. CPU use for all that was only 30%

[tmm1]

Look at htop to see where the load is coming from

[Rice]

@Tex_B

I'm just curious if anyone else running Ubuntu Server has experienced new performance issues in the last week or two?

Just slammed mine for a test with 4 different sources, comskip, and regenerate. No playback freezes or issues that you describe.

[Tex_B]

Thanks for checking it, there is probably just something goofy going on with mine. I will stress it after work and see if I can find the task easting all the resources.

Do you know when you last installed OS updates?

[sdust]

You might benefit from installing netdata

Netdata - real-time system monitoring tool

tl;dr $ wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh $ sudo sh /tmp/netdata-kickstart.sh --disable-telemetry --disable-cloud The web interface should be available now on port 19999 on your channels DVR system http://<DVR IP>:19999 For a live demo:

I am guessing the disk might be an issue.

[Tex_B]

Okay so a little update.

I ran TOP and CPU usage is higher than normal, but whenever I'm experiencing the problem the CPU is never over 50%. I rebooted the server and did a stress test to try and replicate the issue, and it performed just fine initially- I was recording 5 shows at once, playing back another, and even did comskip, and it worked completely fine, and with low resource usage.

Once I stopped watching a show and stopped some recordings however, CPU usage spiked to nearly 50%, and using iotop I found kworker and the process for the external drive both at 100% IO wait, with very little writing to the drive. The drive seems to stall out.

This has me wondering if the 4TB drive is severely fragmented- I recently deleted about 1TB of shows, so fragmentation seems like big possibility. After checking some of the available Linux tools, I don't see a defrag tool for an external NTFS drive, which makes sense. But this has me wondering, is anyone else with a Linux server using a USB drive they just plugged in and started using? Or did everyone format their drive to ext4? I mean in hindsight it seems like a no brainer to use ext4, but I was so focused on just getting the thing to mount that I didn't even notice it was still NTFS. For now I'll plug the drive into my windows PC and see if I can defrag, but long term I may need to backup my data and format the drive as ext4. I'll report back if it is a fragmentation issue

[tmm1]

mount.ntfs indicates FUSEFS which is not good for performance

Newer kernels have a proper ntfs driver built in

[Tex_B]

Edit: It looks like Ubuntu 22.04 is using a newer kernel, but it doesn't include ntfs3 by default, according to first reddit post that came up in google. I'll keep digging.

---

How new are we talking? I downloaded Ubuntu 22.04 LTS last August when I set it up and have been keeping it updated. I'm still learning a lot about Linux servers, so I'm sure I'm overlooking obvious stuff. I'll search for more info on FUSEFS, and probably switch the drive to ext4.

Meanwhile-

[racameron]

@Tex_B

How new are we talking?

The NTFS3 driver contributed by Paragon was added in 5.15 (Sept 2021). Since then, it has had incremental improvements with each new kernel release.

[Tex_B]

Alright I'm back with an update, drive fragmentation appeared to be the issue. Thank you everyone for your assistance!

When I set up my Ubuntu server, my 5 TB usb drive didn't auto mount. So I followed instructions to update /etc/fstab and I did it using ntfs. I didn't realize I should or could use ntfs-3g in the fstab config.

Either way, the 5 TB drive at 75% fragmentation was probably quicker to copy all of my data off the drive and reformat it than it was to defrag. After learning about the benefits of ext4, I just formatted the drive as ext4 and moved on.

After I restored my data, everything is working great- performance seems even snappier than when it was new, and I couldn't be happier.

Moral of the story, if you've noticed some of your recordings are starting to get scrambled, don't forget to check your drive performance and fragmentation. And if you're using Linux, don't use the basic ntfs driver.