Just Tried what you asked me and its acting similar.
on the ON Now the lower I went in stream quality it loaded a bit faster but when it did it would load momentarily and then return to buffer.
on the Recorded shows, it would just hang, didnt load anything. here is the logs while I was attempting different streams.
2017/10/18 00:11:28 [HLS] Starting transcoder for file-11 at 8s (encoder=libx264, resolution=, deinterlacer=linear, bitrate=10000)
2017/10/18 00:11:40 [HLS] Stopping transcoder for file-11 after seek to 0s (out=9.151167s, finished=false)
2017/10/18 00:11:40 [HLS] Starting transcoder for file-11 at 0s (encoder=libx264, resolution=, deinterlacer=linear, bitrate=10000)
Error while decoding stream #0:0: Invalid data found when processing input
2017/10/18 00:11:58 [HLS] Stopping transcoder for file-11 after seek to 43s (out=2.252267s, finished=false)
2017/10/18 00:11:58 [HLS] Starting transcoder for file-11 at 43s (encoder=libx264, resolution=, deinterlacer=linear, bitrate=10000)
Error while decoding stream #0:0: Invalid data found when processing input
2017/10/18 00:13:06 [HLS] Stopping transcoder session file-11 @ 49.489833s
2017/10/18 00:13:11 [HLS] Starting transcoder for file-1 at 57m13s (encoder=libx264, resolution=, deinterlacer=linear, bitrate=10000)
[mpegts @ 0x29d7480] start time for stream 3 is not set in estimate_timings_from_pts
[mpegts @ 0x29d7480] start time for stream 4 is not set in estimate_timings_from_pts
[mpegts @ 0x29d7480] Dropped corrupted packet (stream = 1)
[mpegts @ 0x29d7480] Dropped corrupted packet (stream = 2)
[mpegts @ 0x29d7480] Dropped corrupted packet (stream = 1)
Last message repeated 1 times
[ac3 @ 0x29fe4c0] frame sync error
Error while decoding stream #0:1: Invalid data found when processing input
2017/10/18 00:13:23 [HLS] Stopping transcoder for file-1 after seek to 56m58s (out=57m14.451467s, finished=false)
2017/10/18 00:13:23 [HLS] Starting transcoder for file-1 at 56m58s (encoder=libx264, resolution=, deinterlacer=linear, bitrate=10000)
[mpegts @ 0x1ee8480] start time for stream 3 is not set in estimate_timings_from_pts
[mpegts @ 0x1ee8480] start time for stream 4 is not set in estimate_timings_from_pts
[mpegts @ 0x1ee8480] Dropped corrupted packet (stream = 1)
[mpegts @ 0x1ee8480] Dropped corrupted packet (stream = 2)
[mpegts @ 0x1ee8480] Dropped corrupted packet (stream = 1)
Last message repeated 1 times
[ac3 @ 0x1f4b9c0] frame sync error
Error while decoding stream #0:1: Invalid data found when processing input
2017/10/18 00:13:47 [HLS] Stopping transcoder for file-1 after seek to 11s (out=57m0.469144s, finished=false)
2017/10/18 00:13:47 [HLS] Starting transcoder for file-1 at 11s (encoder=libx264, resolution=, deinterlacer=linear, bitrate=10000)
[mpegts @ 0x3653480] start time for stream 3 is not set in estimate_timings_from_pts
[mpegts @ 0x3653480] start time for stream 4 is not set in estimate_timings_from_pts
[mpegts @ 0x3653480] Dropped corrupted packet (stream = 1)
[mpegts @ 0x3653480] Dropped corrupted packet (stream = 2)
[mpegts @ 0x3653480] Dropped corrupted packet (stream = 1)
Last message repeated 1 times
[ac3 @ 0x36b6b00] frame sync error
Error while decoding stream #0:1: Invalid data found when processing input
2017/10/18 00:13:55 [HLS] Stopping transcoder session file-1 @ 11.867544s
2017/10/18 00:15:53 [TNR] Opened connection to 1311071A for ch501
2017/10/18 00:15:53 [HLS] Starting transcoder for channel 501 (encoder=libx264, resolution=, deinterlacer=linear, bitrate=10000)
2017/10/18 00:16:05 [WRN] Buffer for 1311071A ch501 is more than 50% full (clients=1, len=16793476)
2017/10/18 00:16:10 [WRN] Buffer for 1311071A ch501 is more than 75% full (clients=1, len=25165868)
2017/10/18 00:16:14 [WRN] Buffer for 1311071A ch501 is more than 95% full (clients=1, len=31877468)
2017/10/18 00:16:15 [WRN] Buffer for 1311071A ch501 is more than 99% full (clients=1, len=33219788)
2017/10/18 00:16:21 [HLS] Stopping transcoder session ANY-ch501 @ 1.968633s
2017/10/18 00:16:21 [TNR] Closed connection to 1311071A for ch501
[hls @ 0x306efa0] failed to rename file /volume1/video/Channels DVR/Streaming/ANY-ch501/stream2.ts.tmp to /volume1/video/Channels DVR/Streaming/ANY-ch501/stream2.ts
av_interleaved_write_frame(): No such file or directory
2017/10/18 00:16:34 [TNR] Opened connection to 1311071A for ch501
2017/10/18 00:16:34 [HLS] Starting transcoder for channel 501 (encoder=libx264, resolution=1080, deinterlacer=linear, bitrate=8000)
2017/10/18 00:16:46 [WRN] Buffer for 1311071A ch501 is more than 50% full (clients=1, len=16777684)
2017/10/18 00:16:51 [WRN] Buffer for 1311071A ch501 is more than 75% full (clients=1, len=25165868)
2017/10/18 00:16:55 [HLS] Stopping transcoder session ANY-ch501 @ 1.9019s
2017/10/18 00:16:55 [TNR] Closed connection to 1311071A for ch501
[hls @ 0x1d30b60] failed to rename file /volume1/video/Channels DVR/Streaming/ANY-ch501/stream2.ts.tmp to /volume1/video/Channels DVR/Streaming/ANY-ch501/stream2.ts
av_interleaved_write_frame(): No such file or directory
2017/10/18 00:17:11 [TNR] Opened connection to 1311071A for ch501
2017/10/18 00:17:11 [HLS] Starting transcoder for channel 501 (encoder=libx264, resolution=720, deinterlacer=linear, bitrate=6000)
2017/10/18 00:17:22 [WRN] Buffer for 1311071A ch501 is more than 50% full (clients=1, len=16777684)
2017/10/18 00:17:28 [WRN] Buffer for 1311071A ch501 is more than 75% full (clients=1, len=25165868)
2017/10/18 00:17:32 [WRN] Buffer for 1311071A ch501 is more than 95% full (clients=1, len=31877468)
2017/10/18 00:17:33 [WRN] Buffer for 1311071A ch501 is more than 99% full (clients=1, len=33219788)
2017/10/18 00:17:40 [HLS] Stopping transcoder session ANY-ch501 @ 4.2042s
2017/10/18 00:17:40 [TNR] Closed connection to 1311071A for ch501
[mpegts @ 0x33e33e0] Dropped corrupted packet (stream = 1)
[mpeg2video @ 0x33eab00] invalid cbp -1 at 17 40
[mpeg2video @ 0x33eab00] Warning MVs not available
[hls @ 0x34e1ec0] failed to rename file /volume1/video/Channels DVR/Streaming/ANY-ch501/stream4.ts.tmp to /volume1/video/Channels DVR/Streaming/ANY-ch501/stream4.ts
Error writing trailer of /volume1/video/Channels DVR/Streaming/ANY-ch501/stream.m3u8: No such file or directory
2017/10/18 00:17:52 [TNR] Opened connection to 1311071A for ch501
2017/10/18 00:17:52 [HLS] Starting transcoder for channel 501 (encoder=libx264, resolution=720, deinterlacer=linear, bitrate=4000)
2017/10/18 00:18:04 [WRN] Buffer for 1311071A ch501 is more than 50% full (clients=1, len=16777684)
2017/10/18 00:18:09 [WRN] Buffer for 1311071A ch501 is more than 75% full (clients=1, len=25165868)
2017/10/18 00:18:14 [WRN] Buffer for 1311071A ch501 is more than 95% full (clients=1, len=31894576)
2017/10/18 00:18:15 [WRN] Buffer for 1311071A ch501 is more than 99% full (clients=1, len=33219788)
2017/10/18 00:18:29 [HLS] Stopping transcoder session ANY-ch501 @ 6.139467s
2017/10/18 00:18:29 [TNR] Closed connection to 1311071A for ch501
[hls @ 0x2d23460] failed to rename file /volume1/video/Channels DVR/Streaming/ANY-ch501/stream6.ts.tmp to /volume1/video/Channels DVR/Streaming/ANY-ch501/stream6.ts
av_interleaved_write_frame(): No such file or directory
2017/10/18 00:18:42 [TNR] Opened connection to 1311071A for ch501
2017/10/18 00:18:42 [HLS] Starting transcoder for channel 501 (encoder=libx264, resolution=720, deinterlacer=linear, bitrate=2000)
2017/10/18 00:18:54 [WRN] Buffer for 1311071A ch501 is more than 50% full (clients=1, len=16777684)
2017/10/18 00:18:59 [WRN] Buffer for 1311071A ch501 is more than 75% full (clients=1, len=25165868)
2017/10/18 00:19:04 [WRN] Buffer for 1311071A ch501 is more than 95% full (clients=1, len=31877468)
2017/10/18 00:19:05 [WRN] Buffer for 1311071A ch501 is more than 99% full (clients=1, len=33219788)
2017/10/18 00:19:50 [HLS] Stopping transcoder session ANY-ch501 @ 12.479133s
2017/10/18 00:19:50 [TNR] Closed connection to 1311071A for ch501
[hls @ 0x3786520] failed to rename file /volume1/video/Channels DVR/Streaming/ANY-ch501/stream12.ts.tmp to /volume1/video/Channels DVR/Streaming/ANY-ch501/stream12.ts
av_interleaved_write_frame(): No such file or directory
2017/10/18 00:20:04 [TNR] Opened connection to 1311071A for ch501
2017/10/18 00:20:04 [HLS] Starting transcoder for channel 501 (encoder=libx264, resolution=576, deinterlacer=linear, bitrate=3000)
2017/10/18 00:20:16 [WRN] Buffer for 1311071A ch501 is more than 50% full (clients=1, len=16777684)
2017/10/18 00:20:21 [WRN] Buffer for 1311071A ch501 is more than 75% full (clients=1, len=25165868)
2017/10/18 00:20:29 [WRN] Buffer for 1311071A ch501 is more than 95% full (clients=1, len=31877468)
2017/10/18 00:20:30 [WRN] Buffer for 1311071A ch501 is more than 99% full (clients=1, len=33219788)
2017/10/18 00:20:54 [HLS] Stopping transcoder session ANY-ch501 @ 11.4114s
2017/10/18 00:20:54 [TNR] Closed connection to 1311071A for ch501
[hls @ 0x1a61d00] failed to rename file /volume1/video/Channels DVR/Streaming/ANY-ch501/stream11.ts.tmp to /volume1/video/Channels DVR/Streaming/ANY-ch501/stream11.ts
av_interleaved_write_frame(): No such file or directory
2017/10/18 00:21:11 [TNR] Opened connection to 1311071A for ch501
2017/10/18 00:21:11 [HLS] Starting transcoder for channel 501 (encoder=libx264, resolution=240, deinterlacer=linear, bitrate=360)
2017/10/18 00:21:25 [WRN] Buffer for 1311071A ch501 is more than 50% full (clients=1, len=16785580)
2017/10/18 00:21:29 [WRN] Buffer for 1311071A ch501 is more than 50% full (clients=1, len=16785580)
2017/10/18 00:21:35 [WRN] Buffer for 1311071A ch501 is more than 75% full (clients=1, len=25168500)
2017/10/18 00:21:39 [WRN] Buffer for 1311071A ch501 is more than 95% full (clients=1, len=31877468)
2017/10/18 00:21:40 [WRN] Buffer for 1311071A ch501 is more than 99% full (clients=1, len=33219788)
2017/10/18 00:22:15 [WRN] Buffer for 1311071A ch501 is more than 50% full (clients=1, len=16797304)
2017/10/18 00:22:19 [WRN] Buffer for 1311071A ch501 is more than 75% full (clients=1, len=25166804)
[mpegts @ 0x1a1c3e0] Dropped corrupted packet (stream = 0)
[mpegts @ 0x1a1c3e0] Dropped corrupted packet (stream = 1)
2017/10/18 00:22:22 [HLS] Stopping transcoder session ANY-ch501 @ 35.2352s
2017/10/18 00:22:22 [TNR] Closed connection to 1311071A for ch501
[hls @ 0x1ab59c0] failed to rename file /volume1/video/Channels DVR/Streaming/ANY-ch501/stream35.ts.tmp to /volume1/video/Channels DVR/Streaming/ANY-ch501/stream35.ts
av_interleaved_write_frame(): No such file or directory