Problems with current prerelease

eric · July 15, 2021, 10:23pm

We've released a new prerelease build with a fix. Sorry about that.

mjcxp · July 15, 2021, 10:41pm

Recordings are still not ending or being cleared from the activity menu.

The log shows these errors:

2021/07/15 17:36:55.896395 [ERR] Recovered from panic: runtime error: invalid memory address or nil pointer dereference
github.com/fancybits/channels-server/mpegts.(*StreamIndexer).Index.func1()
:102
github.com/fancybits/channels-server/mpegts.(*FFprobeKeyframeIndexer).Index()
:131
github.com/fancybits/channels-server/mpegts.(*FFprobeKeyframeIndexer).indexPacket()
:167
github.com/fancybits/channels-server/mpegts.(*FFprobeKeyframeIndexer).addDuration()
:190

2021/07/15 17:38:00.961616 [WRN] Buffer for 13279F09 is more than 50% full (clients=1, len=33555368)

eric · July 15, 2021, 11:02pm

Very interesting. What kind of recording is this? TVE or HDHR?

mjcxp · July 15, 2021, 11:14pm

HDHomeRun Prime

Recordings never leave the activity section. The recordings are also glitched too and only record one or two seconds. Remote streaming also doesn’t work properly. It remuxes or transcodes at random bitrates 200kbps - 6mbps even when I have original set as the internet streaming quality. I also have adaptive streaming turned off. The previous releases have all worked perfectly.

nriley · July 15, 2021, 11:24pm

5 TVE recordings stuck today. HDHR recordings seem to be completing (one is in progress now).

Logs submitted — ac5861dc-7fab-4d5d-a6d4-8dc6bfc06f33.

I see an hour ago that you said there’s a new prerelease posted but when I check for updates I don’t see it?

eric · July 15, 2021, 11:47pm

DVR prerelease v2021.07.15.2328 has just been released.

It has:

The fix for the "Recovered from panic" error in the indexer
A fix for all future panics to prevent the buffer full issue that corrupted recordings

This should prevent these sorts of bugs from impacting recordings in the future.

Thank you all for the reports.

chDVRuser · July 16, 2021, 1:50am

Just wanted to say Thanks for reporting the version#.

Many posters say they're running the "latest version", but don't say what version.
Most of the time it's not the latest pre-release which was what was meant by a post.
Add to that the confusion between DVR release and pre-release versions.

You hopefully saved my midnight and early morning recordings from disaster.

Will keep you informed

eric · July 16, 2021, 9:48pm

5 posts were split to a new topic: Potential issue in latest prerelease with audio remuxing

koltasi · July 16, 2021, 2:53am

My ATSC 3.0 channels remuxes this as well on iPadOS and FireTV on away from home. Not sure about Apple. When it remuxes, I lose audio. Watching the recording plays audio, but not live.

hancox · July 16, 2021, 10:59am

Got a newer/different kernel panic when the DVR tried to upgrade to the even-later prelease late last night:

2021/07/15 23:15:53.233404 [SYS] Shutting down for upgrade from v2021.07.15.2328 to v2021.07.16.0117
2021/07/15 23:15:53.235263 [SYS] Bonjour service stopped.
2021/07/15 23:15:53.241953 [DVR] Recording engine stopped.
2021/07/15 23:15:53.261904 [ERR] Recovered from panic: runtime error: invalid memory address or nil pointer dereference
main.StartUpdater.func1()
:55
main.performRestart()
:230
main.Restart()
:238
main.StopDVR()
:93
main.RefreshVirtual()
:453
github.com/fancybits/channels-server/dvr.(*virtualProvider).SetRecorder()
:61

Hard rebooting the pi brought everything back. 57f48e5c-1fce-4204-a19a-d4766d4da36a submitted

mtnbkr60 · July 16, 2021, 11:34am

I ma running 2021.07.16.0117 and I also have shows stuck in record, and commercial detection failing on those even though they are not actually recording.

2021/07/15 20:00:02.344517 [ERR] Recovered from panic: runtime error: invalid memory address or nil pointer dereference
github.com/fancybits/channels-server/mpegts.(*StreamIndexer).Index.func1()
:102
github.com/fancybits/channels-server/mpegts.(*FFprobeKeyframeIndexer).Index()
:129
github.com/fancybits/channels-server/mpegts.(*FFprobeKeyframeIndexer).indexPacket()
:147

DHollenback · July 16, 2021, 12:14pm

How do you downgrade to the stable version? I'm on Pi4 so may be different than your Mac Mini. I update to the pre-release often but would like to know how to get to a stable release if one of the Pre-release versions causes issues.

Macnbaish · July 16, 2021, 12:18pm

I can confirm this happened to me as well last night. Had to stop the package via Synology UI and restart it to get Channels running again. I also submitted diagnostics after.

hancox · July 16, 2021, 12:29pm

Noticed a similarly timed prerelease cut - any chance this is just an inadvertent regression?

hancox · July 16, 2021, 12:49pm

I'll let @tmm1 confirm my logic, so YMMV on trying this, but believe the steps would be (prereq of having ssh set up on your pi, assuming you have the dedicated version)

I also used an earlier prerelease that is known working, not stable. I'm assuming there may be a flag to pull whatever stable is, but i don't know it off hand.

Back up database
run:
distro-upgrade 2021.07.14.1719
systemctl reboot
Inspect and restore db, if need be (would think this is rare, but there would be occasions where prereleases may make restoring backups difficult). You may need an older restore point too, depending on how far back you're going on builds.

chDVRuser · July 16, 2021, 5:00pm

Got this panic updating to the new prerelease this morning on all 3 servers.

2021/07/16 09:51:46.380526 [SYS] Downloading new version v2021.07.16.0117
2021/07/16 09:51:51.306414 [SYS] Update downloaded and verified successfully.
2021/07/16 09:51:51.306448 [SYS] Shutting down for upgrade from v2021.07.15.2328 to v2021.07.16.0117
2021/07/16 09:51:51.309901 [DVR] Recording engine stopped.
2021/07/16 09:51:51.310416 [ERR] Recovered from panic: runtime error: invalid memory address or nil pointer dereference
main.StartUpdater.func1()
    :65
main.performRestart()
    :230
main.Restart()
    :238
main.StopDVR()
    :93
main.RefreshVirtual()
    :453
github.com/fancybits/channels-server/dvr.(*virtualProvider).SetRecorder()
    :61

Logs have been submitted as a55c8f2f-2731-404e-a3c0-6e434d24e1ef

Here's the log from submitting diagnostics

2021/07/16 09:59:33.533765 [TRS] DNS: OK: Resolved host via system DNS
2021/07/16 09:59:34.353953 [TRS] Cloud Reachable: OK: Connected to community.getchannels.com
2021/07/16 09:59:34.447974 [TRS] Local Time: OK: Local clock is accurate
2021/07/16 09:59:34.481480 [TRS] TV Everywhere: OK
2021/07/16 09:59:34.481550 [TRS] Remote Access: OK: Remote access not configured
2021/07/16 09:59:34.481563 [TRS] Disk Permissions: OK
2021/07/16 09:59:36.804674 [DIAG] Could not package /dvr/rules: 404 Not Found
2021/07/16 09:59:36.806180 [DIAG] Could not package /dvr/jobs: 404 Not Found
2021/07/16 09:59:36.807834 [DIAG] Could not package /dvr/groups: 404 Not Found
2021/07/16 09:59:36.809382 [DIAG] Could not package /dvr/files?all=true: 404 Not Found
2021/07/16 09:59:36.810949 [DIAG] Could not package /dvr/programs: 404 Not Found
2021/07/16 09:59:36.812389 [DIAG] Could not package /dvr/queues/items: 404 Not Found


2021/07/16 09:59:36 [Recovery] 2021/07/16 - 09:59:36 panic recovered:
runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:212 (0x43747a)
runtime/signal_unix.go:734 (0x451012)
github.com/fancybits/channels-server/dvr/db_files.go:200 (0x13bb005)
github.com/fancybits/channels-server/http_dvr_scanner.go:72 (0x14f764c)
github.com/gin-gonic/[email protected]/context.go:165 (0x14a3f7b)
github.com/fancybits/channels-server/http.go:234 (0x14a3f5a)
github.com/gin-gonic/[email protected]/context.go:165 (0x14a3ea7)
github.com/fancybits/channels-server/http.go:214 (0x14a3e89)
github.com/gin-gonic/[email protected]/context.go:165 (0x14a2fdb)
github.com/fancybits/channels-server/http.go:206 (0x14a2fba)
github.com/gin-gonic/[email protected]/context.go:165 (0xa960d9)
github.com/gin-gonic/[email protected]/recovery.go:106 (0xa960c0)
github.com/gin-gonic/[email protected]/context.go:165 (0xa95153)
github.com/gin-gonic/[email protected]/logger.go:241 (0xa95112)
github.com/gin-gonic/[email protected]/context.go:165 (0x144e115)
github.com/gin-contrib/[email protected]/sessions.go:65 (0x144e0fd)
github.com/gin-gonic/[email protected]/context.go:165 (0x1446ebd)
github.com/gin-contrib/[email protected]/gzip.go:47 (0x1446ea5)
github.com/gin-gonic/[email protected]/context.go:165 (0x14c39e7)
github.com/fancybits/channels-server/http.go:275 (0x14c39ce)
github.com/gin-gonic/[email protected]/context.go:165 (0xa8be04)
github.com/gin-gonic/[email protected]/gin.go:420 (0xa8bdea)
github.com/gin-gonic/[email protected]/gin.go:376 (0xa8b8db)
net/http/server.go:2887 (0x720622)
net/http/server.go:1952 (0x71ba4c)
runtime/asm_amd64.s:1371 (0x473da0)

2021/07/16 09:59:36.814416 [DIAG] Could not package /dvr/scanner/imports/index: 500 Internal Server Error
2021/07/16 09:59:36.815941 [DIAG] Could not package /dvr/recordings/upnext: 404 Not Found
2021/07/16 09:59:36.817479 [DIAG] Could not package /dvr/guide/channels: 404 Not Found
2021/07/16 09:59:36.846190 [DIAG] Could not package /providers/playon/library: 404 Not Found
2021/07/16 09:59:36.949477 [DIAG] Could not find file: channels-comskip-controller.log: open channels-comskip-controller.log: no such file or directory
2021/07/16 09:59:43.149603 New diagnostic log submitted under a55c8f2f-2731-404e-a3c0-6e434d24e1ef

All 3 are back up and running after restarting their docker containers.

tmm1 · July 16, 2021, 5:34pm

The 2021.07.16.xxxx builds contain fixes for all the panics shown above.

The latest build is 2021.07.16.0117 and is safe for everyone to upgrade to. We're looking into the possible bug around remote streaming quality.

The recording bug was introduced in 2021.07.15.0027 and fixed in 2021.07.15.2058

The crash when upgrading was introduced in 2021.07.15.2058 and fixed in 2021.07.16.0046

If you're on 2021.07.15.2058 or 2021.07.15.2328, the prerelease upgrade will fail and require you to restart the software manually.

chDVRuser · July 16, 2021, 5:47pm

Thanks for the update.
Saw a strange error in the log from a recording this morning and not sure if you're aware of it.
Was running version 2021.07.15.2328

2021/07/16 06:30:30.003405 [ERR] Error during stream 1323AADB/0 ch751 APLHDP: read tcp 172.18.0.2:37518->192.168.1.7:5004: read: connection reset by peer

This was the time the full recording ended

2021/07/16 06:00:00.013747 [DVR] Starting job 1626440400-382 Impossible Planet on ch=[751]
2021/07/16 06:00:00.478967 [TNR] Opened connection to 1323AADB/0 for ch751 APLHDP
2021/07/16 06:00:00.479341 [DVR] Recording for job 1626440400-382 from 1323AADB ch751 into "TV/Impossible Planet/Impossible Planet S01E07 Extraordinary Castles and Creatures 2021-07-16-0600.mpg" for 30m29.98613059s
2021/07/16 06:00:00.603084 [IDX] Generating video index for job 1626440400-382
2021/07/16 06:30:30.003405 [ERR] Error during stream 1323AADB/0 ch751 APLHDP: read tcp 172.18.0.2:37518->192.168.1.7:5004: read: connection reset by peer
2021/07/16 06:30:30.004649 [TNR] Closed connection to 1323AADB/0 for ch751 APLHDP
2021/07/16 06:30:30.082762 [SNR] Statistics for "TV/Impossible Planet/Impossible Planet S01E07 Extraordinary Castles and Creatures 2021-07-16-0600.mpg": ss=90%-91% snq=99%,98%-100% seq=100% bps=3740492,733952-6656704 pps=335,43-583
2021/07/16 06:30:30.086492 [DVR] Finished job 1626440400-382 Impossible Planet

Logs from the HDHR appear normal

20210716-12:59:59 Tuner: tuner0 tuning 751 APLHDP (auto:279MHz-1)
20210716-13:00:00 Tuner: tuner0 streaming http to 192.168.1.4:37518
20210716-13:00:00 Tuner: tuner0 lockkey forced release by 192.168.1.4
20210716-13:00:00 CableCARD: tuner0 751 APLHDP (auto:279MHz-1) access = subscribed
20210716-13:30:29 Tuner: tuner0 http stream ended (stop request)

hancox · July 16, 2021, 6:02pm

Bingo. Thanks. Just successfully finished first TVE recording on
2021.07.16.0117

tmm1 · July 16, 2021, 10:01pm

Connection reset is normal. That message will be omitted in the new prerelease.