2017-01-17 Server Crash Post Mortem

On 8:24pm EST, version 2017.01.17.2332 of Channels DVR was deployed. It was a bad build, and when started, it immediately segfaulted and crashed. This meant that it could not be started.

At 9:14pm EST @matthumble noticed the bad behavior. At 9:42pm EST, @tmm1 pulled the build from the update server.

Channels DVR automatically attempts to update every 2 hours. While the build was only out there for a total of 78 minutes, this put the bad build in a very large window for most of you. It is more than likely that your version of Channels DVR was updated to the bad build and Channels DVR is now stuck crashing.

What Happened?

A component of the software was updated, but its specific part was not completely compiled along with the rest of the software before being sent off for distribution by our updater.

How do I fix my installation?

Channels DVR auto updates when it starts up. So it’s just a matter of restarting your Channels DVR.

  1. Restart Channels DVR
    1a. macOS: The server should restart itself.
    1b. linux & NAS: reboot the computer or NAS.
  2. Once Channels DVR is back up and running, it will auto update itself. If it doesn’t, click “Check for Updates”.
  3. Wait for the page to refresh and ensure the current version is atleast: v2017.01.18.
  4. Re-enable DVR in Settings with the checkbox to turn DVR back on.
  5. Everything should be back to normal.

After re-enabling DVR, your guide data will re-download. It may take some time before your scheduled recordings show up.

If you need any help with this, please use the support thread created for this incident to get help from the community.

Will this happen again?

Since the incident, we’ve put things in place so that a version of Channels DVR can not be deployed and distributed unless it successfully launches and runs as it is supposed to.

Something this bad should not happen again.

Extra Support

We’ve closed comments on this post and have created a support thread for everyone to discuss and help each other with this issue. If you need help, do not know how to get command line access to your NAS, or can provide help for those that need it, this is the thread to post in.

Wrap Up

Thanks so much to everyone that helped with the issue and for being so patient with this problem. As you can see, Public Betas can be rocky. That’s why we’ve chosen to launch with a Public Beta before doing a full scale release.

We’re so sorry about this issue. Thanks!!

1 Like