Linux/Mac script for transcoding and adding to Plex

Playground Hacks
#1

I run both Channels DVR and Plex. I've found it convenient to automatically transcode to an h.264 format under Plex for streaming that avoids the need for live transcoding, which requires a more powerful processing. I figured it might be appreciated to share my work with you. This is current housed at GitHub, here:

https://github.com/karllmitchell/Channels-DVR-to-Plex

together with any documentation that used to be on this post.

Please note that this has not been thoroughly tested on all systems, and that it is to be used at your own risk. It has been confirmed to work on Ubuntu 16.04 Xenial (arm64) and Mac OS 10.12 Sierra (intel), after installing the necessary open-source pre-requisites. It should work on any reasonably modern unix-like system, including Linux.

This is sub-beta quality right now! I will do what I can to address any bugs, but my free time is erratic some please forgive slow responses. Feel free to take it and make it your own, or to contribute directly to the code base.

Features include:

  1. An installation script, for ease of use.
  2. Can be run on the same machine as Channels DVR, or remotely on a different machine.
  3. Command-line or daemon operation (mac-friendly LaunchAgent provided).
  4. Fully configurable via command-line arguments or editing the script header.
  5. Uses ffmpeg for transcoding, and so little need to install extra software.
  6. Primary code is portable bash script.
  7. File renaming as per Plex conventions, and direct deposit into Plex file structure.
  8. Optional phone notifications via IFTTT (iPhone or Android at least).
  9. Optional (actually, preferred) execution via GNU Parallel: send the transcoding job to another machine.
  10. iTunes-style tagging, with enhanced tagging via AtomicParsley.

Default settings are produce efficient, fast, flexible, high quality outputs compatible with most modern devices, but tuned for excellent reproduction on Apple TV 4, iPhone 4+, iPad 3+. Transcodes are also backwards compatible with Apple TV 3, and can be added to iTunes directly if desired.

Code and documentation can be found on the github repository, or you can install from a terminal by following the instructions below.

Pre-requisites and installation

You'll need some tools installed on your system. For Mac users, install MacPorts (http://www.macports.org/) or HomeBrew (http://brew.sh/) then run one of these two commands:

sudo port selfupdate; sudo port upgrade outdated; sudo port install curl jq wget

brew install curl jq wget

For Linux users, similar will work, but you should add realpath too if it doesn't exist, e.g.
sudo apt-get update; sudo apt-get upgrade; sudo apt-get install curl jq realpath wget for Debian or Ubuntu.

Once this is done, you should be able to install by running:

curl https://raw.githubusercontent.com/karllmitchell/Channels-DVR-to-Plex/master/install.sh > install.sh && bash install.sh

TO-DO:

  1. Splat those bugs.
  2. More explicit instructions for GNU Parallel.
  3. Improve handling of tags so that the pre-forked version of AtomicParsley also works.
4 Likes
Raspberry Pi Support
MCEBuddy - Channels DVR files
Feature Request- File Name
Raspberry Pi Setup
Feature Request: transcode after record
Reducing Disk Space Best Practice
#4

Awesome, thanks!

You could try using Gist to host this script as it’s more suited for code than these threads are: https://gist.github.com.

#5

Done as per suggestion. Thanks!

#6

https://gist.github.com/karllmitchell may be the more convenient link than the embedded ones above.

ARM64/AAarch 64 based SOCs
#7

There were some glitches with the pled-transcode.sh script, which I fixed today. Thank you.

#8

You could go even further and put it in a git repo and host it on GitHub. That would allow people to fork it, make changes, and submit their changes back to you.

1 Like
#9

Awesome job! Love how clean and well documented it is. I can think of a few things to enhance it, like options for uploading the resulting file to an FTP o making a copy in a Google Drive/Dropbox directory… The best thing though would be constant monitoring of the DVR folder so the script triggers on every individual file once recording & comskip processing is done…

I would also add --subtitle 1 as default so Handbrake converts closed caption info into regular MP4/iOS/ATV subtitles that can be turned on and off, otherwise caption info is lost when transcoding. Doesn’t hurt and is a nice option in the transcoded file.

Ultimately I see this script somehow integrated into Channels, kind of “Advanced Options” in Settings for file post-processing.

Thanks!

1 Like
#10

Well, I don't know about it being part of the main distribution. It remains a little clunky, and there are still a lot of improvements to make. However, if it demonstrates demand for it maybe it will trigger the Channels DVR team to implement something similar. I consider it more a stop-gap while they focus on other things.

I'm focusing primarily on the core script at the moment, but I don't think it would take someone a lot of effort to come up with an automatically triggered solution. For example, something like this under systemd:

http://www.ocsmag.com/2015/09/02/monitoring-file-access-for-dummies/

or this on Mac:

https://www.macissues.com/2015/02/02/how-to-use-launchagents-to-monitor-folder-contents-in-os-x/

Best wishes,

-Karl

#11

transcode-plex.sh has had some fairly major changes, and so I suggest anyone that uses it downloads the new version.

  1. It now renames shows following Plex conventions (https://support.plex.tv/hc/en-us/articles/200220687-Naming-Series-Season-Based-TV-Shows), and has a fairly friendly section up top that can be edited if the Plex system to look up show names fails, e.g. due to multiple shows having the same name. There are some examples in the code. I will attempt to clean this up further using an array, but for now this works. For those that are interested in developing similar techniques, after some trial and error I came up with a regex method for processing the Channels DVR filenames neatly. It’s pretty messy looking, but does the job well:

regex="([0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{4})\ (.*)\ ([0-9]{4}-[0-9]{2}-[0-9]{2})\ [sS]([0-9]{2})[eE]([0-9]{2})\ (.*)\.(mp4|mkv|mpg|ts|m4v)"

  1. The SOURCE_FILE argument now supports full pathnames in addition to pathnames relative to the SOURCE_DIR and wildcards.

  2. CHAPTERS=1 works again!

  3. Various syntax improvements.

Enjoy!

#12

Minor update uploaded: Added subtitles as per @uspino suggestion.

1 Like
#13

Another update uploaded, correcting a notifications bug. I’ve also added a section of text describing how to set up IFTTT and the app to give you progress notifications on your phone.

#14

A more major update this time, with bug corrections, improved commenting and new features. Given how many changes there are, I recommend keeping a copy of the old version if it was already working for you, just in case this doesn’t.

  1. Replaced numerous instances of obsolete syntax that show how old I am.
  2. Improved handling of HandBrake options.
  3. Added ability to set size directly without degrading encoding preset. e.g. MAXSIZE=720 for 720p. Do not set, or make it >=1080, if you want original size preserved.
  4. Fixed mktemp error on some older versions, including some Mac OS systems.
  5. Added experimental parallel job farming processing capability. Yes, you read that correctly. I’ll document it ASAP.
#15

This project is now hosted on github at https://github.com/karllmitchell/Channels-DVR-to-Plex. Contributions and forks are welcome and encouraged. The top post will be edited down, and has mostly been duplicated on github.

#16

Darwin/Mac OS specific corrections uploaded to github today.
One of those corrections also provides a (rare) workaround for Linux users who do not have “realpath” on their systems, which it turns out is not always installed by default. I recommend installing it in any case (e.g. apt-get install realpath), as I can’t say for sure that the workaround is foolproof, and I’ve added that to the “requirements” list at the top.

#17

Bug correction release last night, due to incorrect file uploaded to prefs. Next job: Movies!

#18

For anyone that is interested, I am working on a new release, and there are some fairly substantial changes coming, thanks in part to some assistance from the Channels devs integrating API calls.

I am warning everyone in advance because the changes are substantial enough that it (a) might introduce more bugs, and (b) will require a little effort to update your prefs file. In other words, you might want to skip this release if everything is working well for you.

Updates include:

  1. A massive rewrite, so some bugs are splatted, but undoubtedly new ones have been reintroduced.
  2. Movies are processed correctly.
  3. More extensive metadata is read directly from the Channels database. TheTVDB API is optionally used for additional data. Installation of JSON parser “jq” is now a requirement. (Put simply: less chance of episode mismatch in Plex).
  4. Past transcodes are now logged within a database to prevent duplication or skipped episodes. The database sets itself up, but you might want to run it manually the first time (instructions will be given) to prevent initial duplication of past transcodes. The option to ignore recordings older then a define number of days remains, as does specifying search terms.
  5. Optional (experimental) addition of metadata for iTunes, using AtomicParsley. (Coming soon: Option to add directly to iTunes, primarily for ATV 1/2/3 sharing).
  6. The script can be run from a remote machine with even less effort than before (comtrim/chapters functionality might not work yet without some effort).

I’ll send out another message when it’s uploaded, and you can decide for yourself whether to take a chance on this major update.

1 Like
#19

New version uploaded. Test away! (Or don’t, if it’s already working for you and you don’t like cutting edge).

#20

Karl, hate to bother you about this. I am not clear on several things from the git link where you list out the instructions. They are geared more toward those who are fairly familiar with these kinds of things. I am familiar with some of it. Would it be possible to get a more detailed how-to to set this up (basics first, then changing preferences later). Would appreciate it so much.

I have Channels DVR set up. I have Plex all set up. Both run magnificently. Would just love to get your transcode functionality to run as well. I am on a 2012 Mac Mini 2.3 Quad Core/16/1TB SSD and several External Drives and a WD EX4100.

#21

No problem. I appreciate that this is not extremely user-friendly. I hope to automate most of this setup soon, but the priority has been to produce something that works and has all the functionality I want, rather than something very user-friendly. That should come with time.

I have some instructions below. The preferences file is perhaps a misnomer; You must edit it, or else it simply will not work. Configuration might be a better term. So, in order:

  1. You’ll need some tools installed on your system. For Mac users, install MacPorts (http://www.macports.org/) or HomeBrew http://brew.sh then run one of these two commands:

sudo port selfupdate; sudo port upgrade outdated; sudo port install curl jq realpath

brew install curl jq realpath

N.B. For Linux users, similar will work, e.g. sudo apt-get install curl jq realpath for Debian or Ubuntu.

If you want to enable iTunes tagging, I’d also recommend adding AtomicParsley to that list, but in most instance the version will be older than recommended, and so does not handle HD tagging correctly. Because of this limitation, you have to set it up explicitly in the prefs file. You would need version >= 0.9.6, which can be obtained directly from: https://github.com/wez/atomicparsley

Then onto installing the software:

  1. Open a web browser to https://github.com/karllmitchell/Channels-DVR-to-Plex

  2. Click on the “Clone or download” button and select download zip to a temporary directory.

  3. Open a terminal in that directory, then install the main script by running the following:
    unzip *.zip
    cd into the new folder
    sudo cp transcode-plex.sh /usr/local/bin/
    sudo chmod 755 /usr/local/bin/transcode.sh

  4. Edit your preferences file. Open transcode-plex.prefs in whatever text editor you prefer (vi or nedit here). The really critical things to edit are these options near the top:

  • SOURCE_DIR is where Channels records to (within that folder should be TV, Movies, Logs, etc.). Make sure you have read access.
  • DEST_DIR is a folder containing your Plex recordings, and it is assumed within there that the standard “TV Shows” and “Movies” folders exists. If not, you’ll need to set up symbolic links. (This will be improved in time). Note that you’ll need write access to this folder and its subfolders.
  • WORKING_DIR could be any arbitrary working directory that you can write to.

Most of the subsequent sections you probably wouldn’t touch. However, if any of the required programs are not found in your PATH on first run, you should fill in the *_CLI preferences with the correct locations. You should definitely do this if you want caffeinate (to prevent system sleep) or AtomicParsley (for iTunes compatible tagging) to work (most simply just =1 will force the code to try to find them in your PATH, or else set to entire path, e.g. "/opt/local/bin/AtomicParsley", if they’re located somewhere strange. You should be able to test whether the software will find them by using the which command, e.g. which AtomicParsley.

  1. Next you’ll need to place the preferences file somewhere useful. Assuming you run it as a regular user, I suggest:

mkdir -p ~/.channels-transcoder/
cp channels-transcoder.prefs ~/.channels-transcoder/prefs

If you are on a Mac, then “${HOME}/Library/Application Support/channels-transcoder/” can be used instead.

Software is now installed.

  1. You’re now ready to start using it. First, initialize the database and perform the first run. Note that this run should also let you know if any critical software can’t be found, and bail before doing anything if it can’t. On the assumption that you only want to transcode new shows:

channels-transcoder.sh CLEAR_DB=1 DAYS=0

This will let you know that all shows before now should be marked as already transcoded. If you have some backlog and want to transcode more shows before getting into routine conversion, you could try e.g. DAYS=3, or even DAYS=10000 (sure to catch any past recordings). Bear in mind that on most systems transcoding takes a lot of time, due to the high quality software-only compression methods that preserve most of the original quality, typically comparable (within a factor of a few) to the entire show length, so if you have 20 hours of backlogged TV, you may wish to check on that job the next day. For your machine you’ll probably go considerably faster than real time.

  1. Once that is complete, any time you run:

channels-transcoder.sh

it will only transcode more recent shows. After that, the best way to set up a regular job depends on what sort of system you have, but on the simplest level a cronjob as described in the README should work whichever system you’re on. Bear in mind that, for the moment, it assumes that the previous job is finished before the next one starts, so in most instances run no more than once/day depending on how fast your system transcodes; I’ll improve handling of multiple instances in time.

Finally, do have a read of that preferences file. I’ve attempted to make the instructions as explicit as possible in there, and simplified it a little tonight. There’s also a section at the bottom for substituting show names in case Plex routinely gets them wrong, which I find happens for about 1-in-10 shows.

I hope that’s useful. If you can’t follow this, get back to me, or wait a few weeks and I’ll have something simpler for you.

-Karl

1 Like
#22

BTW, I highly recommend homebrew (http://brew.sh) over MacPorts

1 Like