Crude XMLTV Export

carterbunch · November 28, 2020, 3:07am

Here is my brute force of an alpha version of XMLTV export. It is written and tested in BASH on MacOS BugSur, ymwv! It runs for about an hour. I hope to get this down if I can crack the BASH "read" command with spaces, see comments in script.

Here is the script:

#!/bin/bash

JQ=/usr/local/bin/jq
CURL=/usr/bin/curl
ECHO=/bin/echo
CAT=/bin/cat
DATE=/bin/date

# Grab RAW Program Data
RAW_EPG=$($CURL -s http://mac-mini.local:8089/devices/ANY/guide > tmpEPG.dat)

# Start the XMLTV Guide
echo "<tv>"

# Get the Channel Data
CHANNELS=$($CAT "tmpEPG.dat" | $JQ '. | length')
for ((h = 0 ; h < $CHANNELS ; h++)); do
    # echo "Starting Channel [${h}] ..."

    # Cannot get this to work with Spaces!!!
    # I have tried All Combinations of
    # IFS="", IFS='', IFS=$"\n" with
    # read -r or read
    read CHANNEL_ID DISPLAY_NAME1 DISPLAY_NAME2 DESCRIPTION IMAGE NUMBER <<<$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Station, .Channel.Name, .Channel.Number, .Channel.Description, .Channel.Image, .Channel.Number')

#    CHANNEL_ID=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Station')
#    DISPLAY_NAME1=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Name')
#    DISPLAY_NAME2=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Number')
#    DESCRIPTION=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Description')
#    IMAGE=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Image')
#    NUMBER=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Number')

    echo "    <channel id=${CHANNEL_ID}>"
    echo "        <display-name>${DISPLAY_NAME1}</display-name>"
    echo "        <display-name>${DISPLAY_NAME2}</display-name>"
    echo "        <desc>${DESCRIPTION}</desc>"
    echo "        <icon src=${IMAGE}/>"
    echo "        <lcn>${NUMBER}</lcn>"
    echo "    </channel>"

    # Get the Program Data
    AIRINGS=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Airings | length')

    for ((i = 0 ; i < $AIRINGS ; i++)); do
        # echo "Starting Airings [${i}] for Channel [${h}] ..."

        # Cannot get this to work with Spaces!!!
        # I have tried All Combinations of
        # IFS="", IFS='', IFS=$"\n" with
        # read -r or read
        # read CHANNEL_ID2 TIME DURATION TITLE SUB_TITLE DESCRIPTION2 AIR_DATE CATEGORIES GENRES IMAGE2 SERIESID EPISODE_NUMBER EPISODE_DATE <<<$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Channel, .Time, .Duration, .Title, .EpisodeTitle, .Summary, .Categories[0], .Genres[0], .Image, .SeriesID, .EpisodeNumber, .OriginalDate')

        CHANNEL_ID2=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Channel')
        TIME=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Time')
        DURATION=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Duration')
        END=$((TIME + DURATION))
        START_DATETIME=$($DATE -uR -r $TIME +"%Y%m%d%H%M%S +0000")
        END_DATETIME=$($DATE -uR -r $END +"%Y%m%d%H%M%S +0000")
        TITLE=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Title')
        SUB_TITLE=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .EpisodeTitle')
        DESCRIPTION2=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Summary')
        AIR_DATE=$($DATE -uR -r $TIME +"%Y%m%d")
        CATEGORIES=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Categories[0]')
        GENRES=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Genres[0]')
        IMAGE2=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Image')
        SERIESID=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .SeriesID')

        if [ -z "${SERIESID##pluto*}" ]
        then
           SERIESID=${SERIESID//pluto\//}
            SYSTEM="pluto"
        else
            SYSTEM=""
        fi

        EPISODE_NUMBER=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .EpisodeNumber')
        EPISODE_DATE=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .OriginalDate')

        echo "    <programme start=${START_DATETIME} stop=${END_DATETIME} channel=${CHANNEL_ID2}>"
        echo "        <title lang=\"en\">${TITLE}</title>"
        echo "        <icon src=${IMAGE2}/>"
        echo "        <sub-title lang=\"en\">${SUB_TITLE}</sub-title>"
        echo "        <desc lang=\"en\">${DESCRIPTION2}</desc>"
        echo "        <date>${AIR_DATE}</date>"
        if [ -z "$CATEGORIES" ]
        then
            echo "        <category lang=\"en\">${GENRES}</category>"
        else
            echo "        <category lang=\"en\">${CATEGORIES}</category>"
        fi
        echo "        <series-id system=\"${SYSTEM}\">${SERIESID}</series-id>"
        echo "        <episode-num system=\"onscreen\">${EPISODE_NUMBER}</episode-num>"
        echo "        <episode-num system=\"original-air-date\">${EPISODE_DATE}</episode-num>"
        echo "    </programme>"

    done # Getting Program Data

done # Getting Channel Data

# Finish XMLTV Guide
echo "</tv>"

crackers8199 · November 28, 2020, 5:06am

nice job getting this far, but getting this to work fully with a bash script is probably going to be way more effort than its worth...it's pretty easy with a language like php that is designed to do this kind of thing. my channel mapper (which is written in php) outputs two weeks of guide from the server in about 30 seconds or so.

the shell just isn't designed to do this kind of thing IMO.

racameron · November 28, 2020, 6:23am

No, but this is exactly the type of thing awk and perl excel at.

There are also a slew of various other tools already available that do nothing but transform JSON into XML.

crackers8199 · November 28, 2020, 7:46am

the problem with that (i know because i just went through it when i wrote my parser) is that the json the dvr outputs for the guide doesn't match the xmltv format. you can't just do a one to one translation.

racameron · November 28, 2020, 11:12am

Exactly. Store the data into internal variables and iterate. Most (all?) of the tools from the XMLTV project are written in perl, including tv_grab_zz_sdjson, which pulls in JSON in nearly (exactly?) the same format that Channels is using.

(I understand that you probably chose PHP for your tool probably because that is what you know and are familiar with. But there are many different languages that can handle the task, and probably more efficiently. I mentioned perl and awk because generating this type of output is what they excel at.

In fact, adapting tv_grab_zz_sdjson—or better yet, tv_grab_zz_sdjson_sqlite—to use Channels as the source instead of Schedules Direct would probably have been a much more straightforward way to get XMLTV from Channels' JSON.)

crackers8199 · November 28, 2020, 8:01pm

I used php because it was part of the mapping app I was already writing in php. I ended up not even needing it once I figured out that I could just use station id to have channels pull the data from gracenote directly...

either way, a shell script isn't the way to do this.

racameron · November 28, 2020, 8:20pm

Depends on the situation. Consider a scenario where you have only shell access, and cannot install binaries. Or perhaps the challenge and learning experience of writing a parser/emitter in shell.

Separately: I was unable to find a pure shell XML emitter—although writing one shouldn't be too difficult—but I was able to locate a pure shell JSON parser: jshn. (As an added bonus, it appears to be written in pure POSIX shell, without any bash-isms, so it ought to be portable to any shell environment.)

kanipek · November 29, 2020, 12:35am

Is there a list of station id(s)? Have been looking but no luck.

racameron · November 29, 2020, 12:57am

They're the same as Schedules Direct. You can get a trial membership, and use a client to retrieve station ids.

crackers8199 · November 29, 2020, 2:10am

they're in the json that the dvr spits out if you ask it for a channels list via api.

coolmaui · November 30, 2020, 7:59pm

After a quick look at tv_grab_zz_sdjson_sqlite I agree that this is the way to go. When I can find the time (ha!) I'll see what I can do. Ideally inside a docker that can live alongside @Maddox Pluto container hosted on the RaspPi image.

crackers8199 · November 30, 2020, 8:44pm

doesn't that require you to have a schedules direct account, though? parsing it from the DVR doesn't require an additional subscription...

racameron · November 30, 2020, 8:54pm

The raw portion of the JSON in the guide data is (likely) how it is given to Channels from Gracenote. This format ought to be close enough to the JSON that Schedules Direct emits. With that established, you can modify tv_grab_zz_sdjson{,_sqlite} to query the Channels DVR server, take the raw object, and use the script to handle the heavy lifting.

I never said use the tv_grab_… scripts from the XMLTV as-is. Rather, I said use them as the starting point, and modify them as needed to handle the JSON that Channels emits. Usually having a foundation to build upon can be easier than trying to do it all from scratch.

crackers8199 · November 30, 2020, 8:56pm

...which is why i used laravel. i never said i did it from scratch.

also, unless i'm missing something tv_grab_zz_sdjson isn't open source. how are you supposed to modify that?

i'm just not understanding how using that helps without having a SD account to attach it to. if i'm missing something obvious, please help me out...

coolmaui · November 30, 2020, 9:16pm

Really? XMLTV isn't open source? The source code is readily available for all the grabbers.

crackers8199 · November 30, 2020, 9:19pm

of course the XMLTV format is readily available, but the source code for the grabbers is too? where?

that's why i said if i am missing something completely obvious please help me out...

coolmaui · November 30, 2020, 9:21pm

crackers8199 · November 30, 2020, 9:24pm

yeah you beat me to it. i was editing my post, i didn't realize the grabbers were in there.

thanks. definitely a valid way to go if you're proficient in perl.

racameron · November 30, 2020, 9:26pm

The main repository of the XMLTV project is hosted at Sourceforge, but can also be found on GitHub. The tv_grab_zz_sdjson grabber is a part of the XMLTV project, and can be found in those repositories.

The tv_grab_zz_sdjson_sqlite grabber is hosted separately on GitHub, and can be found here. (The author of this particular grabber can often be found on the SiliconDust forums under the handle gtb.)

Also, if you're curious about the actual XMLTV format, the you can view the DTD on GitHub.

(Edit: It looks like tv_grab_zz_sdjson_sqlite has been merged into the XMLTV project, as both grabbers are present in the grab subdirectory in the main repo.)

racameron · November 30, 2020, 9:30pm

You modify the grabber, so that instead of querying the Schedules Direct servers, you point it to the endpoint of your own Channels DVR server.