Crude XMLTV Export

Here is my brute force of an alpha version of XMLTV export. It is written and tested in BASH on MacOS BugSur, ymwv! It runs for about an hour. I hope to get this down if I can crack the BASH "read" command with spaces, see comments in script.

Here is the script:

#!/bin/bash

JQ=/usr/local/bin/jq
CURL=/usr/bin/curl
ECHO=/bin/echo
CAT=/bin/cat
DATE=/bin/date

# Grab RAW Program Data
RAW_EPG=$($CURL -s http://mac-mini.local:8089/devices/ANY/guide > tmpEPG.dat)

# Start the XMLTV Guide
echo "<tv>"

# Get the Channel Data
CHANNELS=$($CAT "tmpEPG.dat" | $JQ '. | length')
for ((h = 0 ; h < $CHANNELS ; h++)); do
    # echo "Starting Channel [${h}] ..."

    # Cannot get this to work with Spaces!!!
    # I have tried All Combinations of
    # IFS="", IFS='', IFS=$"\n" with
    # read -r or read
    read CHANNEL_ID DISPLAY_NAME1 DISPLAY_NAME2 DESCRIPTION IMAGE NUMBER <<<$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Station, .Channel.Name, .Channel.Number, .Channel.Description, .Channel.Image, .Channel.Number')

#    CHANNEL_ID=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Station')
#    DISPLAY_NAME1=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Name')
#    DISPLAY_NAME2=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Number')
#    DESCRIPTION=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Description')
#    IMAGE=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Image')
#    NUMBER=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Channel.Number')

    echo "    <channel id=${CHANNEL_ID}>"
    echo "        <display-name>${DISPLAY_NAME1}</display-name>"
    echo "        <display-name>${DISPLAY_NAME2}</display-name>"
    echo "        <desc>${DESCRIPTION}</desc>"
    echo "        <icon src=${IMAGE}/>"
    echo "        <lcn>${NUMBER}</lcn>"
    echo "    </channel>"

    # Get the Program Data
    AIRINGS=$($CAT "tmpEPG.dat" | $JQ --arg h $h '.[$h|tonumber] | .Airings | length')

    for ((i = 0 ; i < $AIRINGS ; i++)); do
        # echo "Starting Airings [${i}] for Channel [${h}] ..."

        # Cannot get this to work with Spaces!!!
        # I have tried All Combinations of
        # IFS="", IFS='', IFS=$"\n" with
        # read -r or read
        # read CHANNEL_ID2 TIME DURATION TITLE SUB_TITLE DESCRIPTION2 AIR_DATE CATEGORIES GENRES IMAGE2 SERIESID EPISODE_NUMBER EPISODE_DATE <<<$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Channel, .Time, .Duration, .Title, .EpisodeTitle, .Summary, .Categories[0], .Genres[0], .Image, .SeriesID, .EpisodeNumber, .OriginalDate')

        CHANNEL_ID2=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Channel')
        TIME=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Time')
        DURATION=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Duration')
        END=$((TIME + DURATION))
        START_DATETIME=$($DATE -uR -r $TIME +"%Y%m%d%H%M%S +0000")
        END_DATETIME=$($DATE -uR -r $END +"%Y%m%d%H%M%S +0000")
        TITLE=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Title')
        SUB_TITLE=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .EpisodeTitle')
        DESCRIPTION2=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Summary')
        AIR_DATE=$($DATE -uR -r $TIME +"%Y%m%d")
        CATEGORIES=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Categories[0]')
        GENRES=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Genres[0]')
        IMAGE2=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .Image')
        SERIESID=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .SeriesID')

        if [ -z "${SERIESID##pluto*}" ]
        then
           SERIESID=${SERIESID//pluto\//}
            SYSTEM="pluto"
        else
            SYSTEM=""
        fi

        EPISODE_NUMBER=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .EpisodeNumber')
        EPISODE_DATE=$($CAT "tmpEPG.dat" | $JQ --arg h $h --arg i $i '.[$h|tonumber] | .Airings[$i|tonumber] | .OriginalDate')

        echo "    <programme start=${START_DATETIME} stop=${END_DATETIME} channel=${CHANNEL_ID2}>"
        echo "        <title lang=\"en\">${TITLE}</title>"
        echo "        <icon src=${IMAGE2}/>"
        echo "        <sub-title lang=\"en\">${SUB_TITLE}</sub-title>"
        echo "        <desc lang=\"en\">${DESCRIPTION2}</desc>"
        echo "        <date>${AIR_DATE}</date>"
        if [ -z "$CATEGORIES" ]
        then
            echo "        <category lang=\"en\">${GENRES}</category>"
        else
            echo "        <category lang=\"en\">${CATEGORIES}</category>"
        fi
        echo "        <series-id system=\"${SYSTEM}\">${SERIESID}</series-id>"
        echo "        <episode-num system=\"onscreen\">${EPISODE_NUMBER}</episode-num>"
        echo "        <episode-num system=\"original-air-date\">${EPISODE_DATE}</episode-num>"
        echo "    </programme>"

    done # Getting Program Data

done # Getting Channel Data

# Finish XMLTV Guide
echo "</tv>"
4 Likes

nice job getting this far, but getting this to work fully with a bash script is probably going to be way more effort than its worth...it's pretty easy with a language like php that is designed to do this kind of thing. my channel mapper (which is written in php) outputs two weeks of guide from the server in about 30 seconds or so.

the shell just isn't designed to do this kind of thing IMO.

No, but this is exactly the type of thing awk and perl excel at.

There are also a slew of various other tools already available that do nothing but transform JSON into XML.

the problem with that (i know because i just went through it when i wrote my parser) is that the json the dvr outputs for the guide doesn't match the xmltv format. you can't just do a one to one translation.

Exactly. Store the data into internal variables and iterate. Most (all?) of the tools from the XMLTV project are written in perl, including tv_grab_zz_sdjson, which pulls in JSON in nearly (exactly?) the same format that Channels is using.

(I understand that you probably chose PHP for your tool probably because that is what you know and are familiar with. But there are many different languages that can handle the task, and probably more efficiently. I mentioned perl and awk because generating this type of output is what they excel at.

In fact, adapting tv_grab_zz_sdjson—or better yet, tv_grab_zz_sdjson_sqlite—to use Channels as the source instead of Schedules Direct would probably have been a much more straightforward way to get XMLTV from Channels' JSON.)

1 Like

I used php because it was part of the mapping app I was already writing in php. I ended up not even needing it once I figured out that I could just use station id to have channels pull the data from gracenote directly...

either way, a shell script isn't the way to do this.

Depends on the situation. Consider a scenario where you have only shell access, and cannot install binaries. Or perhaps the challenge and learning experience of writing a parser/emitter in shell.

Separately: I was unable to find a pure shell XML emitter—although writing one shouldn't be too difficult—but I was able to locate a pure shell JSON parser: jshn. (As an added bonus, it appears to be written in pure POSIX shell, without any bash-isms, so it ought to be portable to any shell environment.)

Is there a list of station id(s)? Have been looking but no luck.

They're the same as Schedules Direct. You can get a trial membership, and use a client to retrieve station ids.

they're in the json that the dvr spits out if you ask it for a channels list via api.

After a quick look at tv_grab_zz_sdjson_sqlite I agree that this is the way to go. When I can find the time (ha!) I'll see what I can do. Ideally inside a docker that can live alongside @Maddox Pluto container hosted on the RaspPi image. :wink:

doesn't that require you to have a schedules direct account, though? parsing it from the DVR doesn't require an additional subscription...

The raw portion of the JSON in the guide data is (likely) how it is given to Channels from Gracenote. This format ought to be close enough to the JSON that Schedules Direct emits. With that established, you can modify tv_grab_zz_sdjson{,_sqlite} to query the Channels DVR server, take the raw object, and use the script to handle the heavy lifting.

I never said use the tv_grab_… scripts from the XMLTV as-is. Rather, I said use them as the starting point, and modify them as needed to handle the JSON that Channels emits. Usually having a foundation to build upon can be easier than trying to do it all from scratch.

1 Like

...which is why i used laravel. i never said i did it from scratch.

also, unless i'm missing something tv_grab_zz_sdjson isn't open source. how are you supposed to modify that?

i'm just not understanding how using that helps without having a SD account to attach it to. if i'm missing something obvious, please help me out...

Really? XMLTV isn't open source? The source code is readily available for all the grabbers.

of course the XMLTV format is readily available, but the source code for the grabbers is too? where?

that's why i said if i am missing something completely obvious please help me out...

yeah you beat me to it. i was editing my post, i didn't realize the grabbers were in there.

thanks. definitely a valid way to go if you're proficient in perl.

The main repository of the XMLTV project is hosted at Sourceforge, but can also be found on GitHub. The tv_grab_zz_sdjson grabber is a part of the XMLTV project, and can be found in those repositories.

The tv_grab_zz_sdjson_sqlite grabber is hosted separately on GitHub, and can be found here. (The author of this particular grabber can often be found on the SiliconDust forums under the handle gtb.)

Also, if you're curious about the actual XMLTV format, the you can view the DTD on GitHub.

(Edit: It looks like tv_grab_zz_sdjson_sqlite has been merged into the XMLTV project, as both grabbers are present in the grab subdirectory in the main repo.)

You modify the grabber, so that instead of querying the Schedules Direct servers, you point it to the endpoint of your own Channels DVR server.