Add support for GZip compressed EPG xml

#3

No error. They're just not being detected.
When I use m3u4u.com to build EPG guides, the compressed version doesn't work. The uncompressed one does. I also tried the EPGs over at https://github.com/epgshare01 and they don't work either. However. They do if I download and extract them. So now I've set up a cronjob to do this once a day.

M3U4U.com playlist not pulling in
#4

Please submit diagnostics from your DVR

#5

Sure. Here you go: ca8382dd-b60d-4bce-a956-ea65bd26bf7e

#6

Looks like you might have reinstalled so I'm not seeing the previous failures with the gzip sources

#7

Right.
Okay I just re-added the xml.gz list and redownloaded the EPG to see if I could resolve the problem.
I skimmed through the log myself. Couldn't find anything other than this:
2021/09/06 21:58:06.880435 [IDX] Pruned 0 expired airings from XMLTV-USLocalTV in 138.973µs..
Not sure why it expired. There's nothing wrong with the EPG. Like I said − when I extract it and use the .xml file, it works fine.
The new log is 975ed49c-5a43-4601-998b-6b780d2c427a

#8

And from browsing the forum, it isn't just me.


And also here: XML Guide for M3U tuner?
AeroR1's reply confirmed it.
So I don't think it's an issue on my end. gzip might not be supported.

1 Like
#9

Thanks I see the issue.

Are the other affected urls ending in .gz also?

#10

Yes.

1 Like
#11

Okay fix is uploading in next prerelease

1 Like
#12

Great. Thanks.
So .gz was not listed as a supported format, huh?
Glad it was that simple.

#13

v2021.09.06.2051 is available now.

1 Like
#14

Thank you!

#15

The web server this was hosted on was not configured properly to tell our client that the file was compressed. We had to implement a work-around to decompress URLs ending in .gz which is against HTTP specs, but is required to make this work here.

2 Likes
#16

autoprocessing of content has nothing to do with http specs. no http specs have been violated :wink:

#17

He was saying that the server was not correctly setting the header to let the client know the content was compressed. Typically, when the HTTP server sends content that has been compressed/gzipped, a header is present letting the client know. From the dev's comments, the server in question was not setting the header, so Channels did not expect the content to be compressed.

#18

That's not what happened. what header btw? :wink:

#19

Content-Encoding

Edit: Here's the section of the HTTP spec. Note the use of the word MUST when compressed data is sent: https://httpwg.org/specs/rfc7231.html#header.content-encoding

#20

You are confused. This header is only used if the data is compressed by the server on the fly. In our case the client requests already compressed data so no extra header needed.

The problem for channels here is there is no uncompressed data available.

Check also https://nginx.org/en/docs/http/ngx_http_gzip_static_module.html

#21

I don't have an account at m3u4u.com, so I can't comment to that. However, using the epgshare01 example:

curl -v https://github.com/epgshare01/share01/raw/master/epg_ripper_CA1.xml.gz
> Host: github.com
> user-agent: curl/7.76.1
> accept: */*
...
< HTTP/2 302 
< server: GitHub.com
< date: Tue, 07 Sep 2021 20:06:13 GMT
< content-type: text/html; charset=utf-8

Notice how the returned content-type is text/html? If Channels were receiving content-type: applicaton/x-gzip, then perhaps it would know it is receiving compressed content, but it's not. Also, there is no content-encoding header, either.

#22

How about you follow the redirection (302) via -L option in curl to see what the content-type is? :wink:

content-type: application/octet-stream

Also, there is no content-encoding header, either.

Because no compression was requested. If compression is needed in curl

-H 'Accept-Encoding: gzip'

option needs to be added. HTTP 101 :wink: