BETA: Smart Commercial Detection for TVE & Pluto

Beta Community DVR Beta
#1

With v2021.12.22.0148 we are testing a new smarter commercial detector, which works on some streaming channels.

Instead of using comskip to scan video, this new detector uses a more traditional approach similar to how Pi-hole works. We can detect patterns on some channels where the advertising content comes from a different place.

I have tested this with some Discovery channels on TVE, and some of the channels on Pluto. Share your own experiences below.

Log output for traditional comskip:

2021/12/12 19:03:00.199076 [MTS] Statistics for "TV/Well Done With Sebastian Maniscalco/Well Done With Sebastian Maniscalco S02E01 The Signature Cocktail 2021-12-12-1829.mpg": skipped=0 unhandled_packets=0 discontinuity_detected=69 transport_errors=0 invalid_pts=0 invalid_dts=0 saw_pcr=true saw_pmt=true highest_pts=2016.845378
2021/12/12 19:03:00.550097 [DVR] Processing file-12716: TV/Well Done With Sebastian Maniscalco/Well Done With Sebastian Maniscalco S02E01 The Signature Cocktail 2021-12-12-1829.mpg
2021/12/12 19:03:01.129762 [DVR] Running commercial detection on file 12716 (TV/Well Done With Sebastian Maniscalco/Well Done With Sebastian Maniscalco S02E01 The Signature Cocktail 2021-12-12-1829.mpg)

Log output for smart detector:

2021/12/21 17:33:00.104223 [MTS] Statistics for "TV/Man v. Food/Man v. Food S05E09 2017-09-04 Des Moines IA 2021-12-21-1659.mpg": skipped=0 unhandled_packets=0 discontinuity_detected=35 transport_errors=0 invalid_pts=0 invalid_dts=0 saw_pcr=true saw_pmt=true highest_pts=2014.881478
2021/12/21 17:33:00.105488 [ADS] Detected 4 commercials in file-12788 TV/Man v. Food/Man v. Food S05E09 2017-09-04 Des Moines IA 2021-12-21-1659.mpg

(Recordings where the new detector kicked in will have "CommercialsTagged": true in their JSON.)

12 Likes
DVR Pre-Release Notes
One channel not detecting commercials
Disable commercial detection
Commercial Skip Requires Manual Detection
Anyone else having issues with no commercial detection?
Commercial Detection Experiences
Comskip Changes
Quality of ChannelsDVR releases is going down
Commercial Detection Problem
Commercial Skipping Choices
Why is commercial skip so inconsistent amongst different channels?
Why is commercial skip so inconsistent amongst different channels?
Rerunning comskip on all recorded shows?
Non-Docker source for PlutoTV and Stirr m3u playlists and EPG
TVE channels with smart ad detection - 2024 update
Commercial detection seemingly won't turn off
#2
#3

I had saved a few shows from CNNGo over the weekend that had particularly bad commercial detection with the intention to tweak comskip, so I'm glad that it's getting some attention. Which channels are currently using the new algorithm? It looks like the two shows that failed commercial detection over the weekend are still using comskip and still failing.

2021/12/21 18:21:09.257617 [DVR] Running commercial detection on file 7421 (TV/Fareed Zakaria GPS/Fareed Zakaria GPS 2021-12-19-0700.mpg)
2021/12/21 18:27:04.571393 [DVR] Commercial detection failed for Fareed Zakaria GPS 2021-12-19-0700.mpg with exit status 1
2021/12/21 18:27:04.959878 [DVR] Running commercial detection on file 7420 (TV/State of the Union With Jake Tapper and Dana Bash/State of the Union With Jake Tapper and Dana Bash S01E50 2021-12-19-0600.mpg)
2021/12/21 18:35:19.658265 [DVR] Commercial detection failed for State of the Union With Jake Tapper and Dana Bash S01E50 2021-12-19-0600.mpg with exit status 1

This is from running "Redetect Commercials", so I don't know if that's supposed to take advantage of the new algorithm.

#4

Only new recordings made with the latest build would use the new system.

I don't think it works for CNN yet but I will take a look to see if that can be added.

2 Likes
#5

It looks like it might limit the new detection to recordings from channels 6000–6999, which are the numbers Channels uses internally for TVE/its own internet streams.

#6

Thanks, that would be great. I find the channels that use the static logo screen for commercial breaks to be quite vexing because it seems like they would be the easiest to separate content from "commercial". My initial attempts to tweak comskip utterly failed, though.

#7

CNN does have a specific pattern they follow when switching to ads. I will add it tomorrow.

The detector is not based on any channel number. It is part of the streaming code and will kick in whenever any of the ad-detect patterns match.

3 Likes
TVE channels with smart ad detection - 2024 update
#8

@tmm1 NICE work. Commercial detection was awful on TVE and now from my initial test works AMAZING! I just deleted my entire COPS episode library from Pluto and are re-recording them all now :slight_smile:
Now if only the same technology could be used for OTA lol

3 Likes
#9

Thanks! There are a few channels that all follow the same pattern - mostly black screen with a big logo in the middle - so hopefully you could match that pattern and handle a bunch of them.

#10

I added CNN. Which other channels?

#11

I just tried History and it looks like it is still using the comskip method

2021/12/22 00:03:30.008925 [TNR] Closed connection to TVE-Cox for ch6044 HISTORY
2021/12/22 00:03:30.018721 [MTS] Statistics for "TV/The Curse of Oak Island/The Curse of Oak Island S09E06 2021-12-07 The Root Cause 2021-12-21-2304.mpg": skipped=0 unhandled_packets=0 discontinuity_detected=0 transport_errors=0 invalid_pts=0 invalid_dts=0 saw_pcr=true saw_pmt=true highest_pts=3538.779978
2021/12/22 00:03:30.021265 [DVR] Finished job 1640145890-ch6044 The Curse of Oak Island
2021/12/22 00:03:30.102119 [DVR] Processing file-7444: TV/The Curse of Oak Island/The Curse of Oak Island S09E06 2021-12-07 The Root Cause 2021-12-21-2304.mpg
2021/12/22 00:03:30.351399 [DVR] Running commercial detection on file 7444 (TV/The Curse of Oak Island/The Curse of Oak Island S09E06 2021-12-07 The Root Cause 2021-12-21-2304.mpg)
2021/12/22 00:05:58.270155 [DVR] Commercial detection for The Curse of Oak Island S09E06 2021-12-07 The Root Cause 2021-12-21-2304.mpg finished with 12 markers in 2m27.918855924s.

#12

History looks like it uses a similar pattern as CNN and will work in the new prerelease

#13

awesome thanks. could you add TCM as well?

#14

So is this enabled by default then? as u do not state how to enable this new feature.

Also, how is this new detector on resources/performance compared to comskip? faster or slower?
I have it set to 3 thread count on my Pi4,which still takes some time for comskip.

i will run it on my recordings from yesterday that i have not gotten to watch yet and see if there is any difference in reported time in the logs. nvm did not see that this only works on new things on certain channels.

#15

No luck with TCM. I suspect this means the same for TBS, TNT, etc.

#16

Yes it kicks in automatically. It uses no resources. It is not possible to run manually or on old recordings.

#18

nat geo wild has very regular pattern of comericals, comskip sometimes messes it up though.

Fox, when i record Family Guy, also, for long time now, it always cuts off the very last short end segment of show, then skips right to the start of the news.

It has been very bad on a few other shows I have recorded...but figured it was one offs.

Wheeler Dealers on MotorTrend, it also has issues and does not trim right at time, showing segments of that "ad break in progress" screens as segments.

If any of those channels can be improved, would be awesome!

#19

Yes selecting Redetect will run comskip and be used to override the smart detector manually.

1 Like
#20

There's a bug causing player to crash on some of these recordings: Playback crash on some smart ad detected recordings - #4 by tmm1

The recordings are fine and using forced comskip will fix the issue for now.

#22

TCM airs movies uninterrupted and without commercials, it only has promotions of its own movies between airings. So are you just trying to make it work so that it skips what's before and after a recorded movie?

1 Like