The manual Local Content: Scan Sources command is ignored (does nothing) in some circumstances. Looks like there is some static guard time either related to the prior Scan Sources command or to emptying the trash-- not sure which. Regardless, it is super annoying especially when trying to automate library importing and you must wait some mystery time (seems like 3-5 minutes) between test runs. Would be great to get this fixed. Thanks.
The scanner runs every five minutes.
From what I can see, the only reason it would ignore the manual API is if it is sent twice while the scanner is already running.
Running on fast hardware and the library is small so don't believe it is busy doing something. The exact failure pattern is not obvious to me, but a quick stopwatch example:
T+0 secs: delete item from trash
T+60 secs: add item to imports and trigger manual scan
T+70 secs: item not in library
T+80 secs: item not in library
T+90 secs: item not in library
...
T+120 secs: trigger second manual scan
T+121 secs: item immediately in library
A slightly different test-- once the old item was trashed, immediately copied a new one into imports (<5 secs) and then manual scans every 10 seconds. Tailied the logfile on a console to see the results. Looks like 30-40 seconds rather than my perceived 3 minutes.
2021/08/31 16:43:31.107292 [DVR] Deleted /var/content/Imports/Movies/hallmark/The Shell Seekers (1989).mp4 2021/08/31 16:44:11.439982 [SCN] Imported movie hallmark/The Shell Seekers (1989).mp4 2021/08/31 16:47:43.476522 [DVR] Deleted /var/content/Imports/Movies/hallmark/The Shell Seekers (1989).mp4 2021/08/31 16:48:24.140316 [SCN] Imported movie hallmark/The Shell Seekers (1989).mp4
Ah, it won't scan recently modified files in case they are still being copied over..
1 Like
Perfect-- that is what I needed to know. Will change the modification time after a file is copied to workaround. Since that wont happen until the copy is complete, no downside. Thanks.
The scanner just skips things modified in the last 30 seconds. So just delaying the scan should do, vs modifying those dates.
1 Like
Trying to determine what import filename format(s) work the best and that requires iteration time. No problem delaying once optimal formats identified, but dozens of manual reloads required while experimenting.
The current Channels import metadata query code is "quirky" to be polite. Movies and series are parsed completely differently. Series allows for flexible filename formatting but discards the most valuable information in favor of the most ambiguous with corresponding results. Movies are way simpler to identify and yet the parser is super rigid. The latter is annoying but just requires workaround code. The former is the cause is multiple forum threads and user angst.
Should you decide to revisit the import metadata matching code, happy to provide detailed feedback. Don't believe you need new data sources or anything exotic-- partial consolidation of a couple code paths and a simplistic tiered query approach.
Please post some examples of filenames and paths that you are having trouble with.
Below are some examples of imported data and the results. My files are named slug-style and such that directories list them in airdate episode order. (The latter is accomplished by placing original airdate prior to SxxExx.) For Channels purposes, I symlink an appropriately named file for its consumption:
src: /var/content/Rewinds/csi-miami-20040202-s02e13-blood-moon.m3u dst: /var/content/Imports/TV/csi-miami/csi-miami-20040202-s02e13-blood-moon.mp4 dst: /var/content/Imports/TV/csi-miami/csi miami 20040202 s02e13 blood moon.mp4 dst: /var/content/Imports/TV/csi-miami/csi miami 20040202 S02E13 blood moon.mp4 dst: /var/content/Imports/TV/csi-miami/csi miami 20040202 S02E13 blood moon.mp4 dst: /var/content/Imports/TV/csi-miami/Csi Miami 20040202 S02E13 Blood Moon.mp4 dst: /var/content/Imports/TV/csi-miami/Csi Miami 2004-02-02 S02E13 Blood Moon.mp4
Every Variation Correctly Identified:
CSI: Miami Blood Moon: Season 2, Episode 13
Horatio suspects revenge is behind the murder of a man who tortured Cuban refugees.
Below is the Channels metadata for the import. All information matches what was provided-- Title (CSI Miami), Episode Title (Blood Moon), OriginalDate (2004-02-02) Season Number (2), Episode Number (13).
{"ID":"577", "GroupID":"184820", "Path":"csi-miami/Csi Miami 2004-02-02 S02E13 Blood Moon.mp4", "CreatedAt":1630510683,"Duration":5,"Completed":true,"Processed":true,"Airing":{"Source":"tms", "OriginalDate":"2004-02-02", "Time":1075680000,"Duration":0,"Title":"CSI: Miami", "EpisodeTitle":"Blood Moon", "Summary":"Horatio suspects revenge is behind the murder of a man who tortured Cuban refugees.", "FullSummary":"Horatio suspects revenge is behind the murder of a man who tortured Cuban refugees.", "Categories":["Episode", "Series"],"Genres":["Crime drama", "Action", "Thriller", "Mystery"],"Tags":["HD", "Stereo"],"SeriesID":"184820", "ProgramID":"EP005240610037", "SeasonNumber":2,"EpisodeNumber":13,"Directors":["Scott Lautanen"],"Cast":["David Caruso", "Emily Procter", "Rory Cochrane"],"ReleaseYear":2004,"Raw":{"startTime":"2004-02-02T00:00Z", "endTime":"", "duration":0,"channels":null,"stationId":"", "qualifiers":null,"ratings":null,"program":{"tmsId":"EP005240610037", "rootId":"2722934", "seriesId":"184820", "entityType":"Episode", "subType":"Series", "title":"CSI: Miami", "titleLang":"en", "episodeTitle":"Blood Moon", "episodeNum":13,"seasonNum":2,"releaseYear":2004,"releaseDate":"2004-02-02", "origAirDate":"2004-02-02", "descriptionLang":"en", "shortDescription":"Horatio suspects revenge is behind the murder of a man who tortured Cuban refugees.", "longDescription":"Horatio suspects revenge is behind the murder of a man who tortured Cuban refugees.", "topCast":["David Caruso", "Emily Procter", "Rory Cochrane"],"directors":["Scott Lautanen"],"genres":["Crime drama", "Action", "Thriller", "Mystery"], "runTime":"PT00H44M"}}},"UpdatedAt":1630510683280,"ImportPath":"Imports/TV", "ImportQuery":"csi-miami", "ImportGroup":"184820", "ImportedAt":1630510683091}
src: /var/content/Rewinds/csi-miami-20040209-s02e16-slow-burn.m3u dst: /var/content/Imports/TV/csi-miami/csi-miami-20040209-s02e16-slow-burn.mp4 dst: /var/content/Imports/TV/csi-miami/csi miami 20040209 s02e16 slow burn.mp4 dst: /var/content/Imports/TV/csi-miami/csi miami 20040209 S02E16 slow burn.mp4 dst: /var/content/Imports/TV/csi-miami/Csi Miami 20040209 S02E16 Slow Burn.mp4 dst: /var/content/Imports/TV/csi-miami/Csi Miami 2004-02-09 S02E16 Slow Burn.mp4
Every Variation Incorrectly Identified:
CSI: Miami Invasion: Season 2, Episode 16
A former surf champion is murdered in his home, while his wife and son are left beaten and bound.
Incorrect Identification (Invasion was production episode 16, Slow Burn was airdate episode 16)
(This was episode 14 by airdate and episode 16 by production date)
The identification code seems to exclusively use SxxExx for episode identification. Because different content providers (via their epg data) and different metadate providers (Gracenote, etc) don't have a standardized definition of SxxExx, you encounter a mismatch possibility anytime the episode number based on airdate vs production date differ.
Using original airdate and/or episode title avoids the issue because those are both precisely defined values with little room for individual interpretation. Am certain there is some bad data associated with those, but that is different than using an ambiguously defined field. Below is the metadata for the wrongly identified episode. It is absolutely obvious its wrong as both the episode title and original airdate are incorrect.
{"ID":"576", "GroupID":"184820", "Path":"csi-miami/Csi Miami 2004-02-09 S02E16 Slow Burn.mp4", "CreatedAt":1630510648,"Duration":5,"Completed":true,"Processed":true,"Airing":{"Source":"tms", "OriginalDate":"2004-02-23", "Time":1077494400,"Duration":0,"Title":"CSI: Miami", "EpisodeTitle":"Invasion", "Summary":"A former surf champion is murdered in his home, while his wife and son are left beaten and bound.", "FullSummary":"A former surf champion is murdered in his home, while his wife and son are left beaten and bound.", "Categories":["Episode", "Series"],"Genres":["Crime drama", "Action", "Thriller", "Mystery"],"Tags":["HD", "Stereo"],"SeriesID":"184820", "ProgramID":"EP005240610040", "SeasonNumber":2,"EpisodeNumber":16,"Directors":["Felix Alcala"],"Cast":["David Caruso", "Emily Procter", "Rory Cochrane"],"ReleaseYear":2004,"Raw":{"startTime":"2004-02-23T00:00Z", "endTime":"", "duration":0,"channels":null,"stationId":"", "qualifiers":null,"ratings":null,"program":{"tmsId":"EP005240610040", "rootId":"2722937", "seriesId":"184820", "entityType":"Episode", "subType":"Series", "title":"CSI: Miami", "titleLang":"en", "episodeTitle":"Invasion", "episodeNum":16,"seasonNum":2,"releaseYear":2004,"releaseDate":"2004-02-23", "origAirDate":"2004-02-23", "descriptionLang":"en", "shortDescription":"A former surf champion is murdered in his home, while his wife and son are left beaten and bound.", "longDescription":"A former surf champion is murdered in his home, while his wife and son are left beaten and bound.", "topCast":["David Caruso", "Emily Procter", "Rory Cochrane"],"directors":["Felix Alcala"],"genres":["Crime drama", "Action", "Thriller", "Mystery"], "runTime":"PT00H45M"}}}, "UpdatedAt":1630510648642,"ImportPath":"Imports/TV", "ImportQuery":"csi-miami", "ImportGroup":"184820", "ImportedAt":1630510648418}
Below is a movie import example and demonstrates a parser issue. Whereas TV allows whitespace and some set of symbols between words, the movie year parser looks exclusively for whitespace immediately prior (but the rest of the name can use symbols). Believe the year must be in parenthesis to be recognized, but my testing was limited.
src: /var/content/Rewinds/hallmark-1989-the-shell-seekers.m3u dst: /var/content/Imports/Movies/hallmark/the-shell-seekers-1989.mp4 dst: /var/content/Imports/Movies/hallmark/the-shell-seekers-(1989).mp4 dst: /var/content/Imports/Movies/hallmark/the shell seekers-(1989).mp4 Unable to identify (First choice of Fix Incorrect Match is correct) dst: /var/content/Imports/Movies/hallmark/the-shell-seekers (1989).mp4 dst: /var/content/Imports/Movies/hallmark/the shell seekers (1989).mp4 dst: /var/content/Imports/Movies/hallmark/The Shell Seekers (1989).mp4 Correctly identified
Unfortunately, no amount of fancy renaming can work around the TV Exx issue. Seems like the fix is easy-- use the season from Sxx but start with the episode title. If no match within the season, try the original airdate. If no match, fall back to Exx and no worse off than today. There will still be a quirk in that the listed episode number may mismatch expectations, but the other metadata will be correct.
My conclusion for TV Shows after doing pretty much the same as what you did
- Gracenote data is used for matching and is inconsistent and often changes after something airs
- S##E## should not be used for matching with Gracenote (or should be last resort after Ep Title)
- Like you said, use the Episode Title for matching
My other issues is when you try to match, the results aren't sorted alphabetically (I presume they're by a search score). If they were alphabetical it would be easier to spot if you had a match available to choose.
Having used Plex before, I name
Movies: Title (Year)/Title (Year).ext and additional info goes after (Year) like Gravity (2013) -h264.ts
TV Shows: Title (Year)/Season ##/Title (Year) - S##E## - Episode Tile (OAD).ext
I haven't had any issues importing TV Series if Gracenote has data for them.
For episodic serials, I want to watch them in order by Season and Episode.
If a show isn't an episodic serial, I could care less about Season and Episode, just give me the Episode Title.
One example I have that's not a serial is Farming the Wild (2018).
It imports as unmatched and I leave it that way so I can see the Episode Titles.
If I match it using SH030187620000, I lose the Episode Titles because Gracenote doesn't identify the Seasons and Episodes https://tvlistings.zap2it.com/overview.html?programSeriesId=SH03018762&tmsId=SH030187620000
I got the Episode Title from Channels DVR when it recorded it and the Season and Episode numbers from the channel that aired it https://app.myoutdoortv.com/shows/farming-the-wild
It doesn’t even bring the episode title in when scanning. Just the episode snd season number for looking it up. So it’s not there to show.
We’ll look at consuming them title, but title isn’t a big requirement for our scanner, so it’s tricky.
I would strongly encourage you to also use original air date as the fallback for episode title (and in preference to Exx episode number). The challenge with episode title is that due to punctuation, abbreviations, etc., a good episode-title compare algorithm is tricky. By contrast, original airdate is both unambiguous and a trivial comparison.
Thanks.
If it helps I have a series pass for that show and there's some upcoming airings.
I marked these as Not Recorded so they'll record again this Saturday and next Monday.
Title "Farming the Wild"
SeriesID "15775579"
EpisodeTitle "Stalking Red Deer Hinds in Scotland"
ProgramID "EP030187620011"
OriginalDate "2020-07-13"
EpisodeTitle "Invasive Species 2. (Sika & Rabbits)"
ProgramID "EP030187620016"
OriginalDate "2020-09-28"
Argh, looks like things have changed since I originally imported that show.
I just did a test and moved the files, Prune Deleted and verified they don't appear in the library.
I then moved the files back and Scan Sources.
Now it imports them with an incorrect match.
Can we get an option to "Unmatch" please.
I resolved the issue for my needs. Not elegant, but nicely self-contained. After doing an import, I run a custom script that downloads the library episode content (http://dvr.lxc.lan:8089/dvr/files) and then performs a two pass analysis:
Pass #1: Create two lookup tables for each series that maps episode-name and air-date to the episode number. Essentially this just harvests the existing metadata that Channels previously downloaded.
Pass #2: For each mismatched episode (i.e., both the episode-name and air-date as encoded in the import filename mismatch the metadata), lookup the episode number using the lookup table from pass 1 (first via episode-name and then via air-date). For episodes with a remap correction, rename them to the new episode number. Then delete the episode (curl -s -X DELETE http://dvr.lxc.lan:8089/dvr/files/$id) and permanently delete the episode (curl -s -X PUT http://dvr.lxc.lan:8089/dvr/files/$id/permanently_delete). After all renames/deletes, trigger a rescan (curl -s -XPUT http://dvr.lxc.lan:8089//dvr/scanner/scan).
Because this uses Channels own metadata to find the mismatches and make corrections, it relies on all the mismatched episodes being imported. Since an entire season (or more) is typically imported at once, this is a non-issue in practice. By renaming the file to the updated episode number prior to the delete/permanently_delete, the latter removes the incorrect metadata without impacting the import file contents.
There is some complexity in that you need to parse the air-date and episode-name from whatever format your import files use. And of course, the "find by episode name" is only as good as whatever fuzzy string match you implement. Hope this helps someone else.
1 Like
I've been struggling with some series matches, when bringing in the first three seasons of Top Gear USA Channels insists on putting some episodes in Season 5, etc. I just renamed the episode titles out of them and they
"Top Gear USA\Season 3\S3e03 - Cult Classics.mp4"
"Top Gear USA\Season 3\S3e02 - Small Cars.mp4"
"Top Gear USA\Season 3\S3e01 - Police Cars.mp4"
turned into
"Top Gear USA\Season 3\S3e02.mp4"
"Top Gear USA\Season 3\S3e03.mp4"
"Top Gear USA\Season 3\S3e01.mp4"
This got Channels to keep them in the rights seasons on the menu, but, of course, the titles are lost now for many of the episodes, I guess Gracenote doesn't have them? Is there a way I can check?
fwiw, I only sorted those files in "Season 3" subdirectories because I read a guide article somewhere here saying that was the way to do it. Now I'm guessing that they could all be in one directory.
So now I get this and strongly favor keying on airdates and episode titles, it would fix this.
Here is an example of one that fails if the title ("RVs") is left in the filename, Channels imports this as Season 5 episode 3, as shown below.
{"ID":"1748","GroupID":"8353602","Path":"Top Gear USA/Season 3/S3e09 - RV's.mp4","CreatedAt":1631336624,"Duration":2657.551,"Completed":true,"Processed":true,"Airing":{"Source":"tms","OriginalDate":"2013-02-12","Time":1360627200,"Duration":0,"Title":"Top Gear","EpisodeTitle":"RVs","Summary":"The guys design an RV to take on a southern road trip and compete in a series of wild challenges.","FullSummary":"The guys design an RV to take on a southern road trip and compete in a series of wild challenges.","Image":"https://tmsimg.fancybits.co/assets/p10777099_b_h9_aa.jpg?w=720\u0026h=540","Categories":["Episode","Series"],"Genres":["Entertainment","Auto","Comedy"],"Tags":["HD","Stereo"],"SeriesID":"8353602","ProgramID":"EP013412100035","SeasonNumber":5,"EpisodeNumber":3,"Cast":["Adam Ferrara","Tanner Foust","Rutledge Wood"],"ReleaseYear":2013,"Raw":{"startTime":"2013-02-12T00:00Z","endTime":"","duration":0,"channels":null,"stationId":"","qualifiers":null,"ratings":null,"program":{"tmsId":"EP013412100035","rootId":"9708917","seriesId":"8353602","entityType":"Episode","subType":"Series","title":"Top Gear","titleLang":"en","episodeTitle":"RVs","episodeNum":3,"seasonNum":5,"releaseYear":2013,"releaseDate":"2013-02-12","origAirDate":"2013-02-12","descriptionLang":"en","shortDescription":"The guys design an RV to take on a southern road trip and compete in a series of wild challenges.","longDescription":"The guys design an RV to take on a southern road trip and compete in a series of wild challenges.","topCast":["Adam Ferrara","Tanner Foust","Rutledge Wood"],"genres":["Entertainment","Auto","Comedy"],"preferredImage":{"uri":"https://tmsimg.fancybits.co/assets/p10777099_b_h9_aa.jpg?w=720\u0026h=540","height":"540","width":"720","primary":"true","category":"Banner-L1","text":"yes","tier":"Season"},"runTime":"PT00H45M"}}},"UpdatedAt":1631925765106,"ImportPath":"/media/publib/TV","ImportQuery":"Top Gear USA","ImportGroup":"8353602","ImportedAt":1631925764563},
![https://tmsimg.fancybits.co/assets/p10777099_b_h9_aa.jpg?w=720\u0026h=540","Categories":["Episode","Series"],"Genres":["Entertainment","Auto","Comedy"],"Tags":["HD","Stereo"],"SeriesID":"8353602","ProgramID":"EP013412100035","SeasonNumber":5,"EpisodeNumber":3,"Cast":["Adam](https://tmsimg.fancybits.co/assets/p10777099_b_h9_aa.jpg?w=720%5Cu0026h=540%22,%22Categories%22:%5B%22Episode%22,%22Series%22%5D,%22Genres%22:%5B%22Entertainment%22,%22Auto%22,%22Comedy%22%5D,%22Tags%22:%5B%22HD%22,%22Stereo%22%5D,%22SeriesID%22:%228353602%22,%22ProgramID%22:%22EP013412100035%22,%22SeasonNumber%22:5,%22EpisodeNumber%22:3,%22Cast%22:%5B%22Adam){kind=link}
You can check the Gracenote data Channels uses for matching at www.zap2it.com
They list three Top Gear shows, click on the Episode Guide tab for each to see the episodes
When you find the right one, you can use the tmsId to match the show you imported
Top Gear America (2017) 2 Seasons https://tvlistings.zap2it.com/overview.html?programSeriesId=SH02730918&tmsId=SH027309180000
Top Gear (2002) 30 Seasons https://tvlistings.zap2it.com/overview.html?programSeriesId=SH00335172&tmsId=SH003351720000
Top Gear (2010) 8 Seasons https://tvlistings.zap2it.com/overview.html?programSeriesId=SH01341210&tmsId=SH013412100000
I'm waiting until the devs work on this Imported TV Show matching issue before importing all my shows.
Thanks! It was Top Gear (2010) 8 seasons. From looking at this, Gracenote's data is simply missing for a bunch of the episodes, so that makes sense. Still not sure what Channels matching did what it did for the rest of them.