What is this new experimental feature? Any user facing functionality?
This is mostly a backend change, but should result in smaller search index on disk which will allow us to add more fields to the database in the future.
To try it out, check the checkbox then go to Guide Database > Maintenance > Delete and Re-create
Please note the size of the database before and after, and then check again in a few days to see how the size compares to what it was before.
The new index may also help with issues like Daily loading of EPG guide data using ~50% CPU, normal?
My results:
~83k listings
before: 644mb
after: 330mb
second day: 442mb
I'll try this out this weekend on both my TVE device NAS and PRIME device NAS.
Right now running 2019.08.17.1650 on both.
TVE
6,832 Shows
58,492 Listings
767.33 MB on disk
PRIME
8,064 Shows
64,145 Listings
867.46 MB on disk
Is 'Defragment Database' something that should be performed periodically?
No not really. The new backend does not need defragment anymore.
Setup the new guide database and deleted and recreated the guide of both my NAS's.
Is there anyway to tell when the morning guide refreshes will kick off?
I would like to monitor the CPU usage on each NAS while the morning guide update is running.
Is it 09:00 + RANDOM() MOD 60 ?
Yep, but with MOD 120
I assume it's calculated daily at or before 09:00.
Any way to peek at the value, so I don't have to stare at a screen for RANDOM() mod 120 minutes tomorrow morning waiting for it to happen?
It's now RANDOM mod 120 = 91 minutes and no update on either NAS yet.
Hmm, there's no easy way to see. I'll look at adding that into the Guide Database section of the web UI.
If you just want to look at CPU usage, you can trigger a guide download manually from the maintenance dropdown.
Before: 400MB
After: 185MB
I'm back up to 511mb after this morning's update. So perhaps not such a big difference over time..
Wondering if you have logic built-in that says if a guide refresh was performed within the last (some amount of time), do not fetch new guide data again.
I installed version 2019.08.23.0024 this morning on both my NAS's and enabled the Experimental > New Guide Database.
Then clicked Guide Database > Maintenance > Delete and Recreate Database.
This was a little after 8am this morning.
I then waited until after 11am thinking the morning guide refresh would kickoff [ 09:00 + ( RANDOM() mod 120 ) ], but it didn't.
So, I clicked Guide Database > Maintenance > Fetch Guide Updates.
It only took a few seconds and didn't result in any new guide data downloading, only pruning of expired groups and airings.
Waiting until tomorrow morning to see if the daily morning refresh happens between 09:00 and 11:00.
P.S.
Old NAS used for TVE
6,832 Shows, 55,191 Listings, 403.16 MB on disk (pre-Experimental)
4,480 Shows, 54,809 Listings, 256.85 MB on disk (post-Experimental)
New NAS used for HDHRPrime
8,064 Shows, 60,479 Listings, 454.40 MB on disk (pre-Experimental)
5,832 Shows, 60,358 Listings, 256.50 MB on disk (post-Experimental)
P.P.S. Makes me wonder where all those shows and airings went. Must be the Summer TV doldrums.
There are two options available in the drop down. One fetches only if the lineup changed and there was no update in the past hour, the other forces a refresh no matter what.
If you force refresh of either kind before 9am, I think the next auto update gets scheduled for tomorrow.
With the new version it's taking less time to refresh the guide each morning.
Didn't get a chance to see CPU usage this morning.
TVE NAS
1 Lineup, 6,832 Shows, 55,191 Listings, 403.16 MB on disk (OLD Guide Database 08/23) last update duration 9:11
1 Lineup, 4,480 Shows, 54,809 Listings, 256.85 MB on disk (DAY 1 NEW Guide Database 08/23) update duration 4:51
1 Lineup, 4,675 Shows, 57,837 Listings, 348.17 MB on disk (DAY 2 NEW Guide Database 08/24) update duration 4:46
1 Lineup, 4,780 Shows, 56,343 Listings, 306.89 MB on disk (DAY 3 NEW Guide Database 08/25) update duration 4:36
1 Lineup, 4,889 Shows, 56,889 Listings, 493.41 MB on disk (DAY 4 NEW Guide Database 08/26) update duration 4:36
2019/08/26 12:26:31 [IDX] Optimized guide database for X-TVE in 3.598µs.
1 Lineup, 4,889 Shows, 54,801 Listings, 245.85 MB on disk (DAY 4 After Defragment Database 08/26)
1 Lineup, 5,023 Shows, 58,271 Listings, 329.90 MB on disk (DAY 5 NEW Guide Databsae 08/27) update duration 4:48
1 Lineup, 5,164 Shows, 57,977 Listings, 566.48 MB on disk (DAY 6 NEW Guide Databsae 08/28) update duration 4:52
2019/08/28 17:12:49 [IDX] Optimized guide database for X-TVE in 5.094µs.
1 Lineup, 5,164 Shows, 54,457 Listings, 257.07 MB on disk (DAY 6 After Defragment Database 08/28)
1 Lineup, 5,368 Shows, 58,162 Listings, 330.44 MB on disk (DAY 7 NEW Guide Databsae 08/29) update duration 4:52
1 Lineup, 5,517 Shows, 58,187 Listings, 344.99 MB on disk (DAY 8 NEW Guide Databsae 08/30) update duration 4:47
1 Lineup, 5,656 Shows, 56,894 Listings, 276.10 MB on disk (DAY 9 NEW Guide Databsae 08/31) update duration 4:53
1 Lineup, 5,716 Shows, 57,903 Listings, 406.19 MB on disk (DAY 10 NEW Guide Databsae 09/01) update duration 4:35
1 Lineup, 5,790 Shows, 58,169 Listings, 396.83 MB on disk (DAY 11 NEW Guide Databsae 09/02) update duration 4:43
1 Lineup, 5,884 Shows, 58,323 Listings, 339.94 MB on disk (DAY 12 NEW Guide Databsae 09/03) update duration 4:49
1 Lineup, 6,005 Shows, 57,742 Listings, 297.18 MB on disk (DAY 13 NEW Guide Databsae 09/04) update duration 4:58
1 Lineup, 6,103 Shows, 58,260 Listings, 373.83 MB on disk (DAY 14 NEW Guide Databsae 09/05) update duration 4:49
1 Lineup, 6,258 Shows, 58,060 Listings, 320.24 MB on disk (DAY 15 NEW Guide Databsae 09/06) update duration 4:52
1 Lineup, 6,391 Shows, 58,045 Listings, 515.56 MB on disk (DAY 16 NEW Guide Databsae 09/07) update duration 4:54
1 Lineup, 6,455 Shows, 56,503 Listings, 316.03 MB on disk (DAY 17 NEW Guide Databsae 09/08) update duration 4:46
1 Lineup, 6,509 Shows, 54,490 Listings, 296.22 MB on disk (DAY 18 ReDownloaded Guide 09/09) update duration 4:47
1 Lineup, 6,604 Shows, 57,890 Listings, 315.47 MB on disk (DAY 19 NEW Guide Databsae 09/10) update duration 5:10
1 Lineup, 6,718 Shows, 57,764 Listings, 367.56 MB on disk (DAY 20 NEW Guide Databsae 09/11) update duration 4:53
1 Lineup, 6,897 Shows, 57,877 Listings, 618.52 MB on disk (DAY 22 NEW Guide Databsae 09/13) update duration 4:55
1 Lineup, 6,992 Shows, 57,763 Listings, 327.15 MB on disk (DAY 23 NEW Guide Databsae 09/14) update duration 5:02
1 Lineup, 7,080 Shows, 56,967 Listings, 305.90 MB on disk (DAY 24 NEW Guide Databsae 09/15) update duration 6:15 during comskip run
1 Lineup, 7,180 Shows, 57,982 Listings, 382.86 MB on disk (DAY 25 NEW Guide Databsae 09/16) update duration 4:47
1 Lineup, 7,281 Shows, 58,672 Listings, 300.93 MB on disk (DAY 26 NEW Guide Databsae 09/17) update duration 4:56
HDHR Prime NAS
1 Lineup, 8,064 Shows, 60,479 Listings, 454.40 MB on disk (OLD Guide Database 08/23) last update duration 4:21
1 Lineup, 5,832 Shows, 60,358 Listings, 256.50 MB on disk (DAY 1 NEW Guide Database 08/23) update duration 2:58
1 Lineup, 6,037 Shows, 63,782 Listings, 362.34 MB on disk (DAY 2 NEW Guide Database 08/24) update duration 2:42
1 Lineup, 6,157 Shows, 62,145 Listings, 360.19 MB on disk (DAY 3 NEW Guide Database 08/25) update duration 2:38
1 Lineup, 6,277 Shows, 63,044 Listings, 322.70 MB on disk (DAY 4 NEW Guide Database 08/26) update duration 2:42
2019/08/26 12:25:12 [IDX] Optimized guide database for USA-CA54023-X in 3.006µs.
1 Lineup, 6,277 Shows, 60,564 Listings, 322.70 MB on disk (DAY 4 After Defragment Database 08/26)
1 Lineup, 6,424 Shows, 64,136 Listings, 419.50 MB on disk (DAY 5 NEW Guide Databsae 08/27) update duration 2:37
1 Lineup, 6,565 Shows, 64,380 Listings, 349.82 MB on disk (DAY 6 NEW Guide Databsae 08/28) update duration 2:44
2019/08/28 17:16:23 [IDX] Optimized guide database for USA-CA54023-X in 3.723µs.
1 Lineup, 6,565 Shows, 60,532 Listings, 349.82 MB on disk (DAY 6 After Defragment Database 08/28)
1 Lineup, 6,749 Shows, 64,355 Listings, 376.25 MB on disk (DAY 7 NEW Guide Databsae 08/29) update duration 2:40
1 Lineup, 6,905 Shows, 64,485 Listings, 479.75 MB on disk (DAY 8 NEW Guide Databsae 08/30) update duration 2:46
1 Lineup, 7,045 Shows, 62,968 Listings, 390.87 MB on disk (DAY 9 NEW Guide Databsae 08/31) update duration 2:40
1 Lineup, 7,120 Shows, 63,024 Listings, 330.09 MB on disk (DAY 10 NEW Guide Databsae 09/01) update duration 2:32
1 Lineup, 7,207 Shows, 63,375 Listings, 408.49 MB on disk (DAY 11 NEW Guide Databsae 09/02) update duration 2:41
1 Lineup, 7,298 Shows, 63,217 Listings, 342.24 MB on disk (DAY 12 NEW Guide Databsae 09/03) update duration 2:36
1 Lineup, 7,455 Shows, 62,721 Listings, 418.76 MB on disk (DAY 13 NEW Guide Databsae 09/04) update duration 2:34
1 Lineup, 7,595 Shows, 63,388 Listings, 346.37 MB on disk (DAY 14 NEW Guide Databsae 09/05) update duration 2:35
1 Lineup, 7,752 Shows, 63,233 Listings, 439.75 MB on disk (DAY 15 NEW Guide Databsae 09/06) update duration 2:37
1 Lineup, 7,889 Shows, 63,099 Listings, 392.75 MB on disk (DAY 16 NEW Guide Databsae 09/07) update duration 2:34
1 Lineup, 7,969 Shows, 63,043 Listings, 306.32 MB on disk (DAY 17 NEW Guide Databsae 09/08) update duration 2:33
1 Lineup, 8,034 Shows, 63,259 Listings, 385.25 MB on disk (DAY 18 ReDownloaded Guide 09/09) update duration 2:34
1 Lineup, 8,147 Shows, 62,379 Listings, 298.50 MB on disk (DAY 19 NEW Guide Databsae 09/10) update duration 2:39
1 Lineup, 8,147 Shows, 62,379 Listings, 298.50 MB on disk (DAY 19 NEW Guide Databsae 09/10) update duration 2:39
1 Lineup, 8,245 Shows, 63,217 Listings, 318.59 MB on disk (DAY 20 NEW Guide Databsae 09/11) update duration 2:36
1 Lineup, 8,479 Shows, 63,110 Listings, 385.54 MB on disk (DAY 22 NEW Guide Databsae 09/13) update duration 2:42
1 Lineup, 8,589 Shows, 62,809 Listings, 294.62 MB on disk (DAY 23 NEW Guide Databsae 09/14) update duration 2:37
1 Lineup, 8,663 Shows, 62,776 Listings, 328.39 MB on disk (DAY 24 NEW Guide Databsae 09/15) update duration 2:41
1 Lineup, 8,771 Shows, 62,146 Listings, 331.37 MB on disk (DAY 25 NEW Guide Databsae 09/16) update duration 2:32
1 Lineup, 8,878 Shows, 63,174 Listings, 322.51 MB on disk (DAY 26 NEW Guide Databsae 09/17) update duration 6:25
Day 4 database size increased quite a bit for my TVE NAS, but decreased on my HDHR Prime NAS.
This new version updates much quicker than the old one.
Just did a Defragment Database on my TVE NAS
2019/08/26 12:26:31 [IDX] Optimized guide database for X-TVE in 3.598µs.
1 Lineup, 4,889 Shows, 56,889 Listings, 493.41 MB on disk (DAY 4 Before Defragment Database 08/26)
1 Lineup, 4,889 Shows, 54,801 Listings, 245.85 MB on disk (DAY 4 After Defragment Database 08/26)
and on my HDHR Prime NAS
2019/08/26 12:25:12 [IDX] Optimized guide database for USA-CA54023-X in 3.006µs.
1 Lineup, 6,277 Shows, 63,044 Listings, 322.70 MB on disk (DAY 4 Before Defragment Database 08/26)
1 Lineup, 6,277 Shows, 60,564 Listings, 322.70 MB on disk (DAY 4 After Defragment Database 08/26)
Defragmented again on Day 6 and it reduced the database size on my TVE NAS, but not the HDHDR Prime NAS.
Day 9 shows database size is shrinking on its own without me doing a defragment.
Day 10 shows database size growing on the TVE NAS and shrinking on the HDHR Prime NAS, both having more shows and listings than Day 9??
Still faster daily guide updates than the old version.
How I calculate update duration, or how long it takes Channels DVR to update the guide info each morning between 09:00 and 11:00.
STARTED DAILY GUIDE REFRESH 2019/09/06 09:44:00
2019/09/05 18:58:28 [SYS] Removing old backup backup-20190810.234112
2019/09/06 09:44:00 [DVR] Fetching guide data for 120 stations in X-TVE @ 2019-09-06 9:30AM
2019/09/06 09:44:04 [DVR] indexed 851 airings (120 channels) [0s fetch, 3s index]
2019/09/06 09:44:05 [DVR] indexed 58 movies (25 channels) [0s fetch, 0s index]
2019/09/06 09:44:05 [DVR] pruned 40 replaced airings in 0s.
---
2019/09/06 09:48:47 [DVR] Fetching guide data for 120 stations in X-TVE @ 2019-09-22 3:30AM
2019/09/06 09:48:51 [DVR] indexed 947 airings (120 channels) [0s fetch, 3s index]
2019/09/06 09:48:52 [DVR] indexed 40 movies (22 channels) [0s fetch, 0s index]
2019/09/06 09:48:52 [IDX] Pruned 0 expired groups from X-TVE in 8.388963ms.
2019/09/06 09:48:53 [IDX] Pruned 3365 expired airings from X-TVE in 410.752024ms.
ENDED 09:48:53 - 09:44:00 = 00:04:53 update duration
Was able to kickoff a guide database rebuild and watch the CPU usage.
TVE NAS (CPU AVG 50% CPU PEAK 71% 4.5-5 mins) Synology DS1513+ Intel Atom Quad Core D2701 @ 2.13GHz
HDHR Prime NAS (CPU AVG 40% CPU PEAK 53% 2.5-3 mins) Synology DS1019+ Intel Celeron Quad Core J3455 @ 1.50GHz
1 Like
My DB has grown to 274MB in 3 days
20190823-1321
-------------
2 Lineups
5,668 Shows
61,377 Listings
323.17 MB on disk
20190827-1033
-------------
2 Lineups
6,157 Shows
61,989 Listings
324.18 MB on diskUpdated my post above to show Day 26 stats and CPU usage.
I've been collecting and reporting info on this for 26 days now.
Is this now in the new versions, or are you still tweaking it?
It's not default yet but we are confident with the results and will be making it default in the near future.
1 Like