1-week gap in fetched guide data?

dbloom · April 11, 2019, 6:21pm

I just manually told my DVR (Apple Macmini7,1, macOS version 10.14.4, DVR version 2019.04.03.1709) to re-fetch guide data, and I can't help but notice that there's a gap in the guide data it fetched between 2019-04-21 11:00AM and 2019-04-28 5:00PM. It seem like this might be a bug.

It's not a huge deal (presumably it will download that data in a few days), but it's kind of a bummer for the guide data 10-17 days in the future to be a little bit stale.

Logs:

2019/04/11 11:14:31 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-11 11:00AM
2019/04/11 11:14:33 [DVR]   indexed 885 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:14:33 [DVR]   indexed 27 movies (12 channels) [0s fetch, 0s index]
2019/04/11 11:14:34 [DVR]   pruned 102 replaced airings in 0s.
2019/04/11 11:14:34 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-11 5:00PM
2019/04/11 11:14:36 [DVR]   indexed 870 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:14:36 [DVR]   indexed 34 movies (14 channels) [0s fetch, 0s index]
2019/04/11 11:14:37 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-11 11:00PM
2019/04/11 11:14:38 [DVR]   indexed 944 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:14:39 [DVR]   indexed 18 movies (10 channels) [0s fetch, 0s index]
2019/04/11 11:14:39 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-12 5:00AM
2019/04/11 11:14:41 [DVR]   indexed 944 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:14:41 [DVR]   indexed 14 movies (8 channels) [0s fetch, 0s index]
2019/04/11 11:14:42 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-12 11:00AM
2019/04/11 11:14:43 [DVR]   indexed 884 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:14:44 [DVR]   indexed 31 movies (14 channels) [0s fetch, 0s index]
2019/04/11 11:14:44 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-12 5:00PM
2019/04/11 11:14:46 [DVR]   indexed 806 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:14:46 [DVR]   indexed 53 movies (21 channels) [0s fetch, 0s index]
2019/04/11 11:14:46 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-12 11:00PM
2019/04/11 11:14:48 [DVR]   indexed 902 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:14:48 [DVR]   indexed 36 movies (16 channels) [0s fetch, 0s index]
2019/04/11 11:14:49 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-13 5:00AM
2019/04/11 11:14:51 [DVR]   indexed 967 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:14:51 [DVR]   indexed 21 movies (11 channels) [0s fetch, 0s index]
2019/04/11 11:14:51 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-13 11:00AM
2019/04/11 11:14:53 [DVR]   indexed 797 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:14:54 [DVR]   indexed 63 movies (25 channels) [0s fetch, 0s index]
2019/04/11 11:14:54 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-13 5:00PM
2019/04/11 11:14:56 [DVR]   indexed 735 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:14:57 [DVR]   indexed 76 movies (27 channels) [0s fetch, 0s index]
2019/04/11 11:14:57 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-13 11:00PM
2019/04/11 11:14:59 [DVR]   indexed 847 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:14:59 [DVR]   indexed 44 movies (22 channels) [0s fetch, 0s index]
2019/04/11 11:14:59 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-14 5:00AM
2019/04/11 11:15:02 [DVR]   indexed 984 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:02 [DVR]   indexed 28 movies (17 channels) [0s fetch, 0s index]
2019/04/11 11:15:02 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-14 11:00AM
2019/04/11 11:15:04 [DVR]   indexed 821 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:05 [DVR]   indexed 60 movies (25 channels) [0s fetch, 0s index]
2019/04/11 11:15:05 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-14 5:00PM
2019/04/11 11:15:08 [DVR]   indexed 791 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:08 [DVR]   indexed 56 movies (22 channels) [0s fetch, 0s index]
2019/04/11 11:15:08 [DVR]   pruned 1 replaced airings in 0s.
2019/04/11 11:15:08 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-14 11:00PM
2019/04/11 11:15:10 [DVR]   indexed 932 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:11 [DVR]   indexed 34 movies (17 channels) [0s fetch, 0s index]
2019/04/11 11:15:11 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-15 5:00AM
2019/04/11 11:15:13 [DVR]   indexed 966 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:14 [DVR]   indexed 15 movies (9 channels) [0s fetch, 0s index]
2019/04/11 11:15:14 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-15 11:00AM
2019/04/11 11:15:16 [DVR]   indexed 877 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:16 [DVR]   indexed 27 movies (13 channels) [0s fetch, 0s index]
2019/04/11 11:15:16 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-15 5:00PM
2019/04/11 11:15:18 [DVR]   indexed 875 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:19 [DVR]   indexed 28 movies (12 channels) [0s fetch, 0s index]
2019/04/11 11:15:19 [DVR]   pruned 1 replaced airings in 0s.
2019/04/11 11:15:19 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-15 11:00PM
2019/04/11 11:15:21 [DVR]   indexed 961 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:21 [DVR]   indexed 15 movies (9 channels) [0s fetch, 0s index]
2019/04/11 11:15:21 [DVR]   pruned 1 replaced airings in 0s.
2019/04/11 11:15:22 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-16 5:00AM
2019/04/11 11:15:24 [DVR]   indexed 952 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:24 [DVR]   indexed 13 movies (7 channels) [0s fetch, 0s index]
2019/04/11 11:15:24 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-16 11:00AM
2019/04/11 11:15:26 [DVR]   indexed 874 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:27 [DVR]   indexed 29 movies (13 channels) [0s fetch, 0s index]
2019/04/11 11:15:27 [DVR]   pruned 1 replaced airings in 0s.
2019/04/11 11:15:27 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-16 5:00PM
2019/04/11 11:15:29 [DVR]   indexed 884 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:29 [DVR]   indexed 33 movies (13 channels) [0s fetch, 0s index]
2019/04/11 11:15:30 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-16 11:00PM
2019/04/11 11:15:32 [DVR]   indexed 966 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:32 [DVR]   indexed 18 movies (10 channels) [0s fetch, 0s index]
2019/04/11 11:15:32 [DVR]   pruned 1 replaced airings in 0s.
2019/04/11 11:15:33 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-17 5:00AM
2019/04/11 11:15:35 [DVR]   indexed 945 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:35 [DVR]   indexed 14 movies (7 channels) [0s fetch, 0s index]
2019/04/11 11:15:35 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-17 11:00AM
2019/04/11 11:15:37 [DVR]   indexed 906 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:38 [DVR]   indexed 27 movies (12 channels) [0s fetch, 0s index]
2019/04/11 11:15:38 [DVR]   pruned 3 replaced airings in 0s.
2019/04/11 11:15:38 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-17 5:00PM
2019/04/11 11:15:40 [DVR]   indexed 860 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:40 [DVR]   indexed 37 movies (15 channels) [0s fetch, 0s index]
2019/04/11 11:15:41 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-17 11:00PM
2019/04/11 11:15:42 [DVR]   indexed 943 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:43 [DVR]   indexed 23 movies (13 channels) [0s fetch, 0s index]
2019/04/11 11:15:43 [DVR]   pruned 1 replaced airings in 0s.
2019/04/11 11:15:43 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-18 5:00AM
2019/04/11 11:15:45 [DVR]   indexed 962 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:45 [DVR]   indexed 17 movies (11 channels) [0s fetch, 0s index]
2019/04/11 11:15:45 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-18 11:00AM
2019/04/11 11:15:47 [DVR]   indexed 903 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:48 [DVR]   indexed 27 movies (12 channels) [0s fetch, 0s index]
2019/04/11 11:15:48 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-18 5:00PM
2019/04/11 11:15:50 [DVR]   indexed 879 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:50 [DVR]   indexed 30 movies (15 channels) [0s fetch, 0s index]
2019/04/11 11:15:50 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-18 11:00PM
2019/04/11 11:15:52 [DVR]   indexed 961 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:52 [DVR]   indexed 20 movies (12 channels) [0s fetch, 0s index]
2019/04/11 11:15:53 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-19 5:00AM
2019/04/11 11:15:54 [DVR]   indexed 962 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:55 [DVR]   indexed 15 movies (8 channels) [0s fetch, 0s index]
2019/04/11 11:15:55 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-19 11:00AM
2019/04/11 11:15:57 [DVR]   indexed 891 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:15:57 [DVR]   indexed 29 movies (14 channels) [0s fetch, 0s index]
2019/04/11 11:15:58 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-19 5:00PM
2019/04/11 11:15:59 [DVR]   indexed 833 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:16:00 [DVR]   indexed 52 movies (23 channels) [0s fetch, 0s index]
2019/04/11 11:16:00 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-19 11:00PM
2019/04/11 11:16:01 [DVR]   indexed 928 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:16:02 [DVR]   indexed 33 movies (16 channels) [0s fetch, 0s index]
2019/04/11 11:16:02 [DVR]   pruned 1 replaced airings in 0s.
2019/04/11 11:16:02 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-20 5:00AM
2019/04/11 11:16:04 [DVR]   indexed 990 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:16:04 [DVR]   indexed 28 movies (16 channels) [0s fetch, 0s index]
2019/04/11 11:16:05 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-20 11:00AM
2019/04/11 11:16:06 [DVR]   indexed 824 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:16:07 [DVR]   indexed 61 movies (25 channels) [0s fetch, 0s index]
2019/04/11 11:16:07 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-20 5:00PM
2019/04/11 11:16:09 [DVR]   indexed 746 airings (127 channels) [0s fetch, 0s index]
2019/04/11 11:16:09 [DVR]   indexed 72 movies (29 channels) [0s fetch, 0s index]
2019/04/11 11:16:09 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-20 11:00PM
2019/04/11 11:16:11 [DVR]   indexed 841 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:16:11 [DVR]   indexed 48 movies (25 channels) [0s fetch, 0s index]
2019/04/11 11:16:12 [DVR]   pruned 1 replaced airings in 0s.
2019/04/11 11:16:12 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-21 5:00AM
2019/04/11 11:16:14 [DVR]   indexed 1015 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:16:14 [DVR]   indexed 23 movies (11 channels) [0s fetch, 0s index]
2019/04/11 11:16:14 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-21 11:00AM
2019/04/11 11:16:16 [DVR]   indexed 856 airings (127 channels) [0s fetch, 1s index]
2019/04/11 11:16:16 [DVR]   indexed 59 movies (26 channels) [0s fetch, 0s index]
2019/04/11 11:16:16 [DVR]   pruned 7 replaced airings in 0s.
2019/04/11 11:16:22 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-28 5:00PM
2019/04/11 11:16:23 [DVR]   indexed 30 airings (47 channels) [0s fetch, 0s index]
2019/04/11 11:16:23 [DVR]   indexed 18 movies (18 channels) [0s fetch, 0s index]
2019/04/11 11:16:24 [DVR]   pruned 81 replaced airings in 0s.
2019/04/11 11:16:24 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-28 11:00PM
2019/04/11 11:16:24 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:24 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-29 5:00AM
2019/04/11 11:16:24 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:25 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-29 11:00AM
2019/04/11 11:16:25 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:25 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-29 5:00PM
2019/04/11 11:16:26 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:26 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-29 11:00PM
2019/04/11 11:16:26 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:26 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-30 5:00AM
2019/04/11 11:16:26 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:27 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-30 11:00AM
2019/04/11 11:16:27 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:27 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-30 5:00PM
2019/04/11 11:16:27 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:28 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-04-30 11:00PM
2019/04/11 11:16:28 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:28 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-05-01 5:00AM
2019/04/11 11:16:28 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:28 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-05-01 11:00AM
2019/04/11 11:16:29 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:29 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-05-01 5:00PM
2019/04/11 11:16:29 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:29 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-05-01 11:00PM
2019/04/11 11:16:30 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:30 [DVR] Fetching guide data for 127 stations in USA-CA04487-X @ 2019-05-02 5:00AM
2019/04/11 11:16:30 [DVR]   indexed 0 airings (0 channels) [0s fetch, 0s index]
2019/04/11 11:16:30 [IDX] Pruned 0 expired groups from USA-CA04487-X in 327.914µs.
2019/04/11 11:16:31 [IDX] Pruned 58 expired airings from USA-CA04487-X in 191.930756ms.
2019/04/11 11:16:32 [DVR] Waiting 3h43m27.008923s until next job 1555020000-7 PBS NewsHour

tmm1 · April 11, 2019, 6:29pm

This is by design, in order to reduce the number of guide data requests per day. The next seven days of data is kept fresh and the data beyond that is not updated as often.

I agree the current implementation is a bit too simplistic and am open to improvements.

racameron · April 11, 2019, 8:08pm

Schedules Direct (another downstream provider of Gracenote's date) offers MD5 hashes of each airing. This allows a request of each day's guide data to be to compared to a previously downloaded version, and only request the changed guide data. Does Gracenote provide similar endpoints to Channels DVR? If so, it might be a way to maintain guide data as far as allowed, and only update on changes or additions rather than bulk downloading unnecessary listings.

tmm1 · April 12, 2019, 6:19pm

Unfortunately the Gracenote API that we have access to doesn't have anything like this. I think SchedulesData is either computing the hashes themselves, or using a different Gracenote product which gives them raw access to data dumps.

tmm1 · April 29, 2019, 1:43am

The guide download behavior has been improved in the v2019.04.28.2113 pre-release. The next 14 days of data will be refreshed every morning.