TV Show Metadata

For discussion related to MythTV which doesn't belong in another forum.

Moderator: Forum Moderators

Post Reply
User avatar
jfabernathy
Senior
Posts: 296
Joined: Wed Feb 18, 2015 2:37 pm
United States of America

TV Show Metadata

Post by jfabernathy » Fri Sep 25, 2020 11:42 am

I have on my server a lot of .m4v videos that originally were TV Shows recorded by
mythtv into .ts mpeg2 files and I removed the commercials and converted
them to h.264 and put them on a server using my version of the sendTVtoNAS
script viewtopic.php?f=2&t=1261

Those files are stored on the server in a directory using the program name
and file names that include the program name - title - season and episode
numbers.
i.e. The Big Bang Theory - s1e1 - Pilot.m4v

So anytime I start over with a new mythconverg database and include these
compressed recordings' directory as a Video Storage Group, I get no metadata as I expected, but any new shows I
record, edit, and compress to that same program directory get metadata on
the next day's "mythutil --scanvideos"

The interesting observation is that some of the old videos eventually get
metadata and some of the newly added videos don't.

SO my question is what is supposed to happen with the metadata. And is it
possible to somehow retrieve all the metadata for all the shows that are
NOT in the current mythDB.

User avatar
paulh
Developer
Posts: 544
Joined: Thu Feb 06, 2014 6:09 pm
Great Britain

Re: TV Show Metadata

Post by paulh » Sat Sep 26, 2020 11:27 am

As a backup for your metadata so you can always restore it you could create an *.mxml file with the same name as the video file but with the extension replaced by .mxml. That's what I do so when you rescan in Myth it will always prefer the metadata from that file before trying to look it up. That way you just have to set it once and forget about it it will always be restored.

https://www.mythtv.org/wiki/MythTV_Univ ... ata_Format

A simple example for a video called 'Dr. No (1962).mp4' would be a metadata file called 'Dr. No (1962).mxml' which could look something like this (there's some more complete examples on the wiki page above)

Code: Select all

<!DOCTYPE MythMetadataXML>
<metadata>
    <item>
        <title>Dr. No</title>
        <description>Sean Connery stars in the movie that began the world famous franchise 50 years ago. Iconic British agent James Bond heads to Jamaica to probe the death of a spy. (1962)(110 mins)

Recorded from Sky 007 HD on Friday 05 October 2012 at 08:00 pm.</description>
        <channum>303</channum>
        <chansign>Sky 007 HD</chansign>
        <startts>Fri, 5 October 2012 20:00:00</startts>
        <runtime>120</runtime>
        <runtimesecs>7200</runtimesecs>
        <categories>
            <category name="Movies" type="genre"/>
            <category name="Bond Films" type="genre"/>
        </categories>
    </item>
</metadata>


User avatar
dnalorernst
Developer
Posts: 20
Joined: Mon Feb 17, 2020 8:03 pm
Austria

Re: TV Show Metadata

Post by dnalorernst » Sat Sep 26, 2020 1:19 pm

Apparently, you have selected this option:

Code: Select all

Setup -> Media Settings -> Video Settings -> Metadata settings -> Perform metadata update after video scan
Please look at the frontend logs after doing this scan.

Then, in MythVideo, select a video that does not have good metadata, and press

Code: Select all

I -> Change Video details -> Reset Details
then

Code: Select all

I -> Change Video details -> Retrieve Details
you may get a list if the search gives multiple results, press enter on your choice.
Then wait until all data are fetched, examine the frontend logs and compare them to the logs generated after a "scan for changes".

Again, the wiki is worth to read:
https://www.mythtv.org/wiki/Video_Library
and
https://www.mythtv.org/wiki/MythVideo_File_Parsing

For the second question,
inside MythVideo, press

Code: Select all

M -> Retrieve All Details
This fetches a video again, if you previously reset its details.

You may hit by bug [Ticket]#13518[/Ticket]

which I am currently investigating.

Also, you might affected by slow response from the metadata sites:
See last commit from Mark_K, which extends the timeout to 3 minutes.
You can try this patch on your system and report

Anyway, deleting the local cache in

Code: Select all

$HOME/.mythtv/cache/pytmdb3.cache
$HOME/.mythtv/cache/tvdb_api3
is a good idea.

A note at the end:
It is worth to compare your 'title', 'season', 'episode', 'subtitle' settings with thetvdb.com for validity.

User avatar
jfabernathy
Senior
Posts: 296
Joined: Wed Feb 18, 2015 2:37 pm
United States of America

Re: TV Show Metadata

Post by jfabernathy » Sun Sep 27, 2020 10:39 am

Thanks for all the good suggestions. Here's what I found.
I do have

Code: Select all

Setup -> Media Settings -> Video Settings -> Metadata settings -> Perform metadata update after video scan
The frontend log after mythvideo M -> Scan for changes is as below:

Code: Select all

Sep 27 05:58:20 Marthas-Ubuntu mythfrontend.real: mythfrontend[137431]: I VideoScanner videoscan.cpp:181 (run) Beginning Video Scan.
Sep 27 05:58:20 Marthas-Ubuntu mythfrontend.real: mythfrontend[137431]: I VideoScanner videoscan.cpp:408 (buildFileList) buildFileList directory = myth://Videos@mythbuntu/mnt/md1/samba/public/media/test-vide
os/
Sep 27 05:58:20 Marthas-Ubuntu mythfrontend.real: mythfrontend[137431]: I VideoScanner dirscan.cpp:250 (ScanVideoDirectory) MythVideo::ScanVideoDirectory Scanning Group (myth://Videos@mythbuntu/mnt/md1/samba
/public/media/test-videos/)
Sep 27 05:58:20 Marthas-Ubuntu mythfrontend.real: mythfrontend[137431]: I VideoScanner videoscan.cpp:408 (buildFileList) buildFileList directory = myth://Videos@mythbuntu/mnt/md1/samba/public/media/tv-shows/
Sep 27 05:58:20 Marthas-Ubuntu mythfrontend.real: mythfrontend[137431]: I VideoScanner dirscan.cpp:250 (ScanVideoDirectory) MythVideo::ScanVideoDirectory Scanning Group (myth://Videos@mythbuntu/mnt/md1/samba
/public/media/tv-shows/)
This scan only finds shows added to the list that were copied to the Video Storage groups since last scan, which I do daily as a cron job.
The use of

Code: Select all

I -> Change Video details -> Retrieve Details
does retrieve data for programs that don't have it. The scan time is about 5 seconds per video. However with hundreds of videos that are missing data I'll need an automated way of doing this.
log after:

Code: Select all

Sep 27 06:17:26 Marthas-Ubuntu mythfrontend.real: mythfrontend[138113]: I VideoScanner videoscan.cpp:181 (run) Beginning Video Scan.
Sep 27 06:17:26 Marthas-Ubuntu mythfrontend.real: mythfrontend[138113]: I VideoScanner videoscan.cpp:408 (buildFileList) buildFileList directory = myth://Videos@mythbuntu/mnt/md1/samba/public/media/test-videos/
Sep 27 06:17:26 Marthas-Ubuntu mythfrontend.real: mythfrontend[138113]: I VideoScanner dirscan.cpp:250 (ScanVideoDirectory) MythVideo::ScanVideoDirectory Scanning Group (myth://Videos@mythbuntu/mnt/md1/samba/public/media/test-videos/)
Sep 27 06:17:26 Marthas-Ubuntu mythfrontend.real: mythfrontend[138113]: I VideoScanner videoscan.cpp:408 (buildFileList) buildFileList directory = myth://Videos@mythbuntu/mnt/md1/samba/public/media/tv-shows/
Sep 27 06:17:26 Marthas-Ubuntu mythfrontend.real: mythfrontend[138113]: I VideoScanner dirscan.cpp:250 (ScanVideoDirectory) MythVideo::ScanVideoDirectory Scanning Group (myth://Videos@mythbuntu/mnt/md1/samba/public/media/tv-shows/)
Sep 27 06:17:38 Marthas-Ubuntu mythfrontend.real: mythfrontend[138113]: E CoreContext xmlparsebase.cpp:367 (ParseChildren) XMLParseBase: Parent is NULL
Sep 27 06:17:45 Marthas-Ubuntu mythfrontend.real: mythfrontend[138113]: I MetadataDownload metadatagrabber.cpp:424 (RunGrabber) Running Grabber: /usr/share/mythtv/metadata/Television/ttvdb.py -l en -a US -N The Big Bang Theory The Fuzzy Boots Corollary
Sep 27 06:17:47 Marthas-Ubuntu mythfrontend.real: mythfrontend[138113]: I MetadataDownload metadatacommon.cpp:1141 (ParseMetadataItem) Result Found, Season 1 Episode 3
Sep 27 06:17:48 Marthas-Ubuntu mythfrontend.real: mythfrontend[138113]: I MetadataDownload metadatagrabber.cpp:424 (RunGrabber) Running Grabber: /usr/share/mythtv/metadata/Television/ttvdb.py -l en -a US -N 80379 The Fuzzy Boots Corollary
Sep 27 06:17:50 Marthas-Ubuntu mythfrontend.real: mythfrontend[138113]: I MetadataDownload metadatacommon.cpp:1141 (ParseMetadataItem) Result Found, Season 1 Episode 3
Sep 27 06:17:50 Marthas-Ubuntu mythfrontend.real: mythfrontend[138113]: I MetadataDownload metadatadownload.cpp:183 (run) Returning Metadata Results: The Big Bang Theory 1 3
Sep 27 06:17:50 Marthas-Ubuntu mythfrontend.real: mythfrontend[138113]: I MetadataImageDownload metadataimagedownload.cpp:236 (run) Metadata Image Download: http://thetvdb.com/banners/series/80379/episodes/5ed5426665558.jpg -> myth://Screenshots@mythbuntu/The Big Bang Theory Season 1x3_screenshot.jpg
next I tried M -> Retrieve All Details and nothing seems to happen. So I then tried

Code: Select all

I -> Change Video details -> Retrieve Details
and it updated the one video. Log for both test is:

Code: Select all

Sep 27 06:31:37 Marthas-Ubuntu mythfrontend.real: mythfrontend[138397]: E CoreContext xmlparsebase.cpp:367 (ParseChildren) XMLParseBase: Parent is NULL
Sep 27 06:31:42 Marthas-Ubuntu mythfrontend.real: mythfrontend[138397]: I MetadataDownload metadatagrabber.cpp:424 (RunGrabber) Running Grabber: /usr/share/mythtv/metadata/Television/ttvdb.py -l en -a US -N The Big Bang Theory The Luminous Fish Effect
Sep 27 06:31:44 Marthas-Ubuntu mythfrontend.real: mythfrontend[138397]: I MetadataDownload metadatacommon.cpp:1141 (ParseMetadataItem) Result Found, Season 1 Episode 4
Sep 27 06:31:44 Marthas-Ubuntu mythfrontend.real: mythfrontend[138397]: I MetadataDownload metadatagrabber.cpp:424 (RunGrabber) Running Grabber: /usr/share/mythtv/metadata/Television/ttvdb.py -l en -a US -N 80379 The Luminous Fish Effect
Sep 27 06:31:46 Marthas-Ubuntu mythfrontend.real: mythfrontend[138397]: I MetadataDownload metadatacommon.cpp:1141 (ParseMetadataItem) Result Found, Season 1 Episode 4
Sep 27 06:31:46 Marthas-Ubuntu mythfrontend.real: mythfrontend[138397]: I MetadataDownload metadatadownload.cpp:183 (run) Returning Metadata Results: The Big Bang Theory 1 4
Sep 27 06:31:46 Marthas-Ubuntu mythfrontend.real: mythfrontend[138397]: I MetadataImageDownload metadataimagedownload.cpp:236 (run) Metadata Image Download: http://thetvdb.com/banners/series/80379/episodes/5ed540776e5fb.jpg -> myth://Screenshots@mythbuntu/The Big Bang Theory Season 1x4_screenshot.jpg
Maybe the one error at the start is a key?

User avatar
jfabernathy
Senior
Posts: 296
Joined: Wed Feb 18, 2015 2:37 pm
United States of America

Re: TV Show Metadata

Post by jfabernathy » Sun Sep 27, 2020 12:15 pm

I've been looking thru log files trying to find something. I did not see anything in either the frontend or backend logs that even acknowledge anything in response to M -> Retrieve All Details
I do find in the frontend log some data when I retrieve the details for an individual video that has no details. I get the details and the log has:

Code: Select all

2020-09-27 08:10:16.097262 E  XMLParseBase: Parent is NULL
2020-09-27 08:10:18.713948 I  Running Grabber: /usr/share/mythtv/metadata/Television/ttvdb.py -l en -a US -N The Big Bang Theory The Hawking Excitation
2020-09-27 08:10:20.815880 I  Result Found, Season 5 Episode 21
2020-09-27 08:10:21.321295 I  Running Grabber: /usr/share/mythtv/metadata/Television/ttvdb.py -l en -a US -N 80379 The Hawking Excitation
2020-09-27 08:10:23.069252 I  Result Found, Season 5 Episode 21
2020-09-27 08:10:23.069662 I  Returning Metadata Results: The Big Bang Theory 5 21
2020-09-27 08:10:23.076236 I  Metadata Image Download: http://thetvdb.com/banners/series/80379/episodes/5eb0b2cece018.jpg -> myth://Screenshots@mythbuntu/The Big Bang Theory Season 5x21_screenshot.jpg
I do see one warning in the backend log related to that event:

Code: Select all

Sep 27 07:57:39 mythbuntu mythbackend: mythbackend[1565]: W ProcessRequest ringbuffer.cpp:664 (Start) RingBuf(/mnt/md0/screenshot//The Big Bang Theory Season 11x18_screenshot.jpg): Not starting read ahead thread, this is a write only RingBuffer

User avatar
jfabernathy
Senior
Posts: 296
Joined: Wed Feb 18, 2015 2:37 pm
United States of America

Re: TV Show Metadata

Post by jfabernathy » Sun Sep 27, 2020 10:04 pm

paulh wrote:
Sat Sep 26, 2020 11:27 am
As a backup for your metadata so you can always restore it you could create an *.mxml file with the same name as the video file but with the extension replaced by .mxml. That's what I do so when you rescan in Myth it will always prefer the metadata from that file before trying to look it up. That way you just have to set it once and forget about it it will always be restored.

https://www.mythtv.org/wiki/MythTV_Univ ... ata_Format

A simple example for a video called 'Dr. No (1962).mp4' would be a metadata file called 'Dr. No (1962).mxml' which could look something like this (there's some more complete examples on the wiki page above)

Code: Select all

<!DOCTYPE MythMetadataXML>
<metadata>
    <item>
        <title>Dr. No</title>
        <description>Sean Connery stars in the movie that began the world famous franchise 50 years ago. Iconic British agent James Bond heads to Jamaica to probe the death of a spy. (1962)(110 mins)

Recorded from Sky 007 HD on Friday 05 October 2012 at 08:00 pm.</description>
        <channum>303</channum>
        <chansign>Sky 007 HD</chansign>
        <startts>Fri, 5 October 2012 20:00:00</startts>
        <runtime>120</runtime>
        <runtimesecs>7200</runtimesecs>
        <categories>
            <category name="Movies" type="genre"/>
            <category name="Bond Films" type="genre"/>
        </categories>
    </item>
</metadata>

I may be missing something, but how do you create the files once all your metadata is created and up to date? I have manually gone through all on one program's episodes and gotten their metadata retrieved. Is there a secret script somewhere that lets you export the metadata from mythtv and create the .mxml files?

User avatar
dnalorernst
Developer
Posts: 20
Joined: Mon Feb 17, 2020 8:03 pm
Austria

Re: TV Show Metadata

Post by dnalorernst » Tue Oct 13, 2020 7:09 pm

Please try and check the attached script fits to your needs.

Roland
Attachments
export_to_mxml.zip
Initial version to export metadata into mxml files
(2.38 KiB) Downloaded 6 times

User avatar
jfabernathy
Senior
Posts: 296
Joined: Wed Feb 18, 2015 2:37 pm
United States of America

Re: TV Show Metadata

Post by jfabernathy » Tue Oct 13, 2020 11:39 pm

Thanks, I'll play with it.

Post Reply