TiVo Community
TiVo Community
TiVo Community
Go Back   TiVo Community > Main TiVo Forums > TiVo Home Media Features & TiVoToGo
TiVo Community
Reply
Forum Jump
 
Thread Tools
Old 07-25-2012, 03:43 PM   #1
Dark Helmet
Registered User
 
Join Date: Sep 2006
Location: Northern Virginia
Posts: 132
Questions about TS streams and TiVo metadata

Hi, I've been working on my own program to streamline my conversion process from my TiVo to iTunes on my Mac, since there isn't a great one-stop solution that does everything I need. I've not been following things lately regarding the latest state of the art on the TiVo front, so I have a few questions about things:
  • As I understand it someone was working on decoding Transport Streams from the TiVo for tivodecode, but it seems like that effort has been abandoned (when I tried that out it didn't work on the streams I tested). Is that correct? And as a related question ... is tivodecode development dead?
  • Is there a description of the metadata format used by the TiVo? I know it's XML, I was wondering if someone had a description I see that there are two "chunk" files that tivodecode produces and they have different information in them. A lot of that is relatively easy to understand but I haven't really seen a reference to what it all means. For example, I gather that "episodeNumber" is really a combination of season number and the actual episode number; is that right?
  • In a related note, I've seen at least one example where the TiVo knows the season and episode number of a show, but there's no corresponding episodeNumber in the metadata. Any idea where the TiVo is storing that information at? I was just interested in trying to automate as much metadata entry as I can.

Thanks for any help you can give me!
Dark Helmet is offline   Reply With Quote
Old 07-25-2012, 05:03 PM   #2
ThAbtO
TiVoholic by the bay
 
ThAbtO's Avatar
 
Join Date: Apr 2000
Location: SF Bay Area
Posts: 4,982
TivoDecode really does not work well with TS as with the regular PS streams. I use KMTTG to get metadata and it is in .tivo.txt or .mpg.txt. text files and only about 4 items are used by the Tivo, namely for title, description etc., everything else that was extracted and re-intracted from the .txt files was ignored by the Tivo.

title:
seriesTitle:
description:
episodeTitle:

I can edit these items and they can show up on the Tivo for the video.
__________________
Series 2 (PLS, sold)
TiVo HD 1TB (157 HD Hrs, PLS, "652")
Roamio! Oh, wait, Its Juliet!
OTA! OTA! OTA!

~~~~~

The TiVo Apocalypse begins on Sat 2013 Feb 16 00:00 GMT!
ThAbtO is offline   Reply With Quote
Old 07-25-2012, 07:33 PM   #3
lpwcomp
Registered User
 
Join Date: May 2002
Location: John's Creek, Georgia
Posts: 5,538
Quote:
Originally Posted by ThAbtO View Post
TivoDecode really does not work well with TS as with the regular PS streams. I use KMTTG to get metadata and it is in .tivo.txt or .mpg.txt. text files and only about 4 items are used by the Tivo, namely for title, description etc., everything else that was extracted and re-intracted from the .txt files was ignored by the Tivo.

title:
seriesTitle:
description:
episodeTitle:

I can edit these items and they can show up on the Tivo for the video.
Why on earth do you state that the rest of the metadata is ignored? It most certainly is not.
__________________
James L. Sutherland
"You know TiVo users. Bunch of b****y little girls" TiVoAxe

"Jessica Fletcher visits Midsomer. Carnage ensues!"
lpwcomp is online now   Reply With Quote
Old 07-25-2012, 08:33 PM   #4
lpwcomp
Registered User
 
Join Date: May 2002
Location: John's Creek, Georgia
Posts: 5,538
Quote:
Originally Posted by Dark Helmet View Post
For example, I gather that "episodeNumber" is really a combination of season number and the actual episode number; is that right?
Not really. It's just a number. Some series do format it that way in the TiVo metadata and it is the standard way for user generated metadata.
Quote:
Originally Posted by Dark Helmet View Post
In a related note, I've seen at least one example where the TiVo knows the season and episode number of a show, but there's no corresponding episodeNumber in the metadata. Any idea where the TiVo is storing that information at? I was just interested in trying to automate as much metadata entry as I can.
If you're talking about what is displayed by the Premiere HDUI, nobody really knows. I have a theory but have not been able to either confirm or disprove it.

What I do know is that the the last portion of the url for an individual episode page at zap2it is the <program> level <uniqueId> in the TiVo XML. The page for a season has a piece for each episode with season #, episode #, and that uniqueId. I had code at one time to get the uniqueID for an episode based on season and episode #. It would be a simple matter to do it in the other direction. What I couldn't figure out was how to navigate to the page for anything other than the most recent season from within a python program.
__________________
James L. Sutherland
"You know TiVo users. Bunch of b****y little girls" TiVoAxe

"Jessica Fletcher visits Midsomer. Carnage ensues!"
lpwcomp is online now   Reply With Quote
Old 07-26-2012, 04:02 AM   #5
wmcbrine
Resistance Useless
 
wmcbrine's Avatar
 
Join Date: Aug 2003
Posts: 9,154
Quote:
Originally Posted by lpwcomp View Post
Why on earth do you state that the rest of the metadata is ignored? It most certainly is not.
He probably only uses push.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
wmcbrine is offline   Reply With Quote
Old 07-26-2012, 04:27 AM   #6
ggieseke
Registered User
 
Join Date: May 2008
Posts: 3,063
Quote:
Originally Posted by wmcbrine View Post
He probably only uses push.
Or a Premiere.
ggieseke is offline   Reply With Quote
Old 07-26-2012, 05:37 AM   #7
wmcbrine
Resistance Useless
 
wmcbrine's Avatar
 
Join Date: Aug 2003
Posts: 9,154
No, a Premiere will transfer all the fields.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
wmcbrine is offline   Reply With Quote
Old 07-26-2012, 10:19 AM   #8
lpwcomp
Registered User
 
Join Date: May 2002
Location: John's Creek, Georgia
Posts: 5,538
Using the <program><uniqueId> to obtain season & episode # should be easy. Just get the xml for the url of form
Code:
http://tvlistings.zap2it.com/tv/<series name>/<program><uniqueid>
where <series name> is all lower case with "-" replacing blanks.

for instance,

Code:
http://tvlistings.zap2it.com/tv/the-big-bang-theory/EP009311820069
is the url for season 4, episode 2 of "The Big Bang Theory".
__________________
James L. Sutherland
"You know TiVo users. Bunch of b****y little girls" TiVoAxe

"Jessica Fletcher visits Midsomer. Carnage ensues!"
lpwcomp is online now   Reply With Quote
Old 07-26-2012, 10:08 PM   #9
Dark Helmet
Registered User
 
Join Date: Sep 2006
Location: Northern Virginia
Posts: 132
Quote:
Originally Posted by lpwcomp View Post
Not really. It's just a number. Some series do format it that way in the TiVo metadata and it is the standard way for user generated metadata.
Yeah, that's what I'm talking about. Ok, thanks for that tip!

Quote:
If you're talking about what is displayed by the Premiere HDUI, nobody really knows. I have a theory but have not been able to either confirm or disprove it.
Okay, I just wanted to be sure I wasn't missing something there.

Quote:
What I do know is that the the last portion of the url for an individual episode page at zap2it is the <program> level <uniqueId> in the TiVo XML. The page for a season has a piece for each episode with season #, episode #, and that uniqueId. I had code at one time to get the uniqueID for an episode based on season and episode #. It would be a simple matter to do it in the other direction. What I couldn't figure out was how to navigate to the page for anything other than the most recent season from within a python program.
I see the uniqueID is included in the second chunk file ... and that works with zap2it!

But I am a bit confused ... you say in your later post to "get the xml for the url of the form ...". That URL you give me has already been formatted into HTML.

Hm, okay, some googling around makes me think that the answer to that is:
  • zap2it used to provide XML data, but doesn't anymore
  • There are a number of screen scrapers that convert the web page to XML format
  • That seems to be the only option, free or paid.

Is that all correct? I actually don't mind paying for a feed (depending on the cost), because I sure do hate entering in metadata.
Dark Helmet is offline   Reply With Quote
Old 07-26-2012, 10:33 PM   #10
moyekj
Registered User
 
Join Date: Jan 2006
Location: Mission Viejo, CA
Posts: 9,299
Quote:
Originally Posted by Dark Helmet View Post
  • In a related note, I've seen at least one example where the TiVo knows the season and episode number of a show, but there's no corresponding episodeNumber in the metadata. Any idea where the TiVo is storing that information at? I was just interested in trying to automate as much metadata entry as I can.
The HDUI gathers extra information from TiVo servers on the fly to display program information - such information is not available when TiVo servers are down. That's part of the reason it makes a zillion nslookups when browsing program info.
__________________
Roamio Pro, Elite, Premiere
Cox - Motorola CableCards & TAs
Slingbox 350 via TiVo Mini & TiVo Stream for remote viewing

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
moyekj is offline   Reply With Quote
Old 07-27-2012, 01:18 AM   #11
lpwcomp
Registered User
 
Join Date: May 2002
Location: John's Creek, Georgia
Posts: 5,538
Quote:
Originally Posted by Dark Helmet View Post
  • zap2it used to provide XML data, but doesn't anymore
  • There are a number of screen scrapers that convert the web page to XML format
  • That seems to be the only option, free or paid.

Is that all correct? I actually don't mind paying for a feed (depending on the cost), because I sure do hate entering in metadata.
It probably always has been HTML, but it doesn't really matter. All you need to do is get the source for the page. You should then be able to search and parse it as a text string.
__________________
James L. Sutherland
"You know TiVo users. Bunch of b****y little girls" TiVoAxe

"Jessica Fletcher visits Midsomer. Carnage ensues!"
lpwcomp is online now   Reply With Quote
Old 07-27-2012, 09:26 AM   #12
Dark Helmet
Registered User
 
Join Date: Sep 2006
Location: Northern Virginia
Posts: 132
Quote:
Originally Posted by moyekj View Post
The HDUI gathers extra information from TiVo servers on the fly to display program information - such information is not available when TiVo servers are down. That's part of the reason it makes a zillion nslookups when browsing program info.
Ahhhh ... So do they actually retrieve the information via DNS? Or is it via some HTTP-based protocol that normal mortals can't use?
Dark Helmet is offline   Reply With Quote
Old 07-27-2012, 09:31 AM   #13
Dark Helmet
Registered User
 
Join Date: Sep 2006
Location: Northern Virginia
Posts: 132
Quote:
Originally Posted by lpwcomp View Post
It probably always has been HTML, but it doesn't really matter. All you need to do is get the source for the page. You should then be able to search and parse it as a text string.
I've done that for other things, but in my experience those things are brittle; server changes can easily break your programs.

I found out about Subscriber Direct, but I don't know if it's a complete replacement because that looks more like a current program guide rather than a lookup for a particular program (which may not have been broadcast recently).
Dark Helmet is offline   Reply With Quote
Old 07-27-2012, 10:24 AM   #14
moyekj
Registered User
 
Join Date: Jan 2006
Location: Mission Viejo, CA
Posts: 9,299
Quote:
Originally Posted by Dark Helmet View Post
Ahhhh ... So do they actually retrieve the information via DNS? Or is it via some HTTP-based protocol that normal mortals can't use?
Well no, the queries are made to TiVo servers by name so DNS is needed to convert to IP #s. It's been noted by those with special router firmware that can track such requests that when using the HDUI and navigating the My Shows list that there is a lot of HTTP and DNS lookup activity happening which is part of the reason HDUI can be so slow. For example the pictures associated with specific shows you see on the right side are all obtained that way while you are navigating the HDUI along with extra info such as episode # etc.
__________________
Roamio Pro, Elite, Premiere
Cox - Motorola CableCards & TAs
Slingbox 350 via TiVo Mini & TiVo Stream for remote viewing

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
moyekj is offline   Reply With Quote
Old 07-27-2012, 10:39 AM   #15
Dark Helmet
Registered User
 
Join Date: Sep 2006
Location: Northern Virginia
Posts: 132
Quote:
Originally Posted by moyekj View Post
Well no, the queries are made to TiVo servers by name so DNS is needed to convert to IP #s. It's been noted by those with special router firmware that can track such requests that when using the HDUI and navigating the My Shows list that there is a lot of HTTP and DNS lookup activity happening which is part of the reason HDUI can be so slow. For example the pictures associated with specific shows you see on the right side are all obtained that way while you are navigating the HDUI along with extra info such as episode # etc.
You would think that some of those DNS queries could be cached, but hey, whatever. Guess it's time to break out a network sniffer ...
Dark Helmet is offline   Reply With Quote
Reply
Forum Jump




Thread Tools


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Advertisements

TiVo Community
Powered by vBulletin® Version 3.6.8
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
vBulletin Skins by: Relivo Media

(C) 2013 Magenium Solutions - All Rights Reserved. No information may be posted elsewhere without written permission.
TiVoŽ is a registered trademark of TiVo Inc. This site is not owned or operated by TiVo Inc.
All times are GMT -5. The time now is 03:15 PM.
OUR NETWORK: MyOpenRouter | TechLore | SansaCommunity | RoboCommunity | MediaSmart Home | Explore3DTV | Dijit Community | DVR Playground |