PDA

View Full Version : extract episode description, genre, rating etc from .Tivo file

DooZo
12-28-2006, 01:30 PM
Hi there, I'm trying to resolve a very simple problem, I've been browsing various Tivo sites for the past week but still unable to find the solution.

I'm trying to create a searchable database for all the .tivo files that I had accumulated. I'm able to get the show name, episode, recorded date and channel name directly from the .tivo file name. However, I'm also hoping to save the detail description, genre, category etc to the database, those information are shown in the TivoDesktop but I could not copy and paste it to my database. Since my collection of .Tivo files are huge, ~1TB or ~1000 program files, opening each of them up in Tivo Desktop and type them in is simply too time consuming. Is there a way I could copy-paste them from Tivo Desktop or better still if there is a program out there that would take in the .tivo file and output those detail description that I needed. As you could imagine, I'm running out of space in my server and planning to burn them into DVD but would like to create the comprehensive database first so that I could search and retrieve them when needed in the future.

Thanks.
classicsat
12-28-2006, 03:38 PM
The details you want are encrypted in the file. If you want to be a hero and figure out the encryption scheme and keys, go ahead.
DooZo
12-28-2006, 04:39 PM
The details you want are encrypted in the file. If you want to be a hero and figure out the encryption scheme and keys, go ahead.

:) thanks for the very helpful advice. The whole point is to be able to copy paste the information already displayed on my screen and not to break the encryption. After I entered my MAK into programs such as Tivo Desktop, Tivo Decode Manager, NowPlaying2.0 etc, I could see those information, but I'm not able to highlight it to be able to copy-paste. So, my point is not to figure out the encryption, or to break it, it is just to be able to copy what I've seen on the screen and paste it into my database. Hope this make it clearer what I'm trying to do and someone will be able to shed some lights on this. Thanks again.
greg_burns
12-29-2006, 09:44 AM
http://www.tivocommunity.com/tivo-vb/showthread.php?p=3791858&&#post3791858
2) While I can't actually extract the data directly from a pre-existing .tivo file it is possible to get it from the TiVo Server application by querying the TiVo Server using HTTP commands. It's not super elegant, but it could be used as a work around to extract info from .tivo files which are on your hard drive but no longer on your TiVo.


I have the syntax for this query at home. I'll try and post it later tonight.
DooZo
12-29-2006, 11:08 PM
I have the syntax for this query at home. I'll try and post it later tonight.

Thanks a lot greg_burns, hopefully this will save me a lot of time and effort.
greg_burns
12-29-2006, 11:30 PM
Thanks a lot greg_burns, hopefully this will save me a lot of time and effort.

I haven't forgotten you, but I currently can't access my drive that has the query saved on it. (long story) :o

Let me do some searching and see if I can find where Dan originally posted that method.
greg_burns
12-29-2006, 11:40 PM
http://www.tivocommunity.com/tivo-vb/showthread.php?p=3792692&&#post3792692
Actually all of the metadata can be retrieved via standard http using a URL like so....

http://tivo:<MAK>@<Computer_IP>:8080/TiVoConnect/TivoNowPlaying/I<filename>.TiVo?Format=text%2Fxml

It's the longer version of the data, but it contains all the information needed for the shorter version as well.

Dan

I've tried this in the past and it does indeed work. IIRC, it may be easier if you temporarily rename the file to something simpler to type.
greg_burns
12-30-2006, 12:00 AM
Definitely works. Unfortunately, the xml comes out flat. You'll want to use something like XMLSpy to make it readable.


<?xml version="1.0" encoding="utf-8"?>
<TvBusMarshalledStruct:TvBusEnvelope xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xmlns:TvBusMarshalledStruct="http://tivo.com/developer/xml/idl/TvBusMarshalledStruct" xmlns:TvPgdRecording="http://tivo.com/developer/xml/idl/TvPgdRecording" xmlns:TvBusDuration="http://tivo.com/developer/xml/idl/TvBusDuration" xmlns:TvPgdShowing="http://tivo.com/developer/xml/idl/TvPgdShowing" xmlns:TvDbShowingBit="http://tivo.com/developer/xml/idl/TvDbShowingBit" xmlns:TvBusDateTime="http://tivo.com/developer/xml/idl/TvBusDateTime" xmlns:TvPgdProgram="http://tivo.com/developer/xml/idl/TvPgdProgram" xmlns:TvDbColorCode="http://tivo.com/developer/xml/idl/TvDbColorCode" xmlns:TvPgdSeries="http://tivo.com/developer/xml/idl/TvPgdSeries" xmlns:TvDbShowType="http://tivo.com/developer/xml/idl/TvDbShowType" xmlns:TvPgdChannel="http://tivo.com/developer/xml/idl/TvPgdChannel" xmlns:TvDbTvRating="http://tivo.com/developer/xml/idl/TvDbTvRating" xmlns:TvDbRecordQuality="http://tivo.com/developer/xml/idl/TvDbRecordQuality" xmlns:TvDbBitstreamFormat="http://tivo.com/developer/xml/idl/TvDbBitstreamFormat" xs:schemaLocation="http://tivo.com/developer/xml/idl/TvBusMarshalledStruct TvBusMarshalledStruct.xsd http://tivo.com/developer/xml/idl/TvPgdRecording TvPgdRecording.xsd http://tivo.com/developer/xml/idl/TvBusDuration TvBusDuration.xsd http://tivo.com/developer/xml/idl/TvPgdShowing TvPgdShowing.xsd http://tivo.com/developer/xml/idl/TvDbShowingBit TvDbShowingBit.xsd http://tivo.com/developer/xml/idl/TvBusDateTime TvBusDateTime.xsd http://tivo.com/developer/xml/idl/TvPgdProgram TvPgdProgram.xsd http://tivo.com/developer/xml/idl/TvDbColorCode TvDbColorCode.xsd http://tivo.com/developer/xml/idl/TvPgdSeries TvPgdSeries.xsd http://tivo.com/developer/xml/idl/TvDbShowType TvDbShowType.xsd http://tivo.com/developer/xml/idl/TvPgdChannel TvPgdChannel.xsd http://tivo.com/developer/xml/idl/TvDbTvRating TvDbTvRating.xsd http://tivo.com/developer/xml/idl/TvDbRecordQuality TvDbRecordQuality.xsd http://tivo.com/developer/xml/idl/TvDbBitstreamFormat TvDbBitstreamFormat.xsd" xs:type="TvPgdRecording:TvPgdRecording">
<recordedDuration>PT28M59S</recordedDuration>
<vActualShowing>
<element>
<showingBits value="515"/>
<time>2006-12-19T02:31:00Z</time>
<duration>PT29M</duration>
<program>
<vActor>
<element>Louis-Dreyfus|Julia</element>
<element>Gregg|Clark</element>
<element>Linklater|Hamish</element>
<element>Gagnon|Trevor</element>
<element>Rutherford|Emily</element>
</vActor>
<vAdvisory/>
<vChoreographer/>
<colorCode value="4">COLOR</colorCode>
<description>Ritchie wants to attend church. Copyright Tribune Media Services, Inc.</description>
<vDirector>
<element>Ackerman|Andy</element>
</vDirector>
<episodeTitle>Oh God, Yes</episodeTitle>
<vExecProducer>
<element>Lizer|Kari</element>
<element>Ackerman|Andy</element>
</vExecProducer>
<vProgramGenre>
<element>Sitcom</element>
</vProgramGenre>
<vGuestStar>
<element>Begley Jr.|Ed</element>
<element>Gross|Mary</element>
<element>Bryan|Benjamin</element>
<element>Connor|Brandon Patrick</element>
</vGuestStar>
<vHost/>
<isEpisode>true</isEpisode>
<originalAirDate>2006-10-09T00:00:00Z</originalAirDate>
<vProducer/>
<series>
<isEpisodic>true</isEpisodic>
<vSeriesGenre>
<element>Sitcom</element>
<element>Comedy</element>
</vSeriesGenre>
<seriesTitle>The New Adventures of Old Christine</seriesTitle>
</series>
<showType value="5">SERIES</showType>
<title>The New Adventures of Old Christine</title>
<vWriter>
<element>Crittenden|Jennifer</element>
</vWriter>
</program>
<channel>
<displayMajorNumber>2</displayMajorNumber>
<callsign>WBOC</callsign>
</channel>
<tvRating value="4">PG</tvRating>
</element>
</vActualShowing>
<vBookmark/>
<recordingQuality value="75">HIGH</recordingQuality>
<showing>
<showingBits value="515"/>
<time>2006-12-19T02:31:00Z</time>
<duration>PT29M</duration>
<program>
<vActor>
<element>Louis-Dreyfus|Julia</element>
<element>Gregg|Clark</element>
<element>Linklater|Hamish</element>
<element>Gagnon|Trevor</element>
<element>Rutherford|Emily</element>
</vActor>
<vAdvisory/>
<vChoreographer/>
<colorCode value="4">COLOR</colorCode>
<description>Ritchie wants to attend church. Copyright Tribune Media Services, Inc.</description>
<vDirector>
<element>Ackerman|Andy</element>
</vDirector>
<episodeTitle>Oh God, Yes</episodeTitle>
<vExecProducer>
<element>Lizer|Kari</element>
<element>Ackerman|Andy</element>
</vExecProducer>
<vProgramGenre>
<element>Sitcom</element>
</vProgramGenre>
<vGuestStar>
<element>Begley Jr.|Ed</element>
<element>Gross|Mary</element>
<element>Bryan|Benjamin</element>
<element>Connor|Brandon Patrick</element>
</vGuestStar>
<vHost/>
<isEpisode>true</isEpisode>
<originalAirDate>2006-10-09T00:00:00Z</originalAirDate>
<vProducer/>
<series>
<isEpisodic>true</isEpisodic>
<vSeriesGenre>
<element>Sitcom</element>
<element>Comedy</element>
</vSeriesGenre>
<seriesTitle>The New Adventures of Old Christine</seriesTitle>
</series>
<showType value="5">SERIES</showType>
<title>The New Adventures of Old Christine</title>
<vWriter>
<element>Crittenden|Jennifer</element>
</vWriter>
</program>
<channel>
<displayMajorNumber>2</displayMajorNumber>
<callsign>WBOC</callsign>
</channel>
<tvRating value="4">PG</tvRating>
</showing>
<startTime>2006-12-19T02:30:58Z</startTime>
<stopTime>2006-12-19T02:59:59Z</stopTime>
<bitstreamFormat>
<vFormat>
<element>
<vByte>
<base64>EjQAAwABAjoBywxXAAAADwAAAAQAAAACAAAAAwAAAA==</base64>
</vByte>
</element>
</vFormat>
</bitstreamFormat>
<expirationTime>2006-12-21T02:31:00Z</expirationTime>
</TvBusMarshalledStruct:TvBusEnvelope>
greg_burns
12-30-2006, 12:07 AM
After I entered my MAK into programs such as Tivo Desktop, Tivo Decode Manager, NowPlaying2.0 etc, I could see those information, but I'm not able to highlight it to be able to copy-paste.

Got any links for those apps? They sound interesting.
DooZo
12-30-2006, 10:24 AM
Got any links for those apps? They sound interesting.

i'm unable to post link as I've not reach my 5th post yet
DooZo
12-30-2006, 10:25 AM
Got any links for those apps? They sound interesting.

try these links:
http://thebenesch.com/tdm/

http://www.dashboardwidgets.com/showcase/details.php?wid=281

About the solution you point me to, I'm still having problem using that as I keep on getting error message, I must be doing something obviously wrong somewhere:

http://tivo:<MAK>@<Computer_IP>:8080/TiVoConnect/TivoNowPlaying/I<filename>.TiVo?Format=text%2Fxml

So if my Tivo internal IP is at 192.168.1.7 and the computer that has the Tivo file is at 192.168.1.4, should I use either of them for the <Computer_IP> above? I actually tried both but was unsuccessful :(
windracer
12-30-2006, 10:30 AM
Got any links for those apps? They sound interesting.
Ah, they're Mac apps ... that's why I hadn't heard of them. ;)
greg_burns
12-30-2006, 10:58 AM
http://tivo:<MAK>@<Computer_IP>:8080/TiVoConnect/TivoNowPlaying/I<filename>.TiVo?Format=text%2Fxml

So if my Tivo internal IP is at 192.168.1.7 and the computer that has the Tivo file is at 192.168.1.4, should I use either of them for the <Computer_IP> above? I actually tried both but was unsuccessful :(

replace <Computer_IP> with your computer's IP, not the Tivo's.

Here is the syntax. Notice I renamed the tivo file to something simple; x.tivo. The file must be in your My Tivo Recordings folder.

http://tivo:1234567890@192.168.1.4:8080/TiVoConnect/TivoNowPlaying/Ix.TiVo?Format=text%2Fxml

I have no idea if this does or doesn't work on a mac. Does Tivo2Go even work on a mac? :confused: Pretty sure not, which means none of this is going to work for you.
DooZo
12-30-2006, 11:16 AM
I'm actually trying it out on PC and Mac, here is the actualy syntax, the only thing removed was MAK:

http://tivo:XXXXXXXXXX@192.168.1.4:8080/TiVoConnect/TivoNowPlaying/ITrue%20Caribbean%20Pirates%20(Recorded%20Dec%2023,%202006,% 20HISTORY).TiVo?Format=text%2Fxml

The IP is my computer IP where the file is located....Is it important to keep the "My Tivo Recordings" folder in a particular directory? Right now, the "My Tivo Recordings" is in "My Documents" folder. I've also disabled all firewall when I tried this out :(
greg_burns
12-30-2006, 11:26 AM
I think it is important that you can "see" the .tivo file in TivoDesktop. In other words, yes, keep (or move) the file back into the directory TivoDesktop normally downloads them to.

I would definitely temporarily rename the file to something easier. Spaces in the filename is just asking for trouble. Although it looks like you are escaping them properly.

I don't believe firewalls will be an issue. The command is calling TivoServer (part of TivoDesktop) which is running on your PC, not the Tivo.

BTW, I've been testing use Firefox 2.0. Just tried with IE7 and got an error about not being able to find the file...

---------------------------
Address Bar
---------------------------
Windows cannot find 'http://tivo:1234567890@192.168.1.4:8080/TiVoConnect/TivoNowPlaying/Ix.TiVo?Format=text%2Fxml'. Check the spelling and try again.
---------------------------
OK
---------------------------
DooZo
12-30-2006, 11:32 AM
I was using IE7, let me remove all the spaces and try again, it was the original file name from Tivo so I thought not to touch it earlier. If it still doesn't work, I will install the Firefox. I've Firefox on my Mac that I used often but not on my PC.
greg_burns
12-30-2006, 11:45 AM
I just tried again with IE7, but this time I left out the tivo:<MAK> portition. This forced IE to prompt for that info, which it seemed to like a lot more.


http://localhost:8080/TiVoConnect/TivoNowPlaying/Ix.TiVo?Format=text%2Fxml

Even seems to work with localhost (and 127.0.0.1), avoiding needing to know the ip address. :up:
DooZo
12-30-2006, 12:38 PM
Cool. It is working well for me now in both IE and Firefox using either method (Comp_IP and localhost), you are right, the spaces in the file name is causing all the trouble. Now I will try to find a XML program to make it is easier to copy-paste the text that I wanted to extract. Thanks for all your help on this. Sorry about the Mac programs and not PC based, PC is way in advance in term of program to process/manipulate TiVo related files and Mac has a lot of catch up to do. :)
DooZo
12-30-2006, 12:46 PM
Just curious about one more thing, you mention that this method is using the TiVo Server running on the PC, does it mean the we are using the TiVo server to decode the information in the .Tivo file or the TiVo server has to retrieve those information from the Tivo database via Internet?
greg_burns
12-30-2006, 12:53 PM
Just curious about one more thing, you mention that this method is using the TiVo Server running on the PC, does it mean the we are using the TiVo server to decode the information in the .Tivo file or the TiVo server has to retrieve those information from the Tivo database via Internet?

Nah, the information is all included (albeit encrypted) in the header of each .tivo file. TivoServer knows how to decrypt it is all. I wonder what could be done using tivodecode's (http://tivodecode.sourceforge.net/) algorithms in this regard?
DooZo
12-30-2006, 01:20 PM
Yeah.....some programs that I used in Mac like TivoDecodeManager (that uses TivoDecode internally) actually are able to show the details (but not able to copy-paste) though. So, I guess somehow the mechanism to decode the header must be included in the TivoDecode.

see screenshot:
http://thebenesch.com/tdm/TDM2.png

If there is a way to run TivoDecode in command line and it will output the episode description etc, it will be much faster as then I can write a script in Mac to automate that instead of doing manually.
greg_burns
12-30-2006, 01:36 PM
Yeah.....some programs that I used in Mac like TivoDecodeManager (that uses TivoDecode internally) actually are able to show the details (but not able to copy-paste) though. So, I guess somehow the mechanism to decode the header must be included in the TivoDecode.

see screenshot:
http://thebenesch.com/tdm/TDM2.png

If there is a way to run TivoDecode in command line and it will output the episode description etc, it will be much faster as then I can write a script in Mac to automate that instead of doing manually.

Maybe, maybe not. If TivoDecodeManager is just downloading the shows from a Tivo via the web interface it could be just getting that info from there. Or does it actually shows that info for files already downloaded to your Mac? Big difference, IMO.

I suspect it is only using tivodecode to remove the DRM from the files once downloaded.
Dan203
12-30-2006, 08:17 PM
Like Greg said there is a very simple way to get the information from the TiVo itself, using a sinilar http command, before it is downloaded. However for shows that are already on the PC and no longer on the TiVo itself the only way to get at the info is to use the TiVo Server, and as you discovered that's got limitations.

It's possible that the algorythm used for tivodecode could be used to get at the metadata as well. However as it is now it's specifically designed to decrypt MPEG data streams only, so someone would have to go to great lengths to adapt it to decoding the metadata. And with the workarounds available it's probably not a high priority project.

Dan
DooZo
12-30-2006, 11:42 PM
Thanks Greg and Dan, I got it. I'm starting to populate my database with the details that I could retrieve, before I remove the shows from the TiVo which I could get relatively easily. For the old shows on hard drives, I will just use the file name to populate fields such as Name, Episode, Channel, Recorded Date, for more detail description like genres, rating etc, I will hold it for now until someone come up with something that I can use easily. Since I'm keeping all the file in the original .TiVo format, I will have the ability to do that when the tool is ready (unless TiVo make significant changes to their file structure). Or if I have more time to spare, I could use the TiVo server on my PC and method by Dan (pointed to me by Greg) to retrieve the information needed. Thanks again for all your help and pointer.
DooZo
01-01-2007, 04:28 AM
Hi..I found a workaround to the problem of extracting metadata from old tivo files on my harddrive by using the TiVo Attach program by Greg. I used the TiVo Attach to attach the TiVo files I had in my harddrive to a small mpg that I created (less than 1M). This way, the metadata in those TiVo files are attached to that small mpg file. Then I used Galleon GoBack feature to upload the newly created TiVo files into TiVo. Then I either use the new version of TiVoDownloadManager that allow me to copy paste the info needed or I can use https://192.168.x.x/nowplaying/index.html to copy the show description. This will save me tons of time typing.