Proof of concept

greg_burns · 01-30-2006, 10:02 PM

I've been thinking of doing this for quite awhile and finally gave it a shot.

I just "removed" the .tivo header portion of a Family Guy episode I transferred using a hex editor and saved it as a file named FG.dat

I then edited the original .tivo file in VideoRedo to remove commercials, saved that away as FG.mpg

Opened up a DOS prompt and typed copy /b FG.dat+FG.mpg FG_edit.tivo

Just transferred FG_edit.tivo back to my TiVo and ... voila ... I have a Family Guy episode with no commercials but all its information intact (and in a folder!).

Now to automate this... and figure out how to alter the xml data inside the .tivo portion of the file.

Edit: We'll I've taken a stab at automating the removal of the header...

TiVo Attach

davezatz · 01-31-2006, 05:14 AM

Sweet!

dmets · 01-31-2006, 09:00 AM

So could this process also be used to add Tivo metadata to video files that did not originate from a tivo. Like Family Videos - So that they could have titles and be placed in folders when transfered to a Tivo via GoBack...

greg_burns · 01-31-2006, 09:35 AM

Quote:

Originally Posted by dmets

So could this process also be used to add Tivo metadata to video files that did not originate from a tivo. Like Family Videos - So that they could have titles and be placed in folders when transfered to a Tivo via GoBack...

First I gotta figure out how the information is stored in the header. It is not in clear text, but it may simple be Base64 or something. I see a hash and salt combination. I assume that is just hashing the MAK and not all the metadata, but I am not sure.

That would be [peter]sweet![/peter] though.

HDTiVo · 01-31-2006, 10:14 AM

Congratulations, you are on the way to providing an enormous benefit to the TiVo World.

gonzotek · 01-31-2006, 12:26 PM

Good work, and good luck on the next steps!!!

davezatz · 01-31-2006, 04:39 PM

Quote:

Originally Posted by greg_burns

First I gotta figure out how the information is stored in the header. It is not in clear text, but it may simple be Base64 or something. I see a hash and salt combination. I assume that is just hashing the MAK and not all the metadata, but I am not sure.

It would be nice if TiVo could point us (YOU) in the right direction for creating custom headers until they create some sort of TiVo Desktop add-on widget to do it. I do remember early on there was a Wiki entry somewhere trying to decode that stuff, but I don't think they got very far. (Their goal was different anyhow.)

It'd be nice to add Dave's Home Video or something and end up with a folder of all my home videos. Not that I have any home videos, but maybe I'd think about getting a camera.

samkuhn · 01-31-2006, 05:38 PM

So if you copy the header back on without changing the file extension to .tivo does the Tivo still recognize the metadata? Is the editing file that has been re-merged playable on a personal computer that doesn't know the MAK?

I sent this thread to the author of VideoRedo asking if he would add the ability to retain the metadata after editing as described in this thread.

videoredo · 01-31-2006, 05:55 PM

This is Dan from VideoReDo. Since the text isn't in clear-text if someone can describe what it looks like and where its located we can look at automating this. I can be reached at the videoredo support email (The board would not let me post our email address).

greg_burns · 01-31-2006, 05:58 PM

Quote:

Originally Posted by samkuhn

So if you copy the header back on without changing the file extension to .tivo does the Tivo still recognize the metadata?

No. You can try this yourself by just renaming an original .tivo file to .mpg. Your Tivo will no longer show the metadata for that file, just its filename. The Tivo box must decide whether to even look for metadata based on file extension. If you do rename .tivo file to .mpg it won't even transfer.

Quote:

Originally Posted by samkuhn

Is the editing file that has been re-merged playable on a personal computer that doesn't know the MAK?

Haven't tried that. Can't image that it would since the part of the file that makes it a .tivo file has been reattached.

greg_burns · 01-31-2006, 06:04 PM

Quote:

Originally Posted by videoredo

This is Dan from VideoReDo. Since the text isn't in clear-text if someone can describe what it looks like and where its located we can look at automating this. I can be reached at the videoredo support email (The board would not let me post our email address).

Well, all I am doing is opening any Tivo file up using a Hex editor. (I am using the one built-into VirtualDub, its on the Tools menu).

You quickly see a disclaimer at the start of the file in clear text\xml format.

From 0x480 to around 0x4000 is data. I assume this must be where the metadata is stored. The regular mpeg file seems to be after this point.

jmemmott · 01-31-2006, 11:14 PM

Quote:

Originally Posted by greg_burns

From 0x480 to around 0x4000 is data. I assume this must be where the metadata is stored. The regular mpeg file seems to be after this point.

I spent some time with the file format when I was learning to extract the Closed Captioning information. Adding what has been posted here to what I already know, I would expect that reusing the existing header is workable but creating something custom will be much more difficult. The file is really two parts. The first part is a header created by Tivo with a few notable objects:

Byte 0 : Fingerprint – “TiVo”
Byte 9, length 4 or Byte 11, length 2 : offset of the MPEG2 Program Stream.
Byte 28 : Start of the xml wrapper containing the metadata.
a. Copy Right Notice is in clear text.
b. The metadata follows and is encrypted with a cipher I believe is similar to Blowfish.

The offset value points to a standard (ISO/IEC 13818) MPEG2 Program Stream. Some parts of the program stream are allowed to be encrypted in 13818 and TiVo uses this. In particular, all P and B pictures as well as the audio blocks are encrypted. Everything before the MPEG2 Program Stream should be copied as metadata.

AV_Novice · 02-01-2006, 12:23 PM

Quote:

Originally Posted by jmemmott

b. The metadata follows and is encrypted with a cipher I believe is similar to Blowfish.

A quick Google of Blowfish cipher took me to a web site that claims to have a list of 150 products that use Blowfish. Tivo, Inc. is listed.

jmemmott · 02-01-2006, 12:38 PM

Quote:

Originally Posted by AV_Novice

A quick Google of Blowfish cipher took me to a web site that claims to have a list of 150 products that use Blowfish. Tivo, Inc. is listed.

Blowfish is conjecture on my part but it does have characteristics that fit such as processor/memory requirements, speed, security, license requirements, etc. However, there is also evidence that they have looked at other related algorithms such as Turing. In any case, it is a modern block or stream cipher, so even if Tivo explicitly told us which one they are using, we would still not be likely to break it. The security is in the key and key expansion algorithms not the cipher and no one except Tivo knows what those are...

greg_burns · 02-01-2006, 12:58 PM

I noticed there is a thread on the other site (which shall remain nameless and unlinked

) that is trying to determine the makeup of .tivo file as well.

"tivo file format (split from: Decrypting TiVo ToGo)"

I'm afraid I am in WAY over my head.

HDTiVo · 02-01-2006, 01:39 PM

Quote:

Originally Posted by greg_burns

I noticed there is a thread on the other site (which shall remain nameless and unlinked

) that is trying to determine the makeup of .tivo file as well.

"tivo file format (split from: Decrypting TiVo ToGo)"

I'm afraid I am in WAY over my head.

What you have done so far is very valuable. It would be nice if, for example, VideoRedo just added the option to preserve the data and output to a .tivo extension.

greg_burns · 02-01-2006, 01:53 PM

Quote:

Originally Posted by HDTiVo

What you have done so far is very valuable. It would be nice if, for example, VideoRedo just added the option to preserve the data and output to a .tivo extension.

Dan's (from VideoRedo) response to my email to him...

Quote:

OK, Seems simple enough. Will see what we can do.

.. Dan

Let's hope he's right.

endicot · 02-01-2006, 06:53 PM

This is awesome!

Is is possible to do the reverse? That is, make up a .tivo from a homemade .mpg - adding the tivo meta data (episode information) to the mix. Is there a utility to do this easy (assuming you have your media access key)?

greg_burns · 02-01-2006, 07:14 PM

Quote:

Originally Posted by endicot

This is awesome!

Is is possible to do the reverse? That is, make up a .tivo from a homemade .mpg - adding the tivo meta data (episode information) to the mix. Is there a utility to do this easy (assuming you have your media access key)?

There is currently no such utility. Unless Tivo makes the software themselves to do this, the chance of one of us figuring it out looks slim. There appears to be somesort of encryption (Blowfish?) on that portion of the file.

megazone · 02-01-2006, 08:57 PM

TiVo has used the El Gamal and Turing algorithms, but I don't know exactly which algorithm is used on TTG.

greg_burns · 02-01-2006, 08:58 PM

Quote:

Originally Posted by megazone

TiVo has used the El Gamal and Turing algorithms, but I don't know exactly which algorithm is used on TTG.

OUt of curiosity, any idea what this is referring to?

megazone · 02-01-2006, 09:00 PM

TiVo® DVRs by TiVo Inc.
Digital Video Recorders. Use Blowfish to safely transmit data to and from the hard drive.
---
The files are encrypted locally on the drive, aren't they? Perhaps that's what this is referring to? Or maybe MRV, since I thought that used a pre-shared key/cert and a symmetric system...

davezatz · 02-02-2006, 11:49 AM

Here's some speculation... The decryption key is buried somewhere within that .dll (and possibly loaded into memory when called). Depending what type of encryption scheme is used, the same key may or may not be used for encryption. In theory one would need to decrypt the .tivo file to understand the syntax of the meta data, then create properly formatted unique meta data, encrypt it, and prepend it to a mpeg. This is mostly beyond what my little brain can comprehend.

HDTiVo · 02-02-2006, 12:15 PM

It sounds like the metadata is encrypted with the MAK - which is unique to each account. That means my metadata would not work - or might be scrambled - with your mpegs. However, if the MAK could be found in the metadata, it is possible replacing it with your own would work.

If it is not the case that the metadata itself is unique to the account, then a "metadata sharing" library could be created to offer known metadata to people that need it. You could create a library of your own metadata and use it as needed. Not that this is going to be very useful.

It might be that there is a universal key, or no encryption key code (like a MAK of all zeroes.) If this could be found then it would be possible to make from scratch metadata with the universal key and "clear text" that any account could use.

davezatz · 02-02-2006, 12:23 PM

Quote:

Originally Posted by HDTiVo

It sounds like the metadata is encrypted with the MAK - which is unique to each account. That means my metadata would not work - or might be scrambled - with your mpegs. However, if the MAK could be found in the metadata, it is possible replacing it with your own would work.

Could be... Though I was assuming the MAK is used to authentic or verify the owner prior to decrypting with a seperate key. It's also possible a decryption key is partially composed of the MAK and partially composed of some universal string. Regardless, it's beyond my skills to analyze.

Quote:

It might be that there is a universal key, or no encryption key code (like a MAK of all zeroes.) If this could be found then it would be possible to make from scratch metadata with the universal key and "clear text" that any account could use.

This is probably where need TiVo's assistance and blessing.

greg_burns · 02-02-2006, 11:54 PM

I built a little app to automate this process of removing the header information.

TiVo Attach

Requires .NET 2.0 Framework to be installed.

Basically, it allows you to remove the header from a .tivo file. You can then edit the .tivo file using VideoRedo (or whatever), reattach the header file back onto the edited .mpg file. Thus preserving the metadata for transferring back to the TiVo.

lafos · 02-05-2006, 10:51 AM

Greg, I tried your app and it works just great. I downloaded an episode of Lost, cut the ads with VideoRedoplus, then put the metafile back on. It would be great if VideoRedo could automate, since the file now has to be written teice, once after editing, and again in your utility.

Very nice.

Grimm1 · 02-05-2006, 07:06 PM

Quote:

Originally Posted by greg_burns

I built a little app to automate this process of removing the header information.

TiVo Attach

Requires .NET 2.0 Framework to be installed.

Basically, it allows you to remove the header from a .tivo file. You can then edit the .tivo file using VideoRedo (or whatever), reattach the header file back onto the edited .mpg file. Thus preserving the metadata for transferring back to the TiVo.

Good job Greg!

I've used it on a few shows and movies and so far it has been working great!

greg_burns · 02-05-2006, 10:16 PM

Another use for the little app; disguising... ahem... shows on your Tivo!

Just attach some boring .tivo headers to your more interesting mpegs and there ya go!

BTW, I posted a slightly streamlined version earlier tonight. Less button clicking.

merrickw · 02-08-2006, 08:05 AM

All we need now is a little editor for those .tivo header files ...