1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Editing .TiVo Metadata

Discussion in 'TiVo Home Media Features & TiVoToGo' started by ggieseke, Apr 2, 2010.

  1. Apr 2, 2010 #1 of 47
    ggieseke

    ggieseke Active Member

    4,023
    12
    May 30, 2008
    I figured out a way to create .TiVo files that aren't encrypted so that you can edit the metadata. The actual metadata is in two XML blocks (hereafter referred to as chunks), that you can extract using tivodecode with the -D option. I will leave documenting the exact XML format to others. The basic format of the file header and chunk headers comes from the tivodecode source, and the rest from trial and error.

    I'm attaching a very basic Windows command prompt exe for anyone that wants to try it. The rest of this post is probably of interest to developers only. The usage is as follows:

    mktivo chunk1 chunk2 mpegfile tivofile

    chunk1 & chunk2 are the XML files extracted by tivodecode which you can edit after extracting them, or create your own from scratch. mpegfile is the unencrypted MPEG from tivodecode, kmttg, VideoReDo etc, and tivofile is the output file that it will write.

    The technical stuff:

    TiVo files normally consist of the file header, followed by chunk #3, chunk #1, chunk #2, and the actual MPEG file. Chunk #3 is not encrypted and contains the disclaimer. Desktop could care less if it's there so I leave it out. The rest of the data is usually encrypted, but by changing a field in the chunk headers from 1 to 0 it works fine without the encryption. All numeric fields in the file and chunk headers use big-endian (or Motorola) format - Windows developers take note. Each chunk consists of a chunk header, the XML data, and some padding.

    The file header is 16 bytes long:
    Bytes 0-3: The text string "TiVo"
    Bytes 4-5: (16-bit) Unknown - set it to 4
    Bytes 6-7: (16-bit) Unknown - set it to 13
    Bytes 8-9: (16-bit) Unknown - set it to zero
    Bytes 10-13: (32-bit) The offset in bytes to the actual MPEG data
    Bytes 14-15: (16-bit) The number of chunks - set it to 2

    Each chunk header is 12 bytes long:
    Bytes 0-3: (32-bit) The chunk size in bytes - more on that later
    Bytes 4-7: (32-bit) The size of the XML file in bytes
    Bytes 8-9: (16-bit) The chunk ID, which will be 1 or 2
    Bytes 10-11: (16-bit) 0=unencrypted, 1=encrypted, so set it to 0

    For some reason the first chunk has 4 zero bytes after it and subsequent chunks have 7 zero bytes of padding, so the size for chunk 1 is the header size+XML size+4 and the size for chunk 2 is the header size+XML size+7.

    After writing the chunks, pad the output file with zero bytes until the file size is an even multiple of 1024 bytes, then pad it with another 1024 zero bytes. After that, append the MPEG file and you have a .TiVo file. I have no idea why any of the padding bytes are necessary but it seems to require them.

    I have also noticed that if you transfer a show back to the TiVo that's in its current database it uses the some of the fields like description from the database, not the metadata once it's transferred. I'm pretty sure that's based on the unique episode ID, but more experimentation is probably needed.

    I hope this helps somebody. I mostly use it to edit the airdate and duration on series shows like Good Eats so they show up in order by episode and have the correct duration after editing out the commercials, but a full-blown GUI program to edit or create metadata would be sweet if anyone wants to tackle it.
     

    Attached Files:

  2. Apr 2, 2010 #2 of 47
    dlfl

    dlfl Cranky old novice

    6,996
    16
    Jul 6, 2006
    Near...
    Nice work!

    Am I correct that .TiVo files created this way can be shared among users since the MAK is no longer involved ?

    I wonder if we now have a new MPEG-2-plus-metadata video file format standard? What are the competitors? Well I guess wtv and dvrms.
     
  3. Apr 2, 2010 #3 of 47
    reneg

    reneg Member

    699
    0
    Jun 19, 2002
    Interesting. I wonder if this can be used to get episode number & genres metadata to display on files transferred back to the Tivo.
     
  4. Apr 2, 2010 #4 of 47
    ggieseke

    ggieseke Active Member

    4,023
    12
    May 30, 2008
    That would break TiVo's Terms & Conditions (which wasn't my intention), but yes. IMO it's hardly worth the effort since tivodecode, kmttg, VideoReDo & DSD could do that already. I just wanted to be able to organize my own files.

    I'm kinda hoping that somebody with more .TiVo XML experience will run with it and come out with a really elegant metadata editor.
     
  5. Apr 2, 2010 #5 of 47
    ggieseke

    ggieseke Active Member

    4,023
    12
    May 30, 2008
    Absolutely. I have 178 Iron Chef Japan episodes and 206 Good Eats episodes that are doing that right now, and I wrote my own XML files from scratch for those series.

    I haven't tested it with pyTivo yet, but in my experience so far anything that transfers back using standard .TiVo files and Desktop transfers back using this method.
     
  6. Apr 2, 2010 #6 of 47
    wmcbrine

    wmcbrine Ziphead

    10,364
    22
    Aug 2, 2003
    Here's some more information on the header, from Dan203:

    P.S. Dan's bit numbering here is high-to-low (backwards from what I'd use). Where he says "D = S3", this corresponds to what ggieseke is saying with "Bytes 6-7: (16-bit) Unknown - set it to 13".
     
  7. Apr 2, 2010 #7 of 47
    ggieseke

    ggieseke Active Member

    4,023
    12
    May 30, 2008
    Thanks for the additional info! I get 13 (D in hexadecimal) from my S2DT TiVos, but the rest is interesting. I don't have a Series 3 - dunno what's up there.

    4-13 is my birthday, so I thought it was just a cool coincidence. :D
     
  8. Apr 2, 2010 #8 of 47
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    37,446
    165
    Apr 17, 2000
    Nevada
    Excellent work! We might be able to do something with this in VideoReDo. Not exactly sure what yet, since it's still illegal for us to decrypt the original metadata, but perhaps we could use it to allow DVR-MS/WTV to TiVo conversions with intact metadata.

    You're right. When I wrote that I was reading the bits from a hex editor from left to right. My bad. :eek:

    Anyway the only important bit in that whole thing, at least for this program, is the SD/HD bit. If you set it to 0, which is SD, and the file is actually HD then S2 units will think that they are capable of transferring these files and will simply error out after the transfer starts. This may confuse users because the message provided by the TiVo is completely nondescript.

    Dan
     
  9. Apr 2, 2010 #9 of 47
    ggieseke

    ggieseke Active Member

    4,023
    12
    May 30, 2008
    Ya gotta love corporate rules. :rolleyes: It's legal to decrypt and edit the video, but not the metadata?

    Maybe if you agree to NOT use the copyrighted TMS description it would be legal....
     
  10. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    37,446
    165
    Apr 17, 2000
    Nevada
    It's more about the encryption then the actual data. We can access the unencrypted video using the DirectShow filter provided with TiVo Desktop. However they do not provide anything similar for accessing the metadata. We can't use tivodecode because it technically circumvents the encryption and as such is a violation of the DMCA.

    Dan
     
  11. reneg

    reneg Member

    699
    0
    Jun 19, 2002
    Played with your utility a bit and made myself a .tivo file from a small clip of an hour long show. First thing I ran into when transferring the file back was pyTivo crashing. Since it was a .tivo file that I was transferring, pyTivo tried to run Tdcat to get the metadata out of the file. I'm guessing Tdcat is assuming the .tivo metadata is encrypted and decrypted the clear text xml into garbage. [Edit] Ran tdcat on the .tivo file and it produced an empty file, not garbage.

    I moved the tdcat utility so pytivo could not find it and transferred the file again. Second time it transferred. The duration was off but it played fine.

    Next I changed the duration and recorded duration xml fields in the chunks and transferred a new .tivo file back. It worked exactly like you would expect which was cool.

    Then I tried adding episodeNumber into the xml as the clip I was using didn't contain an episode number. Unfortunately, the episodeNumber did not show up on the Tivo. On the plus side, genres did display back on the Tivo.
     
  12. reneg

    reneg Member

    699
    0
    Jun 19, 2002
    Following up after playing around some more. I got episode numbers to display back on the Tivo. I started with a recorded show on the Tivo that didn't have an episode number. I created the two xml chunks from the recorded program and created some tests.

    First test was to add <episodeNumber> keyword to both xml chunks. There was no change in behavior on the transferred program. It did not display episode number.

    Then I added <episodeNumber> keyword in both chunks and changed <uniqueId> keyword in chunk2 (not the seriesID uniqueId, but the longer one that has an additional two bytes at the end, I just changed these last two bytes). After these changes, the episode number displayed on the Tivo under both the program and info screen after the file was transferred back to the Tivo.

    Furthermore, it appears that the Tivo is picking up data to display from the <showing> section of chunk 2.

    I wonder if it would be possible for pyTivo to pull files as .tivo files and get complete metadata?
     
  13. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    37,446
    165
    Apr 17, 2000
    Nevada
    pyTiVo should be able to be updated to support this new format. All they have to do is read that encryption bit and decide if the XML needs to be decrypted or not. However I'm not sure it's necessary. pyTiVo already has it's own external metadata format, so why would you want to go through the hassle of creating one of these modified .tivo files instead?

    Dan
     
  14. innocentfreak

    innocentfreak Active Member

    8,950
    3
    Aug 25, 2001
    Florida
    I would definitely be interested in this. The transfer speeds of the Premiere have given me the itch to be able to transfer anything to the TiVo initiated from the TiVo. I looked briefly into it on the TiVo HD, but I couldn't handle the transfer speeds. Now I am just trying to figure out the best way to have my rips and that where I can pull them to the TiVo rather than streaming.

    Does PyTiVo or VideoReDo currently have the ability to convert files in other formats/codecs to .TiVo files that the TiVo can see to pull preferably with guide data?
     
  15. mattack

    mattack Active Member

    20,734
    4
    Apr 9, 2001
    sunnyvale
    I thought they did.. I've been playing with iTivo a bit lately, and it has the option to get metadata out. I just noticed something.. In the metadata for an episode of the tonight show, it ALSO put the metadata for the jimmy fallon show -- presumably because I padded the recording by one minute and so it technically now contains part of the jimmy fallon show.
     
  16. reneg

    reneg Member

    699
    0
    Jun 19, 2002
    Because even with everything that pyTivo currently does, the metadata is incompletely displayed on the Tivo for files pulled back to the Tivo. Specifically, episode number and genres do not display.

    I wasn't thinking of pyTivo creating a modified .tivo file, but rather feeding the Tivo a dynamically generated .tivo data stream on pulls of mpeg2's. pyTivo already deals with xml for metadata and transcoding files to mpeg2.
     
  17. wmcbrine

    wmcbrine Ziphead

    10,364
    22
    Aug 2, 2003
    Yes, that's my plan. But I still have a few more things to figure out -- like, the bitstreamFormat field: What's that about? Can we do without it?

    I also suspect that this might be the only way to feed transport streams (hopefully to include H.264) to the TiVo, but I'm not sure yet.
     
  18. ggieseke

    ggieseke Active Member

    4,023
    12
    May 30, 2008
    I leave out everything between </showing> and </TvBusMarshalledStruct:TvBusEnvelope> and it doesn't seem to affect anything, so I think we can do without bitstreamFormat. Some of the other fields like startTime, stopTime & expirationTime are ignored anyway on transfers. I suspect that the entire vActualShowing section may also be unnecessary since it has the wrong info if you pad a show early.

    I'm attaching some typical chunks that I created from scratch. Episode number & genres display properly on my TiVos before and after transfer. I made up the episode number and the unique episode id since the data from TMS is incomplete for that series.

    I think that the unique episode id in chunk 2 may be the key to how TiVo determines if an episode is already in the NPL, but that's just a guess. From what I've seen it always starts with "EP", then the 6-digit series id, and 4 more digits. On some shows the TMS guide data seems to use the episode number for the last 4 digits (if it exists), and on other shows they just start at 1 and increment it with each new episode.
     

    Attached Files:

  19. ggieseke

    ggieseke Active Member

    4,023
    12
    May 30, 2008
    A very minor patch to tivodecode would also fix it. Glancing at the source for tdcat, adding one 'if' near the bottom should work.

    if ((o_chunk_1 && chunk->id == 1) || (o_chunk_2 && chunk->id == 2))
    {
    if(chunk->type == TIVO_CHUNK_BLOB)
    {
    prepare_frame(&metaturing, 0, 0);
    skip_turing_data(&metaturing, (size_t)(chunk_start - current_meta_stream_pos));
    decrypt_buffer(&metaturing, chunk->data, chunk->data_size);
    current_meta_stream_pos = chunk_start + chunk->data_size;
    }
    if (fwrite (chunk->data, 1, chunk->data_size, ofh) != chunk->data_size)
    {
    perror("write chunk");
    return 8;
    }
    }
     
  20. reneg

    reneg Member

    699
    0
    Jun 19, 2002
    Just a bit of clarification on the 2nd <uniqueId> in chunk 2, I've seen it start with "EP", then the 6-digit series id, and 4 more digits and I've also seen it start with "EP", then contain 8-digit series id, and 4 more digits. At some point in the past, they ran out of unique 6-digit series id numbers.

    An example from a recent episode of The Mentalist:
    Code:
    <uniqueId>EP010587140044</uniqueId>
    Also, reformating the xml with whitespace doesn't seem to hurt anything, but helps readability quite a bit.
     

Share This Page