Better recovery from transport stream sync errors

Discussion in 'TiVo Home Media Features & TiVoToGo' started by mkrebs, Nov 2, 2020.

  1. mkrebs

    mkrebs New Member

    4
    4
    Mar 11, 2013

    Advertisements

    TL;DR: A developer's take on how TivoLibre (and kmttg) can better recover from transport stream sync errors.

    Comcast finally switched all my channels to MPEG-4, which means I now have to transfer everything as a transport stream (TS). As many of you already know, there are often sync errors that occur in these downloads that have plagued us for years, making for errors in the decrypted result. The current best practice seems to be to download it multiple times until one doesn't contain an error.

    Having to deal with these errors for all my downloads finally motivated me to do a deep dive into these things. I recently posted my detailed insights here: Merging TiVo Transport Stream Files (intended for developers).

    In short, some improvements can be made to TiVo decryption for transport streams, plus it's possible to merge multiple downloads. (And, hopefully, kmttg can get a new certificate so it can keep working full-steam after the end of 2020!)

    ----

    Improvements:

    1. When TivoDecoder.jar (aka. TivoLibre, which kmttg uses) encounters a bad sync byte (0x47) in the TS, it skips ahead millions of bytes to re-synchronize the decryption. However, that's not necessary. Every TiVo packet in the stream resets the cipher, so TivoLibre only needs to skip ahead to the next TiVo packet. In a tool that I wrote to decode TiVo files, instead of losing around 45s of video upon a sync error, I only lose between 300ms and 1.6s of video. That's MUCH more bearable!

    2. I noticed that there's a bug in some of my recordings where ALL video packets are marked as encrypted. In reality, though, only 1 out of every 4 video packets is encrypted by the TiVo. (I'm guessing this is Comcast's fault, so this might not be seen by everyone.) With TivoLibre, the result of decrypting such a .TiVo file is that the video output is corrupt, with maybe a valid frame being shown every once in a while. So, in my decoding tool, I just skip decryption for those other 3 video packets and the resultant video is mostly flawless!

    3. I made my tool capable of merging multiple downloaded transport streams of the same program (and not just the same recording). This means I can download a recording 3-4 times and weed out the sync errors. If I really want to increase my chances of decrypting a recording without errors, I'll add a couple of manual recordings of the same program ahead of time -- which can be merged in with the other downloads. This relies on the fact that the recorded TS from multiple tuners is ultimately the same stream of bytes coming from the cable company.

    ----

    In conclusion, I'm hoping someone can do something with this info. To fix the first issue, resetting the cipher, it should be a simple change to TivoLibre. If necessary I can try to put together a pull request for it, but I can't build TivoDecoder.jar so I would be just taking a stab at it.

    Also, as I mention in my blog post, while the decoding tool I wrote does all the cool stuff I mention (like merging), it's not in a releasable state. Sorry. :( I wish I could say it will be released someday, but I don't know that I'll ever have the time to productize it, get a copyright release from my employer, etc.

    I will try to watch this thread for any questions, but please know that I'm bad at that kind of diligence. I'm hoping my insights will prove useful even if I don't respond to folks. :)

    P.S. I want to give great thanks to the makers of tivodecode, TivoLibre, pyTivo, and kmttg -- as well as everybody else who's reverse-engineering made these things possible. Without them, I'd be stuck in the dark ages of TV-watching. Oh, and I'd like to thank TiVo too! ...it may not be what it used to be, but it has certainly had a good run (and here's hoping kmttg can get another certificate and keep that good run going).
     
    slowbiscuit and ClearToLand like this.
  2. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    52,709
    6,719
    Apr 17, 2000
    Nevada
    In my experiments when writing pyTivo Desktop the errors were almost always in the same place on repeated downloads. Sometimes you can download a copy with less errors, or even occasionally no errors, but for the most part the errors in one copy are in the exact same place as the errors in another so merging multiple recordings to get a clean copy requires you to get extremely lucky with the mishmash you download. You're probably more likely to just accidentally get a clean copy.

    Also some others have experimented with artificially throttling their network and have found that at a slow enough speed you're more likely to get a clean file. If you're really serious about getting a clean copy you should play with that and see if you have similar results.

    As for the Tivolibre skipping too many bytes... you're right. VideoReDo does something similar to what you say your tool does and is also able to recover more of the corrupted video.
     
    slowbiscuit and ClearToLand like this.

Share This Page

spam firewall

Advertisements