TL;DR: A developer's take on how TivoLibre (and kmttg) can better recover from transport stream sync errors.
Comcast finally switched all my channels to MPEG-4, which means I now have to transfer everything as a transport stream (TS). As many of you already know, there are often sync errors that occur in these downloads that have plagued us for years, making for errors in the decrypted result. The current best practice seems to be to download it multiple times until one doesn't contain an error.
Having to deal with these errors for all my downloads finally motivated me to do a deep dive into these things. I recently posted my detailed insights here: Merging TiVo Transport Stream Files (intended for developers).
In short, some improvements can be made to TiVo decryption for transport streams, plus it's possible to merge multiple downloads. (And, hopefully, kmttg can get a new certificate so it can keep working full-steam after the end of 2020!)
----
Improvements:
1. When TivoDecoder.jar (aka. TivoLibre, which kmttg uses) encounters a bad sync byte (0x47) in the TS, it skips ahead millions of bytes to re-synchronize the decryption. However, that's not necessary. Every TiVo packet in the stream resets the cipher, so TivoLibre only needs to skip ahead to the next TiVo packet. In a tool that I wrote to decode TiVo files, instead of losing around 45s of video upon a sync error, I only lose between 300ms and 1.6s of video. That's MUCH more bearable!
2. I noticed that there's a bug in some of my recordings where ALL video packets are marked as encrypted. In reality, though, only 1 out of every 4 video packets is encrypted by the TiVo. (I'm guessing this is Comcast's fault, so this might not be seen by everyone.) With TivoLibre, the result of decrypting such a .TiVo file is that the video output is corrupt, with maybe a valid frame being shown every once in a while. So, in my decoding tool, I just skip decryption for those other 3 video packets and the resultant video is mostly flawless!
3. I made my tool capable of merging multiple downloaded transport streams of the same program (and not just the same recording). This means I can download a recording 3-4 times and weed out the sync errors. If I really want to increase my chances of decrypting a recording without errors, I'll add a couple of manual recordings of the same program ahead of time -- which can be merged in with the other downloads. This relies on the fact that the recorded TS from multiple tuners is ultimately the same stream of bytes coming from the cable company.
----
In conclusion, I'm hoping someone can do something with this info. To fix the first issue, resetting the cipher, it should be a simple change to TivoLibre. If necessary I can try to put together a pull request for it, but I can't build TivoDecoder.jar so I would be just taking a stab at it.
Also, as I mention in my blog post, while the decoding tool I wrote does all the cool stuff I mention (like merging), it's not in a releasable state. Sorry.
I wish I could say it will be released someday, but I don't know that I'll ever have the time to productize it, get a copyright release from my employer, etc.
I will try to watch this thread for any questions, but please know that I'm bad at that kind of diligence. I'm hoping my insights will prove useful even if I don't respond to folks.
P.S. I want to give great thanks to the makers of tivodecode, TivoLibre, pyTivo, and kmttg -- as well as everybody else who's reverse-engineering made these things possible. Without them, I'd be stuck in the dark ages of TV-watching. Oh, and I'd like to thank TiVo too! ...it may not be what it used to be, but it has certainly had a good run (and here's hoping kmttg can get another certificate and keep that good run going).
Comcast finally switched all my channels to MPEG-4, which means I now have to transfer everything as a transport stream (TS). As many of you already know, there are often sync errors that occur in these downloads that have plagued us for years, making for errors in the decrypted result. The current best practice seems to be to download it multiple times until one doesn't contain an error.
Having to deal with these errors for all my downloads finally motivated me to do a deep dive into these things. I recently posted my detailed insights here: Merging TiVo Transport Stream Files (intended for developers).
In short, some improvements can be made to TiVo decryption for transport streams, plus it's possible to merge multiple downloads. (And, hopefully, kmttg can get a new certificate so it can keep working full-steam after the end of 2020!)
----
Improvements:
1. When TivoDecoder.jar (aka. TivoLibre, which kmttg uses) encounters a bad sync byte (0x47) in the TS, it skips ahead millions of bytes to re-synchronize the decryption. However, that's not necessary. Every TiVo packet in the stream resets the cipher, so TivoLibre only needs to skip ahead to the next TiVo packet. In a tool that I wrote to decode TiVo files, instead of losing around 45s of video upon a sync error, I only lose between 300ms and 1.6s of video. That's MUCH more bearable!
2. I noticed that there's a bug in some of my recordings where ALL video packets are marked as encrypted. In reality, though, only 1 out of every 4 video packets is encrypted by the TiVo. (I'm guessing this is Comcast's fault, so this might not be seen by everyone.) With TivoLibre, the result of decrypting such a .TiVo file is that the video output is corrupt, with maybe a valid frame being shown every once in a while. So, in my decoding tool, I just skip decryption for those other 3 video packets and the resultant video is mostly flawless!
3. I made my tool capable of merging multiple downloaded transport streams of the same program (and not just the same recording). This means I can download a recording 3-4 times and weed out the sync errors. If I really want to increase my chances of decrypting a recording without errors, I'll add a couple of manual recordings of the same program ahead of time -- which can be merged in with the other downloads. This relies on the fact that the recorded TS from multiple tuners is ultimately the same stream of bytes coming from the cable company.
----
In conclusion, I'm hoping someone can do something with this info. To fix the first issue, resetting the cipher, it should be a simple change to TivoLibre. If necessary I can try to put together a pull request for it, but I can't build TivoDecoder.jar so I would be just taking a stab at it.
Also, as I mention in my blog post, while the decoding tool I wrote does all the cool stuff I mention (like merging), it's not in a releasable state. Sorry.
I wish I could say it will be released someday, but I don't know that I'll ever have the time to productize it, get a copyright release from my employer, etc.I will try to watch this thread for any questions, but please know that I'm bad at that kind of diligence. I'm hoping my insights will prove useful even if I don't respond to folks.
P.S. I want to give great thanks to the makers of tivodecode, TivoLibre, pyTivo, and kmttg -- as well as everybody else who's reverse-engineering made these things possible. Without them, I'd be stuck in the dark ages of TV-watching. Oh, and I'd like to thank TiVo too! ...it may not be what it used to be, but it has certainly had a good run (and here's hoping kmttg can get another certificate and keep that good run going).
