Java port of TivoDecode

Discussion in 'Developers Corner' started by fflewddur, Sep 6, 2015.

  1. Sep 10, 2015 #21 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    I think I fixed this inadvertently while getting another file to decode properly; at any rate, it decodes and plays fine on my computer (Mac OS X 10.10, VLC 2.21, Java 1.8.0_60) with today's 0.5.4 release of TivoLibre. Diff also doesn't show any differences between the file generated by tivodecode-ng and TivoLibre. Not sure what the problem was, though, and that's going to irk me. ;)

    I really appreciate you testing this out on a wider selection of files than I have available. If you find any more that don't decode properly, please let me know.
     
  2. Sep 10, 2015 #22 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    OK, here's an old one that gave the original author of TS tivodecode headaches trying to figure out (and I don't think ever did):
    https://drive.google.com/file/d/0B0SMFC97ymdEanJua1Vad3ZuOHM/view?usp=sharing

    tivolibre, tivodecode-ng and DSD all produce same file size.
    tivolibre and tivodecode-ng files are identical (via diff) but there are several very obvious segments where it's not getting decrypted correctly - you will see square blocks appear all over the video.
    The DSD produced file plays back perfectly.

    From what I recall there were a few other clips such as this one that I had, but I can't seem to find them. Solving this one I think may solve the last type of problem that I recall seeing with tivodecode and TS files.
     
  3. Sep 11, 2015 #23 of 154
    telemark

    telemark Active Member

    1,544
    4
    Nov 12, 2013
    So glad to see revived activity.

    VLC is known as a liberal decoder. If you want to track down unseen bugs, you might see if there's a mpeg validater that could be used.
     
  4. Sep 11, 2015 #24 of 154
    PaulS

    PaulS Active Member

    779
    38
    Sep 16, 2002
    Southern NH
    In my experience previously noodling with tivodecode, I never saw streams that weren't part of the PMT. I can't see how that would impact the playout of the decoded video.
     
  5. Sep 11, 2015 #25 of 154
    PaulS

    PaulS Active Member

    779
    38
    Sep 16, 2002
    Southern NH
    Haha... Join the club! I saw the same thing when I was working on tivodecode. I'd have two sample clips that looked exactly the same at the same spot in the encryption process, excepting one had a packet with encrypted contents and the other did not. Never could figure that one out...
     
  6. Sep 11, 2015 #26 of 154
    PaulS

    PaulS Active Member

    779
    38
    Sep 16, 2002
    Southern NH
    As an aside : how performant do we think this port will be, as compared to the C/C++ implementations ? Quick enough for at least real-time decode ? Do we think that any/all of the fixes here can be backported to the C++ version ?

    I'm thinking about the application of this new version, given the PipedInputStream/PipedOutputStream support. One could probably implement a streaming playback application using this.
     
  7. Sep 11, 2015 #27 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    It's noticeably faster than the C++ version. I just timed it on my machine (Core i7 2.6 Ghz), and a 2GB file took 16.8 seconds to decode; tivodecode took 22.8 seconds on the same file. Definitely fast enough for real-time playback.

    As for back-porting fixes, I'm sure it's possible, but I'll leave that to someone else. I tried to clean up the code during the initial port, and I keep refactoring it into more module chunks as I gain a better understanding of the decoding processes, so there's not a one-to-one correspondence with tivodecode anymore.
     
  8. Sep 11, 2015 #28 of 154
    wmcbrine

    wmcbrine Well-Known Mumbler

    11,698
    810
    Aug 2, 2003
    Well, so far, tivodecode-ng is 2x-3x faster than tivolibre for me, so that's interesting. Core 2 Duo (old MacBook Air).

    I plan to backport anything applicable... of course, I still hope to be first to a complete solution, so the porting will have to go the other way. ;) Currently, I can cleanly decode every sample moyekj posted to this thread (except the last), with the benefit of the one-bit patch in #19. I still have my problem child, which both tivodecode-ng and tivolibre fail on -- seemingly in completely different ways, strangely enough. (They had produced identical output a few tivolibre versions ago.)
     
    Last edited: Sep 11, 2015
  9. Sep 11, 2015 #29 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    I only tested one file, so my numbers may not be typical. I'm trying to get it correctly processing TS files before really looking into performance issues; just wanted to make it clear that it's fast enough for real-time use. :)

    I'll keep you posted on any issues I find that were likely to come from the tivodecode codebase. In addition to the ES bit fix, tivodecode chokes on packets from streams that weren't specified in the PMT (for an example, see the first file moyekj posted), and if the transport stream gets out of sync, it can't recover. I've got a 10 GB file with this problem; if you're interested, I'll try to trim it down to a smaller size that still exhibits the issue and post it online for you. I fixed the later problem by scanning for TS sync bytes (0x47) 188 bytes apart; once TivoLibre finds a few of them, it resyncs at the first and starts reading packets again.
     
  10. Sep 11, 2015 #30 of 154
    wmcbrine

    wmcbrine Well-Known Mumbler

    11,698
    810
    Aug 2, 2003
    I'm not seeing the choking? :confused:

    I welcome any test cases, and BTW, I'll take any size of file. As far as resync goes, tivodecode-ng already has a patch for that, but I'm not sure I've ever had a test case for it.
     
  11. Sep 11, 2015 #31 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    From my testing last night with latest version of tivodecode-ng as wmcbrine stated all my testscases posted here so far save the last one are now working with tivodecode-ng. For this particular problem you mention above tivodecode-ng will spit out warnings to stderr about it but the decrypt completes anyway.
     
  12. Sep 11, 2015 #32 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Ah, it outputs errors, but the decode works successfully. The decode finished so quickly, I assumed it was stopping after the error.

    I'm uploading the sync test case to my web server right now, but it estimates another 6 hours till completion; I'll PM you the link once it's done.
     
  13. Sep 11, 2015 #33 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    William, any idea how to generate static tivodecode-ng binary for Windows? I used to do it with cygwin back when they supported static libraries, but now on cygwin one can only compile using dynamic libraries, so generating an executable without any cygwin dll dependencies is not possible. I've been using cygwin to compile/test so far. A while back I briefly looked into using mingw but didn't get very far.
     
  14. Sep 11, 2015 #34 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Got it. This file has PES headers that extend across multiple TS frames, but the decoding processes wasn't accounting for that. We only want to decrypt stream data, not stream headers. The latest version in GitHub results in a file that plays properly.

    I did notice that some TS frames are out of order in the resulting file, although that doesn't seem to impact playback. I'll try to get that fixed up this weekend, regardless. I also want to take another look at the first file you sent to see if we can figure out what those mystery streams are, or at least, how to ensure they end up in the right place in the output file

    Any other files that still have playback issues?
     
  15. Sep 11, 2015 #35 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    Wow, that was qiuck! Well done! Binary diff vs DSD does show difference probably because as you said right now some TS frames are out of order.

    I've got some longer clips that I'm getting messages like this:
    Sep 11, 2015 7:37:38 PM net.straylightlabs.tivolibre.TransportStreamPacket checkHeader
    SEVERE: Invalid TS packet header for packet 10896845

    tivodecode-ng spits out messages to stderr for the above clip such as:
    loss_of_sync
    skipped 4 bytes, found a SYNC
    found 3 syncs in a row, loss_of_sync = 0
    globalBufferLen > TS_FRAME_SIZE, pulling packet from globalBuffer[]
    globalBufferLen > TS_FRAME_SIZE, pulling packet from globalBuffer[]

    These aren't fatal errors and the decrypt does complete generating a file of different size than DSD (normally larger). Just wondering if the above are expected and can be safely ignored? Since these are long clips it's difficult to watch the whole thing looking for problems.

    FYI: tivodecode-ng and tivolibre for the above generate identical files (per diff).
     
    Last edited: Sep 11, 2015
  16. Sep 11, 2015 #36 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Cool, thanks for testing it! Yeah, that error probably doesn't need to be reported to the end user; it's in there right now because I wasn't positive I'd fixed the out-of-sync problem. I've got a file that throws those errors as well, so I'll take a look at why the file size is different than DSD's.
     
  17. Sep 12, 2015 #37 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
  18. Sep 12, 2015 #38 of 154
    PaulS

    PaulS Active Member

    779
    38
    Sep 16, 2002
    Southern NH

    WOW!!!! GREAT JOB!!! That was EXACTLY the problem that tormented me for so long. I could never figure out how to fix that one...

    Congrats!

    Just as long as the packets for each stream as still in the proper order, it shouldn't have any adverse effects on playback.

    The development of this app has been going in spurts for a long while, and it's exciting that we're finally so close to the finish line.
     
  19. Sep 12, 2015 #39 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Thanks, it's been really fun learning about MPEG streams and getting this to work :)

    I'm not sure my earlier fix is exactly right, though. The file Kevin posted works properly if I go back to the old way of ignoring PES header lengths when calculating where to start decrypting each packet. Right now my hope is that the MpegParser isn't calculating PES header length correctly on the latest file; if it's reporting a header that's slightly too long, it messes up the next packet's decryption. This could also explain the out-of-order packets I've seen in hex dumps; if the PES header is reported as the length of a packet's payload, it gets buffered so that we can keep reading the rest of the header in the next packet. If a second stream has a packet interleaved with the first streams' packets and it *doesn't* have a PES header >= the packet length, it would be output before the buffered packet. That shouldn't have any impact on playback (as long as the packet's aren't encrypted) because each stream is still in the same logical order; it only becomes a problem if one of the buffered packets is encrypted and we start decrypting it at the wrong offset. That seems to match up with the behavior I'm seeing, but it's still just a hypothesis.

    Now it's time to see what the MPEG standard says about how PES header lengths should be calculated, and see if that matches what the code is currently doing. :)
     
  20. Sep 13, 2015 #40 of 154
    wmcbrine

    wmcbrine Well-Known Mumbler

    11,698
    810
    Aug 2, 2003
    I'd definitely go with MinGW -- it does pretty much everything Cygwin does (more so now than ever), minus the problems. Only, I'm finding there are some current issues that keep me from building tivodecode-ng with it. Working on that now.

    I also like OpenWatcom, although I haven't even tried tivodecode-ng with it yet. And of course there's a free subset of Visual C++, which used to be compatible with tivodecode, so I'll have to at least make sure that works again, even if I don't care much for it.
     

Share This Page