Java port of TivoDecode

Discussion in 'Developers Corner' started by fflewddur, Sep 6, 2015.

  1. Sep 6, 2015 #1 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    In case this is of any interest: I recently ported tivodecode 0.4.4 to Java and implemented support for Transport Stream files. Both the source code and precompiled JARs are available on GitHub. Any feedback/bug reports are welcome!

    Update: You can download the latest stable version at https://github.com/fflewddur/tivolibre/releases.
     
    Last edited: Sep 21, 2015
  2. Sep 7, 2015 #2 of 154
    telemark

    telemark Active Member

    1,544
    4
    Nov 12, 2013
  3. Sep 7, 2015 #3 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Thanks for posting the link. I just updated the code with some fixes and a much-improved command-line application; usage instructions are included in the README file.
     
    mlippert likes this.
  4. Sep 8, 2015 #4 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    And as of today, TivoLibre also supports Program Stream files, bringing it up to feature parity with tivodecode.

    I also added support for PipedInputStream sources via an internal buffer, which allows TivoLibre to act as a pass-through while downloading. For example, you can setup a PipedInputStream and PipedOutputStream pair before downloading and create an instance of TivoDecoder with the PipedInputStream as its input and the destination file as its output. Each time you read a packet from the network, you write it to the PipedOutputStream. TivoLibre will decode it in real-time and write it to the output file, saving you the need to save the entire file to disk before beginning the decoding process.

    As always, bug reports are welcome!
     
    mlippert likes this.
  5. Sep 8, 2015 #5 of 154
    moyekj

    moyekj Well-Known Member

    12,161
    811
    Jan 23, 2006
    Mission...
    Wow, thanks for sharing, this could be very useful.

    I have a TS .TiVo file that doesn't decrypt either with tivodecode-ng or tivolibre. The tivolibre error is:
    SEVERE: Error: No TransportStream exists with PID 0x0633

    tivodecode-ng just gives a seg fault with no useful information.
    NOTE: There are several other clips that tivodecode-0.44 and tivodecode-ng have trouble with as well. Haven't tried them all with tivolibre yet.
    I did get tivolibre to work with at least 1 short H.264 TS .TiVo file.

    DirectShowDump works without any issues at all.

    I was able to make a very small 20 sec clip using VideoRedo that still has the problem and that you can download from here:
    https://drive.google.com/file/d/0B0SMFC97ymdEdXdUaWw2eVRqdUE/view?usp=sharing

    If you need my MAK for further debugging please send a PM, though you will probably need to get your post count up before you can do that.
     
  6. Sep 8, 2015 #6 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Awesome, thanks for including a file so I can investigate. I'll shoot you a PM for the MAK.

    EDIT: Or not; the forum won't let me send PMs without 10 posts. Could you send it to me instead?
     
  7. Sep 9, 2015 #7 of 154
    moyekj

    moyekj Well-Known Member

    12,161
    811
    Jan 23, 2006
    Mission...
    OK, PM sent. I assume you can at least read PMs sent to you.
     
  8. Sep 9, 2015 #8 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Yep, got it. Thank you!

    It looks like TivoLibre is also choking on a recording of Ken Burn's Civil War documentary from last night, so I've got a couple of test cases to fix now :)
     
  9. Sep 9, 2015 #9 of 154
    wmcbrine

    wmcbrine Well-Known Mumbler

    11,704
    812
    Aug 2, 2003
    I get the following message when I try to clone the repo from the command line:

    Code:
    $ git clone https://github.com/fflewddur/tivolibre.git
    Initialized empty Git repository in /Users/wmcbrine/tivolibre/.git/
    Cannot get remote repository information.
    Perhaps git-update-server-info needs to be run there?
    I got the same thing on another GitHub repo a few days ago, so it may be something GitHub is doing wrong. "Download as .Zip" works.
     
  10. Sep 9, 2015 #10 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Odd, it's working fine for me. Just ran this a second ago:

    Code:
    todd@dathyl:tmp$ git clone https://github.com/fflewddur/tivolibre.git
    Cloning into 'tivolibre'...
    remote: Counting objects: 121, done.
    remote: Compressing objects: 100% (84/84), done.
    remote: Total 121 (delta 44), reused 92 (delta 19), pack-reused 0
    Receiving objects: 100% (121/121), 523.19 KiB | 0 bytes/s, done.
    Resolving deltas: 100% (44/44), done.
    Checking connectivity... done.
    todd@dathyl:tmp$ git --version
    git version 2.3.2 (Apple Git-55)
    todd@dathyl:tmp$ 
    
    Maybe it was a temporary issue with GitHub?
     
  11. Sep 9, 2015 #11 of 154
    wmcbrine

    wmcbrine Well-Known Mumbler

    11,704
    812
    Aug 2, 2003
    Well, this seems to explain my problem:

    https://code.djangoproject.com/ticket/17220

    Sure enough, I tried it with "git://", and it worked fine. The odd thing is that I'm sure I cloned repos much more recently than the cut-off date they talked about there. Hmpf.

    Sorry.
     
  12. Sep 9, 2015 #12 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Thanks again for sending that file. I found a couple of obvious problems and fixed them, and decided to let the library discard packets for streams that weren't specified in the file's PMT. I'm not certain it's the right way to handle that situation--there are still a couple of fields in the PMT packet that I don't understand, and I'm guessing they describe the missing stream. Still, discarding the packets doesn't seem to hurt, and the resulting MPEG plays properly.

    I'm still getting "Invalid TS header" errors on a separate file, so these fixes didn't take care of all of the issues.
     
  13. Sep 9, 2015 #13 of 154
    moyekj

    moyekj Well-Known Member

    12,161
    811
    Jan 23, 2006
    Mission...
    Thanks for that fix and I confirmed my short clip now decrypts OK, though it comes out smaller file size than DSD decrypt and obviously diff shows a difference, but playing it there are no obvious glitches.

    For the full clip decrypt I'm getting a different error now:
    Sep 09, 2015 7:01:30 PM net.straylightlabs.tivolibre.TransportStreamDecoder process
    SEVERE: Unsupported packet type: NONE

    Looks like this happens at the very end of the decrypt since time wise the full decrypt does seem to happen. Again the full clip is smaller by several bytes from DSD version (2906039704 bytes vs 2909128544 bytes from DSD).

    I'll try and see if I can duplicate the problem with a short clip of the end of the large clip so that I can upload it for you to take a look at.
     
  14. Sep 9, 2015 #14 of 154
    moyekj

    moyekj Well-Known Member

    12,161
    811
    Jan 23, 2006
    Mission...
  15. Sep 9, 2015 #15 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    This should be fixed now, too. That's from the same recording as the first clip you sent, right? I ask because it also has lots of packets associated with stream 0x0633, just like the first clip, but the header for the file doesn't setup a stream with that ID. I haven't seen that behavior on any files off of my TiVo, which is the only device I have to test with. If you have any thoughts on what the extra data might be (different language audio? Subtitles?), I'm all ears. But since they don't seem related to the main audio and video streams, there doesn't seem to be much harm in discarding those packets, hence the smaller file size.
     
  16. Sep 10, 2015 #16 of 154
    moyekj

    moyekj Well-Known Member

    12,161
    811
    Jan 23, 2006
    Mission...
    Yes, it's from same recording and confirm it's now fixed. I tried a couple more clips that I never got to work properly with tivodecode/tivodecode-ng and they are working now with tivolibre, so it looks like a big improvement already to me over tivodecode at least for TS TiVo files, so great job!
     
  17. Sep 10, 2015 #17 of 154
    moyekj

    moyekj Well-Known Member

    12,161
    811
    Jan 23, 2006
    Mission...
  18. Sep 10, 2015 #18 of 154
    wmcbrine

    wmcbrine Well-Known Mumbler

    11,704
    812
    Aug 2, 2003
    Can't win, can we?

    Meanwhile, I'm trying to figure out why your first two samples work better with tivolibre... The other day, I felt I was tantalizingly close to solving the whole thing -- my model "problem child" clip has large blocks that decode properly if I just shift the Turing data by one byte; the only trick is figuring out when to do that -- but these clips are a completely different set of problems. Bleah.
     
  19. Sep 10, 2015 #19 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    I think the first samples are working because 1) TivoLibre discards packets with an unknown PID (as suggested here) and 2) it fixes a problem handling what should be the 12 ES Info Length bits in the PMT packet, but the original tivodecode code treated them as 13 bits. That extra bit is almost always 0, so usually no harm done, but every now and then it's 1, and that screws up handling the rest of the PMT packet. I think you can fix that in tivo_decoder_ts.cxx by changing the line
    Code:
    es_info_length = pmt_field & 0x1fff;
    to
    Code:
    es_info_length = pmt_field & 0x0fff;
    Hope that helps!
     
  20. Sep 10, 2015 #20 of 154
    wmcbrine

    wmcbrine Well-Known Mumbler

    11,704
    812
    Aug 2, 2003
    It did indeed! Thanks. :up:
     

Share This Page