Java port of TivoDecode

Discussion in 'Developers Corner' started by fflewddur, Sep 6, 2015.

  1. Sep 18, 2015 #61 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    OK thanks, that clip now works. This evening I found several places in the big file with binary differences, some right at the beginning of the file - so I simply made a quick cut using "head --bytes=60000K" to produce a clip and the resulting clip decrypted with DSD plays back fine, but has several blocky segments as decrypted by tivolibre:
    https://drive.google.com/file/d/0B0SMFC97ymdEakstWEZ2YW9BOEk/view?usp=sharing
     
  2. Sep 18, 2015 #62 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Thanks, that clip should work properly now. Does the rest of the file decode as well?
     
  3. Sep 18, 2015 #63 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    Excellent! Yes that clip and now the whole 7 GB file decrypt identically to DSD. I just ran all my checks again and they are all binary identical to DSD now (I'm up to 18 test files now).

    Do you have any clips of your own that are still problematic?
     
  4. Sep 18, 2015 #64 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Awesome! No, I don't know of any files that are still problematic. I just finished a batch test of my own: 91 files, all of which are identical to the DirectShow filter's output. I'm starting to download all of my TiVo Suggestions just to have something else to throw at it (and... really TiVo? These are your suggestions? It's almost as if you don't know me at all...). I'll post the results tomorrow, but I think we're getting close to something release-worthy...
     
  5. Sep 19, 2015 #65 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    That's truly amazing, job well done!
     
  6. Sep 19, 2015 #66 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    tivodecode has an option to just retrieve/dump the XML metadata from a .TiVo file. Is it possible to add a Java method for doing that to tivolibre?
     
  7. Sep 19, 2015 #67 of 154
    HerronScott

    HerronScott Well-Known Member

    7,420
    1,065
    Jan 1, 2002
    Staunton, VA
    Congratulations and I agree with job well done! It's been fun lurking and watching the progress! :)

    Scott
     
  8. Sep 19, 2015 #68 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Thanks, this has been a fun project!

    I checked the output of the test I ran overnight... every file decrypted perfectly except the very last. Figures, right? Still not quite ready for prime-time, but getting close. :)

    And yeah, there's no reason we couldn't add a method for getting the XML data. I've added it to the list of issues blocking a public release.
     
  9. Sep 19, 2015 #69 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Just checked in code to do this: there's now a getMetadata() method that returns a List of XML Document objects. You'll need to call either decode() or the new method decodeMetadata() before calling getMetadata(); the latter method will exit after processing the metadata, it won't decode the video itself.

    I also added the same -D and -x command-line parameters tivodecode uses to dump the metadata files and disable video processing. And, the command-line app will automatically save the user's MAK now, so you only need to provide it the first time its executed.
     
  10. Sep 19, 2015 #70 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    Awesome, thanks! I've been working on integrating tivolibre into kmttg and just replaced use of tivodecode with tivolibre for metadata generation from .TiVo files as well as adding option to use tivolibre for decrypt and/or combined download/decrypt. Things are working very nicely.
    I take it you solved the problem with your last file that didn't decrypt properly with this latest version?
     
  11. Sep 19, 2015 #71 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Great, can't wait to try it out! No, I haven't cracked that last file, but it plays properly; there's only one byte that's different from the reference file, and it's in that loss-of-sync area that gets skipped over during playback. Until I see the same problem in more files, it'll be hard to figure out what the pattern is that causes it.
     
  12. Sep 21, 2015 #72 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Just a heads-up: I put together a new release today (version 0.7.0), and a couple of important things have changed in the API.

    First, I added a compatibility-mode switch for maintaining binary compatibility with the TiVo DirectShow filter. Compatibility mode is disabled by default, which means TivoLibre will try to clean up obvious errors in TS files. Right now that means dropping out-of-sync frames, frames that even the DLL leaves encrypted, null frames, and orphaned packets. In the future, I'd also like to fix out-of-order packets and correct timestamps.

    Second, I've moved to the Builder pattern for creating TivoDecoder objects and deprecated the public constructor. This makes adding options (like the compatibility-mode switch) easy to do without constantly adding newer, more complicated constructors. The API examples in the README file has been updated to illustrate the new usage.
     
  13. Sep 21, 2015 #73 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    Correcting timestamp errors would be huge - many TiVo downloads for me have timestamp issues that require VideoRedo to clean them up. AFAIK fixing timestamp issues requires a remux doesn't it?
     
  14. Sep 22, 2015 #74 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    Having trouble with this clip:
    https://drive.google.com/file/d/0B0SMFC97ymdEdGVoTFFwdWF2cmM/view?usp=sharing
    Code:
    Sep 22, 2015 5:09:11 AM java.util.prefs.WindowsPreferences <init>
    WARNING: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(...) returned error code 5.
    Encryption by QUALCOMM
    05:09:11.933 [main] ERROR c.s.tivolibre.TivoDecoder - Problem parsing Turing header
    05:09:11.935 [main] ERROR c.s.tivolibre.TivoDecoder - Decrypting packet failed
    Exception in thread "main" java.lang.RuntimeException: Decrypting packet failed
            at net.straylightlabs.tivolibre.TransportStream.decryptPacket(TransportStream.java:170)
            at net.straylightlabs.tivolibre.TransportStream.processPacket(TransportStream.java:98)
            at net.straylightlabs.tivolibre.TransportStreamDecoder.decryptAndWritePacket(TransportStreamDecoder.java:404)
            at net.straylightlabs.tivolibre.TransportStreamDecoder.process(TransportStreamDecoder.java:123)
            at net.straylightlabs.tivolibre.TivoStream.processVideo(TivoStream.java:131)
            at net.straylightlabs.tivolibre.TivoStream.process(TivoStream.java:77)
            at net.straylightlabs.tivolibre.TivoDecoder.decode(TivoDecoder.java:76)
            at net.straylightlabs.tivolibre.DecoderApp.decode(DecoderApp.java:168)
            at net.straylightlabs.tivolibre.DecoderApp.run(DecoderApp.java:131)
            at net.straylightlabs.tivolibre.DecoderApp.main(DecoderApp.java:52)
    
    If it makes a difference, the above was uploaded to TiVo using pyTivo and then transferred back. The original clip was longer, but I managed to replicate problem on truncated smaller file.
     
  15. Sep 22, 2015 #75 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Interesting, that file is the first example I've seen of decryption on a packet prior to reading an encryption key from the data stream, and it also sheds some light on what's happening inside the doHeader() method! The crash is fixed and the file plays back properly, though it's not a bit-for-bit copy with the DLL's output. I'll keep working on it to see if I can figure out how the DLL is setting up its decryption keys prior to the first private data packet.
     
  16. Sep 22, 2015 #76 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    I'm not sure; to be honest, this project is my first experience working with MPEG streams. But now that I've written an entire frame-handling pipeline, plus a good chunk of the necessary PES handling code, it seems like going a step further and correcting the data flowing through the pipeline should be feasible.

    On the other hand, I've been playing around with recent ffmpeg builds and they seem to fix the timestamp issues too, so there may not be much need for TivoLibre to handle this.
     
  17. Sep 24, 2015 #77 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    In latest version I'm not allowed to create a TivoDecoder instance with null output stream. I was doing that because for case when I'm only interested in getting metadata there's no need for an output stream, so I was using:
    Code:
    TivoDecoder decoder = new TivoDecoder.Builder().input(inputStream).output(null).mak(MAK).build();
    if (decoder.decodeMetadata()) {
       docList = decoder.getMetadata();
    ...
    ...
    
    Is there a better way to obtain just XML metadata without having to create a (temporary) output stream?
     
  18. Sep 24, 2015 #78 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Ah, I hadn't thought of that. I fixed up the API so that it doesn't require an OutputStream when building a TivoDecoder. It'll still throw an exception if you try to decode a video without an OutputStream set, but it should let you decode the metadata without one.
     
  19. Sep 24, 2015 #79 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    Thanks, working well with those changes.
     
  20. Sep 27, 2015 #80 of 154
    Allanon

    Allanon Active Member

    909
    151
    Nov 2, 2005
    Recently my cable company informed me that my Tivo HD will no longer support decoding some of the HD channels because they are changing the video compression format. The only option was to upgrade to a Tivo series 4 or above or use their cable box which is a Tivo Premiere. I opted not to upgrade because I'm moving soon and will just upgrade to their cable boxes when I do move.

    But I do have one of their cable boxes now and I can see and record all HD channels with it. But when I download the .TiVo file from one of those newly encoded channels and try to decode it, all I get is audio. I tried both tivodecode and and tivo-libre but both just produce an audio only mpeg. Also when I transfer it from the cable box to my Tivo HD all I get is audio.

    Do you think you could figure out how to decode one of these files? Below is a link to a .Tivo file with about 1 minute of recorded video. I will PM my MAK if needed.

    test.zip (3.42 MB)
     
    Last edited: Sep 27, 2015

Share This Page