Java port of TivoDecode

Discussion in 'Developers Corner' started by fflewddur, Sep 6, 2015.

  1. Sep 13, 2015 #41 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    I just committed a large rewrite of the PES parser and packet handling code; now every file posted in this thread not only decrypts and plays properly, but is also binary identical to the output of the TiVo DirectShow filter (except the last 120 bytes of sample4.TiVo; they don't form a complete 188-byte TS packet so TivoLibre discards them).

    The PES handling is a little tricky: the DirectShow filter doesn't seem to count all of the different types of PES headers when calculating decryption offsets, so without extensive testing it's impossible to say whether TivoLibre is handling the correct subset of PES headers now. I'm going to try running this on all of the files from my TiVo over the next few days to see if there are any other binary differences from the DirectShow filter's output. If anyone else has problematic files to test it on, I'd love to hear the results.
     
    Last edited: Sep 13, 2015
  2. Sep 13, 2015 #42 of 154
    wmcbrine

    wmcbrine Well-Known Mumbler

    11,692
    808
    Aug 2, 2003
    Amazing work, fflewddur.
     
  3. Sep 13, 2015 #43 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    Awesome progress fflewddur, a big step forwards! Being binary identical to DSD gives good assurances and makes it much easier to test. As you say, a couple of my samples are a few bytes shorter than DSD because of the end of the file.

    Unfortunately now a couple of samples I was testing with that were working previously now do not with the new code:
    1st one is:
    The Neighbors - (s02e10) Supreme Like Me_ts.TiVo
    https://drive.google.com/file/d/0B0SMFC97ymdEc0ZZbmh0Ny1CVlk/view?usp=sharing
    Code:
    Sep 13, 2015 5:52:28 PM net.straylightlabs.tivolibre.PesHeader parsePesHeaderExtension
    SEVERE: PES header extension starts with invalid bits: 0x1
    Sep 13, 2015 5:52:28 PM net.straylightlabs.tivolibre.TransportStream getPesHeaderLength
    SEVERE: Exception: java.lang.RuntimeException: PES header extension start with invalid bits
    java.lang.RuntimeException: PES header extension start with invalid bits
            at net.straylightlabs.tivolibre.PesHeader.parsePesHeaderExtension(PesHeader.java:225)
            at net.straylightlabs.tivolibre.PesHeader.parsePesHeader(PesHeader.java:217)
            at net.straylightlabs.tivolibre.PesHeader.parseBytes(PesHeader.java:81)
            at net.straylightlabs.tivolibre.PesHeader.<init>(PesHeader.java:45)
            at net.straylightlabs.tivolibre.PesHeader.createFrom(PesHeader.java:54)
            at net.straylightlabs.tivolibre.TransportStream.getPesHeaderLength(TransportStream.java:108)
            at net.straylightlabs.tivolibre.TransportStream.calculatePesHeaderOffset(TransportStream.java:95)
            at net.straylightlabs.tivolibre.TransportStream.writePacket(TransportStream.java:69)
            at net.straylightlabs.tivolibre.TransportStreamDecoder.addPacketToStream(TransportStreamDecoder.java:351)
            at net.straylightlabs.tivolibre.TransportStreamDecoder.process(TransportStreamDecoder.java:113)
            at net.straylightlabs.tivolibre.TivoStream.process(TivoStream.java:85)
            at net.straylightlabs.tivolibre.TivoDecoder.decode(TivoDecoder.java:58)
            at net.straylightlabs.tivolibre.DecoderApp.decode(DecoderApp.java:123)
            at net.straylightlabs.tivolibre.DecoderApp.run(DecoderApp.java:95)
            at net.straylightlabs.tivolibre.DecoderApp.main(DecoderApp.java:40)
    
    2nd one is:
    h264_tivo_sample2.TiVo
    https://drive.google.com/file/d/0B0SMFC97ymdESnc0ci1QdTF6eDg/view?usp=sharing
    Code:
    Sep 13, 2015 5:54:09 PM net.straylightlabs.tivolibre.TransportStreamPacket readHeader
    SEVERE: Found private adaptation field data; we don't know how to handle this
    Exception in thread "main" java.lang.IndexOutOfBoundsException
            at java.nio.Buffer.checkIndex(Unknown Source)
            at java.nio.HeapByteBuffer.get(Unknown Source)
            at net.straylightlabs.tivolibre.TransportStreamDecoder.createPacketAtNextSyncByte(TransportStreamDecoder.java:154)
            at net.straylightlabs.tivolibre.TransportStreamDecoder.process(TransportStreamDecoder.java:63)
            at net.straylightlabs.tivolibre.TivoStream.process(TivoStream.java:85)
            at net.straylightlabs.tivolibre.TivoDecoder.decode(TivoDecoder.java:58)
            at net.straylightlabs.tivolibre.DecoderApp.decode(DecoderApp.java:123)
            at net.straylightlabs.tivolibre.DecoderApp.run(DecoderApp.java:95)
            at net.straylightlabs.tivolibre.DecoderApp.main(DecoderApp.java:40)
    
    So currently I have 10 test clips total:
    6 produce binary identical to DSD
    2 are binary identical to DSD except at very end of file
    2 posted above have problems.
     
  4. Sep 13, 2015 #44 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Yeah, I just noticed this on one my test cases as well; it's fixed in my local repo and should be committed later this evening. It looks like I missed at least one PES header because I'm also seeing some decryption problems in the same test case, and want to get that ironed out before committing.
     
  5. Sep 13, 2015 #45 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Thanks for posting those files, they're both decoding properly now :)
     
  6. Sep 14, 2015 #46 of 154
    PaulS

    PaulS Active Member

    779
    38
    Sep 16, 2002
    Southern NH
    Impressive work guys. Simply amazing.... Kudos!
     
  7. Sep 14, 2015 #47 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    Fantastic! Now 8 out of 10 are identical to DSD output and none have any playback issues. Will test some more over next few days. Thank you very much for all your work on this!
     
  8. Sep 14, 2015 #48 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    It's been my pleasure! Quick question: do you test with the DirectShow filter from the last free release of TiVo Desktop, or the current release of TiVo Desktop Plus (assuming that includes a different filter)?

    I ask because I've been investigating how the DirectShow filter handles out-of-sync streams, and from what I can tell it not only includes all of the unsynchronized packets in the output, it also stops decrypting the stream for quite a while, then jumps back and outputs decrypted packets again. On this test file I'm seeing about 24MB worth of junk data from just one loss of synchronization: 1 MB of out-of-sync frames, then 23 MB of encrypted MPEG data. VLC plays the file perfectly, but QuickTime keeps losing audio sync and ffmpeg cries bloody murder about the file not being a proper MPEG-2 stream.

    I'm striving for binary compatibility, so I want to make sure this isn't just a bug in the DirectShow DLL I'm using (which is from TiVo Desktop 2.8.3). Is this the recommended source for the DirectShow filter?
     
  9. Sep 14, 2015 #49 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    TiVo Desktop Plus uses same filter as regular TiVo Desktop. I'm using 2.8.3 as well. Generally speaking at least for me the .TiVo files (and resulting decrypted files) have lots of timestamp problems and have to be run through VRD QS Fix to fix them up before you can take them through encoding with something like ffmpeg. After fixing with VRD there are generally no problems for me.
     
  10. Sep 15, 2015 #50 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Ah, good to know. I think a --compatibility flag might end up being necessary; it's really nice to have binary compatibility with the DirectShow filter (especially for testing), but if we can fix up the file before writing it to disk, we probably should. I started working on this code so that I wouldn't have to rely on a Windows box with the DirectShow filter to archive recordings off of my TiVo... if we can also remove the Video ReDo dependency, so much the better :)
     
  11. Sep 15, 2015 #51 of 154
    wmcbrine

    wmcbrine Well-Known Mumbler

    11,692
    808
    Aug 2, 2003
    I'm pretty sure those are the same thing (2.8.3). It hasn't been updated in a long time (except for the expired cookie patch, but they didn't even bump up the version for that).
     
  12. Sep 15, 2015 #52 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    More progress! I just committed some big improvements toward full binary compatibility with the DirectShow filter: every file posted in this thread is now a bit-for-bit replica of the filter's output.

    I also put together a script to automate batch testing, it's located in the root of the Git repository. Given a directory full of .TiVo files and MPEGs decoded with the DirectShow filter, the script will decode each .TiVo file, compare its output with the reference MPEG, and print a nice list at the end of all of the files that didn't decode to a bit-for-bit copy of the reference MPEG. The test script requires Python 3.5 and a directory of consistently named files, where every TiVo file has the same extension and every reference file has the same name as the corresponding TiVo file, plus a consistent extension (I used ".ref.mpg", but you can specify your own). Run it with the -h parameter for a full list of configurable options.

    I'm running the test script right now on every file from my TiVo, about 200GB worth. If anyone else cares to run it on a large directory, I'd love to know the results. :)
     
  13. Sep 15, 2015 #53 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    Excellent!
    Funny about your script. I setup a similar one to run DSD and tivolibre on a bunch of .TiVo files and then run a diff (I used Perl). Indeed all of my .TiVo clips decrypted by tivolibre are now identical to DSD ones so I'd say it's working perfectly now based on the small set of files I'm testing with so far. At some point I need to download and run many more just to check things out.

    One question for you. Have you tested downloading from a TiVo and piping the inputStream to TiVoDecoder? I have no trouble when the inputStream comes from a .TiVo file, but so far when it's from inputStream of a URLConnection I'm getting very strange results - improperly decrypted and truncated resulting file. Could well be I'm just doing something wrong and I haven't actually looked at your source code yet to check how reads from inputStream are being done, but figure I would check with you to see if you have tested something along those lines.
     
  14. Sep 15, 2015 #54 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    This should work, but if the InputStream in question is an instance of PipedInputStream, the current code will have problems dealing with it. The trouble is in the CountingDataInputStream class; for PipedInputStreams, it uses a variable-length buffer to prevent the pipe's circular buffer from filling up and overwriting itself. Something isn't right though, because sometimes memory usage shoots through the roof and the CPU gets pegged at 100% usage (for comparison, the command-line app never pushes my CPU higher than 25%; it's definitely IO-bound). I'm planning to look into it once these compatibility issues are all worked out. When the InputStream provided to TivoDecoder is not an instance of PipedInputStream, the CountingDataInputStream class just wraps the stream in a DataInputStream, which doesn't have the issue.

    You can find an example of how TivoDecoder is supposed to work with a network connection at https://github.com/fflewddur/archiv...lightlabs/archivo/controller/ArchiveTask.java, particularly the way-too-long handleResponse() method. Again, it doesn't work properly at the moment, but it'll get there :)

    EDIT: Now that I think more about it, the problem might be with the DataInputStream; the read() methods don't verify that the requested number of bytes were returned, and with a network connection they might be returning early. That would definitely screw up the rest of the decoding process.
     
    Last edited: Sep 15, 2015
  15. Sep 16, 2015 #55 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    I found 2 samples that give binary differences vs DSD. I've uploaded the short one. The other one is 7 GB so don't want to upload the whole thing. If needed I can try and determine where differences are exactly to try and make a small sample. The shorter one is here:
    https://drive.google.com/file/d/0B0SMFC97ymdEUGNZMFZ1dE1KTTQ/view?usp=sharing

    NOTE: I deleted most of the samples I'd uploaded previously to google drive so most previous links to my files will no longer work.
     
  16. Sep 16, 2015 #56 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Great, thanks for the clip. Don't worry about the longer one right now; I'm still working on fixing all of the issues with my own collection of TiVo files, so let's test it again after these are working.
     
  17. Sep 16, 2015 #57 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Woohoo! The current code successfully decodes every file on my TiVo (about 200 GB), all in TS format, with only three files that show 1-byte differences from the DirectShow filter's output. The last file moyekj posted triggered a bug in the PS decoding, which has also been fixed.

    I'm having trouble tracking down exactly how to fix this (hopefully final) bit difference--the differing bits all follow a loss of TS sync but before decryption has kicked in again, so they appear to be placeholders of some sort. They're always 188 bytes after file positions evenly divisible by 0x100000, e.g., 0x2ee000bf, 0x697000bf, and 0x376000bf (but a different position in each file, and at different offsets from the loss of sync, re-establishment of sync, and offset for encryption to restart). The DirectShow filter applies a mask of 0xC0 to these bytes, resulting in the two highest bits being turned off. This is exactly what the filter does at *every* file position divisible by 0x100000 for some time after a loss of sync, but it doesn't do it consistently 188 bytes after that position. Anyone else have an idea of what may be going on here?

    (Regardless, resolving this issue will only improve binary compatibility--the masked bits are in a region of the file that has no impact on playback, and in fact should be removed from the file in order to make it a standards-compliant MPEG.)
     
  18. Sep 17, 2015 #58 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    Thanks for the update. The update makes the short one I posted above identical to DSD. The 7 GB clilp still has ~250MB bytes differences (cmp -l file1 file2 | wc -l). If you have suggestions on how to easily narrow in on where the differences are I can try and generate a short clilp around that point.
     
  19. Sep 17, 2015 #59 of 154
    moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    Was trying to narrow in on where the difference may be in the large clip using VRD. This short clip generated by cutting out a 1 min segment using VRD now crashes tivolibre:
    https://drive.google.com/file/d/0B0SMFC97ymdEd2JBSkFhYlpUM00/view?usp=sharing
    Code:
    Sep 17, 2015 7:52:48 AM net.straylightlabs.tivolibre.TransportStream processPacket
    SEVERE: Exception while calculating PES header offset: Unknown PES extension header type
    java.lang.RuntimeException: Unknown PES extension header type
            at net.straylightlabs.tivolibre.PesHeader.parseExtensionHeader(PesHeader.java:293)
            at net.straylightlabs.tivolibre.PesHeader.parseBytes(PesHeader.java:87)
            at net.straylightlabs.tivolibre.PesHeader.<init>(PesHeader.java:46)
            at net.straylightlabs.tivolibre.PesHeader.createFrom(PesHeader.java:55)
            at net.straylightlabs.tivolibre.TransportStream.getPesHeaderLength(TransportStream.java:149)
            at net.straylightlabs.tivolibre.TransportStream.calculatePesHeaderOffset(TransportStream.java:132)
            at net.straylightlabs.tivolibre.TransportStream.processPacket(TransportStream.java:87)
            at net.straylightlabs.tivolibre.TransportStreamDecoder.decryptAndWritePacket(TransportStreamDecoder.java:391)
            at net.straylightlabs.tivolibre.TransportStreamDecoder.process(TransportStreamDecoder.java:122)
            at net.straylightlabs.tivolibre.TivoStream.process(TivoStream.java:85)
            at net.straylightlabs.tivolibre.TivoDecoder.decode(TivoDecoder.java:58)
            at net.straylightlabs.tivolibre.DecoderApp.decode(DecoderApp.java:125)
            at net.straylightlabs.tivolibre.DecoderApp.run(DecoderApp.java:95)
            at net.straylightlabs.tivolibre.DecoderApp.main(DecoderApp.java:40)
    
     
  20. Sep 17, 2015 #60 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Thanks for sending me that clip, it should decode properly now.

    I use Hex Fiend to find the exact location of differences in binary files; it displays a side-by-side comparison of the files with differences highlighted and searchable. If you don't have access to a Mac, VBinDiff should work, or any hex editor that supports large files.

    For what it's worth, if you have a place to store a 7 GB file on the Internet, I don't mind the large download and would be happy to test with it. If it's easier than you hunting through it for the problem area, we could even setup a Torrent to transfer it directly between our computers.
     

Share This Page