Java port of TivoDecode

Discussion in 'Developers Corner' started by fflewddur, Sep 6, 2015.

  1. Feb 3, 2016 #141 of 154
    moelleref

    moelleref New Member

    7
    0
    Jan 27, 2016
    Thanks fflewddur for all your help fixing my tivolibre issue!
     
  2. Feb 3, 2016 #142 of 154
    HerronScott

    HerronScott Well-Known Member

    7,419
    1,065
    Jan 1, 2002
    Staunton, VA
    Thanks for the explanation/education. :)

    Scott
     
  3. mackworth

    mackworth cTiVo Developer

    97
    3
    Jul 31, 2006
    Hi @fflewddur. I was having a problem with certain shows and tivoLibre, and it took me quite a while to track down. These shows (e.g. Daily show on Comedy Central) sometimes download with a sync problem that causes tivoLibre 0.7.3 to lose track of the process, dropping about 43M before continuing, and the resulting mpg file is unplayable/unconvertible beyond that point. (That "sometimes" is why it was hard to track down. Most of the time, the file downloads fine from TiVo, but I caught it doing badly twice.)

    When I run the same .tivo file through tivodecode-ng, it finds the same problem, but the recovery seems much better: the file are only missing 23K; while there's still a glitch in Quicktime player at that location (22:47), it can be played beyond that point and the file runs through ffmpeg without a problem.

    There's a sample at this location:
    https://dl.dropboxusercontent.com/u/21507587/DailyShowBad.tivo
    and a correctly downloaded version at
    https://dl.dropboxusercontent.com/u/21507587/DailyShowGood.tivo

    I'll PM the MAK

    Here's the debug output for 0.7.3:

    For your interest, here's the verbose output from tivodecode-ng:
     
  4. fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Hi mackworth, thanks for reporting this and providing the sample files! I took a look at them tonight and compared the output of TivoLibre with TivoDesktop, and the output is binary identical between the two, both for the good and bad sample files. This is using the --compat-mode parameter with TivoLibre. Without --compat-mode, TivoLibre drops the 41 MB you mention, but this is by design; those packets are encrypted, and generally cause problems during playback. I'll investigate more this weekend to see if it's dropping some valid packets by mistake, but both files you sent me play back fine with VLC.

    As an aside, QuickTime has problems playing both of them back, including the versions decoded with TivoDesktop; I haven't used QuickTime for video playback in years, so I'm not sure if this is a common problem anymore, but it used to have all sorts of issues playing .mpg files, particularly keeping the video and audio in sync.
     
  5. mackworth

    mackworth cTiVo Developer

    97
    3
    Jul 31, 2006
    Thanks for looking into this. Although losing 40M isn't good, the main problem isn't QuickTime, it's that ffmpeg rejects the resulting file for further video processing. I'm curious; once you've found the sync again, why is it necessary to then skip to the "next position evenly divisible by the TS frame size"?
     
  6. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    50,957
    6,024
    Apr 17, 2000
    Nevada
    Not trying to sell you anything here but have you tried running these files through VideoReDo Quick Stream Fix? It can probably fix up these bad files and get you something you can process with ffmpeg. (or you can process directly in VRD since it has a full set of transcoding capabilities)
     
  7. mackworth

    mackworth cTiVo Developer

    97
    3
    Jul 31, 2006
    Sounds good; I look forward to trying out your Mac version!
     
  8. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    50,957
    6,024
    Apr 17, 2000
    Nevada
    Didn't realize you were a Mac guy. (you spelled it wrong in your user name ;)) Although you could use BootCamp or Parallels. :)
     
  9. mackworth

    mackworth cTiVo Developer

    97
    3
    Jul 31, 2006
    Well, having worked at Apple in the early 80's, I'm unlikely to change now :). A little more seriously, I think you're correct; do you know any similar Mac programs to validate the streams?
     
  10. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    50,957
    6,024
    Apr 17, 2000
    Nevada
    I am not aware of any, sorry. Based on the number of requests we get to port VideoReDo to Mac I'm not sure there are any. Unfortunately we're a small company (2 developers) so we just don't have the resources to port VRD to Mac.
     
  11. fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    After a sync loss, TiVo Desktop stops decrypting all of the packets until the next position in the byte stream that's evenly divisible by the TS frame size. To ensure binary compatibility, TivoLibre does the same thing. From my testing, dropping those packets completely seemed to improve the file's compatibility with ffmpeg and comskip (since they're still encrypted, they shouldn't have any bearing on the file's playback), but the file you provided shows that it doesn't always work that way.

    I'm going to try testing a release that preserves these packets; if it works with the recent releases of ffmpeg and comskip, it should probably be the default behavior. In the meantime, you can pass the "--compat-mode" parameter to TivoLibre to prevent it from dropping these packets.
     
  12. May 1, 2016 #152 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    I tried this for the 1.0 release of Archivo, but the result was a slew of people reporting problems with audio/video synchronization. Reverting to dropping these packets in the 1.0.1 release solved all of those issues, so I think the smartest default behavior is to keep dropping the encrypted packets. Again, you can change this behavior by passing the --compat-mode argument to TivoLibre from the command-line.
     
  13. May 1, 2016 #153 of 154
    fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Just a heads up that TivoLibre 0.7.4 is available at https://github.com/fflewddur/tivolibre/releases. This release fixes a rare decoding problem and, for anyone compiling from source, migrates to the Gradle build system. I also began releasing two JARs: tivo-libre.jar is intended for software developers to embed in their own projects, and doesn't include any dependencies. TivoDecoder.jar is an executable JAR that does include dependencies; it's intended for people who want to run TivoLibre directly from the command-line.
     
  14. NMUser

    NMUser New Member

    1
    0
    Oct 11, 2017
    Hello fflewddur,

    I am having an issue running tivolibre 0.7.4. All of my programs download from the TiVo, but some of them do not decode correctly. When bringing in programs from CNBC I get the following debug output:

    java -jar TivoDecoder.jar -d -i Fast* -o eric.mpg -m XXXXXXXXXX dir = 'null', file = 'eric.mpg'
    Header: TivoStreamHeader{ fileType=TiVo (54:69:56:6F), mpegOffset=0x3400, numChunks=3}
    Chunk 0: TivoStreamChunk{chunkSize=1140, id=3, type=PLAINTEXT}
    Chunk 1: TivoStreamChunk{chunkSize=4768, id=1, type=ENCRYPTED}
    Chunk 2: TivoStreamChunk{chunkSize=5640, id=2, type=ENCRYPTED}
    File format: TRANSPORT_STREAM
    Starting TS processing at position 13312
    Creating a new stream for PMT PID 0x0064
    Creating a new VIDEO stream for PID 0x0011 (type=0x1b)
    Creating a new AUDIO stream for PID 0x0014 (type=0x81)
    Creating a new PRIVATE_DATA stream for PID 0x0015 (type=0x97)
    Unsupported packet type: RESERVED2
    Closing CountingDataInputStream. Final read position: 13876


    On the programs to decode correctly the output looks like:
    java -jar TivoDecoder.jar -d -i The* -o eric.mpg -m XXXXXXXXXX dir = 'null', file = 'eric.mpg'
    Header: TivoStreamHeader{ fileType=TiVo (54:69:56:6F), mpegOffset=0x5400, numChunks=3}
    Chunk 0: TivoStreamChunk{chunkSize=1140, id=3, type=PLAINTEXT}
    Chunk 1: TivoStreamChunk{chunkSize=9012, id=1, type=ENCRYPTED}
    Chunk 2: TivoStreamChunk{chunkSize=9888, id=2, type=ENCRYPTED}
    File format: TRANSPORT_STREAM
    Starting TS processing at position 21504
    Creating a new stream for PMT PID 0x0064
    Creating a new VIDEO stream for PID 0x1291 (type=0x02)
    Creating a new AUDIO stream for PID 0x1294 (type=0x81)
    Creating a new PRIVATE_DATA stream for PID 0x1296 (type=0x97)
    PacketId: 100,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x1291 Position after read
    ing: 18,821,504
    PacketId: 200,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x1291 Position after read
    ing: 37,621,504
    PacketId: 300,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x1291 Position after read
    ing: 56,421,504
    PacketId: 400,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x1291 Position after read
    ing: 75,221,504
    .
    .
    .

    Please let me know if you need more info

    Thanks
     

Share This Page